JNGP postproc

[WaterKnight]

JNGP postproc is being developed for Windows at the moment. It most likely won't work on other platforms. And anyway, I only have a Win7 system to test it.

Exposition

What I have long been wishing for is that we distinguish between the development map/project file and the compiled/playable map result, like any compiled language does. vJass only works without that concept because the game does not use the war3map.wct and war3map.wtg but rather the editor generates the war3map.j every save anew. If we wrote directly to the war3map.j, jasshelper would translate it once and that's it, now we would not possess the source vJass code any longer.

There are several handy applications in finalizing a map. Ex: converting objects to slk, optimization (like discarding editor-only files) or recently asked path filling

Strategy

JNGP/wursteditor:

JNGP uses a file wehack.lua from its main directory and hooks onto the save and testmap procedures of the editor. That's how jasshelper gets called or the map is run with additional parameters like -window. I edited the file a bit, so it redirects to my postproc Lua scripts.

The 2nd issue is how to access the compiled map. As said above, we do not want to operate on the original map. That's why the save procedure actually makes a duplicate and that is the one that gets passed to tools by $MAP$ placeholder. This also means if you are using vJass, wurst or whatever, the compiler for this language should be invoked with postproc.

The arising question is how to comfortably test the map using the Playtest button from editor. JNGP hooks on that and postproc alters the -loadfile parameter under certain conditions. A problem is that I have not found a way to detect when a new map has been loaded within the editor or what was the original path of the map being playtested, so I can only guess if the last of my own copies postproc created beforehand actually corresponds to the one being playtested. By guessing I mean I have at least inserted a condition that the first 256 bytes (the header) of both files have to match, since that's unlikely to be the target of modifications. This feature "auto-redirect-testmap" feature can however be deactivated in the postproc menu and this menu also contains a function to run the last compiled map.

Download and documentation

Download the installer or the raw bundle from here.

For setup instructions, usage, the API and included tools, view the manual.

Credits

postproc (and friends) uses Lua and some addons: LuaFileSystem, Rings, Lua BitOp, LuaSocket
The updater uses libcurl and wxWidgets, the Windows API. The md5 algorithm used to compare local and remote files was taken from here.
The JNGP setup uses wxLua, a Lua binding for wxWidgets.

 

[Xonok]

There is a discussion going on in the JNGP2 issues that actually is about plugins. I am not entirely sure, but maybe it would be possible to include this kind of framework together with the plugin framework(as the same entity).

 

[WaterKnight]

I will look into it. The current plugin menu seemed like for interactive functionality inside the editor view.

I want to address a couple more ideas. A basic problem when dealing with maps is to get through the mpq packaging. It seems ineffective to have each tool do it on its own which is why I suggested to not pass the map around but extract all the files once centrally, then pass the folder, finally repack it. However, I want to support tools that do not align with this as well. Since we already have this Lua entrypoint, you could run tools with .lua extension directly and/or provide libs for dealing with the wc3 formats and porting.

 

[WaterKnight]

Making some progress but still unsure about some formalities.

Xonok requested that there should be a command/option to auto-pass $MAP$ and $LOOKUP_PATHS$ as 1st/2nd argument. I do not really like to do stuff hiddenly.

Possibilities:

a) differ between "post" and "postproc"

//! post tool arg3 arg4 ...
//! postproc tool $MAP$ $LOOKUP_PATHS$ arg3 arg4 ...

I think that's still pretty hidden and confusable.

b) make some option suffix

//! post[stdArgs] tool arg3 arg4 ...

c) write it like a function

//! post(tool, stdArgs, arg3, arg4)

Also another request was to be able to target the source map. This can either by done via another replacement $DEV_MAP$ or another command e.g. "postmanip". Dunno what's best.

Furthermore I introduced custom variables.

You can write

//! $COMMON_J$ = jasshelper\common.j
//! $BLIZZARD_J$ = jasshelper\blizzard.j

//! post Jasshelper --debug $COMMON_J$ $BLIZZARD_J$ $MAP$

but there are currently three configuration files: the one inside the postproc directory, the one from JNGP if you use it and the internal .wct of the map, alternatively a file (instructionFile) you pass to postproc.

I guess you should only be able to call tools from the latter one because else it's questionable what should be the order. What may be possible, however, is to insert declarations in the map/instructionFile and to overwrite declarations with the priority map>JNGP>postproc. An issue that needs to be addressed is that the variables in the example above denote paths. Paths specified in the map/instructionFile should be relative to the working directory, paths from JNGP relative to the JNGP directory, paths from postproc relative to the postproc directory. So do I have to classify the variables to be able to transform them or do something like

//! $COMMON_J$ = toAbsPath(jasshelper\common.j)

? Basically not sure if laying things out as a database or executed script.

 

[WaterKnight]

update:
-bundled everything together waterlua+wc3libs+procTools+postproc, with the paths pre-aligned
-added tools embedBuildNumber, pathFiller, vjassImport
-lua tools are now executed in separate lua states via luaRings, so they cannot influence each other or the postproc framework, package paths are forwarded to them, so they can find waterlua/wc3libs
-tool error messages are directed to the error popup in JNGP
-tools can be configured not to throw those prompts (maybe they already show a notification like jasshelper)
-postproc stops executing tools on first tool error
-fixes and stuff

 

[WaterKnight]

update:
-added a postproc.exe launcher, so you can finally invoke postproc directly without getting to it via a Lua script
-so you can now start it for example from Eclipse (wrote a "tutorial" above) or another IDE
-when started from JNGP, there is now another ring spawning a separate Lua state in which the postproc framework is done -> in case of errors, it will less likely affect the editor behavior
-added wehack.lua for wursteditor (which is very similar to JNGP)
-added a postproc menu in JNGP with several functions, rather than writing the //! noDefaultTools line, you may/should check "Block other tools" there
-fixed a problem that occured when the map header (as instruction input) was empty
-the tool definitions are now in postproc\configTools.slk, it still supports the old variant but I think I will deprecate that
-some other fixes

 

[Chaosy]

Am I seeing things? I am pretty sure I am understanding something wrong. The editor only compiles the vjass code once and not over and over?

*crosses fingers*

 

[WaterKnight]

What are you referring to? JNGP normally calls jasshelper once per save.

 

[Chaosy]

If we wrote directly to the war3map.j, jasshelper would translate it once and that's it, now we would not possess the source vJass code any longer.

Indeed. But this says otherwise?

 

[WaterKnight]

If there was no .wtg and .wct, the editor's files to store triggers, and we would directly write our vJass code in the .j file, which wc3 uses, then using jasshelper on the map would translate the vJass .j to plain jass .j and the next time we opened the map in the editor, we would not have the vJass code any longer (since the only script file was converted), it would be irreversible. I am inducing that there are transformations to map files that are valid to do in order to play them but which can ruin the map for editing.

Another example would be if I had a tool that rotates the terrain. Terrain is only stored in the .w3e, so the tool would have to affect the .w3e but I do not want to rotate my map with every compilation further and further, only basing it on my initial editor data.

 

[WaterKnight]

New version:

• new functionality: logTracker
• wc3libs: wc3jass, wc3objMerge
• new tools: jassAid, objForJass, exportFile, importFile
• some JNGP menu changes

I am planning to collect explanations for the tools in the first post.

edit: done

 

[WaterKnight]

New version:

• fixed the 'lfs not found' bug, it should now properly find all libraries without you having to have Lua installed
• the LogTracker no longer needs a waterlua on its own -> less size, better to maintain
• added "use console log" option to open a logging command window when postproc is running
• added a manual.html
• added some JNGP menu actions
• jasshelper.conf no longer requires logPath and outputPathNoExt, when not set, it takes the default paths
• instructions are now done via Lua script: Press >postproc >Edit instructions in JNGP, so it creates a repository for your map and opens it in explorer, the _current file contains which instruction file is currently used (release.lua on default), edit that instruction file (see the manual for API) with your favorite editor, save it and the next time you compile the map, it will use that file, also it gets injected into the map, if the map has no repository but contains postproc files, 'Edit instructions' will pull them but you should rather just maintain the files externalized
• added the functions 'unwrap' and 'wrap' to export all files from a map/import all from a directory (see manual)
• updated objForJass to optionally take an extra path to a Lua script for advanced control

The manual also describes what to do to get postproc working initially.

 

[Zwiebelchen]

Wow, so you actually made this a reality now, huh?

Amazing, especially for the postproc pathFiller tool (which imho is the first tool for this that has a high relevancy for every mapper, regardless of programming skill level... you should definitely write a tutorial for how to use it!).
Does the pathFiller tool consider destructable pathing, or only doodad/terrain pathing?

I'm asking this because I used mainly destructable pathing blockers, for performance reasons (I have a run-time script that removes them on map init and uses the pathing native to write the pathing info permanently into the pathing map --> less doodads on the map, but the same amount artistic pathing information).


Can you write a tool with this that allows manipulating the shadow map? I know I can just extract and convert the shadow map, then paint it manually in any picture editor, but there is a significant artistic problem with this:
To properly "paint" a shadowmap, one must have some kind of "overlay" to know where the large/high objects are.

A possible solution to this could be:
- Use the postproc to "scan" the map for all doodads and destructables
- Each doodad/destructable type rawcode can be assigned a defined shadow map look - so you basicly feed the algorithm with small pixel-images (kind of like stamps) for each doodad/destructable rawcode in a database
- the postproc then uses the doodad/destructable positions to "stamp" the shadowmap

Limits of this approach: obviously, it won't take terrain and doodad height/scaling information into account. A properly calculated shadowmap will still look better than this, but this should at least get you into the ballpark and be much much faster to computate than the WC3 default shadowmap calculation algorithm.



EDIT:
Just so you know why this is important: Calculating shadows for a large-scale 256x256 map with thousands of doodads (even when removing all doodads temporarily that don't cast a significant shadow) can take up to several days even on modern CPUs.
What I currently do to fight this is to use Newgen to adjust the map size, to split the map up into smaller "chunks", computate shadows for them, then connect the chunks back together in an image editor. This is extremely cumbersome, as you can imagine.

EDIT2:
Now that I think about it, this workaround I do should actually be the one that should be automated, as it's more precise and doesn't require user input ...

 

[WaterKnight]

Wow, so you actually made this a reality now, huh?

more like some months ago

Amazing, especially for the postproc pathFiller tool (which imho is the first tool for this that has a high relevancy for every mapper, regardless of programming skill level... you should definitely write a tutorial for how to use it!).

Well you place the entry points with rects prefixed by "wpm_" in the editor and run the tool. That's all, there is no other parametrization.

Does the pathFiller tool consider destructable pathing, or only doodad/terrain pathing?

Currently it only considers terrain pathing aka terrain+doodads. That makes more sense because destructables can potentially be destroyed, therefore the areas are accessible. I would rather suggest another tool that strikes out specific doodads/manipulating preplaced objects, so you do not have to worry about doing it ingame.

postproc is only the toolchain framework btw.

Can you write a tool with this that allows manipulating the shadow map? I know I can just extract and convert the shadow map, then paint it manually in any picture editor, but there is a significant artistic problem with this:
To properly "paint" a shadowmap, one must have some kind of "overlay" to know where the large/high objects are.

Your described shadow script is surely possible although wc3 shadows suck hard.

Limits of this approach: obviously, it won't take terrain and doodad height/scaling information into account. A properly calculated shadowmap will still look better than this, but this should at least get you into the ballpark and be much much faster to computate than the WC3 default shadowmap calculation algorithm.

Well that's normally not something essential to do on every save.

What I currently do to fight this is to use Newgen to adjust the map size, to split the map up into smaller "chunks", computate shadows for them, then connect the chunks back together in an image editor. This is extremely cumbersome, as you can imagine.

And that's faster?

Now that I think about it, this workaround I do should actually be the one that should be automated, as it's more precise and doesn't require user input ...

Well I have no way to control the World Editor shadow calculation feature.

 

[Zwiebelchen]

Well you place the entry points with rects prefixed by "wpm_" in the editor and run the tool. That's all, there is no other parametrization.

I see... nice idea with the rects btw!

Currently it only considers terrain pathing aka terrain+doodads. That makes more sense because destructables can potentially be destroyed, therefore the areas are accessible. I would rather suggest another tool that strikes out specific doodads/manipulating preplaced objects, so you do not have to worry about doing it ingame.

That's a bummer then. I think this feature should be added as an optional toggle, simply because the default WC3 invisible pathing blockers are destructables, not doodads.

Your described shadow script is surely possible although wc3 shadows suck hard.

Agreed, but unfortunately, the map still looks better with shadows calculated than without.

And that's faster?

Not if you want to recalculate the whole map... but it allows you "saving" already calculated parts of your map.

So, for example, if you have one area on your map already calculated and didn't make any new terrain changes, it would be stupid to calculate shadows for this area again. So basicly, this allows you to only calculate shadows in areas that you changed in the mapping process, saving lots of time.
Also, again, calculating shadowmaps for large-scale projects takes multiple days. So splitting the process up into smaller chunks is a good idea.
Especially as you can then effectively use all your CPU cores instead of just one by multiboxing the map editor...

Well I have no way to control the World Editor shadow calculation feature.

Bummer.

 

[WaterKnight]

I have written some support for the doo and dooUnits file yesterday, for shd as well. The striking out tool was therefore very simple/I made it generic as a script-controled dooEdit, for the shadowStamp tool I still need to know how exactly the input data should look like.

To convert the pathing of destructables, I would need some more action, especially a tga reader.

 

[Zwiebelchen]

I have written some support for the doo and dooUnits file yesterday, for shd as well. The striking out tool was therefore very simple/I made it generic as a script-controled dooEdit, for the shadowStamp tool I still need to know how exactly the input data should look like.

To convert the pathing of destructables, I would need some more action, especially a tga reader.

I thought about this "stamping" thing again and the more I think about it, the more I realized that it probably wouldn't solve anything.
The problem is that you can't just ignore terrain or doodad height when doing this. Using vertical terrain height modification is just way too popular.

So unless you can also make this read out the terrain heightmap AND the doodad scaling values, it's probably not worth the effort.
And at this point, we basicly replicated the editor-internal shadowmap calculation algorithm anyway; hard to expect that a custom one to be any faster then, though I might be wrong.


But if you still want to do this for shits and giggles, here's how I could imagine the input data looking like:
You basicly have a formatted text file that holds all the doodad rawcodes you want to read, then basicly a 4x4 or 8x8 ascii char grid for every rawcode (let the user specify what to input here). if possible, let it accept a sequence of different rawcodes.

< 'D000','D001','D002' - [4x4] >
< ooxo >
< oxxo >
< xxxo >
< oXoo >

< 'D004','D005' - [8x8] >
< oooooooo >
< ooooooxo >
< oooooxxx >
< ooooxxxx >
< oooxxxxx >
< ooxxxxxo >
< ooxxXxxo >
< oooxxxoo >

In the above example, x mark the areas which have a shadow, o mark areas without. A capital letter X or O mark the stamp pivot point, where the doodad center is located.



But as I said, at this point, it's probably easier to just write a 3rd party tool that allows easy import/export of shadowmaps and calculating them in chunks.

Actually, there is:
http://www.hiveworkshop.com/forums/tools-560/shdconvertor-1-0-a-226717/

But it can only convert pictures to shadowmaps, not the opposite. A tool that allows converting the files back and forth would already be a huge worksaver. Bonus points for a tool that could do all the "puzzle work" automaticly, in that it takes a sequence of map files instead of images.

The workflow could then look like this:
-> You reduce the map size via Newgen
-> You calculate shadows
-> You upload the now calculated sub-map into the tool
-> repeat until you have all parts of the map you want calculated and entered into the tool as a list of filepaths
-> You select the "output" map file
-> The tool merges all the shadow maps from the sub-map files into the output map.

This should be easily doable, now that I think about it; you just read out the borders of your map file and sub map files and basicly have an algorithm that merges all the SHD images into one.
It's basicly the stamp approach again, just that it takes map inputs, not text input and has a larger scale.

Check out the picture below for a visual representation of what I mean.

A tool like this would take a series of maps (sub 1 to sub 5) and then merge the SHD files into an output map.

 

[WaterKnight]

Reading terrain height and doodad scale is no problem but shadow calculation in World Editor involves the model, which I have little knowledge of, and basically you have to determine which parts are blocked by the model's shapes from the light vector.

Maybe the hacker brigade, SharpCraft and company, could extract the shadow calculation function and what belongs to it from wc3.

 

[Zwiebelchen]

Reading terrain height and doodad scale is no problem but shadow calculation in World Editor involves the model, which I have little knowledge of, and basically you have to determine which parts are blocked by the model's shapes from the light vector.

Maybe the hacker brigade, SharpCraft and company, could extract the shadow calculation function and what belongs to it from wc3.

Yeah I figured that.

But I actually like that multi-map approach more anyway. Seems more practical from a user-perspective (and can be completely decoupled from the WC3 editor, so basicly a standalone tool kind of like MapOptimizer).
I updated my post with an example image.

Can you do this? This would be extremely useful for basicly every large-scale project.

Also, sounds pretty easy to code. Just needs a very basic GUI and an algorithm that reads the hex values from the .shd maps and merges them with the output file.

 

[WaterKnight]

Do the chunks have the same coordinate system as the original/output? Should be no problem then. But I do not see how calculating the shadows on small maps would attain the same as applying it to the whole because all the shadows on the edges would be cut off. When you have 10ks of doodads allocated, no way you won't slice it.

 

[Zwiebelchen]

Do the chunks have the same coordinate system as the original/output? Should be no problem then.

Yes. In Newgen, you can basicly adjust the map size and camera bounds in steps of 32 cells within a range of -32768 to +32768 freely (even in the open map!).
This way, a map can have very odd "uneven" map bounds like -1024 to +16384 or +2048 to +4096 in JNGP, depending on how you extend it. The 0,0 coordinate might not even be in your playable map area.

Basicly, what you do when creating these chunks is that you create a copy of your map, scroll to the area you want to calculate and then reduce the map borders (everything else on the map gets cut off). Then you save&calculate.

If you want to get a little bit more fancy, you can allow manual user input to override the sub-map bounds, so that he can "copy and paste" shadow layouts from other areas. But this is totally optional.

In the workflow I envision with this, the coordinate system is always correct (in other words: 0,0 is always at the same location).

But I do not see how calculating the shadows on small maps would attain the same as applying it to the whole because all the shadows on the edges would be cut off. When you have 10ks of doodads allocated, no way you won't slice it.

The edges are not really a problem if you can provide overlapping maps, as shown in the example (sub map 2, 3 and 5 are overlapping).

If a coordinate is present in two overlapping maps, simply apply an OR logic to it; if any of the maps has the particular coordinate shaded, then it should also be shaded in the output map.
A coordinate should only be unshaded in the output map if it is unshaded in all maps.

This way, you can get rid of any potential seams by simply having all maps overlap slightly.


... so, could I convince you creating such a tool? That would be extremely useful!

 

[oger-lord]

You want a tool that can update the shadows of a map in a specified area?
So you would first generate a copy of the map without unnecessary terrain and doodads,
calculate the shadows in the editor and then copy the shadows with a tool into the old map?

You remind me that I wanted to write a tool that can calculate shadows faster than the editor.
I mean the doodad merger can already load doodads, I would need to add the terrain and a clever function to calculate the shadows.

 

[Zwiebelchen]

You want a tool that can update the shadows of a map in a specified area?
So you would first generate a copy of the map without unnecessary terrain and doodads,
calculate the shadows in the editor and then copy the shadows with a tool into the old map?

Yes, but with the added feature that it can also combine multiple map SHD files into one, based on the map internal coordinate system.

You remind me that I wanted to write a tool that can calculate shadows faster than the editor.
I mean the doodad merger can already load doodads, I would need to add the terrain and a clever function to calculate the shadows.

I highly doubt thats possible, as you have to take the polygons of all the doodads into account aswell.

Basicly, you'd manually write a raytracing algorithm, which is not feasable. What you could do, however, is using an existing raytracing algorithm based on directX ... however you could possibly apply that to the game without some serious hacking.

 

[WaterKnight]

mini update:

• added support for .shd
• importing files with the portLib should now compress the archive to a reasonable hashtable size
• other compilation tools in JNGP are now executed before postproc, I did it earlier the other way around because the blocking instruction was identified when postproc was already running but now you can simply check it in the plugIn menu

Attached is the shdMerger tool Zwiebelchen requested, which hopefully works. It does not use postproc but the libs included therein. See readme.

 

[Zwiebelchen]

Attached is the shdMerger tool Zwiebelchen requested, which hopefully works. It does not use postproc but the libs included therein. See readme.

Awesome! I'll check it out this afternoon and give you my feedback.

Just a question btw; does it consider the existing .shd file from the target map aswell, or will the target map be completely overwritten by the auxiliary shds?

 

[Zwiebelchen]

Can't get it to work; whenever I try to to run shdMerger.exe, it tells me that it "could not find or load the Qt platform plugin "windows"".
When I try to execute Lualauncher.exe, it tells me that "libgcc_s_dw2-1.dll" is missing.

Do I need to download and install Qt for this?

Also, I couldn't get postproc to work either, as I'm using JNGP 2.0.7 for multiplayer emulation, not the outdated 5d and 5e versions and when I'm overwriting wehack.lua for the new JNGP build, it actually breaks all the features of JNGP 2.0.7

 

[WaterKnight]

Do you have Windows? I cleared the Path environment variable so I was pretty sure it would not need anything besides from system32.

 

[Zwiebelchen]

Do you have Windows? I cleared the Path environment variable so I was pretty sure it would not need anything besides from system32.

Yes, using W7.

 

[WaterKnight]

Third time is the charm.

 

[Zwiebelchen]

Third time is the charm.

Alright, it works now.

Gonna check it out by trying to add a small extra shadow to my map; i'll edit in the results.


EDIT: Wow, almost forgot how even a very small piece of the map takes years to calculate shadows ... Only 96x128 map dimensions of the sub map and still only halfway through after 15 minutes.

EDIT2: Okay, when trying to apply a sub-map to my target map by hitting "start", it shows me an error message "Error in process:".
The target map has the dimensions 480x288 (x: -30720 to +30720; y: -5632 to +31232) and the sub map is 96x128 (x: +18432 to +30720; y: -5632 to +10752).
GaiasNew3_2 (82).w3x is my original map file.
GaiasShadow1.w3x is my reduced shadow map file.

 

[WaterKnight]

mini update:

• optimized the .shd read/write algorithm
• added a wehack.lua for JNGP207

 

[Zwiebelchen]

mini update:

• optimized the .shd read/write algorithm
• added a wehack.lua for JNGP207

Thanks for the JNGP207 update!

So did you finish your work on the shdMerger? Does it work with large maps now?

 

[Zwiebelchen]

Awesome, thanks for the update!

Got it working and the result looks great.
However, there's one thing about the functionality that is bugging me:

The "target map" still keeps it's shadows it had before you added the sub maps into it, so there's no way of "removing" unwanted shadows again. You should make it so that the shd map file from the target map is actually deleted before the sub maps get added.

Now if people actually want to keep the target map's current shadows, they can always just add it as a sub map input, so that it writes into itself again.

 

[WaterKnight]

New version:

• Added an updater for Windows, you can now download a client and it pulls all the other files from a server and whenever there is an update, it detects the differences and compensates them. You should not put own files in the postproc directory as those might be deleted in the process.
• In the JNGP subdirectory, there is now a setup.bat file that you can execute to connect your JNGP editor to postproc, it replaces the lua51.dll, the wehack.lua and adds a postproc.conf in your JNGP's main directory.
• The wehack.lua is basically outsourced to postproc, so even if you update postproc from within JNGP (menu bar >postproc >Update), unless there is a change to the init script, you won't have to restart the editor.
• The wc3libs were re-arranged for more uniformity and modularity. Support for a couple of wc3 formats were added. At some point, I want to wrap all wc3 formats.
• new tool: pathPainter: The areas of specifically prefixed GUI rects are added to terrain pathing. "wpma_" -> blocks air, "wpmg_" -> blocks ground, "wpmb_" -> blocks air and ground.
• new tool: tagReplacer: Literalizes object editor like string tags in the object data in order to fix the original wc3 mechanism. ex: "<A000,dur1>" will become "5" if the "dur1" field of the "A000" object contains that value.

The first post now points to a website where you can download the installer or postproc (and friends) rawly. Also I will just link the manual for instructions instead of trying to keep the first post in sync.

I do not expect everything to work right off the bat but at least things look good on my end. If there is a problem, just chat me up (I am usually sitting in the Hive chat).