LUA wc3binaryFile

[WaterKnight]

A package of lua scripts that shall help to modify the binary files of map archives. You can read from files, the values get stored in a table under identifiers, can be edited from there and written into a file again in the end.

This allows you to work with standardized mask functions for both reading and writing, thus you need to care less for the sequential format of the binary files. Still, the masks are less suitable for manipulation of the data, which is why you may coonsider converting to another format temporarily.

Included masks:

obj (w3u, w3t, w3b, w3h)
objEx (w3d, w3a, w3q)
env (w3e), though it's very slow to read large terrains
guiTrig (wtg), you need to copy the latest TriggerData.txt from your War3Patch.mpq\UI\ to the working directory in order to use this since it contains information about the arguments different functions take
wct
doo
cam (w3c)

Sample code:

require "wc3binaryFile"
require "wc3binaryMaskFuncs"

root = wc3binaryFile.create()

root:readFromFile('war3map.wtg', guiTrigMaskFunc)

root:getSub('trig1'):setVal('enabled', 0)

root:getSub('trig1'):print()

root:print("print.txt")

root:writeToFile('output.wtg', guiTrigMaskFunc)

Besides lua, you currently need 3 things to use it:

LuaFileSystem (lfs) (google for it if you do not have it already)
waterlua https://github.com/WaterKnight/waterlua, see the readme
wc3binary https://github.com/WaterKnight/wc3binary

 

[Nestharus]

Why is this in Lua over c/c++?...

 

[WaterKnight]

Because my c/c++ sucks even more. This and lua is easy to handle for the user. Of course it's very slow to process large quantities of data like the w3e (environment) contains. I will do an update this weekend because the w3e for example uses other data types I have not included yet.

 

[Nestharus]

lol, there's a dll online you can use to open the wc3 map archive. From there, you can literally typecast the data into unions >.>. It's really simple, heh.

I was working on a c++ library for reading wc3 maps at one point. Would you like the project? It already has everything setup, all you gotta do is the typecasts.

I have another project that embeds Lua into a c++ project, so you can pass the Lua scripts to it.

 

[moyackx]

One question: the idea is that I just copy this lua file, load them with Dofile(...) and can use them pretty straight??

@Nestharus: a dll to open maps?? I'm interested. Is there a list of public functions list to check it out??

 

[Nestharus]

@Nestharus: a dll to open maps?? I'm interested. Is there a list of public functions list to check it out??

Yea, you can just google it if you know how to use dlls

 

[WaterKnight]

One question: the idea is that I just copy this lua file, load them with Dofile(...) and can use them pretty straight??

Yes, you put the libs in the same directory as the application file and modify and execute latter.

Extracting map archives/repacking is a separate task but of course you could optionally implement it. I myself use Ladik's but unpack the whole map into a directory, so I can run different tools on it and merge it with external sources, only in the end do I build the playable map file from everything.

 

[Nestharus]

That'd be a good idea if we had a terrain editor as good as WE :\

 

[FRENGERS]

Nice TQ Lovely!!

 

[WaterKnight]

Before I can release the new version, I have a problem with understanding the format of shorts in wc3 to be able to correctly read and write them.

According to http://www.wc3c.net/tools/specs/index.html

shorts consist of 2 bytes, the highest 2 bits are unused for the number value. The range is from -16k to +16k, which would be a size of 32k. How is 2^(16-2)=32k?

Also 0x3FFF represents the max value and 0xC000 the min value. But 0xC000 in binary is 1100 0000 0000 0000, so in this case the 2 highest bit would have to be considered. Only two fields use the short type in map archives, both being in w3e (groundHeight and waterLevel) and only waterLevel makes use of its highest bit for an additional flag. Furthermore the editor allows heights only between -256 and 2048, which corresponds to 0x2000 to 0x3FFF. That's why I can badly check this.

The formula is

finalHeight = (groundHeight - 0x2000 + (cliffLevel - 2)*0x0200)/4

also depicted within the reference above.

Any idea?

 

[WaterKnight]

Update:

-rewrote everything
-added nodes structure to avoid prefix mess
-should now use the <:> operator between root/node and method instead of <.> in order to pass the object being manipulated
-typeDefs were exported to an own file
-added short data type, renamed real to float, setting the value now checks for senseful values
-introduced bitMasks replacing intflags: any type of definite size can be added using a table of named bits, those bit indexes are ignored when reading the parent data but extract their own value, so the different information does not get in conflict with each other, they are merged again when outputting, well it's practically only needed in the w3e
-added mask functions for w3e (environment) and wtg (gui triggers), latter needs the latest TriggerData.txt from wc3
-implemented a loading display since the w3e requires a lot of time to be processed
-fixed some bugs
-now checking if file format matches

 

[Ralle]

Thread opened on author request.

 

[Deaod]

I have a problem with understanding the format of shorts in wc3 to be able to correctly read and write them.

If im understanding the spec correctly, it makes no sense.
The highest bit of a short is not 0x4000, its 0x8000. 0x2000 is 8192, which also rules out having the short scale from 0b00_0000_0000 to 0b11_1111_1111, because thats not a range of 2^15 (ie. 32768).

So, to make everything fit, shorts have 16 bits, of which only the 16th bit is reserved. Short have values ranging from 0b000_0000_0000 (0) to 0b111_1111_1111 (32767, 0x7FFF), and a bias of 0b100_0000_0000 (16384, 0x4000).

Alternatively, the two highest bits are reserved, and the spec is wrong about the range of values allowed, the corrected range being from 0 to 16383, with 8192 being ground level.

 

[WaterKnight]

update:
-fixed a problem with writing floats of value 0 (uses another return instruction but forgot to write the bytes for 0)
-added masks for doo (doodad placements) and w3c (cameras)
-included the latest TriggerData.txt (for gui triggers) in the package
-added a doc.txt for the api and the sample code from above

 

[WaterKnight]

update:
-added wct mask
-added an application to transform wct (custom script trigger data) with help of the wtg (gui trigger data) to a file system corresponding to the view in trigger editor: you need to pass the .wct as first parameter and the .wtg needs to be within the same directory and the same file name except extension

 

[WaterKnight]

update:
-added a version of Ladik's mpq editor to the package
-added portLib.lua, which offers an interface for said mpq editor, so you can easily extract/add files from/to a map archive
-added wtsParser.lua, which translates TRIGSTR_ expressions to the corresponding strings in the war3map.wts file, the war3map.wts should have been built from the World Editor though because the parsing pattern does not allow for every malformed-ness, I do not possess the exact file specs
-common basic functions have been outsourced into the 'waterlua' package, a set of global modules that should be placed in %LUA_DEV%\lua\waterlua, so any lua script can load it by

require 'waterlua'

, it also requires 'LuaFileSystem' (lfs), a very common library for lua, which may help with platform-independence

Both 'waterlua' and 'wc3binary' are now available on gitHub

https://github.com/WaterKnight

 

[WaterKnight]

update:
-multiple new unrefined stuff, basically this is not only about the binary files anymore but dealing with the wc3 formats in general, I do not have too much spare time in the near future, so I pushed the repositories as is
-added wc3profile (the object text files under Units\), wc3skin (is just a wrapper for wc3profile atm because they are similar, for war3mapSkin.txt/war3mapMisc.txt/war3mapExtra.txt), wc3objMod (map object files like war3map.w3u, basically a better interface for wc3binaryFile)
-in applications I have built a converter for objMod to slk/profile (using meta slks) since someone requested it
-another application to create a custom TriggerData.txt from source TriggerData.txt or jass files for the World Editor
-refactored the portLib a bit due to instabilities of Ladik's mpq editor, he also popped a new version for me
-general refactoring, lots of asserts inserted

Maybe at some other point I will be presenting the ways of mapmaking I had in mind before in detail because it basically uses these tools instead of World Editor work.

 

[Nestharus]

I'll use these tools in my new map framework thing.


Extremely useful, exactly what I was looking for, especially the TriggerData thing, which I assume adds new triggers from JASS files. +rep

 

[WaterKnight]

As I said, it's still very unrefined, most of this stuff stems from situational necessity, like someone asked what's wrong with his map, so I mostly used it for debugging.

The createGUITriggerData is currently only for the common.j (natives), if you want to add custom .js, you have to make an alternate regex analogue to line 866.

 

[moyackx]

Awesome!!! I have evil intentions with this concept

muahahahhhhaaaaaaa!!!