1. Find your way through the deepest dungeon in the 18th Mini Mapping Contest Poll.
    Dismiss Notice
  2. A brave new world lies beyond the seven seas. Join the 34th Modeling Contest today!
    Dismiss Notice
  3. Check out the Staff job openings thread.
    Dismiss Notice
Dismiss Notice
Hive 3 Remoosed BETA - NOW LIVE. Go check it out at BETA Hive Workshop! Post your feedback in this new forum BETA Feedback.
Dismiss Notice
60,000 passwords have been reset on July 8, 2019. If you cannot login, read this.

LUA wc3binaryFile

Discussion in 'Warcraft Editing Tools' started by WaterKnight, Apr 28, 2013.

  1. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,035
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    A package of lua scripts that shall help to modify the binary files of map archives. You can read from files, the values get stored in a table under identifiers, can be edited from there and written into a file again in the end.

    This allows you to work with standardized mask functions for both reading and writing, thus you need to care less for the sequential format of the binary files. Still, the masks are less suitable for manipulation of the data, which is why you may coonsider converting to another format temporarily.

    Included masks:

    obj (w3u, w3t, w3b, w3h)
    objEx (w3d, w3a, w3q)
    env (w3e), though it's very slow to read large terrains
    guiTrig (wtg), you need to copy the latest TriggerData.txt from your War3Patch.mpq\UI\ to the working directory in order to use this since it contains information about the arguments different functions take
    wct
    doo
    cam (w3c)

    Sample code:

    Code (Text):
    require "wc3binaryFile"
    require "wc3binaryMaskFuncs"

    root = wc3binaryFile.create()

    root:readFromFile('war3map.wtg', guiTrigMaskFunc)

    root:getSub('trig1'):setVal('enabled', 0)

    root:getSub('trig1'):print()

    root:print("print.txt")

    root:writeToFile('output.wtg', guiTrigMaskFunc)
    Besides lua, you currently need 3 things to use it:

    LuaFileSystem (lfs) (google for it if you do not have it already)
    waterlua https://github.com/WaterKnight/waterlua, see the readme
    wc3binary https://github.com/WaterKnight/wc3binary
     
    Last edited: Oct 10, 2014
  2. Nestharus

    Nestharus

    Joined:
    Jul 10, 2007
    Messages:
    6,146
    Resources:
    8
    Spells:
    3
    Tutorials:
    4
    JASS:
    1
    Resources:
    8
    Why is this in Lua over c/c++?...
     
  3. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,035
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    Because my c/c++ sucks even more. This and lua is easy to handle for the user. Of course it's very slow to process large quantities of data like the w3e (environment) contains. I will do an update this weekend because the w3e for example uses other data types I have not included yet.
     
  4. Nestharus

    Nestharus

    Joined:
    Jul 10, 2007
    Messages:
    6,146
    Resources:
    8
    Spells:
    3
    Tutorials:
    4
    JASS:
    1
    Resources:
    8
    lol, there's a dll online you can use to open the wc3 map archive. From there, you can literally typecast the data into unions >.>. It's really simple, heh.

    I was working on a c++ library for reading wc3 maps at one point. Would you like the project? It already has everything setup, all you gotta do is the typecasts.

    I have another project that embeds Lua into a c++ project, so you can pass the Lua scripts to it.
     
  5. moyackx

    moyackx

    Joined:
    Feb 15, 2006
    Messages:
    801
    Resources:
    7
    Maps:
    4
    Spells:
    2
    Tutorials:
    1
    Resources:
    7
    One question: the idea is that I just copy this lua file, load them with
    Dofile(...)
    and can use them pretty straight??

    @Nestharus: a dll to open maps?? I'm interested. Is there a list of public functions list to check it out??
     
  6. Nestharus

    Nestharus

    Joined:
    Jul 10, 2007
    Messages:
    6,146
    Resources:
    8
    Spells:
    3
    Tutorials:
    4
    JASS:
    1
    Resources:
    8
    Yea, you can just google it if you know how to use dlls ;)
     
  7. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,035
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    Yes, you put the libs in the same directory as the application file and modify and execute latter.

    Extracting map archives/repacking is a separate task but of course you could optionally implement it. I myself use Ladik's but unpack the whole map into a directory, so I can run different tools on it and merge it with external sources, only in the end do I build the playable map file from everything.
     
  8. Nestharus

    Nestharus

    Joined:
    Jul 10, 2007
    Messages:
    6,146
    Resources:
    8
    Spells:
    3
    Tutorials:
    4
    JASS:
    1
    Resources:
    8
    That'd be a good idea if we had a terrain editor as good as WE :\
     
  9. FRENGERS

    FRENGERS

    Joined:
    Aug 4, 2012
    Messages:
    970
    Resources:
    4
    Icons:
    3
    Maps:
    1
    Resources:
    4
    Nice TQ :D :D Lovely!!
     
  10. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,035
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    Before I can release the new version, I have a problem with understanding the format of shorts in wc3 to be able to correctly read and write them.

    According to http://www.wc3c.net/tools/specs/index.html

    shorts consist of 2 bytes, the highest 2 bits are unused for the number value. The range is from -16k to +16k, which would be a size of 32k. How is 2^(16-2)=32k?

    Also 0x3FFF represents the max value and 0xC000 the min value. But 0xC000 in binary is 1100 0000 0000 0000, so in this case the 2 highest bit would have to be considered. Only two fields use the short type in map archives, both being in w3e (groundHeight and waterLevel) and only waterLevel makes use of its highest bit for an additional flag. Furthermore the editor allows heights only between -256 and 2048, which corresponds to 0x2000 to 0x3FFF. That's why I can badly check this.

    The formula is

    finalHeight = (groundHeight - 0x2000 + (cliffLevel - 2)*0x0200)/4

    also depicted within the reference above.

    Any idea?
     
  11. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,035
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    Update:

    -rewrote everything
    -added nodes structure to avoid prefix mess
    -should now use the <:> operator between root/node and method instead of <.> in order to pass the object being manipulated
    -typeDefs were exported to an own file
    -added short data type, renamed real to float, setting the value now checks for senseful values
    -introduced bitMasks replacing intflags: any type of definite size can be added using a table of named bits, those bit indexes are ignored when reading the parent data but extract their own value, so the different information does not get in conflict with each other, they are merged again when outputting, well it's practically only needed in the w3e
    -added mask functions for w3e (environment) and wtg (gui triggers), latter needs the latest TriggerData.txt from wc3
    -implemented a loading display since the w3e requires a lot of time to be processed
    -fixed some bugs
    -now checking if file format matches
     
  12. Ralle

    Ralle

    Owner

    Joined:
    Oct 6, 2004
    Messages:
    11,545
    Resources:
    21
    Tools:
    2
    Maps:
    5
    Tutorials:
    14
    Resources:
    21
    Thread opened on author request.
     
  13. Deaod

    Deaod

    Joined:
    Nov 18, 2007
    Messages:
    805
    Resources:
    12
    Maps:
    1
    Spells:
    11
    Resources:
    12
    If im understanding the spec correctly, it makes no sense.
    The highest bit of a short is not 0x4000, its 0x8000. 0x2000 is 8192, which also rules out having the short scale from 0b00_0000_0000 to 0b11_1111_1111, because thats not a range of 2^15 (ie. 32768).

    So, to make everything fit, shorts have 16 bits, of which only the 16th bit is reserved. Short have values ranging from 0b000_0000_0000 (0) to 0b111_1111_1111 (32767, 0x7FFF), and a bias of 0b100_0000_0000 (16384, 0x4000).

    Alternatively, the two highest bits are reserved, and the spec is wrong about the range of values allowed, the corrected range being from 0 to 16383, with 8192 being ground level.
     
  14. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,035
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    update:
    -fixed a problem with writing floats of value 0 (uses another return instruction but forgot to write the bytes for 0)
    -added masks for doo (doodad placements) and w3c (cameras)
    -included the latest TriggerData.txt (for gui triggers) in the package
    -added a doc.txt for the api and the sample code from above
     
  15. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,035
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    update:
    -added wct mask
    -added an application to transform wct (custom script trigger data) with help of the wtg (gui trigger data) to a file system corresponding to the view in trigger editor: you need to pass the .wct as first parameter and the .wtg needs to be within the same directory and the same file name except extension
     
  16. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,035
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    update:
    -added a version of Ladik's mpq editor to the package
    -added portLib.lua, which offers an interface for said mpq editor, so you can easily extract/add files from/to a map archive
    -added wtsParser.lua, which translates TRIGSTR_ expressions to the corresponding strings in the war3map.wts file, the war3map.wts should have been built from the World Editor though because the parsing pattern does not allow for every malformed-ness, I do not possess the exact file specs
    -common basic functions have been outsourced into the 'waterlua' package, a set of global modules that should be placed in %LUA_DEV%\lua\waterlua, so any lua script can load it by
    Code (Text):
    require 'waterlua'
    , it also requires 'LuaFileSystem' (lfs), a very common library for lua, which may help with platform-independence

    Both 'waterlua' and 'wc3binary' are now available on gitHub

    https://github.com/WaterKnight
     
  17. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,035
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    update:
    -multiple new unrefined stuff, basically this is not only about the binary files anymore but dealing with the wc3 formats in general, I do not have too much spare time in the near future, so I pushed the repositories as is
    -added wc3profile (the object text files under Units\), wc3skin (is just a wrapper for wc3profile atm because they are similar, for war3mapSkin.txt/war3mapMisc.txt/war3mapExtra.txt), wc3objMod (map object files like war3map.w3u, basically a better interface for wc3binaryFile)
    -in applications I have built a converter for objMod to slk/profile (using meta slks) since someone requested it
    -another application to create a custom TriggerData.txt from source TriggerData.txt or jass files for the World Editor
    -refactored the portLib a bit due to instabilities of Ladik's mpq editor, he also popped a new version for me
    -general refactoring, lots of asserts inserted

    Maybe at some other point I will be presenting the ways of mapmaking I had in mind before in detail because it basically uses these tools instead of World Editor work.
     
  18. Nestharus

    Nestharus

    Joined:
    Jul 10, 2007
    Messages:
    6,146
    Resources:
    8
    Spells:
    3
    Tutorials:
    4
    JASS:
    1
    Resources:
    8
    I'll use these tools in my new map framework thing.


    Extremely useful, exactly what I was looking for, especially the TriggerData thing, which I assume adds new triggers from JASS files. +rep
     
  19. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,035
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    As I said, it's still very unrefined, most of this stuff stems from situational necessity, like someone asked what's wrong with his map, so I mostly used it for debugging.

    The createGUITriggerData is currently only for the common.j (natives), if you want to add custom .js, you have to make an alternate regex analogue to line 866.
     
  20. moyackx

    moyackx

    Joined:
    Feb 15, 2006
    Messages:
    801
    Resources:
    7
    Maps:
    4
    Spells:
    2
    Tutorials:
    1
    Resources:
    7
    Awesome!!! I have evil intentions with this concept :)

    muahahahhhhaaaaaaa!!!