1. Updated Resource Submission Rules: All model & skin resource submissions must now include an in-game screenshot. This is to help speed up the moderation process and to show how the model and/or texture looks like from the in-game camera.
    Dismiss Notice
  2. DID YOU KNOW - That you can unlock new rank icons by posting on the forums or winning contests? Click here to customize your rank or read our User Rank Policy to see a list of ranks that you can unlock. Have you won a contest and still haven't received your rank award? Then please contact the administration.
    Dismiss Notice
  3. Weave light to take you to your highest hopes - the 6th Special Effect Contest is here!
    Dismiss Notice
  4. Lead your forces to battle in the 15th Techtree Contest. The call is yours, commander!
    Dismiss Notice
  5. Check out the Staff job openings thread.
    Dismiss Notice
Dismiss Notice
60,000 passwords have been reset on July 8, 2019. If you cannot login, read this.

MPQ Compressor

Discussion in 'Warcraft Editing Tools' started by LeP, Nov 16, 2014.

  1. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    465
    Resources:
    0
    Resources:
    0
    This is beta software
    Always make backups, it could easily eat your kitten, etc.


    What is it


    WarCraft 3 stores files inside the mpq compressed. The algorithm used is deflate.
    This tool uses a deflate implementation by Google which takes around 100 times longer but also produces something between
    3% and 10% smaller files.
    For most of you this probably is pretty useless but i assume that for people who scratch at the 8MB map limit every saved byte
    is precious.
    So this tool is no wonder tool so first use other mechanisms of lowering your maps size such as model and image compression.


    How to use it


    In the best case it's very easy to use as it expects only two files: the input map-file and the output map-file.
    Code (Text):
    $ compress.exe mymap.w3x mymap_min.w3x
    But you probably want to use it in conjunction with other tools such as the w3mapoptimizer.
    The w3mapoptimizer removes the listfile which is essential1 for this tool to work.
    If no listfile is found or your listfile isn't sufficient it will say so.
    So to use it with w3mapoptimizer you can provide an additional listfile:
    Code (Text):
    $ compress.exe -l /path/to/listfile.txt mymap.w3x mymap_min.w3x
    Additional options and tweaks are explained below.




    If you want to use this tool you should avoid map protectors.
    First of all: they don't work, if WarCraft can read your files so can everybody else.
    But secondly either if you use this tool after any protectors it probably wont work as i only parse valid MPQs as written by worldedit.
    Or if you use this tool first and then some protector it will probably undo the compression made by this.
    So you should use this tool as your last step in publishing your map.


    Options



    Option Default Explenation
    --threads, -t
    2
    The number of threads that are started. A good value would be the number of
    cores your CPU has.


    --iterations, -i
    15
    How many iterations are spent on compressing every file. Increasing this
    slows down the tool even further.


    --listfile, -l
    not set
    Additional listfile if the map internal listfile is not sufficient/non
    existent.


    --shift-size, -s
    15
    Sets the mpqs blocksize to 512*2shiftsize. Blizzard uses 3 and
    w3mapoptimizer recomends 7.

    --block-splitting-max
    15
    Maximum amount of blocks to split into (0 for unlimited, but this can give
    extreme results that hurt compression on some files).



    Internals



    MPQs have the ability to store files as a single block instead. If so the flag 0x01000000 is set.
    An earlier version of this tool used this flag but WarCraft wouldn't open the maps. So either this flag is not used
    for maps or i made a mistake. If someone knows about this please tell me.

    Update 12.02.2017:
    Updated zopfli and finally released version which can read any wc3 map.
    All the decompression routines except for zlib taken from StormLib.
    Zlib-decompression via miniz.

    ______________

    ^1: It could work without listfile but it would take quite some extra time implementing which i don't feel is neccessary as this tool is intended to be a dev-tool for mappers who have access to their maps listfile.
     

    Attached Files:

    Last edited: Feb 14, 2017
  2. Ezekiel12

    Ezekiel12

    Joined:
    Mar 13, 2012
    Messages:
    1,058
    Resources:
    0
    Resources:
    0
    Are you talking about the version flag? Wc3 uses version 1, version 2 is used since WoW.
     
  3. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    465
    Resources:
    0
    Resources:
    0
    No, i mean the per file flags stored in the blocktableentry. With 0x01000000 being MPQ_FILE_SINGLE_UNIT.
     
  4. PurgeandFire

    PurgeandFire

    Code Moderator

    Joined:
    Nov 11, 2006
    Messages:
    7,427
    Resources:
    18
    Icons:
    1
    Spells:
    4
    Tutorials:
    9
    JASS:
    4
    Resources:
    18
    Nice work. :)

    3-10% is pretty damn useful. It seems promising.

    In the mean time, it could be interesting to post benchmarks or examples of compression results (using random maps from the map section). It would be purely aesthetic info, just to show off your tool. Seeing the results directly is often more influential than seeing the %'s.

    edit: I would +rep, but I must spread.
     
  5. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    465
    Resources:
    0
    Resources:
    0
    I would love to test it on "real" maps as i only tested it on my small testmaps without much substance but most maps here are already optimized or worse so i lack the listfiles in the best case.

    If w3mapoptmizer doesn't encrypt the files i could add support for listfile-less operation but i'd have to check it first as i'd think thats the most common use case.

    Listfileless operation here would be ok as w3mapoptimizer already rebuilds the hash- and blocktable afaik and only encryption would be a hassle.

    But tbh. as already stated it shouldn't be neccessary as this is a dev-tool.
     
  6. IcemanBo

    IcemanBo

    Joined:
    Sep 6, 2013
    Messages:
    6,488
    Resources:
    22
    Maps:
    3
    Spells:
    11
    Template:
    1
    Tutorials:
    4
    JASS:
    3
    Resources:
    22
    Opened.
     
  7. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    465
    Resources:
    0
    Resources:
    0
    Updated. See first post. New source will come soon(tm).
    Dunno if this is still needed in the times of 128mb maps but whatever.
    Here's an example:
    Code (Text):

    * DotA v6.83d.w3x 8218959 bytes
    * minimized:      8040024 bytes
     
     
  8. pyf

    pyf

    Joined:
    Mar 21, 2016
    Messages:
    2,550
    Resources:
    2
    Tutorials:
    2
    Resources:
    2
    (note: I see the Puss in Boots trick is very effective)
    Thanks @LeP :thumbs_up:


    Quickly tested this new version with (12)DivideAndConquer.w3m, and now it works. The previous version of the compressor was crashing at step 13, that is with the file war3mapMap.blp.

    Stats:
    (12)DivideAndConquer.w3m
    - original: 349294 bytes
    - optimized: 277962 bytes
    HDD space gain: 20.43% (-71332 bytes), using default settings


    Out of curiosity, did you take some inspiration from the AdvanceCOMP source code?

    128 MB ought to be enough for anybody?
    ...
    Hmm... I remember reading something that sounded similar, a very very long time ago... :wink:
     
  9. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,996
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    Might be worth noting that MPQ compression was not really meant to obtain high compression ratios, just some decrease in file size. The way it uses compressed blocks for a file means that one either needs to discard random access performance by making a huge block size or suffer decreased compression ratios as a result of finite block size.
     
  10. DracoL1ch

    DracoL1ch

    Joined:
    Dec 12, 2010
    Messages:
    1,984
    Resources:
    2
    Tutorials:
    2
    Resources:
    2
    with the amount of bugs which came along with 27b its easy to say long live 26, especially if there's not that many special snowflakes with win10
    although I cant make this tool run over my dota, 'cause it says there's non-zipped files inside :C
     
  11. pyf

    pyf

    Joined:
    Mar 21, 2016
    Messages:
    2,550
    Resources:
    2
    Tutorials:
    2
    Resources:
    2
    This is unexpected...
    Which DotA version are you using?

    Did you try out the new, updated compress.exe?
     
  12. DracoL1ch

    DracoL1ch

    Joined:
    Dec 12, 2010
    Messages:
    1,984
    Resources:
    2
    Tutorials:
    2
    Resources:
    2
    nvm, it worked on vexorian'd map, but not the one I working on. difference is the lack of .j file inside (local files). anyway testing now
    result: 8084 vs 7968
    116Kb saved
    not bad

    the error I said about:
    [​IMG]
    disregard listfile
     
  13. pyf

    pyf

    Joined:
    Mar 21, 2016
    Messages:
    2,550
    Resources:
    2
    Tutorials:
    2
    Resources:
    2
    You can use the --listfile optional parameter to use an additional external listfile, if the internal listfile is not sufficient/non existent.
     
  14. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    465
    Resources:
    0
    Resources:
    0
    I have attached the new src in the OP. Also added a new .exe compiled with -flto which may result in a bit faster computation.


    That error looks like the old version.
     
  15. DracoL1ch

    DracoL1ch

    Joined:
    Dec 12, 2010
    Messages:
    1,984
    Resources:
    2
    Tutorials:
    2
    Resources:
    2
    basically it's just the computing algo change, and it can't affect user-side anyhow, right? just to clear things out
     
  16. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    465
    Resources:
    0
    Resources:
    0
    Yes, only the way the files are stored in the mpq archive is changed. The file content should stay the same (otherwise it's a bug)
     
  17. TriggerHappy

    TriggerHappy

    Code Moderator

    Joined:
    Jun 23, 2007
    Messages:
    3,761
    Resources:
    22
    Spells:
    11
    Tutorials:
    2
    JASS:
    9
    Resources:
    22
    Any idea why it says missing "libwinpthread-1.dll" when I try to run it?
     
  18. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    465
    Resources:
    0
    Resources:
    0
    Because i develop mostly on unix systems and cygwin on windows. To avoid such issues i cross-compile to windows from linux and/or freebsd. That seemed to have failed.
    I think i had a working setup once but i can't really get it to work now.
    Installing cygwin and running it from there could solve it. Or compile it yourself on a mingw/unix-system.
    The source is available.
    If you've got problems compiling feel free to message me.
     
  19. GhostWolf

    GhostWolf

    Joined:
    Jul 29, 2007
    Messages:
    4,922
    Resources:
    2
    Tools:
    1
    Tutorials:
    1
    Resources:
    2
    Maps can't have MPQ_FILE_SINGLE_UNIT files, but normal archives can.
    Maps themselves cannot be compressed and must be stored with FILE_EXISTS (e.g. in a campaign).
    Why? Blizzard.
     
  20. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,996
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    I think that flag was added in newer MPQ specifications. I recall only seeing it appear in the specifications around D3/SC2 time period.
    Probably due to how their MPQ API works. If a file is not compressed then it is stored as a continuous block inside the archive file. By using a file position offset one can load it from the host MPQ file like the host MPQ archive was loaded, just at a different offset. Otherwise they would need an API that abstracts file systems and supports logically nesting a file system archive inside a file system archive.

    Even some compression APIs might have difficulty handling the nesting efficiently. Not that it even makes sense to nest compression like that in the first place from a compression logic point of view.
     
    Last edited: May 4, 2020