MPQ Compressor

[LeP]

This is beta software
Always make backups, it could easily eat your kitten, etc.

What is it

WarCraft 3 stores files inside the mpq compressed. The algorithm used is deflate.
This tool uses a deflate implementation by Google which takes around 100 times longer but also produces something between
3% and 10% smaller files.
For most of you this probably is pretty useless but i assume that for people who scratch at the 8MB map limit every saved byte
is precious.
So this tool is no wonder tool so first use other mechanisms of lowering your maps size such as model and image compression.

How to use it

In the best case it's very easy to use as it expects only two files: the input map-file and the output map-file.

$ compress.exe mymap.w3x mymap_min.w3x

But you probably want to use it in conjunction with other tools such as the w3mapoptimizer.
The w3mapoptimizer removes the listfile which is essential¹ for this tool to work.
If no listfile is found or your listfile isn't sufficient it will say so.
So to use it with w3mapoptimizer you can provide an additional listfile:

$ compress.exe -l /path/to/listfile.txt mymap.w3x mymap_min.w3x

Additional options and tweaks are explained below.

---

If you want to use this tool you should avoid map protectors.
First of all: they don't work, if WarCraft can read your files so can everybody else.
But secondly either if you use this tool after any protectors it probably wont work as i only parse valid MPQs as written by worldedit.
Or if you use this tool first and then some protector it will probably undo the compression made by this.
So you should use this tool as your last step in publishing your map.

Options

Option	Default	Explenation
--threads, -t	2	The number of threads that are started. A good value would be the number of cores your CPU has.
--iterations, -i	15	How many iterations are spent on compressing every file. Increasing this slows down the tool even further.
--listfile, -l	not set	Additional listfile if the map internal listfile is not sufficient/non existent.
--shift-size, -s	15	Sets the mpqs blocksize to 512*2shiftsize. Blizzard uses 3 and w3mapoptimizer recomends 7.
--block-splitting-max	15	Maximum amount of blocks to split into (0 for unlimited, but this can give extreme results that hurt compression on some files).

Internals

MPQs have the ability to store files as a single block instead. If so the flag 0x01000000 is set.
An earlier version of this tool used this flag but WarCraft wouldn't open the maps. So either this flag is not used
for maps or i made a mistake. If someone knows about this please tell me.

Update 12.02.2017:
Updated zopfli and finally released version which can read any wc3 map.
All the decompression routines except for zlib taken from StormLib.
Zlib-decompression via miniz.

______________

^1: It could work without listfile but it would take quite some extra time implementing which i don't feel is neccessary as this tool is intended to be a dev-tool for mappers who have access to their maps listfile.

 

[Ezekiel12]

MPQs have the ability to store files as a single block instead. If so the flag 0x01000000 is set.

Are you talking about the version flag? Wc3 uses version 1, version 2 is used since WoW.

 

[LeP]

Are you talking about the version flag? Wc3 uses version 1, version 2 is used since WoW.

No, i mean the per file flags stored in the blocktableentry. With 0x01000000 being MPQ_FILE_SINGLE_UNIT.

 

[PurgeandFire]

Nice work.

3-10% is pretty damn useful. It seems promising.

In the mean time, it could be interesting to post benchmarks or examples of compression results (using random maps from the map section). It would be purely aesthetic info, just to show off your tool. Seeing the results directly is often more influential than seeing the %'s.

edit: I would +rep, but I must spread.

 

[LeP]

Nice work.

3-10% is pretty damn useful. It seems promising.

In the mean time, it could be interesting to post benchmarks or examples of compression results (using random maps from the map section). It would be purely aesthetic info, just to show off your tool. Seeing the results directly is often more influential than seeing the %'s.

edit: I would +rep, but I must spread.

I would love to test it on "real" maps as i only tested it on my small testmaps without much substance but most maps here are already optimized or worse so i lack the listfiles in the best case.

If w3mapoptmizer doesn't encrypt the files i could add support for listfile-less operation but i'd have to check it first as i'd think thats the most common use case.

Listfileless operation here would be ok as w3mapoptimizer already rebuilds the hash- and blocktable afaik and only encryption would be a hassle.

But tbh. as already stated it shouldn't be neccessary as this is a dev-tool.

 

[IcemanBo]

Opened.

 

[LeP]

Updated. See first post. New source will come soon(tm).
Dunno if this is still needed in the times of 128mb maps but whatever.
Here's an example:

* DotA v6.83d.w3x 8218959 bytes
* minimized:      8040024 bytes

 

[pyf]

(note: I see the Puss in Boots trick is very effective)
Thanks @LeP


Quickly tested this new version with (12)DivideAndConquer.w3m, and now it works. The previous version of the compressor was crashing at step 13, that is with the file war3mapMap.blp.

Stats:
(12)DivideAndConquer.w3m
- original: 349294 bytes
- optimized: 277962 bytes
HDD space gain: 20.43% (-71332 bytes), using default settings


Out of curiosity, did you take some inspiration from the AdvanceCOMP source code?

Dunno if this is still needed in the times of 128mb maps but whatever.

128 MB ought to be enough for anybody?
...
Hmm... I remember reading something that sounded similar, a very very long time ago...

 

[Dr Super Good]

Might be worth noting that MPQ compression was not really meant to obtain high compression ratios, just some decrease in file size. The way it uses compressed blocks for a file means that one either needs to discard random access performance by making a huge block size or suffer decreased compression ratios as a result of finite block size.

 

[DracoL1ch]

Updated. See first post. New source will come soon(tm).
Dunno if this is still needed in the times of 128mb maps but whatever.
Here's an example:

* DotA v6.83d.w3x 8218959 bytes
* minimized:      8040024 bytes

with the amount of bugs which came along with 27b its easy to say long live 26, especially if there's not that many special snowflakes with win10
although I cant make this tool run over my dota, 'cause it says there's non-zipped files inside :C

 

[pyf]

although I cant make this tool run over my dota, 'cause it says there's non-zipped files inside

This is unexpected...
Which DotA version are you using?

Did you try out the new, updated compress.exe?

 

[DracoL1ch]

This is unexpected...
Which DotA version are you using?

Did you try out the new, updated compress.exe?

nvm, it worked on vexorian'd map, but not the one I working on. difference is the lack of .j file inside (local files). anyway testing now
result: 8084 vs 7968
116Kb saved
not bad

the error I said about:

disregard listfile

 

[pyf]

You can use the --listfile optional parameter to use an additional external listfile, if the internal listfile is not sufficient/non existent.

 

[LeP]

I have attached the new src in the OP. Also added a new .exe compiled with -flto which may result in a bit faster computation.

nvm, it worked on vexorian'd map, but not the one I working on. difference is the lack of .j file inside (local files). anyway testing now
result: 8084 vs 7968
116Kb saved
not bad

the error I said about:

disregard listfile

That error looks like the old version.

 

[DracoL1ch]

basically it's just the computing algo change, and it can't affect user-side anyhow, right? just to clear things out

 

[LeP]

basically it's just the computing algo change, and it can't affect user-side anyhow, right? just to clear things out

Yes, only the way the files are stored in the mpq archive is changed. The file content should stay the same (otherwise it's a bug)

 

[TriggerHappy]

Any idea why it says missing "libwinpthread-1.dll" when I try to run it?

 

[LeP]

Any idea why it says missing "libwinpthread-1.dll" when I try to run it?

Because i develop mostly on unix systems and cygwin on windows. To avoid such issues i cross-compile to windows from linux and/or freebsd. That seemed to have failed.
I think i had a working setup once but i can't really get it to work now.
Installing cygwin and running it from there could solve it. Or compile it yourself on a mingw/unix-system.
The source is available.
If you've got problems compiling feel free to message me.

 

[GhostWolf]

Maps can't have MPQ_FILE_SINGLE_UNIT files, but normal archives can.
Maps themselves cannot be compressed and must be stored with FILE_EXISTS (e.g. in a campaign).
Why? Blizzard.

 

[Dr Super Good]

Maps can't have MPQ_FILE_SINGLE_UNIT files, but normal archives can.

I think that flag was added in newer MPQ specifications. I recall only seeing it appear in the specifications around D3/SC2 time period.

Maps themselves cannot be compressed and must be stored with FILE_EXISTS (e.g. in a campaign).
Why? Blizzard.

Probably due to how their MPQ API works. If a file is not compressed then it is stored as a continuous block inside the archive file. By using a file position offset one can load it from the host MPQ file like the host MPQ archive was loaded, just at a different offset. Otherwise they would need an API that abstracts file systems and supports logically nesting a file system archive inside a file system archive.

Even some compression APIs might have difficulty handling the nesting efficiently. Not that it even makes sense to nest compression like that in the first place from a compression logic point of view.