Tool to extract w3x file, replace/edit any file, then create new w3x file.

[sethmachine]

Hi,

I want to be able to edit/replace specific files within a Warcraft III TFT map file, i.e. .w3x. Ideally this runs on command line and is platform independent (Mac support).

For example, I tried using MPQExtractor (see: GitHub - Kanma/MPQExtractor: A command-line tool to extract files from MPQ archives). It let me list and extract files from a w3x file, but I couldn't edit the files and create a new w3x file with it.

war3map.w3e
war3map.w3i
war3map.wtg
war3map.wct
war3map.wts
war3map.j
war3map.shd
war3mapMap.blp
war3map.mmp
war3map.wpm
war3map.doo
war3mapUnits.doo
war3map.w3r
war3map.w3c
war3mapMisc.txt
war3mapSkin.txt

What tool exists that will let me extract these files, swap in my own (or edit them, e.g. the war3map.j file) and then create a new valid w3x file that can be played or opened in the World Editor?

 

[HappyTauren]

get Ladik's MPQ editor

 

[sethmachine]

Looks like a Windows based GUI. I'm looking for easy to understand code with an example that replaces the war3map.j in an existing w3x file.

 

[HappyTauren]

Ah, I didn't see the CLI part. Well, if you need command line multi platform access, there's no better than this.

 

[sethmachine]

Yes, that looks very promising.

I went through the painstaking process of getting the C++ API to work. I was able to replace the war3map.j file by removing it, then adding a new one and then I saw the changes take place in the map I played (I changed the start location by some random offset).

What I also saw is that the MPQ size (w3x) increased from 21 KB to 26 KB, even though I didn't add any new lines to the war3map.j. However, I believe I opted for zero compression.

Does it matter what compression flags are used when writing to an MPQ? I notice this Java library uses "int dwCompression = StormLib.MPQ_COMPRESSION_ZLIB;" even though the Storm Lib API lists several possible compression flags:
MPQ_COMPRESSION_HUFFMANN
(0x01) Use Huffman compression. This bit can only be combined with MPQ_COMPRESSION_ADPCM_MONO or MPQ_COMPRESSION_ADPCM_STEREO.
MPQ_COMPRESSION_ZLIB
(0x02) Use ZLIB compression library. This bit cannot be combined with MPQ_COMPRESSION_BZIP2 or MPQ_COMPRESSION_LZMA.
MPQ_COMPRESSION_PKWARE
(0x08) Use Pkware Data Compression Library. This bit cannot be combined with MPQ_COMPRESSION_LZMA.
MPQ_COMPRESSION_BZIP2
(0x10) Use BZIP2 compression library. This bit cannot be combined with MPQ_COMPRESSION_ZLIB or MPQ_COMPRESSION_LZMA.
MPQ_COMPRESSION_SPARSE
(0x20) Use SPARSE compression. This bit cannot be combined with MPQ_COMPRESSION_LZMA.
MPQ_COMPRESSION_ADPCM_MONO
(0x40) Use IMA ADPCM compression for 1-channel (mono) WAVE files. This bit can only be combined with MPQ_COMPRESSION_HUFFMANN. This is lossy compression and should only be used for compressing WAVE files.
MPQ_COMPRESSION_ADPCM_STEREO
(0x80) Use IMA ADPCM compression for 2-channel (stereo) WAVE files. This bit can only be combined with MPQ_COMPRESSION_HUFFMANN. This is lossy compression and should only be used for compressing WAVE files.
MPQ_COMPRESSION_LZMA
(0x12) Use LZMA compression. This value can not be combined with any other compression method.


Which compression flag should I use if I also edit the terrain or doodad file? Or does it only matter if I want to keep the w3x file size smaller?

 

[GhostWolf]

tl;dr zlib is fine.

MPQ archives are containers, much like zip archives.
They are not compressed themselves, but rather the data inside them, represented as files, can be compressed, using all sorts of supported algorithms (or can be left flat with no compression at all).
It is usually a lot easier to support reading different compression types than writing, so this specific library might have opted to only support compressing with zlib, which is fairly simple.
It might also support more, I don't know, but even if it does, it will not necessarily be better than using zlib, it completely depends on the data.
If more than one compression type is supported, ideally you still don't need to change it in any way, since the library needs to try all supported types to see which one would yield the smallest result (a part of why compressing is harder than decompressing).
In the extreme cases where the compressed data is bigger than the uncompressed data for all supported compression types, the data must be saved flat as-is. This will generally never happen, since you need really small files (or rather sectors, since the files are generally saved in chunks of a constant size, e.g. 4KB) for that, on the range of a few bytes.

Just as a side tidbit, other libraries (e.g. mine ) usually don't support all types, Ladik's I believe does read all of them. Zlib is the easiest one to support, so probably every library does. If this matters to you, take it into consideration.

 

[sethmachine]

I'm confused, @GhostWolf, or I misunderstand compression.

If I compress an individual file foobar.j (within the archive) using compression algorithm A, then whatever program that wants to read foobar.j needs to know which algorithm was used to compress it. So shouldn't the choice matter, e.g. if Blizzard isn't expecting the war3map.j file to be compressed with MPQ_COMPRESSION_SPARSE, then the map won't load?

Unless what is typical is try each (un)compression algorithm and see which one works? Otherwise there needs to be metadata telling the program "here is the algorithm used to compress this file". Is that written in the header of the compressed file?

 

[HappyTauren]

You needn't concern yourself with any of that, just test to see what works and what doesn't. Using default settings on most things like this is just fine, anyways.

 

[GhostWolf]

Each sector contains metadata in a single byte, there is no guesswork involved.
If you want to get even more technical, the byte is actually a bitmask that can define a chain of algorithms, and thus you see comments of things you can combine or can't combine.

On a side note, BZIP2, Sparse, and LZMA, are all actually for MPQ2, formerly used by WoW.

In the end of the day, for your typical files, zlib is all you need, and it's the safest bet.

 

[sethmachine]

I've put together a Python library to wraps both the StormLib C++ API and also ports some of the work @eejin. I've been able to read and write to MPQ/W3X files without issue. The Python code is not yet complete, but I find it much easier to work with than the C++ or Java code.

See: GitHub - sethmachine/hive-world-editor: Python translation of key parts of the C++ HiveWE project.

I created a context manager for handling the opening and closing of MPQ Archives. I think this would be the Pythonic way to handle such an object. For example, look how simple it is to extract all the files from an MPQ archive:

import archive

infile = 'data/test/Test.w3x'
mode = 'r' #use 'w' to enable writing

with archive.open_archive(infile, mode) as a:
    listfile = 'list.txt'
    a.extract_list_file(listfile)
    outdir = 'archive-extracted'
    a.extract_all_files(outdir, listfile)