Updated JMPQ

[Frotty]

Warcraft-ready, native java mpq editor.
Supports opening and modifying mpq archives. Warcraft3 maps and mpqs are confirmed working.

Most notable features:

• Supports decompressing all files from wc3 game data
• Supports building mpqs
• Avoids most map protection with FORCE_V1 mode

Artifact: JitPack | Publish JVM and Android libraries
Source: GitHub - inwc3/JMPQ3: Native java MoPaQ implementation.

Feel free to comment.

 

[TriggerHappy]

Good job.

Unfortunately you don't support PKWARE which I think is used for some files in the main MPQs and for every RoC beta map. Also, as far as I know bzip2 is used by Starcraft II maps.

 

[Dr Super Good]

Seems he has not included my rough PKWare decompressor, but then again it was incomplete (could not decompress text files, but WC3 release never had any...).

It also still pointlessly makes a temporary copy of the archive when one might just want to read from it. Any speed improvements from copying are fake, it just seems to make it faster due to the file being in the file cache as it was recently written. Read only mode is kind of important to make sure one does not accidently make changes to a MPQ archive when not desirable, and also because one might not have write access on some MPQ files but can still read them.

Although memory mapping may seem a clever solution to reading chunks, it is not. The actual memory mapping calls are very slow and so only trade off if one maps entire large files with a long map retention time (eg for a high performance database engine). For small reads such as involved here it would be better to read into a freshly allocated ByteBuffer. This also removes the dependency on memory mapping support from the file system, which would help make way for it to read directly from the file.

Manipulating the actual files should be done using some sort of FileChannel or SeekableByteChannel. This allows for partial reads to only have to process the required file blocks instead of all file blocks. It also fits with a lot of standard IO code. When writing all writes are pushed directly to a temporary file, and all reads come from the temporary file (only if at least 1 byte was written). On closure the temporary file is then written to a file in the MPQ.

Unfortunately MPQs are not really designed for random modifications. This is one of the reasons Blizzard moved to CASC for all their modern game data.

 

[Frotty]

@TriggerHappy I get no exceptions on the main MPQs for wc3, so I don't think pkware is used (but the ADPCM/huff is probably faulty. exported wavs are playable, but not identical to what ladik exports). When I need sc2 stuff I might add it

@Dr Super Good The way jmpq works is horrid and I would take a different route were I to make it from scratch. But I was too lazy for that and just fixed crigges' mess into something somewhat usable.

 

[Dr Super Good]

@TriggerHappy I get no exceptions on the main MPQs for wc3, so I don't think pkware is used (but the ADPCM/huff is probably faulty. exported wavs are playable, but not identical to what ladik exports). When I need sc2 stuff I might add it

It is possible they have phased out PKWare compression with the recent patches as that would mean they do not need to deal with the licence nonsense associated with it, and free/open source compression types are better anyway.

Are you are using something based on my implementation of ADPCM/huff decompressor? If there is considerable difference between it and what MPQEdit exports then there might be a bug with the ADPCM code. It is unlikely there is a bug with the Huffman decompression code as that is lossless compression stacked on other compression so any error there would result in garbage out. It is also possible that MPQEdit performs some kind of filtering or audio processing on extracted sounds, who knows.

@Dr Super Good The way jmpq works is horrid and I would take a different route were I to make it from scratch. But I was too lazy for that and just fixed crigges' mess into something somewhat usable.

I could look into improving it. At least removing the required copy to open nonsense.

EDIT:
Cannot build it. It appears to not be self-contained and requires other external libraries to build, which are not mentioned clearly eg "com.jcraft".

 

[Frotty]

@Dr Super Good ever heard of maven/gradle?
It uses gradle and is completely contained, as you can see at the CI here Travis CI - Test and Deploy Your Code with Confidence
If you use a proper IDE, you can import it as gradle project.
Otherwise just use the wrapper (type "gradlew build" in terminal to build classes and get jar)

Same goes for test coverage, calculated at every commit inwc3/JMPQ-v3 | Coveralls - Test Coverage History & Statistics

If you push a git tag it will then deploy a jar to the github releases page I linked above.

And yes, I took your decompression code from some zip you uploaded some time (I can credit you if you desire. I just looked if someone had already done this and found your stuff)

e: If you need any more help getting this to work feel free to let me know.

 

[Dr Super Good]

@Dr Super Good ever heard of maven/gradle?

I know of maven, but not gradle. I was trying to import it into Eclipse for editing/testing purposes as otherwise editing in notepad++ or such is a pain. I know I could just modify the files out of eclipse but that sort of defeats the purpose of having IDEs. I also know that there are better Java IDEs but I am kind of used to Eclipse.

If you use a proper IDE, you can import it as gradle project.

Apparently Eclipse can support both maven and gradle via plugins. I will update my historic version of Eclipse and then get the plugins and try again. Chances are it will work perfectly then.

If you push a git tag it will then deploy a jar to the github releases page I linked above.

Not too sure what you mean by this. Do I not have to send a pull request first to merge in any changes I make to master?

 

[Frotty]

Yes, recent versions of eclipse come with buildship iirc. It worked for me when I was still using eclipse, the old gradle plugin was quite painful though.

Dunno why I added the random tag thought.
Maybe to emphasize that it's all an automated process. If you make a PR, travis will build it, run tests and coverage report and mark validity on the github thread iirc, but not release it.

 

[Dr Super Good]

I got everything working eventually.

I have pushed some commits that allow JMpqEditor to be able to open archive files without having to make a copy of the archive file first. Other components still use a lot of temporary files but that is a job for another day.

I also altered the API slightly to allow for open options to specify read only mode (JMpqEditor will not alter the archive file in any way, useful for the WC3 MPQ files) and to force opening as MPQ format version 0 (now required to open map archives that have been deliberately damaged, eg by Spazzler). It is not possible to have MPQ format version 0 apply automatically as that would break future support for MPQ format version 1, 2 and 3.

Internally some ground work was laid for full large sized file support. MPQ user data headers are roughly parsed and used to locate the MPQ archive header. Some values from MPQ format versions newer than 0 are now loaded, but currently cannot be written out.

 

[Frotty]

Thanks to DSG jmpq now supports the features mentioned, namely readonly mode and userdata parsing.
I updated the first post.

 

[Dr Super Good]

It seems that StormLib updated their ADPCM decompressor since I ported it. I will see if I can revise it.

EDIT: There are 2 opcode commands present in their old reverse engineered ADPCM decompressor that are missing from their new one. However next to those the code seems functionally equivalent. Could some example files be posted for hexing?

 

[Frotty]

Quick update:

• jmpq is available as artifact again via Jitpack JitPack | Publish JVM and Android libraries
• some critical bugs handling 0byte files and incompressible files have been fixed (thanks to WaterKnight)
• jmpq now supports optional recompression via zopfli which can save about 2-5% of space

@Dr Super Good I will look into pushing an example for the wavs soonish :}

 

[Dr Super Good]

Finally got around to rewriting the rest of the MPQ cryptographic stuff to be more Java conformant, complete with documentation. The classes should perform comparably but now have the advantage of being able to process buffers containing only part of the data as well as having far fewer magic numbers and constants.

Until the wav example is up I will look into cleaning up/rewriting the HashTable class so at the very least it no longer needs to generate a temporary file.