Java MPQ Library, Anyone?

[Retera]

Does anybody know of a Java MPQ library?

The recent versions of the Matrix Eater -- my Java warcraft 3D modeling software -- rely on JStormLib to open MPQ files, and the version of JStormLib that I had appears to not work on my Technical Preview of Windows 10. I was thinking a native Java MPQ reader would be a fantastic replacement for my Matrix Eater software, as this would allow it to function properly in any environment that was capable of turning on the program.

Thanks!

 

[PurgeandFire]

scotta made one for bnubot:
https://code.google.com/p/bnubot/source/browse/trunk/BNUBot/src/net/bnubot#bnubot/util/mpq

But when people write their own mpq wrappers, they tend to be very limited in functionality. They'll often support limited functionality, and not much else. But definitely give that a look and see if it has the functions you need.

Peeeeq on our site also wrote a CLI Java wrapper for StormLib. It is cross-platform (for the major platforms at least--it has StormLib compiled for linux/windows/os x), but it isn't a port. It's just used for extraction/injection for Wurst, so that is what it supports. But they have bindings for the other functions, so you may be able to expand it!

Which mpq functions do you need in particular?

 

[Crigges]

Well there are actually three java libarys out there:
The first one is JMpq-v1 written by peq using Stormlib via JNI. This one made some problems so peq and me wrote JMpq-v2. A pure java based mpq implementation. This version works fine as long as you only use ZLib compression. It also uses a lot of RAM, this is why i started to develop JMpq-v3. This one is still under development and only file extract works so far.

If you are just going to extract files out of a mpq you should use v3 otherwise you need to use v2.

If you got any problems you can contact peq or me anytime.

Edit:
@PurgeandFire bnubot will not work, since it only impements the imploded compression algorithm

 

[Dr Super Good]

I own a Java MPQ library (this is possibly an old version, I can check if people show interest). It can extract all WC3 files very efficiently (includes decompressors for all the crazy compression types written in JAVA). It can also extract files from SC2 map MPQs (although the HET and BET tables are not supported).

Some of the decompression libraries are incomplete or poorly optimized. However the not implemented features were not used in WC3 and also they work fast enough to decompress one of the main MPQ files in a matter of seconds once the archive file is cached in memory.

In retrospect I messed up the API. It should be mounted as a file system using the standard file system API so accessing files in a MPQ should be no harder than reading a file from disk. This would use the NIO interface obviously (the old IO interface is too limited and has inbuilt adapters anyway).

The key aspect for performance is to try and cache buffers. The same two buffers can be used to decompress all blocks of a file.

 

[Crigges]

@Dr Super Good

Your libary looks nice so far. I guess i will use your compression implementations in JMpq-v3, if you don't mind. So pls check if everything is up to date

 

[Dr Super Good]

Your libary looks nice so far. I guess i will use your compression implementations in JMpq-v3, if you don't mind. So pls check if everything is up to date

Only decompression is implemented, and the PKWare one lacks text decompression (WC3 does not use that, not sure if older games also do not and newer ones do not use PKWare). The Huffman also is missing some kind of lookup table optimization (which was practically impossible to translate to Java due to bad "assembly" based C++ source code). Most of the other compression and decompression (LZMA, BZip2, etc) can be taken from standard libraries scattered around the internet.

I have come to the conclusion that a Java MPQ implementation should be based around a File System. This is so that standard I/O techniques (nio) can access the files inside a MPQ archive as if they were files on the default file system. This also means that MPQ support can be used without the need to import any non-standard classes since the file system API would cover it.

Obviously one can expect it to have unusual behaviour. For example if no listfile is present then directory streams and other folder/file search methods would not work. If no attribute file is present then no files will have attributes. Folders cannot have attributes. Each MPQ archive would probably be a single File Store in the File System.

By the way the link provided for "JMpq-v3" is broken. I do not think the URL is correct.

 

[Retera]

Thanks to all of you guys! It looks like you have quite the sort of things I was looking for. Essentially, the matrix eater has a wrapper class that has a "hasGameFile" and "getGameFile" function that look through the various war3x/war3/war3xlocal/war3patch archives, and any library for grabbing files that I can use to implement that class will work.

The .dll JNI stuff was giving me trouble (issues with UnsatisfiedLinkError due to a different version of windows not having certain linked DLL API stuff I'm not familiar with), so the pure-Java alternatives are probably the way I'm going to go.

 

[WaterKnight]

As long as nobody offers a full solution, I will stick to Ladik's. Unless you want to process lots of data, there is no problem to have it separated.

 

[oger-lord]

Crigges is working on the full version, he is just very slow.

 

[LeP]

Crigges is working on the full version, he is just very slow.

just like his library ☜(ﾟヮﾟ☜)

 

[Dr Super Good]

A full solution would involve the nio FileSystem interface scheme. This would allow you to mount MPQs as a file system and use them without dealing with any non-standard classes.

There are two problems doing such a thing. First is coming up with a sensible URI scheme. The second is trivializing multi-threaded read and writing so that multiple files can be written seemingly in parallel (not necessarily truly in parallel) while still producing consistent results.

 

[Retera]

I would just like to reiterate how much I appreciate your guys' help. I was looking at trying to do some updates to the Matrix Eater MDL modeling software I wrote a while ago, which uses a Java interface that essentially consists of two functions:

	File getFile(String filepath);
	boolean has(String filepath);

... and after borrowing Dr Super Good's MPQ source code from up there, I was able to implement a working version of this interface that does not require DLLs, .NET libraries, or other total nonsense, and the library even allows read-only opening of the MPQs (so that the program does not need extra privileges to extract files from the Warcraft installation).

I may take some time and investigate the other options listed as well, but one way or the other you guys have done an amazing job solving the issue I was facing.

 

[PurgeandFire]

Crigges updated his library:
http://www.hiveworkshop.com/forums/warcraft-editing-tools-277/jmpq-v3-pure-java-mpq-library-270680/

I know you got DSG's code working, but I just thought I'd mention it.

 

[Dr Super Good]

I know you got DSG's code working, but I just thought I'd mention it.

Still not a FileSystem provider though...

I do admit that is on my list of things to do if I can ever get blp implemented properly.