MPQ compression when adding a file

[Barade]

Hey,
at the moment I am implementing the method Archive::addFile() in wc3lib to add write support for MPQ archives. Uncompressed and encrypted files seem to work so far.
My question is: Is there any simple way to detect which compression I should use when adding a file to an MPQ archive?
For example by its extension or content.
In the StormLib code I found:

static DWORD DefaultDataCompression = MPQ_COMPRESSION_PKWARE;

Of course you can always define the compression yourself but when you use an MPQ Editor and add a whole directory it would be annoying as hell.

 

[TriggerHappy]

The only file type I know of that uses a specific compression is .wav files.

Aren't the rest of the files compressed however the user compressed it in whatever the program their using?

Can't just default with gzlib?

 

[Barade]

That's the question. I will probably use some default compression like StormLib does. Other things might be too complicated. I guess zip compression is very useful if I can detect a text file.

 

[PurgeandFire]

Feel free to look up benchmarks comparing them, but roughly: bzip2 usually takes longer, but it can provide better compression.

I would say use zlib as a default. I honestly use it for everything. I would only use a longer compression if I'm publishing my project (I think LeP--or someone else?--made a tool that does exactly that).

You can't use LZMA or SPARSE because that is sc2. As TriggerHappy mentioned, there are some specific lossy compression algorithms for wav. And I think PKWARE DCL was only used in Diablo I. So that just leaves zlib (gzip) and bzip2.

Both are very good and have similar goals (pattern matching), so I don't think you can say one is better for a particular wc3 format. That is an interesting question though. I don't know too much into bzip2, but I'll look into it.

 

[LeP]

Yeah just go with zlib. It's the best of the supported formats.
Maybe the new patch will introduce new compression methods but for now zlib should be the best almost all the time and it is fast enough to just use it.

And yeah, if you want maximum reduction i have written this but you don't want to run it everytime you add a file to your mpq.

 

[pyf]

Fyi, for those curious about zlib and / or data compression
http://zlib.net/

 

[Dr Super Good]

It is illegal for you to use MPQ_COMPRESSION_PKWARE as you lack the licence for it. In most regions you can decompress for compatibility purposes but compression is now allowed. Or at least was not allowed at some stage.

You are meant to use ZIP and other open source compression. For wav files use the PWM based compression with Huffman output. BLP files of JPEG content gain very little from compression so can be uncompressed (only the headers compress, as the JPEG content is already compressed).

And I think PKWARE DCL was only used in Diablo I. So that just leaves zlib (gzip) and bzip2.

WC3 used it for most files. It was part of their copy protection. Just like how most MPQ files are encrypted. It is illegal in Europe and America to use PKWARE compression due to licencing, or at least was back when WC3 was made. MPQEdit gets around it because I think they are Russian/from that area so not subject to such restrictions. It is legal to decompress, but only decompress like how 7Zip does rar files. For all normal purposes ZIP obsoleted PKWARE compression since it was made by the same guys who realized they made a big mistake with regard to licencing, it is only used due to its legal obligations in commercial development.

MPQ compression in version 0 are flags. This means you can run all the compression algorithms you want and even chain them. WAV files do this in that they PWM compress then use Huffman on the output for better compression. Newer, but mostly obsolete, MPQ versions specify strict compression flags which mean certain compression orders and types so you cannot chain every compression together. Chaining compressions can be useful and is commonly used for artistic assets, however chaining all compressions together is not useful as it will mostly result in larger file sizes.

For Diablo files the compression used was declared in the file table with a special flag which specifically meant PKWARE compression was used. WC3 compression uses a different system with a different flag in the file table representing compression and the first byte of every block representing a compression flag array which is used to select which decompression algorithms to use. Modern MPQ versions retain the WC3 approach and file table flag but the first byte compression flag array has a different meaning.

Off topic... Your BLP component states the following...

RGBA color space

BLP files, at least in WC3, are in linear RGB colour space. This distinction is important because normal display colour space is sRGB which is non-linear (~2.2 gamma). Alpha is always linear and not visible so is not really considered part of a colour space (humans cannot see alpha, only the results of alpha). A BLP file needs to undergo colour space conversion for it to be displayed correctly. Additionally normal images need to undergo a colour space conversion to be made into correct blp files.

 

[Barade]

You realize that StormLib uses pkware by default as I found it in the source code? Besides I use the same source code for pkware compression as StormLib (copy paste). I'd like to contact the author and ask for permission directly.

I know about the chain. That's why I am asking which ones should be used. Are you sure not every chain can be used? I thought the only exception is when a file is imploded.

What color space conversion are you talking about? I thought about the binary representation which is RGBA or which I did read somewhere. Anyway the BLP display works perfectly.

@LeP:
In the MPQMaster there is something like Flush MPQ archive which compresses the archive? I want to implement something similiary in my MPQ Editor so it is indeed useful.
My console tool "mpq" is like "tar" and should support such options as well.

Any input is welcome. Maybe I will post some screenshots soon. The editor can be used on Linux and is based on Qt/KDE.

 

[GhostWolf]

Not sure what's legal or not as far as chaining algorithms goes, but I only saw sound files using chaining at all in Blizzard files.

About the colors, it's just some fetish he has, that's not relevant to anything, but he keeps posting it in every thread that has the word blp in it.

 

[Dr Super Good]

You realize that StormLib uses pkware by default as I found it in the source code?

StormLib is maintained by Russians so they do not care?

I'd like to contact the author and ask for permission directly.

What good would that be? He does not own PKWARE compression intellectual property. The PKWARE compression algorithm is owned by PKWARE Incoperated who are the only people who can legally sell/distribute licences to use the PKWARE compression. Blizzard paid/pays them in order to use it, hence why their new games no longer use PKWARE compression.

It is almost certainly illegal in the USA and Europe to use PKWARE compression on data without a licence from PKWARE Incorporated, even if your implementation is open source.

Are you sure not every chain can be used?

Only in version 2/3 MP files is not every combination permitted. Although it retains its flag based layout mostly, only certain combinations are valid. This was to add some form of sense to the compression used since chaining every compression is pointless functionality.

What color space conversion are you talking about? I thought about the binary representation which is RGBA or which I did read somewhere.

A color space cannot be RGBA because alpha is not a colour. Alpha is metadata to help process a colour. In real life nothing has an "alpha" colour.

BLP textures are in a linear RGB colour space. Alpha is always linear as it is computational metadata so can be ignored (you never see alpha, only its affects). Your display is in a non-linear RGB colour space. In a BLP texture a component value of 128 is twice as bright as a component value of 64. In your display space a component value of 128 is ~4.59 times as bright as a component value of 64.

Does this make a difference? Yes it does because if you do not interpret the image correctly the colours will be wrong due to the RGB components having a completely different mix. A good example would be the mipmaps would look wrong as the averaging used would be biased towards dark colours.

Why is your display non-linear? Because your eyes are non-linear (like most natural sensors) and so it gives much better dynamic range for the bandwidth used. In a non-linear colour space with 8 bits per channel a 255 value channel is 255 times as bright as a 1 value channel. In the 2.2 gamma display space with 8 bit per colour channel, which most display manufacturers approximate, a 255 value channel is 196,965 times as bright as a 1 value channel. This means BLP files actually have very bad colour range when it comes to dark colours (cannot represent any dark details).

How do I know this is right? I looked at the OpenGL API calls WC3 makes and what textures it loads into memory when in OpenGL mode. All textures correspond to their BLP pixel values meaning no correction is performed. All textures are loaded as linear RGB textures both due to observation of the API calls made and the logical reasoning of them only being able to be loaded as linear RGB textures. When WC3 was made the sRGB texture format, which performs automatic correction, did not exist.

WC3 also lacks a colourizer/HDR stage as it uses the legacy fixed shader pipelines. As such a gamma of 2.2 (setting 44) is needed so that the output matches approximately your display response. Modern games like SC2, HotS and Diablo III have their gamma correction built into the colourizer/HDR shader stage so that the in game gamma is more a further correction for displays that do not follow a 2.2 curve (eg MAC displays with 1.9).

Anyway the BLP display works perfectly.

What evidence do you have for this?

About the colors, it's just some fetish he has, that's not relevant to anything, but he keeps posting it in every thread that has the word blp in it.

It is because people are ignorant to the facts about how displays work. Its like they did not bother spending 5 minutes to read the professional documentation about the subject.

 

[edo494]

if I may ask, what the fuck has the RGB shit anything to do with the original question of file compression? You guessed it, completly nothing

 

[PurgeandFire]

@LeP:
In the MPQMaster there is something like Flush MPQ archive which compresses the archive? I want to implement something similiary in my MPQ Editor so it is indeed useful.
My console tool "mpq" is like "tar" and should support such options as well.

It might be compacting the archive:
http://www.zezula.net/en/mpq/stormlib/sfilecompactarchive.html

When you remove a file from an archive through StormLib, it leaves a gap (the space is still used--it just adds a delete marker to the file's flags).

Compacting the archive will get rid of those gaps and rebuild. I'm guessing MPQMaster's "flush" does the same thing, but I haven't checked.

 

[Barade]

Ah I see, yes that's another thing which I have to take care of, reordering the blocks to remove unused space.
I will add a dialog maybe when you import a file or directory where you can select compression options.

About PKWare: It would not matter if they are russians when the software is used by others. I thought many wc3tools are using StormLib (https://launchpad.net/smpq for example). Btw. on the website it says Czech which is part of the EU since 2004 and the code was created in 2003 so maybe that does matter. I am not even sure if software patents are allowed in Germany as they are in the USA. I thought as long as you write the code by yourself and dont copy it, nobody can do anything. Actually I dont care as long as nobody is complaining.

About the color space: Does it matter if I call it RGBA color space or not? Of course only RGB are colors but I guess everyone understands it. The textures look like the textures from the game. If there is no correction by the game why should I do any correction? I dont get your point.

 

[Dr Super Good]

About PKWare: It would not matter if they are russians when the software is used by others. I thought many wc3tools are using StormLib (https://launchpad.net/smpq for example). Btw. on the website it says Czech which is part of the EU since 2004 and the code was created in 2003 so maybe that does matter. I am not even sure if software patents are allowed in Germany as they are in the USA. I thought as long as you write the code by yourself and dont copy it, nobody can do anything. Actually I dont care as long as nobody is complaining.

They support writing PKWARE compression purely so they can claim they support it. There is no guarantee the compression algorithm they use produces as good results as WC3's licenced implementation. Additionally there was no real motivation to optimize it.

The default compression used should be one of the supported zip formats. Sure people could request PKWARE compression if they wanted, but this way you are not passively forcing them to use it.

I thought as long as you write the code by yourself and dont copy it, nobody can do anything.

The code is not written by themselves as far as I can tell. It was reverse engineered from the code written by PKWARE used in WC3. Over time it may have been cleaned up but ultimately they did not magically write compatible code, they reverse engineered compatible code. This used to be best seen in their implementation of Huffman compression where they openly admitted in comments that they ported the assembly into C statements.

Reverse engineering is permitted for compatibility purposes, hence reading PKWARE stuff is pretty solid (how 7Zip can read RAR files). However writing it is a different matter entirely, especially if it is done to a similar quality as the actual library.

As such I would say by default zip all files with unknown extensions then PMC and Huffman compress all WAV files. This is similar behaviour to how WE operates except the zip part might be a little more efficient than WE due to better implementations.

Huffman compression is quite special in that it can compress seemingly random data due to statistical variance with byte value occurrence. This means that in theory it can compress already compressed data. There are also 4 or so Huffman tables to choose from, each producing different compression results (different compressed size).

Unfortunately Huffman is applied before zip and deflate compression meaning it cannot be used to compress such blocks. However compression flags are declared per block, a brute force Huffman could be applied to every compressed file block to see if it beats other compression algorithms for the data block. This may be useful with JPEG content BLP files as the JPEG content is itself already compressed so higher compression rates may be obtainable than with other compression types.

If there is no correction by the game why should I do any correction?

WC3 has gamma correction, it is located in the game options. Just for some stupid reason it defaults to 1.0 and is never explained to the users how it should be used other than incorrectly labeling it "brightness". Automatic conversion was not possible back when WC3 was made as sRGB texture support only became mainstream with D3D9 and its OpenGL counter part.

More recent Blizzard games like StarCraft II/HotS may appear to no gamma correction however they actually do as part of the HDR Colourizer which performs purposely skewed correction for visual effects.

It only matters for texture processing, specifically mipmap production. Standard sRGB algorithms for downsizing (used by most image processors) will produce wrong mipmap results. Instead one literally must average the channel values between pixels.