• 🏆 Texturing Contest #33 is OPEN! Contestants must re-texture a SD unit model found in-game (Warcraft 3 Classic), recreating the unit into a peaceful NPC version. 🔗Click here to enter!
  • It's time for the first HD Modeling Contest of 2024. Join the theme discussion for Hive's HD Modeling Contest #6! Click here to post your idea!

BLP Specifications (WC3)

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
What is a BLP file?

This manual covers the specification of Warcraft III BLP files. World of Warcraft BLP files are not fully covered but they are specified quite well by other sources. It is an update of existing specifications based on observations and experimentation results to tidy it up.

A BLP (extension .blp) file is an image container. Each BLP file contains a single image with an optional full set of mipmap images. The format supports images up to 65,535 pixels in either dimension. BLP0 files are logically limited to at most 512 pixels in either dimension. BLP1 used to have the same dimension limit as BLP0 until Warcraft III The Frozen Throne patch 1.27b. BLP1 also used to limit the largest useful mipmap level to at most 512 pixels in any dimension until Warcraft III The Frozen Throne patch 1.29.

Pixels produce linear RGB colour components and optionally an alpha component. The size of colour components varies depending on the encoding type. Some types of decoded component data are intended to be directly loaded as a linear colour space textures in by a graphics API.

It is important to note that BLP files technically contain linear RGB colour space images. Most images such as from JPEG files or drawn using art tools are manipulated in the sRGB or other non-linear colour spaces. This means that in theory a colour space conversion should be performed in order for the BLP image to represent the correct colours. However there is strong evidence to suggest that Blizzard treated the image components as sRGB when encoding the assets of Warcraft III, possibly due to a lack of understanding about colour space management. So that encoded and decoded assets generally appear as intended is not recommended to perform colour space correction when processing BLP files. If a colour space is required for output pixel components then it is recommended that sRGB is used as that will likely be most visually correct. When encoding it is recommended that pixel component values are used directly with color space being ignored if possible.

In Warcraft III one can apply an approximate colour space correction after scene rendering as part of the "gamma" setting. Warcraft III gamma runs approximately from exponent 1.2 (minimum, setting 0) to exponent 0.2 (maximum, setting 100, gamma 5.0). For semi-accurate results gamma could be set to ~2.2 (registry setting 74-75). The way WC3 generates the gamma correction LUTs appears to suffer from minor rounding error or possibly uses a more correct sRGB curve rather than a plain gamma curve. It is worth noting that despite this being technically correct, the game will look terrible and over exposed hinting that textures were not correctly colour space managed.

File Structure

BLP header file structure is stream based. This means that it is recommended to read it 1 field at a time into an object rather than mapping directly into a structure. Do note that the below code is pseudo code of sorts so not intended for direct use by a programming language, it is only to give an idea of the file stream structure. All fields are in little-endian byte order.

.blp File Structure
Code:
// from start of file stream the following objects are present.
BLPHeader_t blpHeader;;
if (version >= 1) { // determined by BLPHeader
    MipmapLocator_t mipmapLocator;
}
if (content == CONTENT_DIRECT) {
    DirectContentHeader_t contentHeader;
} else {
    JPEGContentHeader_t contentHeader;
}

BLPHeader_t Structure
Code:
// BLPHeader_t
uint32_t magic; // determines version
uint32_t content;
if (version >= 2) {
    uint8_t encodingType; // not documented
    uint8_t alphaBits;
    uint8_t sampleType; // not documented
    uint8_t hasMipmaps;
} else {
    uint32_t alphaBits;
}
uint32_t width;
uint32_t height;
if (version < 2) {
    uint32_t extra;
    uint32_t hasMipmaps;
}

// valid magic values
uint32_t const MAGIC_BLP0 = decodeMagic("BLP0"); // version = 0
uint32_t const MAGIC_BLP1 = decodeMagic("BLP1"); // version = 1
uint32_t const MAGIC_BLP2 = decodeMagic("BLP2"); // version = 2

// valid content values
uint32_t const CONTENT_JPEG = 0; // JPEG encoded content.
uint32_t const CONTENT_DIRECT = 1; // Directly encoded content

// default values if invalid or new
content = CONTENT_JPEG;
alphaBits = 0;
extra = 5;

MipmapLocator_t Structure
Code:
// MipmapLocator_t
uint32_t mmOffsets[16];
uint32_t mmSizes[16];

DirectContentHeader_t Structure
Code:
// DirectContentHeader_t
uint32_t cmap[256];

BLP File Header

All fields assume little-endian byte order unless otherwise stated.

The BLP file header...
Code:
uint32_t magic;
uint32_t content;
if (version >= 2) {
    uint8_t encodingType; // not documented
    uint8_t alphaBits;
    uint8_t sampleType; // not documented
    uint8_t hasMipmaps;
} else {
    uint32_t alphaBits;
}
uint32_t width;
uint32_t height;
if (version < 2) {
    uint32_t extra;
    uint32_t hasMipmaps;
}

The magic field is used to identify files as BLP image files and what format version they are. It consists of 4 ASCII characters that sequentially represent something human readable. The characters are read from lowest byte to highest byte when forming the strings. This should be decoded into a constant version number to make subsequent reading and processing easier.

Code:
// converts a 4 character ASCII string into a single 32 bit unsigned integer.
uint32_t decodeMagic(char const * magicstr);

// Version 0 used by Warcraft III: Reign of Chaos beta.
uint32_t const MAGIC_BLP_V0 = decodeMagic("BLP0");
// Version 1 used by Warcraft III.
uint32_t const MAGIC_BLP_V1 = decodeMagic("BLP1");
// Version 2 used by World of Warcraft.
uint32_t const MAGIC_BLP_V2 = decodeMagic("BLP2");

uint32_t const BLP_VERSION_MAP[] = {MAGIC_BLP_V0,
    MAGIC_BLP_V1, MAGIC_BLP_V2};

The content field determines how the image data is stored. CONTENT_JPEG uses non-standard JPEG (JFIF) file compression of BGRA colour component values rather than the usual Y′CbCr color component values. CONTENT_DIRECT refers to a variety of storage formats which can be directly read as pixel values. If content field is invalid then CONTENT_JPEG must be assumed and it is recommended a warning be generated.
Code:
// JPEG content.
uint32_t const CONTENT_JPEG = 0;
// Pixmap content.
uint32_t const CONTENT_DIRECT = 1;

The encodingType and sampleType fields determine the encoding used for CONTENT_DIRECT images. The exact mechanics of the fields are not documented here. For version 0 and 1 these can be assumed to correspond to indexed encoding.

The alphaBits field determines the alpha component precision of the stored image. CONTENT_JPEG must have either 0 or 8 bits due to the mechanics of how JPEG stores components, and a JPEG alpha component must still physically exist even for 0 bits. Direct content has been known to support values of 8, 4, 1 or 0 bit and depends on the encoding type. Invalid values of alphaBits for the content or encoding used must evaluate as 0 bit and it is recommended a warning be generated.

Warcraft III has inconsistent processing for CONTENT_JPEG with 0 bit alpha as UI images are correctly opaque but alpha component values are still used when blending model textures. As such when writing CONTENT_JPEG with 0 bit alpha it is required that the alpha band be assigned opaque component values (0xFF).

With some direct encoding types in version 2 it is possible the alphaBits field takes on another, unknown meaning according to various World of Warcraft BLP2 specifications.

The width and height fields define the pixel dimensions of the full scale image stored in the BLP file. Although often a power of two, it is not required to be such.

The extra field appears to serve no purpose at all. There is no particularly strong correlation between BLP file usage and value. Testing also shows the value to have no visual impact on how the textures is processed. Some legacy documentation stated that it effected the team colour blending on unit models in Warcraft III however tests have been unable to recreate any such effect. Other people speculate that it might be a version field for the encoder used to create the BLP files. Another guess might be that it was used internally by Blizzard to perform some kind of image classification during development. Images extracted without processing the field show no noticeable artefacts. A recommended value to use would be 5 as that is used by the WorldEdit generated war3mapMap.blp (mini-map) file.

The hasMipmaps field is a boolean for if mipmaps are present for the image. If 0 then no mipmaps exist and the image will be present at full resolution at mipmap level 0. If non 0 then a full compliment of mipmaps exist ending with a 1*1 pixel image at the highest mipmap level (eg for the maximum size 65,535 x 65,535 image all 16 mipmaps are required). It is recommended that an exception or error be generated when trying to access mipmap levels which do not logically exist. It is also recommended that an exception or error be generated when trying to encode or decode an image with mipmaps that has any dimension larger than 65,535 pixels. Mipmaps may be treated as thumbnails when it comes to previewing the image content.

Mipmaps are required by textures which use mipmaps for filtering, such as used by models. Warcraft III does not automatically generate mipmaps for BLP textures, it assumes 0 value RGBA components (transparent black) for all non-0 mipmap levels when using an image with hasMipmaps set to 0. Mipmaps are not required by textures which do not need mipmaps (only tested on windows) such as command card buttons, shadows and minimap images. Mipmaps are also needed by textures that are to scale with Warcraft III texture quality.

Mipmap Location

If version is greater than or equal to 1 then a mipmap chunk location structure is present after the BLP header.

The mipmap location header...
Code:
if (version >= 1) {
    uint32_t mmOffsets[16];
    uint32_t mmSizes[16];
}

The mmOffsets array field determines the file stream offset where the mipmap data chunk is located counting from the start of the file. The mmSizes array field determines how many bytes must be read for the mipmap data chunk. With both arrays the index refers to the mipmap level with mipmap 0 being the full sized image and higher indexes representing smaller mipmap levels.

If hasMipmaps is 0 then only index 0 is used. If hasMipmaps is not 0 then the number of indices used should be the number of mipmap levels required for the full sized image which is determined by the maximum of height and width. There are no strict positioning or ordering requirements with mipmap data chunks, and padding can even be mixed between. However it is recommended that mipmap data chunks be placed sequentially in ascending mipmap level order with no padding in between.

Mipmap data chunks must be sourced from within the file. WC3 does not bounds check the buffers used after reading past EOF of a BLP file resulting in undefined and crash prone behaviour. To allow some image data to be read from technically malformed files it is recommended that if mipmap data exceeds EOF then the EOF cutoff be used to truncate the mipmap data chunk and a warning be generated.

attachment.php

Warcraft 3 1.27 example of Indexed Color Model texture where pixel values are sourced unsafely from outside buffer bounds. The circular features are likely memory garbage leftover from processing parts of the clock UI graphics.

If version is 0 then the mipmap data chunks for different mipmap levels are stored in separate files. The full scale image, mipmap 0, is in a file in the same directory with the same root name as the BLP file but with extension "b00". Each mipmap level is assigned a unique file extension derived from its level in the form of "bXX" where "XX" are replaced with the 0 leading mipmap level number such as "b08" for mipmap level 8 and "b10" for mipmap level 10. The mipmap data chunk size is the file size of the file it is located in.

The way version 0 stores mipmap data chunks might not be directly compatible with some image IO APIs. To decode them the API must support processing or acquiring a file system path rather than only dealing with a file stream to the BLP file. Due to this and how uncommon version 0 files are support is considered optional. With exception of how mipmap data chunks are sourced version 0 and 1 are mostly identical so a lossless version 0 to 1 converter could be made to provide version 0 support to a version 1 reader.

Content Headers

Next in the stream is the content header which is determined by the content field. The content header is the final sequential structure that can be read from a BLP file.

Content headers are used to setup the decoder for mipmap data chunks.

JPEG Content

For CONTENT_JPEG BLP files the following header...
Code:
uint32_t jpegHeaderSize;
uint8_t jpegHeaderChunk[JpegHeaderSize];

The jpegHeaderSize field determines the byte size of a header chunk appended to the beginning of all mipmap data blocks to produce a valid JFIF image. It is followed immediately by the jpegHeaderChunk itself. The mechanics of the chunk concatenation mean that it is not required that jpegHeaderSize be non-0 as the mipmap data chunk can incorporate a complete JPEG file. When dealing with blp files with hasMipmaps equal to 0 the mipmap data does not require a non-0 size as the jpegHeaderChunk itself could be a complete JPEG file.

The maximum valid value of jpegHeaderSize is 624 bytes (0x270). Larger values are prone to causing image corruption and crashes in some BLP reader implementations like Warcraft III 1.27b where buffer bounds are not strongly enforced. This limit applies especially when generating a BLP file with JPEG content and without mipmaps as it can prevent dumping the entire full scale image JPEG file into jpegHeaderChunk and using an empty mipmap block. If values larger than 624 are encountered it is recommended that a warning be generated and loading continues as normal using the larger size.

If jpegHeaderSize causes jpegHeaderChunk to extend past EOF or is stupidly large then Warcraft III is susceptible to a buffer over-run crash. In such a case the JPEG header chunk size should be reduced to fit within the file and it is recommended a warning be generated.

Direct Content

For CONTENT_DIRECT BLP files the following header...
Code:
uint32_t cmap[256];

The cmap field array is the colour look up table used for an indexed colour model. Each element represents 24 bit RGB colour component values in the order of 0xBBGGRR. The final byte is alignment padding and will not alter the decoded image in any way. One might be able to improve the file compressibility by carefully choosing padding values.

It is important to note that cmap must always be present for CONTENT_DIRECT even if it is not required to decode an image. This can happen for version 2 files where non index colour model based direct formats exist. In such a case cmap should be left zeroed to allow for better compression.

Mipmap Data Chunks

Mipmap data chunks contain the image data which produces a mipmap level when decoded.

JPEG Mipmap Data

CONTENT_JPEG mipmap data chunks have the following format...
Code:
uint8_t jpegChunk[mipmapSize[this]];
Component values can be obtained by appending the jpegChunk to the jpegHeaderChunk and decoding the resulting JPEG file with a compliant JPEG codec. The JPEG file contains 8 bit BGRA colour components. This is non-standard for JPEG files which usually consist of 8 bit Y′CbCr colour components. As such the stored components have to be used directly before codecs perform Y′CbCr colour conversion, something that many codecs will do automatically. For example when programming in JAVA the standard ImageIO JPEG decoder requires that the JPEG be decoded as a raster and not as a BufferedImage to avoid automatic colour space conversion. Blizzard uses the now discontinued Intel Imaging Framework to produce/decode the JPEG content meaning that each jpegHeaderChunk of official BLP files contains its signature. The use of an embedded signature is optional and likely up to the JPEG encoder used.

Chroma subsampling is not possible or will likely produce bad results since BGRA colour components are used instead of the normal Y′CbCr. Due to the linear colour space used by BLP files the lossy compression of JPEG should visually affect high component values less than low component values which is not ideal and likely why BLP2 files never contain JPEG content.

Since jpegChunk is a JPEG image it defines its own width and height separate from the width and height fields of the BLP header. The internal height and width should match the expected dimensions for the mipmap level the jpegChunk is for. Some badly written BLP composition tools fail to do this for high mipmap levels resulting in decoded mipmap images that fail to comply with mipmap scaling logic. Such mipmaps are known to cause a fatal error on some Mac versions. For feature parity with the Windows version of Warcraft III such images should be loaded with the right and bottom edges being used to crop or expand the image into appropriate dimensions. Padding is done with transparent black (0, 0, 0, 0).

Indexed Mipmap Data

CONTENT_DIRECT mipmap data chunks representing an indexed color model have the following format...
Code:
uint8_t indexedRGB[getMipmapPixelCount(this)];
uint8_t indexedAlpha[(getMipmapPixelCount(this) * alphaBits + 7) / 8];
The function getMipmapPixelCount returns the product of height and width field after being scaled for the appropriate mipmap level using mipmap scaling logic. In Warcraft III the number of bytes read for the chunk does not depend on mipmapSize[this], with it always reading the number of bytes required, even if those bytes are unsafely outside buffer bounds. The total length in bytes of both array fields should match mipmapSize[this]. If the expected chunk size does not match mipmapSize it is recommended that the underlying buffer be resized in an implementation specific way to the expected size and a warning is generated.

BGR component values can be obtained by using indexedRGB values as an index in lutBGR. When producing such values using color matching be aware of the linear nature of the color space. For best results it is recommended that color matching be performed in sRGB or other perceptual color spaces.

Alpha component can be obtained by breaking indexedAlpha into a bit field of alphaBits bit length fragments and then using the bit fragment as the alpha value for the pixel. The alpha pixel components are ordered from least significant to most significant bit with bytes following the same pixel order as indexedRGB. Since the alpha is to alphaBits precision it may need to be resampled to 8 bits be useful depending on the imaging framework used.

Example of different alpha packing in a byte.
Code:
MSB <-> LSB where number indicates the sequential pixel the bits belong to
ALPHA_8B -> 11111111
ALPHA_4B -> 22221111
ALPHA_1B -> 87654321

Pixels in the arrays run in scan lines left to right from top to bottom. The scan line length and number of scan lines is determined from the width and height fields after having mipmap scaling logic applied.

When producing mipmap data an algorithm appropriate for the source colour space should be used. Simple averaging is only suitable for the linear RGB colour space.

For most purposes high level mipmaps are usually so visually insignificant that visual correctness can be sacrificed for filesize. In the case of indexed content this can be achieved by pointing the image data chunk inside one of the lower level mipmaps. In the case of JPEG content it may be possible to use standard jpegChunk fields which have been hand optimized for file size. In all cases it is required that mipmapSize be correct, the appropriate number of mipmaps exist and that the mipmaps are the correct dimensions.
 

Attachments

  • WC3 BLP Buffer Overflow.jpg
    WC3 BLP Buffer Overflow.jpg
    58.9 KB · Views: 1,120
Last edited:
Level 12
Joined
Mar 13, 2012
Messages
1,121
(Disclaimer: as of now I just skimmed over it, so you might dismiss the following. I will read it in detail soon)
You might want to include that whole gamma/color space post you made, to clarify things if it's not already in there. Btw, is this just one of your personal crusades or a real issue :p? To remind you, this is how wc3 looks and should look.


Soooo, the next thing to do is make super awesome complete specs for .mpq, .mdx/.mdl, saved game, replay and other file types in maps and bundle it all in one space^^?
 

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
To remind you, this is how wc3 looks and should look.
Until they officially post that the gamma of every texture in WC3 is wrong and the shader system is messed up, what I post is how it is meant to look to the average user.

You can read both old and modern professional level programming guides for OpenGL/D3D and they all touch gamma correction, at least briefly. Lighting/pixel shader maths works in linear colour space but displays show and most art programs drawn in non-linear space. In modern (not at time of WC3) OpenGL and D3D you can load textures as sRGB which automatically, in real time, for free and more precisely converts the textures to linear colour space as required by the shaders. Such textures can also be used for results allowing for basic automatic gamma correction. Without working in a linear colour space the output pixel results will be wrong for operations such as blending and lighting since they assume linear colour space.

However StarCraft II/Heroes of the Storm also use linear textures but they have a HDR colourizer to apply the gamma correction with some other purpose made distortions (to enhance perceived colour depth, the point of HDR). This is why their Gamma setting needs such a low value around 1.0 as there already is a lot of gamma correction and that the gamma setting is only there to accommodate displays which have varying gamma. As such I doubt they were unaware of the issue of gamma correction back when they made WC3, just for some silly reason a default of 1.0 (setting 20) was choosen instead of the more universally correct setting 2.2 (setting 44).
 
Level 29
Joined
Jul 29, 2007
Messages
5,174
Soooo, the next thing to do is make super awesome complete specs for .mpq, .mdx/.mdl, saved game, replay and other file types in maps and bundle it all in one space^^?

http://www.wc3c.net/tools/specs/
http://www.zezula.net/en/mpq/mpqformat.html
http://www.hiveworkshop.com/forums/programming-714/mdx-specifications-240487/
http://www.hiveworkshop.com/forums/warcraft-editing-tools-277/jmpq-v3-pure-java-mpq-library-270680/ (uhh, source appears to not be public anymore?)
https://github.com/flowtsohg/w3m-w3x-viewer (not specs, but implements handlers for mpq, slk, mdx, blp, etc.)
etc.
 
Level 12
Joined
Mar 13, 2012
Messages
1,121
Until they officially post that the gamma of every texture in WC3 is wrong and the shader system is messed up, what I post is how it is meant to look to the average user.
So you are judging the image I posted as wrong? If yes:
It does not matter what the gamma should be or what the color space should be, every player who has played wc3 has seen it as it is in the image, people are accustomed to it and like it that way.

Yes I know those of course. Those are often incomplete or outdated though. To get the actual current knowledge of the community one has to combine that info with info from many more places (correction posts, new info, code in github, etc.).
 

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
It does not matter what the gamma should be or what the color space should be, every player who has played wc3 has seen it as it is in the image, people are accustomed to it and like it that way.
Yes but arguing that a colour movie should be viewed in black and white because most people have seen it that way does not mean the movie was not originally in colour.

http://www.wc3c.net/tools/specs/ is wrong about BLP spec. It fails to mention mipmap flag, 4 and 1 bit alpha for indexed colour (palleted) textures and makes the nonsense about team colour (model team colour is supported with blending a transparent texture over a team colour texture band).

https://github.com/flowtsohg/w3m-w3x-viewer implementation for BLP is wrong. It fails to support 1 and 4 bit alpha when in indexed colour mode. It does nonsense with pallet alpha (BLP has no alpha pallet support, the last byte for each colour appears unused with default value of 0 which might be why they thought it had inverted alpha). It also cannot process mipmaps but one can overlook that as intentional design (since mipmap 0 is always present and I think it only cared about the full size image), however that will produce inaccurate results as WC3 does not automatically create mipmaps.

Hence why I think a new post about the BLP specification is justified. All others are quite rubbish for some reason. This is not the case with MPQ and other files, those documents seem considerably more well written.

I plan to eventually (soonTM) release a Java ImageIO implementation for BLP files. Just it is constantly being delayed as I run into more and more previously unmentioned behaviour. I am having to rewrite quite a lot to support the 4 and 1 bit alpha for indexed colour model (needs its own SampleModel implementation), something I only discovered after hours of hacking around with a hex editor. I am also having to rewrite the I/O system to be more robust under certain inputs because WC3 will try to load even the most insanely malformed textures with unsafe results as shown in an image of the main post.
 

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
Whether these flags exist is questionable, since literally not a single Warcraft 3 texture uses them.
The "hasMipmaps" boolean field is provable. If you sample all WC3 BLP files (ROC+TFT+PATCH) then there is a perfect correlation between the presence of mipmap data for all mipmap levels and not. Further more if one toggles the boolean you can visually see the mipmaps failing on 512^2 or 256^2 textures (Windows version of WC3) where if near the screen they will show but further away (where mipmap levels above 0 are used) the texture goes black. UI seems to not use mipmaps, it instead always uses bilinear filtering of the lowest mipmap level (full sized image) even when bilinear filtering no longer produces meaningful results due to an output sample covering the area of 9 or more texels. The field is proved as a boolean as setting any of the 32 bits to 1 causes it to assume behaviour of having mipmaps (logical 1) and only when it is 0 (logical 0) does it not assume mipmaps.

The alpha is more questionable I do agree. It even came as a surprise to me, hence why I need a major code rewrite to support the it. No WC3 texture uses alpha bits of 4 or 1. All are either 8 or 0 (the files are quite strict mind you, if they have to alpha they always seem to be 0). However 4 and 1 alphaBits are valid as I have been able to create alpha pattens in an intended way with them, any other non valid alpha bit values count as 0 bits so the image is always completely opaque.

Unfortunately I have mangled my icons a bit when I discovered the buffer overflow error with BLP textures (showen in the main post). I used a hex editor to create a non-mipmaped (as I had proof of that functionality already) 8*8 image of checkered green and red pixels. The alpha then was used in a repetitive pattern to mask alternating columns of pixels, allowing one to detect ordering to some extent. When set to 8 bit the mask of alternating FF and 00 bytes produced the desired pattern. When set to any other value than 8, 4 and 1 it produced the full checkered pattern with no transparency. At 4 it produced 2 columns opaque followed by 2 columns transparent, which makes sense seeing the FF00FF00 pattern. At 1 bit it produced transparent rows as opposed to columns. I tested patterns to try and recreate the desired alternating transparent columns and came up with the bit order.

If you want I can create demonstration images for Indexed with alpha 4 and 1. Why it is supported but never used I do not know however in theory it could allow better compression with some image types, especially overlay mask images.
 
Level 29
Joined
Jul 29, 2007
Messages
5,174
Well, assuming mipmaps add the usual ~1/3 in memory, and a given 512x512 texture:
8bit alpha channel + mipmaps = ~341KB
4bit alpha channel + mipmaps = ~170KB
1bit alpha channel + mipmaps = ~43KB

So the most you could practically save is...nothing compared to JPG!

And since most of the indexed textures in the WC3 MPQs are small icons, the difference becomes a whole lot more negligible.

Supporting a useless feature that complicates your code base, just for the sake of being 100% true to the source (which you are not, because you are fixing their bugs), seems kind of pointless.
 

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
So the most you could practically save is...nothing compared to JPG!
You are not factoring in that depending on content the alpha content can be lossless compressed quite well. JPEG alpha has the potential to suffer from losses which can introduce artefacts if sharp alpha edges are needed (eg masking for a sign).

I do agree that it is kind of a useless feature. However it is still a feature non the less, and one nobody apparently knew about.

which you are not, because you are fixing their bugs
I am not fixing their bugs, I just have to have a decent level of robustness because there are potentially some terribly malformed BLP files out there. The fact WC3 loads anything from them is even kind of a bug. My intention is the loader would log a warning when loading such files (in case people wonder why something breaks), and any saver will not produce them.

And since most of the indexed textures in the WC3 MPQs are small icons
There are 235 BLP files with indexed content and alpha that are 256 wide or more. 77 indexed content BLP files which have no alpha are 256 wide or more. Sure there are more JPEG images in that size range, however the number is not insignificant.

For people interested I have attached 8*8 indexed content demonstration icons for 8, 4, 1 and 0 alphaBits. These can be used in game as buttons to show that the feature exists and works. Anyone implementing a BLP library might want to use them as test images to check if their loader works correctly.

What is funny is BLP Lab fails to correctly read the alpha for 1, 4 and 8 bit alpha versions of the image yet WC3 has no problem with them. Attached is a map demonstrating the icons.
 

Attachments

  • TEST88A0.blp
    1.2 KB · Views: 229
  • TEST88A1.blp
    1.2 KB · Views: 265
  • TEST88A4.blp
    1.2 KB · Views: 210
  • TEST88A8.blp
    1.3 KB · Views: 219
  • BLP Test Map.w3x
    12.5 KB · Views: 272
Last edited:

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
Been revising the specification a bit, merging in some BLP2 stuff and new limitations as I find them.

It appears that WC3 defaults to CONTENT_JPEG and ALPHA_NONE in the case of an invalid value in those fields.

Invalid values of the jpegHeaderSize field can cause WC3 to crash with a "memory could not be read" error, most likely as the result of a buffer over run. So far I have managed to achieve this by accidently entering a large value that massively exceeded the file length.

Garbage JFIF data does not seem to crash WC3. Instead it creates a blank texture (zeroed components).

WC3 alpha logic is a bit inconsistent when it comes to CONTENT_JPEG. When using 0 bits the alpha channel is correctly ignored for UI elements (no visual transparency). However when used as a texture the alpha values are still used for blending (visual transparency). As such when saving JFIF content one must probably store an alpha component of 255 for all pixels even if no alpha is specified in the image. I have yet to test what happens when a JFIF with only 3 components is used.

JPEG images appear to be resized to the header specified height and width. For example a 512*512 JFIF image file (in this case it was Griffon.blp) will load fine as a 384*384 image. The image rect is positioned against the upper left corner with cropping behaviour for the bottom and right edges. I have not tested trying to enlarge a JPEG image. This behaviour has been reported to cause Mac clients to crash in the past (the "fake mipmap" problem), however Windows seems to tolerate it perfectly. This discovery is not reflected in the specification yet.
 
Last edited:

Deleted member 219079

D

Deleted member 219079

Good effort, I will be using this.

And these as well :)

You could promote hex editor such as XVI32 so the reader can experience first-hand how the specification holds true.

Also I may add that adding screenshots is no longer a hassle as this takes only Alt+PrtScr > Ctrl+V in Hive 2:
upload_2016-12-11_12-40-27.png
 

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
You could promote hex editor such as XVI32 so the reader can experience first-hand how the specification holds true.
The specification was reverse engineered using that hex editor. Still does not explain why no one bothered doing it before me though, especially for a game as popular as Warcraft III. All those smart guys over at WC3C and all the BLP stuff they created seems to be wrong.

Also I may add that adding screenshots is no longer a hassle as this takes only Alt+PrtScr > Ctrl+V in Hive 2:
I am not sure it is that easy. Sure getting an image is but a non-annotated image is likely as confusing as the pure text already is.

For some reason the data inspector does not work properly for me, the numeric values it returns are wrong for multi-byte types even though endian is set correctly. It is like it stops reading bytes if it encounters a 0 byte. Maybe this was fixed in a newer version or some setting is wrong, I will need to check.

I really wanted some proper structure diagrams, such as those found in profession standards manuals. However I have no idea how to quickly create them and so my time and effort is better spent writing the BLP Java ImageReader implementation than the diagrams. After all, one does not really need the standard if there is a proper, well documented open source implementation.
 

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
May I ask what on earth is "pixmap"? what is that name even supposed to mean?
Wikipedia
In computing, a bitmap is a mapping from some domain (for example, a range of integers) to bits, that is, values which are zero or one. It is also called a bit array or bitmap index.
The more general term pixmap refers to a map of pixels, where each one may store more than two colors, thus using more than one bit per pixel. Often bitmap is used for this as well. In some contexts, the term bitmap implies one bit per pixel, while pixmap is used for images with multiple bits per pixel.
I did not make up the word. Pixmap seemed to be the correct choice as the pixels are multiple bits. One could argue that index colour supports a bitmap alpha band (1 bit per pixel) however the index band is still 8 bits and results in at least 24 bit colour out.
You also don't make it clear where the BLP0 format ends.
Does it end at the mipmap offsets?
Only difference between BLP0 and BLP1 is that the mipmap data chunks are sourced externally from separate files rather than internally from some place within the same stream. The same logic applies to both BLP0 and BLP1 mipmap data chunks. It also appears that the same logic applies to the BLP2 mipmap data chunks under certain conditions, even the indexed colour header is present despite it sometimes not being used. Sadly I lack means to test or develop for BLP2 so that part of the specification will remain under developed.

You can look at my BLP0/1 ImageReader for Java if implementation details are needed. It seems to load all BLP files from Warcraft III and has been tested on a few BLP0 files as well.

What is interesting is that some Warcraft III JPEG content BLP files are malformed. Specifically they declare 0 bit alpha (opaque) yet still contain some alpha content (not opaque alpha band). For some it is all mipmap levels. With others only one or two mipmap levels.

A bug with WC3's BLP reader is that it will correctly process alpha bits in some uses (eg 0 bit alpha JPEG is always opaque when used on UI, alpha band is ignored completly) but not for others (eg 0 bit alpha JPEG can still be transparent when used as a model texture, alpha band is incorrectly processed as if 8 bit alpha).
 
Last edited:
Level 29
Joined
Jul 29, 2007
Messages
5,174
...what? I never heard that name before, no idea where you pulled that from, but you are also not using it correctly. Wikipedia clearly talks about an image, not an indexed color palette.
I was ok with your lutBGR naming in my parser, but this is just wrong.
 
Last edited:

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
Wikipedia clearly talks about an image, not an indexed color palette.
In theory it is a pixmap since it is 1-2 arrays of multi bit components representing pixels. The indexed color palette is part of the color model used to decode those pixels into color components for display. The reason I named it pixmap content is because in BLP2 it is used for multiple different pixel packing forms such as direct RGBA, DXT and many other forms of pixel component packing.

In any case I will look into revising some of the names, and removing some of the BLP2 stuff because I really cannot complete it.

EDIT: I revised the first half.
EDIT2: Heavily restructured and revised the document.
 
Last edited:
Level 29
Joined
Jul 29, 2007
Messages
5,174
I would really appreciate it if you put the whole format in one code tag (and maybe later copy snippets of it to explain it), and if you know what the known and/or default values are for things, put them in the code as comments.

And again, about BLP0 - where does it end? Does it also contain a JPEG header when using JPEG content? does it even support JPEG?
Does it have lutBGR?
What are the file names?
How does it work?
Not that I am ever going to support it (if only because it will add even more asynchronous headaches in my JS code), but it's very unclear how it works, from reading your text.
 

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
And again, about BLP0 - where does it end? Does it also contain a JPEG header when using JPEG content? does it even support JPEG?
I think you are over thinking BLP0. It literally is BLP1 without the Mipmap Location object and instead sourcing the mipmap data chunks from other files. Hence why it is practically never mentioned anywhere outside the Mipmap Location section since everything else applies to it like it does BLP1.
If version is 0 then the mipmap data chunks for different mipmap levels are stored in separate files. The full scale image, mipmap 0, is in a file in the same directory with the same root name as the BLP file but with extension "b00". Each mipmap level is assigned a unique file extension derived from its level in the form of "bXX" where "XX" are replaced with the 0 leading mipmap level number such as "b08" for mipmap level 8 and "b10" for mipmap level 10. The mipmapSize field of each mipmap data chunk is the size of the file it is located in.

The way version 0 stores mipmap data chunks might not be directly compatible with some image IO APIs. To decode them the API must support processing or acquiring a file system path rather than only dealing with a file stream to the BLP file. Due to this and how uncommon version 0 files are support is considered optional. With exception of how mipmap data chunks are sourced version 0 and 1 are identical so a version 0 to 1 converter could be made to provide version 0 support to a version 1 reader.

I would really appreciate it if you put the whole format in one code tag (and maybe later copy snippets of it to explain it), and if you know what the known and/or default values are for things, put them in the code as comments.
I am not too sure how one would do that in a way that is meaningful. I will try however.

EDIT: Battling the markup a bit. Is this sort of what you wanted? Or do you want the different objects unwound into a single stream structure sequence?
 
Last edited:

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
Updated with limit on JPEG content shared header length. There seems to be a maximum valid limit of 624 (0x270) on shared header length.

Tested in Warcraft III 1.27b. Shared headers bigger than 624, eg 1024 (0x400), are prone to loading corrupted. Very big shared headers, eg 32768 (0x8000), are prone to crashing on load. These have the symptoms of buffer overflow. As such I am guessing Warcraft III 1.27b must allocate a 624 byte buffer for JPEG content shared headers and not check it is big enough to hold the file chunk. The actual limit could be slightly higher, however proving it is difficult. 624 is the biggest observed value in BLP files produced by Blizzard and hence the biggest value that is known to be completely safe from buffer overflow.

I really need to rewrite this some day...
 
Level 29
Joined
Jul 29, 2007
Messages
5,174
I re-read this thread, and you indeed made it a lot better. It can still be cleaned some more. :)

Regarding JPEG and wrong alpha - the results you got make me assume that the game loads the JPEG data directly and the alpha bits are only used for the lookup table.
The difference between it being used for UI or a game model can be as simple as the game turning off blending for UI.

I am thinking about adding mipmap support from the texture, rather than the driver making them. Is there anything to look out for as far as corrupted textures go?
Could you perhaps summarize in the opening post all of your findings on what bad things can happen?
 

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
I re-read this thread, and you indeed made it a lot better. It can still be cleaned some more.
I suck at writing this kind of thing lol.

Regarding JPEG and wrong alpha - the results you got make me assume that the game loads the JPEG data directly and the alpha bits are only used for the lookup table.
The difference between it being used for UI or a game model can be as simple as the game turning off blending for UI.
It always loads all 4 channels. As it is non-standard (no standard defines 4 channel JPEG) it loads the values directly without colour conversion. Alpha bits is most likely used to control logic in some cases, eg turning on/off blending for UI elements. Models probably do not use such logic.

When using a JPEG content BLP as a model texture it must be taking the output of the JPEG decoder library directly, without bothering to check if the alpha components should be ignored (alpha blending turned off). It is likely done this way to avoid having to repack the pixel data, which might have been less trivial back when WC3 was made.

Is there anything to look out for as far as corrupted textures go?
With JPEG content BLP files that have mipmaps, if any mipmap level is invalid (decoding to an image fails, usually with an error code or exception) then the entire texture can be considered invalid and does not load any image. In WC3 case this means any model that uses it will fail to load due to a missing texture. This behaviour is why many old BLP files no longer work in WC3 as some old BLP creation tools did not generate all mipmap images for some reason.

When processing the JPEG header block, it is important to test/manage its size. WC3 allocates a finite buffer size for the JPEG header block, probably the size of the Blizzard generated BLP JPEG header block. If a BLP image has a JPEG header block bigger than this it is technically subject to buffer overflow and so will act really weird in game, including some pixel components not loading and even virtual memory access exception crashes if really large. When such an image is encountered it might be a good idea to produce a warning, or at least not load it so as to signify to the user that something is really wrong with it.

Some third party BLP files seem to have inconsistent mipmap data. This is where features in mipmap levels above 0 do not reflect the features in the full scale image, there are discontinuities between mipmap levels. As WC3 loads mipmaps from the BLP file this means that using such a file as a texture will result in the model looking strange, discontinuous and as if corrupted where mipmap levels above 0 are used. I do not know which BLP creation tool was responsible for this but at least 1 skin I have encountered suffered from this. Technically this is nothing to care about (not your problem someone messed with the mipmaps) but it is worth mentioning as it might flag up as a false positive bug report at some stage.

Although unrelated to BLP, WC3 has no problem loading tga files as textures. It can load and use tga textures of practically any size (tested with 1920*1080, full scale image appeared in use when appropriate with bilinear mipmap discontinuities visible) and will automatically generate mipmap data for such files as required. Terrain tile textures will automatically check for an accompanying .tga file to the specified .blp file. Models are more strict, only checking the specified texture path, which must be either a .blp or .tga file. The game seems to load the textures based on the file suffix, so you cannot trick it into loading a tga file at a .blp path or a blp file at a .tga path.

Could you perhaps summarize in the opening post all of your findings on what bad things can happen?
Not too sure where to put it, or even what sort of "bad things" you might want mentioned.

@Dr Super Good (since it's not likely you'll see my post)
Since I am subscribed to this thread it actually does flag up in my notifications when someone posts. In fact if someone posts an @Dr Super Good it even flags up twice.
 
Last edited:
Level 29
Joined
Jul 29, 2007
Messages
5,174
By bad things I mean that list above and others - what is known to mess up.

A neat comparison between BLP and TGA as far as the game cares would also be nice.
POT vs NPOT, mipmaps in the file or generated by the driver, and other things that you know of.
 
Last edited:

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
Real limit can be 4096x4096 and bigger. ( In my case eax have 0x2000 value, 8192x8192 )
As far as I am aware WC3 can load whatever size texture that the graphic drivers let it. For example I had no problem using a 1920*1080 TGA file as a unit texture and that is not even a power of 2.

However for some silly reason Blizzard added a limit to the maximum resolution texture WC3 will load from a BLP file. In more recent versions of Warcraft III (1.28) it will even load BLP files larger than 512*512 pixels but it will only use the lowest mipmap level that is at most 512*512 pixels. This behaviour is kind of stupid as the game does not apply any such texture size limits to TGA files.
 

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
Why did they add these artificial limitations for BLP? I try use big BLP textures and no have problems. (126a with limit removed)
You do not have problems because you removed the limit by modifying the executable...

If you are asking why the limit exists because WC3 does support large textures in the form of TGA then I am guessing it is because back in 2003 when WC3 was being developed the graphic drivers did not support such large textures so they had to manually limit it and TGA allowing such textures to be loaded is then technically a bug that would have crashed the game at the time.

Part of the reason why people think power of 2 textures do not work in WC3 is because old GPUs did not support them. All modern GPUs do and WC3 does not stop them from trying however if run on an old GPU the game would likely either crash, or garbage/no texture would load.
128 too have this limit located in : 6F1484C0 function
The way it is implemented in 1.28 is different as in earlier versions it would not try to load BLP files when larger than 512*512 but now it will load such files but not load the lower level mipmaps larger than 512*512.

I have repeatedly brought to attention this nonsense in various patch threads.

While they are at it they should consider adding support for some of the modern real time compression formats. JPEG compression for mipmaps is not ideal, as it is not even JPEG2000.
 
Hey, I'm trying to save images to blp. But have some issues with mmOffsets and mmSizes.
Is it correctly interpreted that mmOffsets is the indexes of the bytes where each color starts, and mmSizes is how many bytes is that colors range? So the data for the red color spans mmOffset till mmOfset + mmSizes?
And the same for the other colors and alpha?
 

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
Is it correctly interpreted that mmOffsets is the indexes of the bytes where each color starts, and mmSizes is how many bytes is that colors range? So the data for the red color spans mmOffset till mmOfset + mmSizes?
And the same for the other colors and alpha?
mmOffsets describes the physical location of the mipmap data chunk within the file, in bytes offset from the beginning of the file. mmSizes describes the size of the mipmap data chunk within the file, in bytes. The mipmap data chunk contains all the data to fully decode a complete (appropriate size with all 4 colour channels) image for the selected mipmap level.

A mipmap data chunk is either a non-standard BGRA encoded JPEG image, or is an uncompressed palleted bitmap image encoded as described later in the document. Bitmap based images use the pallet table to decode the BGR colour channels and the alpha bits field to decode alpha channel. The way the data is packed is such that there is a full continuous channel of pallet index bytes followed by a full continuous channel of packed alpha bytes based on the alpha bits value.
 

Dr Super Good

Spell Reviewer
Level 63
Joined
Jan 18, 2005
Messages
27,180
Since the color map (cMap) is only 255 big, does it mean a texture can only have 255 unique colors?
A paletted bitmap texture can have at most 256 unique BGR colours as the palette is 256 entries long and a byte is used to index it. Alpha is separate so you can have the same palette colour with different alpha blend values, allowing for soft edges around blended textures without sacrificing palette colours.
And am I right in thinking they are packed as four Uint8's each in a BGRA pattern?
Palette entries are packed as 3 x unit8 colour component values for BGR and 1 x unit8 that is padding (unused). I am not sure of the byte order off the top of my head however it is BGR packed and not RGB packed. Alpha always comes from the alpha channel which packing depends on the alpha bits selected. This was often incorrectly described in old references, as was most of the paletted bitmap specification.

I think many of these limitations are similar to PNG and GIF so to generate the pallets and encode the images resources related to those might help.
 
Last edited:
Top