BLP1 and Mac Computers

[PurgeandFire]

BLP1 and Mac Computers:

• Testing
• Conclusions
• New Rule (Complete)

No one has formally written what crashes and what doesn't.
I am willing to test a variety of things, so if you have any requests you can
make a request here. The current established rule is the 7-mipmap rule:

I downloaded Button Manager and played around with it in
WINE, and finally got it to produce compatible icons. In the end, it didn't have
anything to do with JPEG vs. paletted, or quality %.

You need to change the Mipmap count to 7. Then, change whatever settings
you like; compressed, paletted, any % or number of colors. Then, press the
button "For all borders". That is critical: without pressing that, it won't use
the same settings when it automatically creates a disabled button.

Source: Mac + certain map = crash on selection

This seems to work so far. However, for developers who want to support
mac players, some extra information may be needed. I'll provide some based
on whatever tests I have been running.
​

Testing:

• I converted an icon using Button Manager (by Spec).
  Spec's Button Manager is on a default setting of automatic quality definition.
  To find the mipmap count, I used BLP Lab. This is how it is displayed:
  
  As you can see, it has Mipmap count: 7 (5 fake). From playing with the
  settings, it seems that BLP Lab and Button Manager will maintain this count
  of 7, and supplement some "fake" mipmaps which preserve the size of the last mipmap.
  You can get how many mipmaps were used when saved by doing 7 - 5 = 2.
  The odd thing is that this worked for one icon, but not for another. Both were
  png files of 256x256 size. The cartman icon did not work. The other did.
  EDIT: I realized the cartman one was under JPEG compression. The other one
  was under paletted.
  .
• Upon testing, the cartman icon seemed to only cause the crash when I first saw it.
  I assigned it to the holy light ability. When I learned the skill (the research icon was unchanged)
  it crashed (which was immediately before the cartman icon would be displayed).
  .
• When I saved the cartman icon with a mipmap count of 7 (using BLP Lab),
  it worked on my mac without any crashes or issues.
  .
• I downloaded a sorceress skin and set its mipmap count to 1. I imported it
  into a test map (one that works on mac) and I tested it on Windows. No crashes. (albeit the texture was messed up)
  When I transferred the map to my mac, it crashed on the loading screen.
  .
• Using that sorceress skin (default mm count of 9), I changed its count
  to 7 with the paletted option. It worked just fine on my mac. With JPEG compression
  at 100% quality, and mipmap count of 7, it crashed. It worked, however, with a mipmap
  count of 9.
  .
• Using that sorceress skin, I changed its mm count to 6 with the paletted
  option checked, as well as its sub-options. It worked without any crashes.
  Same for count of 5, with paletted. And 4. It worked for 1 as well without
  any crashes, despite the model being horribly disfigured even more than on
  my Windows computer. Tested the same for another skin, which was for Lord
  Garithos. It has a default mipmap count of 12, and I used 4 with paletted, and it
  worked without any crashes!
  .
• Next the Lord Garithos skin (512x256); it is under JPEG compression. I compressed it
  with a mipmap count of 11, 10, and 9. They all worked
  without any crashes. A mipmap count of 8 crashed.
  .
• Next a fel guard skin (hq 512x512); it is also under JPEG compression. With the
  settings it had as I downloaded it, it had 12 mipmaps (2 fake), therefore 10 real.
  It worked without any crashes. I dropped it to 9, and shizam it crashed.
  .
• I tried that fel guard skin with 5 mipmaps and paletted under 128 colors. Worked.
  Same for 16.
  .
• Using the fel guard skin, I adjusted the quality to 50%, 25%, and 1%. They all worked
  without crashing. Both with and without the JPEG sub-options enabled.
  .
• Sure enough, the cartman icon worked once I changed it to paletted.

Conclusions:

• First, this crash does not apply only to icons. It applies to both regular textures
  and, most likely, any BLP for that matter. (confirmed only for skins and icons)
  .
• The "fake" mipmaps that are generated by BLP lab/Button manager do not
  seem to fix the crash in most cases. On default settings, button manager can
  potentially cause the map to crash on a mac.
  On default settings, button manager will use the paletted option. It will not
  cause a crash on a Mac.
  Button Manager may choose settings that crash on a Mac.
  .
• The magic number isn't necessarily 7. Skins will work at other mip map values even under JPEG compression.
  .
• You can fix BLP1 files that were already converted by increasing its mip
  map count to the appropriate value.

New Rule:

After a lot of testing, I found a rule that seems to be consistent with my results.
For testing, I used BLP Lab, BLPaletter, ButtonManager, and three skins:
http://www.hiveworkshop.com/forums/skins-552/sorceress-blp-230570/?prev=search=sorceress&d=list&r=20
http://www.hiveworkshop.com/forums/...al-blp-125990/?prev=search=knight&d=list&r=20
http://www.hiveworkshop.com/forums/...ht-blp-152087/?prev=search=knight&d=list&r=20

I'll first explain how mip maps seem to work. The default count of mip maps
will depend on the dimensions of the BLP. That is why I chose the skins above;
the first is 256x256, the second is 512x256, and the third is 512x512.
Now, when you open up a BLP in BLP Lab, the mipmap count will be displayed:

As you can see, there are 9 for this 256x256 texture. Each mip map has its
own dimensions. BLP Lab organizes it in descending order. The first (#0) has
dimensions of 256x256, the second (#1) is 128x128, the third (#2) is 64x64,
the fourth is 32x32, fifth 16x16, sixth 8x8, seventh 4x4, eighth 2x2, ninth 1x1.

When you decrease the mip map count, a number of "fake" mip maps will be created.
They will point to the same offset and size as the previous mip map, and have the same dimensions:

As you can see, mipmap #7 and #8 have the same values as #6. Those are the
two "fake" mip maps.

---

Well, that explains mip maps a little bit. Now for the new rule. It seems that
with JPEG compression, there must be a mip map with a dimension of 1,
either for height or width. This seemed to be consistent with my results.
At first I thought you had to reach 1x1 or have the max mip map count for that size,
but the knight/garithos skin worked with 9 mip maps -> lowest dimension is 2x1.
The quality control and the sub-options for JPEG compression have no effect
on whether the texture crashes.

However, paletted textures work with any mip map count. They will not crash,
and the options (colors/encoding) will not cause the map to crash.

So how did Weep come to his conclusion? He tested it for icons, and he found
out what worked. Thank goodness too, because this would've taken
even longer otherwise. The reason why the magic number "7" works for icons
is that icons are 64x64, which means the mip map count is 7. This follows my
rule that the final mip map must contain a dimension of 1.

For regular (a x a) dimension textures, you must have the max mip map
count if you use JPEG compression. To calculate this, 2^n-1 = dimension.
For example, 2^n-1 = 64, ... ln(64)/ln(2) + 1 = n = 7

For irregular ( a x b) dimension textures, you would use the same equation, but
you would plug in the smaller dimension for the equation. For example, for garithos:
2^n-1 = 256, ... ln(256)/ln(2) + 1 = 9.
​

Tools:

• BLP Lab - Will allow you to set the mip map count of a BLP.
• ButtonManager - Can work if the settings are adjusted properly.
  If you use JPEG compression, it will work so long as you adjust the settings as Weep described (see the quote above).
  For good measure, always open up the icon in BLP lab and check the mipmap count.
• Wc3Borderizer Online - Appears to produce the appropriate icons with 7 mm count.
• BLPaletter - Confirmed that this works. Use the paletted option. Note that this does not
  work on BLP's and PNG files; those formats are not supported by the tool.

------

Thanks to Weep for the preliminary information, Shadow Daemon for BLP Lab
and ButtonManager, Cohadar for requesting whether his tool works (I ended up testing this afterward),
PitzerMike for BLPaletter.

 

[PurgeandFire]

Sorry for the double post, just posting that I have finished documenting this and have formed a new rule.

 

[Dr Super Good]

Sounds like the problem is related to the jpeg block decoding on lower mip-maps. If "fake" mipmaps are used then it will allocate nonsense offsets and sizes. If the game tries to decode these properly, something bad will happen.

On Windows, it must try and decode them and fail (high robustness).
On Mac, it must try and decode them and crash (low robustness).

In either case, fake mipmaps are an error as you are generating invalid mipmap data and it is a reasonable expectation for the program to not behave well in response to this. I doubt any Blizzard textures have "fake" mipmaps.

Palleted and uncompressed will obviously not suffer this problem as long as a reasonable offset and size is chosen since it will read the file stream as a sequence of pixels (thus maybe the garbage graphics?).

Does BLP require mipmaps or full mipmaps? If there is a field that tells the game how many mipmaps to use then it could be possible to eliminate them altogether as often the GPU will generate them when loaded if absent.

Alternatively, specifying the fake mipmaps to target valid mipmaps of garbage might also work. If each mipmap has a header and a series of JPEG blocks then the series of JPEG blocks could be chosen that is compressed the smallest.

Ultimately it may be best to generate full mipmaps. What few people know is there is no logical constraint on what visual data a mipmap uses, as long as it is the expected format. This means mipmap sequences of different colours could be used to make the surface change colour in-game once it falls below a certain scale. Turning on Anisotropic filtering should make it appear to change colour gradually as scale changes.

 

[PurgeandFire]

Seems like a reasonable explanation.

Does BLP require mipmaps or full mipmaps? If there is a field that tells the game how many mipmaps to use then it could be possible to eliminate them altogether as often the GPU will generate them when loaded if absent.

Mipmap count is determined by dimensions, so I assume the game uses that to determine it. I could be wrong though. Although, if it were as easy as changing a field, I think the compressors would have done that instead of generating fake mipmaps.

edit: There is no field in the blp format to show the number of mip-maps.

Ultimately it may be best to generate full mipmaps. What few people know is there is no logical constraint on what visual data a mipmap uses, as long as it is the expected format. This means mipmap sequences of different colours could be used to make the surface change colour in-game once it falls below a certain scale. Turning on Anisotropic filtering should make it appear to change colour gradually as scale changes.

That would be cool.

Overall, people choose to lower mipmaps because it offers great compression (30% for me once, going 10 to 1 mm, probably looked buggy ingame though). People also probably choose JPEG because it appears to offer better compression. It is true that it will usually give a lower file size than paletted, but paletted compresses better in the map archive and will often lead to lower overall size. It is good to know this, since my initial impression was that no one would fix their BLP's since it could sacrifice a chunk of file size. With the paletted option, they may not have to.

 

[Dr Super Good]

Overall, people choose to lower mipmaps because it offers great compression (30% for me once, going 10 to 1 mm, probably looked buggy ingame though).

That is because mipmaps increase texture size by approximately 33%

With the paletted option, they may not have to.

But instead they are sacrificing colour depth since jpeg is 8 bbp RGB where as palleted is limited to 256 colours maximum.

 

[Footman16]

I was talking to ralle about some icons from this site crashing and how it might be due to compressing icons but now that this has been discovered maybe ralle should make a rule about mip maps etc
It would allow mac users to use all icons from this site. Someone should show this to ralle.

 

[Dr Super Good]

I was talking to ralle about some icons from this site crashing and how it might be due to compressing icons but now that this has been discovered maybe ralle should make a rule about mip maps etc

Or he could code a system that automatically corrects all such icons. Until he has time, moderators could do it manually and explain to people what they did wrong.

 

[PurgeandFire]

Sorry for the necro-bump. I just noticed that there is one particular file that kind of evades this rule:
war3mapMap.blp

Oddly, it has 1 mip map and is JPEG compressed and doesn't crash. I'll probably have to do some tests on it. For now, I assume that this crash only occurs when it repeats offsets. In war3mapMap.blp, it doesn't repeat offsets (it literally has 1 mip map, even though *technically* it should have 8 as a 256*256 image). And it doesn't have an alpha channel. But I tried importing a BLP without an alpha channel and with improper mipmap count and it still crashed.

Interestingly enough, there have been reports on macrumors of mini-map bugging (it blackening out). Not sure if these two things are linked, but maybe. I haven't had that issue yet, so I wouldn't know.

 

[Dr Super Good]

I think with recent discoveries this topic disserves a necro, especially since you advertise it so diligently PurgeandFire.

Oddly, it has 1 mip map and is JPEG compressed and doesn't crash.

BLP files have a field which I call "hasMipmaps" which is used to determine mipmap availability. Mipmaps are only used/loaded if this field is set, in which case it will try to load a full set of mipmap images (down to the 1*1 pixel level). If not set then only the full sized image (mipmap 0) will be loaded.

The mipmap data chunk table plays no part in deciding if a mipmap exists or not, instead it is assumed to always be correct for the number of mipmaps needed. Number of mipmaps needed is determined by the hasMipmaps field and the height and width field. Entries in the table for mipmap levels that are not needed are ignored.

If the mipmap image chunk data is invalid then any kind of nonsense can occur. For example if an indexed BLP file has the chunk length specified shorter than what is required for a mipmap image chunk then the result is the mipmap pixel data is sourced from out of buffer bounds memory in an unsafe way (memory garbage, possibly even a fatal error). Mipmaps are an all or nothing affair, you cannot only have some mipmap levels and it is pointless providing extra.

Mipmaps are only needed when the texture is set to use mipmaps. In the Windows version of WC3 it appears that all UI elements such as minimap and command card buttons undergo bilinear filtering but do not use mipmaps. Hence why you can get away with 1 mipmap level for them.

However when mipmaps are needed, such as by units and other ingame 3D models, then it is important to use mipmaps. Warcraft III will not automatically generate mipmaps for such images, instead it will load the mipmaps as blank textures (default of transparent black (0,0,0,0) pixels) which is very impractical. Models using a no-mipmap textures will only appear correctly where the full sized mipmap (level 0) is used, with areas using higher mipmap levels appearing either transparent (if applicable) or black.

When the JPEG decoder encounters an error the game should load the mipmap level as a blank texture. If the JPEG image is the wrong dimensions it should automatically be subsampled through truncation/expansion towards the lower right. My guess is the MAC crash is a result of a non-robust JPEG image decoder which crashes the game on error (trying to decode a JPEG image with just the header and no component data) rather than failing and letting the game use a blank texture.

Interestingly enough, there have been reports on macrumors of mini-map bugging (it blackening out).

That would be the result of the minimap texture being marked to use mipmaps (something that really does not make sense). In Windows none of the buttons and minimap UI elements use mipmaps so they can in theory be created without them. This would save ~30% file space or allow 30% more icons to be packed into a map.

With the recent MAC patch this may need some retesting. Has anyone confirmed this crash still even exists?

 

[PurgeandFire]

Cool info, definitely plausible--especially considering the BLP2 header also has a flag that determines whether mipmaps are used. Thanks for the tests.

With the recent MAC patch this may need some retesting. Has anyone confirmed this crash still even exists?

It still exists, yeah.