Getting rid of DISBTN ... some theorycrafting

[Zwiebelchen]

So, has anyone found a good solution to get rid of DISBTNs yet?

As it's very annoying to have to import buttons as active and greyed-out variants in order to prevent the ugly green boxes and also wastes a lot of file space, I'm looking for a way to get rid of the DISBTNs, without looking ugly.


I know that you can actually name all your buttons the same, but place them in different folders, so that you can use a uniform DISBTN for all of them:

Example:

Active buttons:
"war3mapImported\Armors\001\BTNArmor.blp"
"war3mapImported\Armors\002\BTNArmor.blp"
"war3mapImported\Armors\003\BTNArmor.blp"
"war3mapImported\Armors\004\BTNArmor.blp"

And the shared disabled button for all of them:
"ReplaceableTextures\CommandButtonsDisabled\DISBTNArmor.blp"


However, this obviously comes with the caveat of all buttons looking the same when disabled. Now this is not a real problem if all you want is avoiding the green box. But it's definitely not the optimal solution either, especially if you have visible inventories for other players enabled (they will always show the disabled icon).


Is it possible to trick or fool WC3 into thinking your BTN is also a DISBTN at the same time by clever use of the file structure? So, for example, placing your button in "ReplaceableTextures\CommandButtonsDisabled\" by default, naming it DISBTN?

Active icon: "ReplaceableTextures\CommandButtonsDisabled\DISBTNArmor.blp"?
Disabled Icon: "ReplaceableTextures\CommandButtonsDisabled\DISBTNArmor.blp"

I suppose it will try to look for a "DISDISBTNArmor.blp", but I'm not sure on this one?


Another thought:
If it's only about filesize, will ZLib compression effectively remove the filesize footprint of a disabled button that is exactly the same icon as the enabled button? So my idea was to import the same icon for both paths; theoretically, the whole icon should be recognized as a "word" then, effectivly compressing the 6+6kb for both icons into 6kb for a single icon.

 

[jopi]

I'm really not sure if this is right, but as there are 'On-Cooldown-Models' maybe the DISBTN is also applied as an overlay on top of the actual icon? in That case you could go with your solution naming all BTNs the same and use a general DISBTN texture with a little bit of alpha so the icon below it is visible through it.

I can remember that I read something about DISBTNs and file size reduction over on playdota.com. But this was years ago and I can't remember any actual facts about it.

 

[Zwiebelchen]

I'm really not sure if this is right, but as there are 'On-Cooldown-Models' maybe the DISBTN is also applied as an overlay on top of the actual icon? in That case you could go with your solution naming all BTNs the same and use a general DISBTN texture with a little bit of alpha so the icon below it is visible through it.

I can remember that I read something about DISBTNs and file size reduction over on playdota.com. But this was years ago and I can't remember any actual facts about it.

Interesting idea; but I doubt that the DISBTNs are just overlays. Otherwise, the green boxes wouldn't appear, right?
Also, I'm not even sure buttons can even have transparency or a working alpha layer.

 

[LeP]

Another thought:
If it's only about filesize, will ZLib compression effectively remove the filesize footprint of a disabled button that is exactly the same icon as the enabled button? So my idea was to import the same icon for both paths; theoretically, the whole icon should be recognized as a "word" then, effectivly compressing the 6+6kb for both icons into 6kb for a single icon.

No, files are compressed one by one - not the whole mpq as one.

 

[jopi]

yes, that would be way to easy. I think what they did in dota, was to take the BTN icon, lessen the saturation a lot (nearly to greyscale) and apply a different icon border than the normal DISBTN icon border. I think it was with a lot less to no fade from the black border to the icon texture. This might save up some filesize but I'm not sure.

 

[Zwiebelchen]

yes, that would be way to easy. I think what they did in dota, was to take the BTN icon, lessen the saturation a lot (nearly to greyscale) and apply a different icon border than the normal DISBTN icon border. I think it was with a lot less to no fade from the black border to the icon texture. This might save up some filesize but I'm not sure.

I'm not asking how to create DISBTNs, I'm asking how to get rid of actually needing them.

 

[PublishedShadow]

I wonder if that ugly green texture is actually set somewhere as a path that an icon will default to if a DISBTN can't be found.

If that's the case then you could import a single DISBTN (lets just say a completely black icon or maybe even a transparent icon) and then rename that to OVERWRITE the green 'default' DISBTN.

Surely that has to be somewhere?

 

[PurgeandFire]

It *might* be possible. I don't think any API's directly support it, but judging by the mpq file definition:

struct MPQFILE {
	MPQFILE * lpNextFile; // Pointer to the next FILEREC struct. Pointer to addresses of first and last files if last file
	MPQFILE * lpPrevFile; // Pointer to the previous FILEREC struct. 0xEAFC5E13 if first file
	char szFileName[260]; // Filename of the file
	HANDLE hPlaceHolder; // Always 0xFFFFFFFF
	MPQARCHIVE * lpParentArc; // Pointer to the ARCHIVEREC struct of the archive in which the file is contained
	BLOCKTABLEENTRY * lpBlockEntry; // Pointer to the file's block table entry
	DWORD dwCryptKey; // Decryption key for the file
	DWORD dwFilePointer; // Position of file pointer in the file
	DWORD dwUnk1; // Seems to always be 0
	DWORD dwBlockCount; // Number of blocks in file
	DWORD * lpdwBlockOffsets; // Offsets to blocks in file. There are 1 more of these than the number of blocks
	DWORD dwReadStarted; // Set to 1 after first read
	DWORD dwUnk2; // Seems to always be 0
	BYTE * lpLastReadBlock; // Pointer to the read buffer for file. Only used for incomplete reads of blocks
	DWORD dwBytesRead; // Total bytes read from open file
	DWORD dwBufferSize; // Size of the read buffer for file. Only used for incomplete reads of blocks
	DWORD dwConstant; // Seems to always be 1
	HASHTABLEENTRY *lpHashEntry;
	LPSTR lpFileName;
};

struct BLOCKTABLEENTRY {
	DWORD dwFileOffset; // Offset to file
	DWORD dwCompressedSize; // Compressed size of file
	DWORD dwFullSize; // Uncompressed size of file
	DWORD dwFlags; // Flags for file
};

struct HASHTABLEENTRY {
	DWORD dwNameHashA; // First name hash of file
	DWORD dwNameHashB; // Second name hash of file
	LCID lcLocale; // Locale ID of file
	DWORD dwBlockTableIndex; // Index to the block table entry for the file
};

You could create a copy of the file where only the name is changed (and the dwName hashes). Untested, but I assume it would work.

The main issue is that most API's don't support modifying these flags directly. So it requires a bit of knowledge to write something up that supports it. On this forum, DSG and Barade probably would know how to implement something like this (I think I recall DSG mentioning this idea in the first place, and Barade wrote his own MPQ lib for wc3lib). Zezula also responds fairly often to his issue tracker on StormLib's github, he seems like a pretty chill dude overall to ask questions to.

Anyway, it is a cool project nonetheless. I would love to write it up, but I am not good at following up quickly on promises. So I'll just leave this info here in case anyone else wants to take a whack at it.

 

[LeP]

[...]
Anyway, it is a cool project nonetheless. I would love to write it up, but I am not good at following up quickly on promises. So I'll just leave this info here in case anyone else wants to take a whack at it.

I coded a quick Proof of Concept and it does seem to work. In practice one would have to check some more things and stuff but here it is.

The zip contains a basefile.w3x, the .exe and the source.
If you want to see what it does check the Makefile.

The .w3x contains an imported icon and uses the icon for some spell. Then another spell uses another path which ofc isn't found in the .w3x. But after adding an entry to the hashtable the second path will also be found.

 

[PurgeandFire]

Awesome! Nice work, and thanks for actually including the source.

 

[Zwiebelchen]

I coded a quick Proof of Concept and it does seem to work. In practice one would have to check some more things and stuff but here it is.

The zip contains a basefile.w3x, the .exe and the source.
If you want to see what it does check the Makefile.

The .w3x contains an imported icon and uses the icon for some spell. Then another spell uses another path which ofc isn't found in the .w3x. But after adding an entry to the hashtable the second path will also be found.

Wait, so if I have to replace any game files for this, I can always just extend the 8MB map size limit with the removal tool, right? So the whole point kind of gets moot?

 

[LeP]

Wait, so if I have to replace any game files for this, I can always just extend the 8MB map size limit with the removal tool, right? So the whole point kind of gets moot?

wat? I don't really understand where you're coming from.
the tool allowes to have the same file be available under two (or more) different paths without consuming more memory.
it's not about replacing any game files. it works solely on the map mpq.

 

[Zwiebelchen]

wat? I don't really understand where you're coming from.
the tool allowes to have the same file be available under two (or more) different paths without consuming more memory.
it's not about replacing any game files. it works solely on the map mpq.

Wait, what? Okay, in this case, I probably couldn't follow you here.
So what do I actually have to do to make this work? Import the extra files directly into the map MPQ with an MPQ editor?

 

[LeP]

Wait, what? Okay, in this case, I probably couldn't follow you here.
So what do I actually have to do to make this work? Import the extra files directly into the map MPQ with an MPQ editor?

No, looking at my zip would be a good first step.
The tool takes three parameters: the map, the path of a file already in the mpq and the new path.
After running it the file should be availavle under both pathes.

But as I said it's a proof of concept.
It only works when there's still free space in the hash table and the file shouldn't be encrypted or both files need to have the same filename (but not the same path overall).
A more polished version could handle both cases.

 

[Zwiebelchen]

No, looking at my zip would be a good first step.
The tool takes three parameters: the map, the path of a file already in the mpq and the new path.
After running it the file should be availavle under both pathes.

But as I said it's a proof of concept.
It only works when there's still free space in the hash table and the file shouldn't be encrypted or both files need to have the same filename (but not the same path overall).
A more polished version could handle both cases.

I see; this would actually be pretty cool as a standalone finalizer tool.

Possibly one that reads out all imported blps from an input map with BTN in it that doesn't have a DISBTN equivalent automaticly and sets the alternate path accordingly.

This could even be integrated into MapOptimizer as a checkbox: "Create dummy DISBTN file-paths"

 

[jopi]

So, with this tool you could use one and the same icon.blp as BTN and DISBTN (without importing it two times, obviously), right? That sounds pretty cool.

 

[LeP]

I see; this would actually be pretty cool as a standalone finalizer tool.

Possibly one that reads out all imported blps from an input map with BTN in it that doesn't have a DISBTN equivalent automaticly and sets the alternate path accordingly.

This could even be integrated into MapOptimizer as a checkbox: "Create dummy DISBTN file-paths"

I wont write such a "map optimzer" but i'll hopefully release a tool which can "link" (that's what i call it now) and check for all the corner cases.

So, with this tool you could use one and the same icon.blp as BTN and DISBTN (without importing it two times, obviously), right? That sounds pretty cool.

Yes, that works if you want it. I tested that. Personally i don't like the look of normal icons for DISBTNS. If you want smaller maps i have developed some other tools. [1] [2]
Maybe there are other uses? I don't know.

Awesome! Nice work, and thanks for actually including the source.

np. the code was already available (not licenced though); i just copied some code from my mpqcompressor.

 

[Dr Super Good]

It is important to note that this will not work on encrypted files. Encrypted files use the file name as a key and hence having a different file name will result in a different file block. Most map protectors and optimizers make all files encrypted.

The demo source code provided is not portable either. It depends on specific struct packing behaviour which is compiler dependant. MSVC for example might build the struct types with more padding so members no longer align with the data. Hence why read/write serialization methods are required which explicitly translate data to and from members. Let us not forget about the endian of the system.

If MPQ ever gets a proper file system implementation this would be supported as "Symbolic link" and require the destination file not be encrypted. Either an additional meta-data file would be used to keep track of the "original" file or every time the file system (archive) is mounted one is chosen in an implementation dependant way with all others linking to it.

 

[PurgeandFire]

That's good info to know. I suppose the process would have to go -> Optimize (without deleting listfile) -> Decrypt icon blps in mpq (i.e. read file to buffer, remove file, add again without encryption) -> and then implement the changes (and leave them unencrypted) -> remove listfile.

 

[Zwiebelchen]

That's good info to know. I suppose the process would have to go -> Optimize (without deleting listfile) -> Decrypt icon blps in mpq (i.e. read file to buffer, remove file, add again without encryption) -> and then implement the changes (and leave them unencrypted) -> remove listfile.

... which raises the question is someone is able to or has the time to write such a thing?

Obviously, for multiplayer maps, this could be extremely interesting, as the only two cases in which disabled icons are needed in multiplayer maps is for:
1) being out of range of shops
2) checking out the inventory of other player's heroes

And I'd say in both cases it doesn't really matter if the icons are greyed out or not.



In singleplayer maps, the disabled icons will also show for abilities whenever the option menu is opened, but you don't care for filesize in singleplayer, so why would you care?


Considering the average icon BLP is 4-6 kilobytes, this could save up several hundreds of KB if someone has roughly 100 icons imported, which isn't all that rare. I probably use about 60 or 70 for items alone.

 

[PurgeandFire]

Yeah it should be doable. LeP already did the hard part. The rest is boring. The optimizer stage would be done with Vex's optimizer. I think there is a flag so that it doesn't remove the listfile, but I don't remember honestly. Technically you don't need a listfile, but if you use StormLib or something similar, it makes it easier (since you can just filter out the files with a *.blp mask and then go through the icons one by one). Reading the file would be done with SFileReadFile, removal would be done with SFileRemoveFile, and adding with SFileAddFileEx without the encryption flag. Then you would run LeP's file and it would be done. (and you can proceed to remove the listfile if you desire)

At least, that is how I'd approach it. LeP probably has his own funcs for his mpq compressor so he might take a more direct approach. Regardless, it'd probably take longer to test than it would take to code.

 

[Rui]

If your problem is about space and you can opt to just not have a DISBTN icon, then you could try doing what they did in "The Kingdom of Kaliron". They replaced the 64x64 lime/green icon itself for a transparent one. Though a few of the buttons still have a disabled version, practically all ability icons, when disabled, just appear as if they were an empty command card template. This is less noticeable because they have a talent tree system of their own — it'd look bad on regular heroes learning new abilities.

 

[Ezekiel12]

If your problem is about space and you can opt to just not have a DISBTN icon, then you could try doing what they did in "The Kingdom of Kaliron". They replaced the 64x64 lime/green icon itself for a transparent one. Though a few of the buttons still have a disabled version, practically all ability icons, when disabled, just appear as if they were an empty command card template. This is less noticeable because they have a talent tree system of their own — it'd look bad on regular heroes learning new abilities.

That was already proposed, the info missing is what path the green texture has.
Do you know?

 

[Rui]

I'm afraid I don't. I thought it could be set on Gameplay Interface options?

 

[Zwiebelchen]

This is not about changing the DISBTNs to something less intrusive. There are already ways to do that, as described in the TO (clever path selection for enabled icons).

This is about using the corresponding BTN icon for DISBTN, as just removing DISBTNs completely looks dumb.