1. Updated Resource Submission Rules: All model & skin resource submissions must now include an in-game screenshot. This is to help speed up the moderation process and to show how the model and/or texture looks like from the in-game camera.
    Dismiss Notice
  2. DID YOU KNOW - That you can unlock new rank icons by posting on the forums or winning contests? Click here to customize your rank or read our User Rank Policy to see a list of ranks that you can unlock. Have you won a contest and still havn't received your rank award? Then please contact the administration.
    Dismiss Notice
  3. We have recently started the 16th edition of the Mini Mapping Contest. The theme is mini RPG. Do check it out and have fun.
    Dismiss Notice
  4. Dismiss Notice
  5. The Highway to Hell has been laid open. Come along and participate in the 5th Special Effect Contest.
    Dismiss Notice
  6. Check out the Staff job openings thread.
    Dismiss Notice
Dismiss Notice
60,000 passwords have been reset on July 8, 2019. If you cannot login, read this.

Getting rid of DISBTN ... some theorycrafting

Discussion in 'The Lab' started by Zwiebelchen, Jul 1, 2015.

  1. Zwiebelchen

    Zwiebelchen

    Joined:
    Sep 17, 2009
    Messages:
    6,793
    Resources:
    12
    Models:
    5
    Maps:
    1
    Spells:
    1
    Tutorials:
    1
    JASS:
    4
    Resources:
    12
    So, has anyone found a good solution to get rid of DISBTNs yet?

    As it's very annoying to have to import buttons as active and greyed-out variants in order to prevent the ugly green boxes and also wastes a lot of file space, I'm looking for a way to get rid of the DISBTNs, without looking ugly.


    I know that you can actually name all your buttons the same, but place them in different folders, so that you can use a uniform DISBTN for all of them:

    Example:

    Active buttons:
    "war3mapImported\Armors\001\BTNArmor.blp"
    "war3mapImported\Armors\002\BTNArmor.blp"
    "war3mapImported\Armors\003\BTNArmor.blp"
    "war3mapImported\Armors\004\BTNArmor.blp"

    And the shared disabled button for all of them:
    "ReplaceableTextures\CommandButtonsDisabled\DISBTNArmor.blp"


    However, this obviously comes with the caveat of all buttons looking the same when disabled. Now this is not a real problem if all you want is avoiding the green box. But it's definitely not the optimal solution either, especially if you have visible inventories for other players enabled (they will always show the disabled icon).


    Is it possible to trick or fool WC3 into thinking your BTN is also a DISBTN at the same time by clever use of the file structure? So, for example, placing your button in "ReplaceableTextures\CommandButtonsDisabled\" by default, naming it DISBTN?

    Active icon: "ReplaceableTextures\CommandButtonsDisabled\DISBTNArmor.blp"?
    Disabled Icon: "ReplaceableTextures\CommandButtonsDisabled\DISBTNArmor.blp"

    I suppose it will try to look for a "DISDISBTNArmor.blp", but I'm not sure on this one?


    Another thought:
    If it's only about filesize, will ZLib compression effectively remove the filesize footprint of a disabled button that is exactly the same icon as the enabled button? So my idea was to import the same icon for both paths; theoretically, the whole icon should be recognized as a "word" then, effectivly compressing the 6+6kb for both icons into 6kb for a single icon.
     
  2. jopi

    jopi

    Joined:
    Aug 30, 2007
    Messages:
    267
    Resources:
    0
    Resources:
    0
    I'm really not sure if this is right, but as there are 'On-Cooldown-Models' maybe the DISBTN is also applied as an overlay on top of the actual icon? in That case you could go with your solution naming all BTNs the same and use a general DISBTN texture with a little bit of alpha so the icon below it is visible through it.

    I can remember that I read something about DISBTNs and file size reduction over on playdota.com. But this was years ago and I can't remember any actual facts about it.
     
  3. Zwiebelchen

    Zwiebelchen

    Joined:
    Sep 17, 2009
    Messages:
    6,793
    Resources:
    12
    Models:
    5
    Maps:
    1
    Spells:
    1
    Tutorials:
    1
    JASS:
    4
    Resources:
    12
    Interesting idea; but I doubt that the DISBTNs are just overlays. Otherwise, the green boxes wouldn't appear, right?
    Also, I'm not even sure buttons can even have transparency or a working alpha layer.
     
  4. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    455
    Resources:
    0
    Resources:
    0
    No, files are compressed one by one - not the whole mpq as one.
     
  5. jopi

    jopi

    Joined:
    Aug 30, 2007
    Messages:
    267
    Resources:
    0
    Resources:
    0
    yes, that would be way to easy. I think what they did in dota, was to take the BTN icon, lessen the saturation a lot (nearly to greyscale) and apply a different icon border than the normal DISBTN icon border. I think it was with a lot less to no fade from the black border to the icon texture. This might save up some filesize but I'm not sure.
     
  6. Zwiebelchen

    Zwiebelchen

    Joined:
    Sep 17, 2009
    Messages:
    6,793
    Resources:
    12
    Models:
    5
    Maps:
    1
    Spells:
    1
    Tutorials:
    1
    JASS:
    4
    Resources:
    12
    I'm not asking how to create DISBTNs, I'm asking how to get rid of actually needing them.
     
  7. PublishedShadow

    PublishedShadow

    Joined:
    Jun 2, 2013
    Messages:
    599
    Resources:
    2
    Maps:
    2
    Resources:
    2
    I wonder if that ugly green texture is actually set somewhere as a path that an icon will default to if a DISBTN can't be found.

    If that's the case then you could import a single DISBTN (lets just say a completely black icon or maybe even a transparent icon) and then rename that to OVERWRITE the green 'default' DISBTN.

    Surely that has to be somewhere?
     
  8. PurgeandFire

    PurgeandFire

    Code Moderator

    Joined:
    Nov 11, 2006
    Messages:
    7,426
    Resources:
    18
    Icons:
    1
    Spells:
    4
    Tutorials:
    9
    JASS:
    4
    Resources:
    18
    It *might* be possible. I don't think any API's directly support it, but judging by the mpq file definition:
    Code (Text):
    struct MPQFILE {
        MPQFILE * lpNextFile; // Pointer to the next FILEREC struct. Pointer to addresses of first and last files if last file
        MPQFILE * lpPrevFile; // Pointer to the previous FILEREC struct. 0xEAFC5E13 if first file
        char szFileName[260]; // Filename of the file
        HANDLE hPlaceHolder; // Always 0xFFFFFFFF
        MPQARCHIVE * lpParentArc; // Pointer to the ARCHIVEREC struct of the archive in which the file is contained
        BLOCKTABLEENTRY * lpBlockEntry; // Pointer to the file's block table entry
        DWORD dwCryptKey; // Decryption key for the file
        DWORD dwFilePointer; // Position of file pointer in the file
        DWORD dwUnk1; // Seems to always be 0
        DWORD dwBlockCount; // Number of blocks in file
        DWORD * lpdwBlockOffsets; // Offsets to blocks in file. There are 1 more of these than the number of blocks
        DWORD dwReadStarted; // Set to 1 after first read
        DWORD dwUnk2; // Seems to always be 0
        BYTE * lpLastReadBlock; // Pointer to the read buffer for file. Only used for incomplete reads of blocks
        DWORD dwBytesRead; // Total bytes read from open file
        DWORD dwBufferSize; // Size of the read buffer for file. Only used for incomplete reads of blocks
        DWORD dwConstant; // Seems to always be 1
        HASHTABLEENTRY *lpHashEntry;
        LPSTR lpFileName;
    };

    struct BLOCKTABLEENTRY {
        DWORD dwFileOffset; // Offset to file
        DWORD dwCompressedSize; // Compressed size of file
        DWORD dwFullSize; // Uncompressed size of file
        DWORD dwFlags; // Flags for file
    };

    struct HASHTABLEENTRY {
        DWORD dwNameHashA; // First name hash of file
        DWORD dwNameHashB; // Second name hash of file
        LCID lcLocale; // Locale ID of file
        DWORD dwBlockTableIndex; // Index to the block table entry for the file
    };
    You could create a copy of the file where only the name is changed (and the dwName hashes). Untested, but I assume it would work.

    The main issue is that most API's don't support modifying these flags directly. So it requires a bit of knowledge to write something up that supports it. On this forum, DSG and Barade probably would know how to implement something like this (I think I recall DSG mentioning this idea in the first place, and Barade wrote his own MPQ lib for wc3lib). Zezula also responds fairly often to his issue tracker on StormLib's github, he seems like a pretty chill dude overall to ask questions to.

    Anyway, it is a cool project nonetheless. I would love to write it up, but I am not good at following up quickly on promises. So I'll just leave this info here in case anyone else wants to take a whack at it.
     
  9. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    455
    Resources:
    0
    Resources:
    0
    I coded a quick Proof of Concept and it does seem to work. In practice one would have to check some more things and stuff but here it is.

    The zip contains a basefile.w3x, the .exe and the source.
    If you want to see what it does check the Makefile.

    The .w3x contains an imported icon and uses the icon for some spell. Then another spell uses another path which ofc isn't found in the .w3x. But after adding an entry to the hashtable the second path will also be found.
     

    Attached Files:

  10. PurgeandFire

    PurgeandFire

    Code Moderator

    Joined:
    Nov 11, 2006
    Messages:
    7,426
    Resources:
    18
    Icons:
    1
    Spells:
    4
    Tutorials:
    9
    JASS:
    4
    Resources:
    18
    Awesome! Nice work, and thanks for actually including the source. :)
     
  11. Zwiebelchen

    Zwiebelchen

    Joined:
    Sep 17, 2009
    Messages:
    6,793
    Resources:
    12
    Models:
    5
    Maps:
    1
    Spells:
    1
    Tutorials:
    1
    JASS:
    4
    Resources:
    12
    Wait, so if I have to replace any game files for this, I can always just extend the 8MB map size limit with the removal tool, right? So the whole point kind of gets moot? ;)
     
  12. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    455
    Resources:
    0
    Resources:
    0
    wat? I don't really understand where you're coming from.
    the tool allowes to have the same file be available under two (or more) different paths without consuming more memory.
    it's not about replacing any game files. it works solely on the map mpq.
     
  13. Zwiebelchen

    Zwiebelchen

    Joined:
    Sep 17, 2009
    Messages:
    6,793
    Resources:
    12
    Models:
    5
    Maps:
    1
    Spells:
    1
    Tutorials:
    1
    JASS:
    4
    Resources:
    12
    Wait, what? Okay, in this case, I probably couldn't follow you here.
    So what do I actually have to do to make this work? Import the extra files directly into the map MPQ with an MPQ editor?
     
  14. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    455
    Resources:
    0
    Resources:
    0
    No, looking at my zip would be a good first step.
    The tool takes three parameters: the map, the path of a file already in the mpq and the new path.
    After running it the file should be availavle under both pathes.

    But as I said it's a proof of concept.
    It only works when there's still free space in the hash table and the file shouldn't be encrypted or both files need to have the same filename (but not the same path overall).
    A more polished version could handle both cases.
     
  15. Zwiebelchen

    Zwiebelchen

    Joined:
    Sep 17, 2009
    Messages:
    6,793
    Resources:
    12
    Models:
    5
    Maps:
    1
    Spells:
    1
    Tutorials:
    1
    JASS:
    4
    Resources:
    12
    I see; this would actually be pretty cool as a standalone finalizer tool.

    Possibly one that reads out all imported blps from an input map with BTN in it that doesn't have a DISBTN equivalent automaticly and sets the alternate path accordingly.

    This could even be integrated into MapOptimizer as a checkbox: "Create dummy DISBTN file-paths"
     
  16. jopi

    jopi

    Joined:
    Aug 30, 2007
    Messages:
    267
    Resources:
    0
    Resources:
    0
    So, with this tool you could use one and the same icon.blp as BTN and DISBTN (without importing it two times, obviously), right? That sounds pretty cool.
     
  17. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    455
    Resources:
    0
    Resources:
    0
    I wont write such a "map optimzer" but i'll hopefully release a tool which can "link" (that's what i call it now) and check for all the corner cases.

    Yes, that works if you want it. I tested that. Personally i don't like the look of normal icons for DISBTNS. If you want smaller maps i have developed some other tools. [1] [2]
    Maybe there are other uses? I don't know.

    np. the code was already available (not licenced though); i just copied some code from my mpqcompressor.
     
    Last edited: Jul 2, 2015
  18. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,704
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    It is important to note that this will not work on encrypted files. Encrypted files use the file name as a key and hence having a different file name will result in a different file block. Most map protectors and optimizers make all files encrypted.

    The demo source code provided is not portable either. It depends on specific struct packing behaviour which is compiler dependant. MSVC for example might build the struct types with more padding so members no longer align with the data. Hence why read/write serialization methods are required which explicitly translate data to and from members. Let us not forget about the endian of the system.

    If MPQ ever gets a proper file system implementation this would be supported as "Symbolic link" and require the destination file not be encrypted. Either an additional meta-data file would be used to keep track of the "original" file or every time the file system (archive) is mounted one is chosen in an implementation dependant way with all others linking to it.
     
  19. PurgeandFire

    PurgeandFire

    Code Moderator

    Joined:
    Nov 11, 2006
    Messages:
    7,426
    Resources:
    18
    Icons:
    1
    Spells:
    4
    Tutorials:
    9
    JASS:
    4
    Resources:
    18
    That's good info to know. I suppose the process would have to go -> Optimize (without deleting listfile) -> Decrypt icon blps in mpq (i.e. read file to buffer, remove file, add again without encryption) -> and then implement the changes (and leave them unencrypted) -> remove listfile.
     
  20. Zwiebelchen

    Zwiebelchen

    Joined:
    Sep 17, 2009
    Messages:
    6,793
    Resources:
    12
    Models:
    5
    Maps:
    1
    Spells:
    1
    Tutorials:
    1
    JASS:
    4
    Resources:
    12
    ... which raises the question is someone is able to or has the time to write such a thing?

    Obviously, for multiplayer maps, this could be extremely interesting, as the only two cases in which disabled icons are needed in multiplayer maps is for:
    1) being out of range of shops
    2) checking out the inventory of other player's heroes

    And I'd say in both cases it doesn't really matter if the icons are greyed out or not.



    In singleplayer maps, the disabled icons will also show for abilities whenever the option menu is opened, but you don't care for filesize in singleplayer, so why would you care?


    Considering the average icon BLP is 4-6 kilobytes, this could save up several hundreds of KB if someone has roughly 100 icons imported, which isn't all that rare. I probably use about 60 or 70 for items alone.