1. Updated Resource Submission Rules: All model & skin resource submissions must now include an in-game screenshot. This is to help speed up the moderation process and to show how the model and/or texture looks like from the in-game camera.
    Dismiss Notice
  2. DID YOU KNOW - That you can unlock new rank icons by posting on the forums or winning contests? Click here to customize your rank or read our User Rank Policy to see a list of ranks that you can unlock. Have you won a contest and still havn't received your rank award? Then please contact the administration.
    Dismiss Notice
  3. Dismiss Notice
  4. We have recently started the 16th edition of the Mini Mapping Contest. The theme is mini RPG. Do check it out and have fun.
    Dismiss Notice
  5. Dismiss Notice
  6. The Highway to Hell has been laid open. Come along and participate in the 5th Special Effect Contest.
    Dismiss Notice
  7. Check out the Staff job openings thread.
    Dismiss Notice
Dismiss Notice
60,000 passwords have been reset on July 8, 2019. If you cannot login, read this.

Stop the CASC madness

Discussion in 'Patch & Reforged Discussion' started by Selaya, Jul 6, 2018.

  1. Frotty

    Frotty

    Wurst Reviewer

    Joined:
    Jan 1, 2009
    Messages:
    1,449
    Resources:
    11
    Models:
    3
    Tools:
    1
    Maps:
    5
    Tutorials:
    1
    Wurst:
    1
    Resources:
    11
    Sounds like a you problem.
    The only reason *you* can run .exe files like magos without "installing anything" is, because windows ships the necessary libraries.
    MacOS and most linux distributions ship with Java, thus you can run wurst and matrix eater out of the box, but can't run .exe files like Magos without installing extra libraries.
    You really don't know this?
     
  2. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,033
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    Which is why despite many tests, there are still users getting errors when dealing with software. There are just too many variables and eventualities that derive from resources outside of the system. Much more so for developers. Things in Warcraft III are not really straightforward, more so with fluctuating versions. Next to writing tool XYZ, we have to educate people.
     
  3. Death Adder

    Death Adder

    Joined:
    Nov 3, 2013
    Messages:
    886
    Resources:
    0
    Resources:
    0
    Couldn't they just add one of those checkboxes to optionally install other software alongside the game?

    Like it's not uncommon to have "recommended" "minimal" and "full" (as well as custom) installation options.
     
  4. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,683
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    That does not solve the problem of file duplication between the standard MPQs and the patch MPQ. Additionally BattleNet Application probably does not support MPQ anymore seeing how Blizzard migrated all their games from it years ago.
    What drawbacks?
    Probably... Most third party tools would probably not notice the difference. You could even dump all the files into a single MPQ with the others empty and they would work.
    I do not see why moving to CASC would prevent one from making models. CASC and MPQ have nothing to do with MDX, MDL and BLP.
    From what I can tell it is nowhere near as hard as one thinks. Most of the CASCVIEW library is aimed at interfacing with BattleNet to query and pull archive information such as encryption keys and other files. To read data from an already built archive that is not encrypted I do not think a quarter of the amount of code is needed.

    The main problem is that one literally has to port parts of the source code as all documentation on CASC is a load of gibberish.
    You can check texture paths using notepad... They are like the only plain text strings near the top of the file.
    I plan to look into a Java CASC library some time in the future. Just currently I am trying to optimize something completely different and unrelated.
    WC3 will never need Java. Third party tools might but that is hardly a problem as Java really should be installed on every computer...
    The game installs everything it needs... If it does not, report it as a bug to Blizzard with their installer.

    Third party tools however might have different requirements and some do not have installers. For example Java for some tools.
     
  5. deepstrasz

    deepstrasz

    Map Reviewer

    Joined:
    Jun 4, 2009
    Messages:
    12,045
    Resources:
    1
    Maps:
    1
    Resources:
    1
    Why? I'm not using a single program which needs it.
     
  6. Chaosy

    Chaosy

    Joined:
    Jun 9, 2011
    Messages:
    10,654
    Resources:
    18
    Maps:
    1
    Spells:
    11
    Tutorials:
    6
    Resources:
    18
    A lot of games in java, it really shouldn't come as a surprise that some programs require it.
    Besides, who cares it is not a 100gb download.
     
  7. Kaijyuu

    Kaijyuu

    Joined:
    Jun 2, 2004
    Messages:
    848
    Resources:
    0
    Resources:
    0
    Ugh, I hate java and deliberately uninstalled it.
     
  8. my man
     
  9. Chaosy

    Chaosy

    Joined:
    Jun 9, 2011
    Messages:
    10,654
    Resources:
    18
    Maps:
    1
    Spells:
    11
    Tutorials:
    6
    Resources:
    18
    I cannot understand that.

    If I like a game/program I could not care less about what it was built on.
     
  10. deepstrasz

    deepstrasz

    Map Reviewer

    Joined:
    Jun 4, 2009
    Messages:
    12,045
    Resources:
    1
    Maps:
    1
    Resources:
    1
    [​IMG]
     
  11. Kaijyuu

    Kaijyuu

    Joined:
    Jun 2, 2004
    Messages:
    848
    Resources:
    0
    Resources:
    0
    The main reason to not have java on your computer is its long, long history of security vulnerabilities. To be fair though, vulnerabilities mostly come from java browser plugins, not standalone programs, but still.

    Also it's an awful language.
     
  12. Selaya

    Selaya

    Joined:
    May 30, 2013
    Messages:
    186
    Resources:
    0
    Resources:
    0
    These few MiB will probably not kill anyone. They haven't killed anyone from 2002-2018 when people had much worse internet; unlikely these will kill anyone going from there.

    Also, I literally cannot see any kind of causality nor correlation between the fact that the battle.net app not supporting mpqs to war3 having to migrate away from it. Why would it be necessary for the app to make sense of any of war3's game contents?
     
  13. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,683
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    How else would it be able to programmatically update the patch archive? Also how would pre-downloading patches work?
    Java is pretty quick at having vulnerabilities patched. Also it being an "awful" language is an opinion...
     
  14. deepstrasz

    deepstrasz

    Map Reviewer

    Joined:
    Jun 4, 2009
    Messages:
    12,045
    Resources:
    1
    Maps:
    1
    Resources:
    1
    Like they did it before, downloading a 50mbs patch and applying it on the game through the patch's .exe? Then there would be no need for Java and CASCO or whatever it's called.
     
    Last edited: Jul 8, 2018
  15. Retera

    Retera

    Tool Reviewer

    Joined:
    Apr 19, 2008
    Messages:
    853
    Resources:
    25
    Models:
    17
    Tools:
    2
    Maps:
    6
    Resources:
    25
    That was definitely my bad for trying to remember what was said in a PM from half a year ago. No offense intended. Perhaps it was me that felt my codebase was unprofessional, and I was simply telling myself what I expected to hear as I read your words at that time.

    @deepstrasz I am not sure you understand how Java is relevant to this discussion. I was the first one who mentioned it, and I only did so because I use it. Nothing in the game uses Java and nothing is going to.
    But I have been working on a library of Warcraft 3 modding code for myself and others to use that is written entirely in Java, and only functions by loading game data. It uses several libraries written by Dr Super Good. Frotty, for example, is another Java developer who is a part of a team that made a similar library. They are more aware of how to work as a team, and use build systems that allow them to reference each other's projects across the internet, so they rewrote most of the Warcraft 3 libraries that I use with their own versions. That's just a matter of personal preference.
    "CASC" is the name used for a storage system that is similar to MPQ. I always visualized MPQ files as something similar to ZIP files when I was first learning about them. "CASC" is somewhat similar. It is a storage system.

    "Downloading a 50 mbs patch and applying it to the game" only ever worked because they gave you a program that downloaded new data -- stored in an MPQ storage system -- and saved that on your computer, replacing old data from older patches. The code for how to parse and understand the MPQ storage format was included inside of Warcraft III itself. But, it was binary, so fans weren't sure how to use it originally. It took them years to figure it out. Initially they discovered that the library called "Storm.dll" from the game was able to open this storage format. They hacked this library in many ways to obtain standard game models, like the Footman, etc, until Blizzard released patches so that it could no longer be hacked. However, at that point, the fans already had "Storm.dll" with a working MPQ library, and the storage format did not change. So, they continued to modify it and developed fan-made derivative works until they created "StormLib.dll" over the years. Eventually, even more years after that, people ported the StormLib to Java, and so for a while my code used "JStormLib". Then, eventually, Dr Super Good and others entirely replaced this with complete rewrites of MPQ storage system parsers.

    The concern here is that Blizzard invented a new storage system. "CASC" is similar to ZIP, again, in that it is a way of storing data. However, the data is stored in many files instead of one file. This means that Blizzard can and will still give you a patch and apply it on the game through an exe. But the EXE will be invisible to you, inside of Battle.net code, because they want it to be easy for users. In addition, there's a new change to the system now. With CASC, instead of downloading a 26 MB "War3Patch.mpq" archive, as the game did many years ago, they can probably download 3 MB of "changes" into the CASC system and have it update itself, rather than replace itself with a whole new 26 MB download. This will greatly improve update bandwidth times for users as Blizzard continues to patch and improve Warcraft III.

    The reason I am concerned is that any software developers, including Java developers like myself, need to relearn and rewrite all of our Warcraft III data parsing libraries if we want to load data out of the game itself.
    Edit: And, actually, rewriting all of that is probably not that bad. If Frotty and the Wurst team get hit by a car tomorrow, all of their code is still around and is open source and somebody like Dr Super Good or me could fix it. If I get hit by a car tomorrow, somebody like Frotty's team could take up the Matrix Eater code and fix it if they really wanted to.

    But Vexorian was already hit by a car. So, nobody is going to update his old map optimizer or his other systems. The guy who wrote the War3ModelEditor is gone, along with the ability to compile his source code. So nobody is going to update his old stuff.
    Blizzard should not be held to the whimsical designs of people who are long gone, but I think that we should try not to let the modding community lose too much of what it can do. And that's why I think a standalone program that can view Warcraft 3 models using data stored in the CASC system should be released by Blizzard, so that guys like me can have several months to catch up with CASC.
     
    Last edited: Jul 8, 2018
  16. deepstrasz

    deepstrasz

    Map Reviewer

    Joined:
    Jun 4, 2009
    Messages:
    12,045
    Resources:
    1
    Maps:
    1
    Resources:
    1
    I get it but it's just that modders have to find workarounds yet again. If that Java stuff works to bring back the soon-dead tools to life, then go ahead.
    So, they're unpacking each time you're executing the game?
    Yeah, exactly. Would Java solve this? I'm not quite sure...
    Well, if they do it, at least, they should do it well to be on par with the oldies. It's them, who broke compatibility anyway.
     
  17. Selaya

    Selaya

    Joined:
    May 30, 2013
    Messages:
    186
    Resources:
    0
    Resources:
    0
    now if you would explain to me why you would want to do that when you can just rewrite 30 mib worth
    war3patch.mpq
    of instead and stop this asinine madness
    :thinking:

    given that most of the updates nowadays will just be incremental changes to
    blizzard.j
    ,
    common.j
    and the various slks it'd just be endlessly overwriting the exact same stuff in
    war3patch.mpq
    so it wouldn't even bloat! (in general you're all vastly exaggerating the bloat potential there; after even significant amounts of content update through the decades it and
    war3xlocal.mpq
    together weren't even remotedly getting close to 100 mib)
     
  18. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,683
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    With CASC the patches would likely be <<20MB.
    Java has nothing to do with Warcraft III. It is the programming language used by some third party (NOT BLIZZARD) tools.

    CASC is Blizzard's current archive system. All BattleNet application games use it from StarCraft to even non Blizzard games like Destiny.
    Yes, the zlib compressed files are unpacked into memory every time the file is loaded. Been done this way since MPQ so nothing new with CASC.

    Decompression is very efficient. It decreases the IO bandwidth needed to read files, which is often the largest loading time bottleneck, at the cost of some CPU time. Since many files have to be loaded at the same time the decompression CPU time can be efficiently masked by multi threading.
    A third party tool written in Java, such as the various ones that already exist, would.
    That is not their problem... It is more on the stupid third party developers for not making their tools future proof by doing stupid things like always expecting the game data to come in MPQs.
    In order for Warcraft III to be compatible with the BattleNet application? How many of the games in the BattleNet application still use MPQ for the main game data? 0?
    So? We can just use CASC and it will work out the same from a user experience.

    Honestly I did warn people this day was coming many years ago already. The fact no one has prepared is the surprising part.
     
  19. Selaya

    Selaya

    Joined:
    May 30, 2013
    Messages:
    186
    Resources:
    0
    Resources:
    0
    you're still missing the point, there is literally no gain from
    integrating
    war3 into the app like that, it doesnt use bnet2.0, nor will it (closed ecosystem will kill game just like sc2, besides the fact that it'd require an extensive rewrite of the engine), and it's not like war3's got a 30 gib download.

    it's probably (much) smaller than a modern games patch
     
  20. Naze

    Naze

    Arena Moderator

    Joined:
    Nov 12, 2007
    Messages:
    2,299
    Resources:
    6
    Icons:
    4
    Skins:
    1
    Maps:
    1
    Resources:
    6
    This would be optimal, but perhaps there would be a drawback on Blizzard's side by doing this? I mean, I don't know how much are they willing the userbase to decrypt their CASC files, considering it is the same filetype used in other games that are not open to modding like Overwatch (and a game that's not even from Blizzard - Destiny). Still I'm hoping your suggestion will be taken in consideration, though deep down I'm skeptical.