Research into Warcraft III Reforged's Classic Mode (SD models and visuals)

[StormKnight]

In our research regarding Standard Definition visuals, also commonly referred to as Classic graphics.
• It is evidently proveable that post 1.32 textures of models and special effects models have artifacts/pixelation that the originals
did not contain.

- Which means all of the visuals in S~D have lesser quality compared to any given pre 1.32 patch.

• The methods used to confirm this included:
- The source Blizzard textures available in the War3ArtTools zipfile.
- Setup of an ingame map with no interruption or involvement of 3rd party tools, including loading ingame textures as images
flat on the ground to compare, with a pitch black tileset background for maximum possible clarity.
- Extracting textures for Blp and DDS files, converting them to the loseless PNG format and comparing them side by side.

• Finding the problem is obvious, but suggesting solutions is another story, these should be good solutions to the problem:

- We are highly suspecting that using the very format of DDS itself produces bad results for the S~D model textures, hence...
one solution is to reimporting all of them back into the game using blp format.
- There is also suspecion that the files may have been directly converted from blp to dds (lossy on lossy image format) that
might have caused this, while taking into consideration the textures might have actually not gotten a possible retouch after
conversion which is usually a standard. we also don't know what type of DDS format Blizzard used, if they are able to keep the
DDS format but do a proper conversion with manual retouching with internal tools to make sure artifacting and pixelation
is reduced, it is obviously the best solution.
- A very humble solution should resources and manpower be lacking is also to only, specifically, convert all of the possible
special effect peices attached to certain models such as Elementals and Phoniex and Team glows/hero glows which are
mostly made out of sfx particles and standalone SFX models themselves with a Source to Blp conversion. while leaving the rest
of the SD models on DDS. since the most obvious of this problem is visible ~ on those pieces.

Credits to: BogdanW3, Retera, Kacpa2 and Ghostwolf for assistance and insight.
Affiliated Link (Twitter)

Small Note

I originally decided to bring out the big guns and do an exceptionally massive image comparison document, but, i'm not sure if Blizzard will even respond to it, but you may find some files, here, however, i will put effort into greatly expanding this should they show signs of communication ~ any update to the file will be documented by date.

October 3rd 2021: The only update prior to any news or signs of further development of the game by Blizzard.
Now includes terrain texture comparison folder, and an additional video of the Water Elemental, credits to Retera.

 

[Kam]

Thank you for putting this together!

 

[Retera]

If it helps, looking at typos and mistakes in the various WC3 builds and patches leads me to the following conclusions about the technicalities of how this stuff probably works at the office at Blizzard:

1. There is some build system to "compile a War3 build" that can be released to players. At one point some stuff I read in the game data suggested it used Jenkins build system, so that might be what it still uses, although I obviously have no way of knowing for certain.
2. This build system was not used for the patches prior to 1.27 (prior to "Classic Games Team"), which were instead built off of some local windows mount located at E:\Drive1\Temp\BuildWar3x\War3\Source\ or something like this, but even long prior to Reforged the new team was changing to the build systems like Jenkins
3. On the older builds, the version-controlled source assets for the game were stored as .tga images which is an old format that was not lossy to my knowledge. So, the .tga images are larger files than their compiled, often JPG-compressed BLP counterparts (which is lossy). So, already since 2002 there was a compile process when things would build and convert all TGAs to BLPs during the process of compiling Warcraft III. This would slightly reduce their quality, but notably there were a lot of fine-tuned details, such as how the BLP textures on units have many repeating smaller copies of the same texture inside of them called MipMaps that are used for 3d Rendering, whereas a lot of the in-game BLP icons did not include this same metadata. My experience with Hive Workshop BLP tools showed me that people in our community believe there are 2-3 different "codecs" basically in the BLP format (@DrSuperGood might know more on this) and basically one of the BLP codecs is a lossy JPG compressed image whereas the other BLP codec is a paletted image stored a totally different way.
4. As just a community guy and a fan, I did not ever see any leaks regarding how the old system would decide whether a TGA compiling to BLP during the game compilation process would decide whether to be a JPG compressed (lossy) BLP format file, or a paletted BLP (maybe paletted was less lossy??? honestly I'm not 100% on that detail). So, although I don't know how this was decided, I gather that something during the original compile process might even have been deciding to preserve fine qualities in some textures while compressing others to save storage space in the ~1 GB resulting game binary release build in the 2000s.
5. Enter Reforged -- a lot of guys in a hurry to get novel ideas running who are well trained in general but not well versed in the above system that I just described. Instead of TGA, they used .TIF as their non-lossy format to store all of their source assets within the office.
6. It was obvious to Reforged developers that the .TIF assets exported from their Reforged HD new asset pipeline (Keith Seizemore's batch Maya exporter, according to notes included within Reforged itself) needed to compile to something compressed so that the Reforged binary release build would not be too big. So, they decided to use .DDS which is common in industry and at Blizzard nowadays (it is also used on SC2).
7. While the guys were getting the compile process going for the new Reforged assets, somebody probably said, "Hey! Why do we have 4+ file formats for textures? Let's try to clean this up a bit!" and so they changed the build process for legacy content so that it would convert legacy assets (probably still TGA inside the War3 source control systems at Blizz??) now to be DDS assets in the release build.
8. Of course as modders this looks at first like a jarring decision to us because we have 20 years of custom maps with all manner of file paths directly referencing "Path\To\Ingame\Legacy\Texture.blp" and now all these file paths were wrong because Reforged's binary release build had all legacy assets as .DDS, so the Reforged devs who knew us and were good to us made the game always do a branching check on every file load that if we ask the game engine for a .BLP that does not exist, the the game engine gives us a .DDS with the same name instead. (Alternatively, maybe this had nothing to do with us modders and was actually a lazy solution to solve internal systems that referenced BLP files extensively in many places also, and were too numerous to change.) So from a useability standpoint outwardly when modding Reforged, the "give the user a DDS if they ask for a BLP" check in the Reforged game engine generally solves all the problems at a cost of some negligable check during in-game asset loading.
9. This brings us to what we are reporting on today. It's not an issue of useability -- generally these assets are "functioning" in the Reforged combined client when it emulates legacy render mode -- but the problem is one of quality. It may be as simple as a case that came up when the programmer changed the build pipeline. He had a simple goal. He wanted to unify the output format (so legacy TGA source files, and also new TIF Reforged source files, both compiled to .DDS in the binary release builds). But what we see is that players are visually able to see the .DDS compression algorithm's artifacts in the Reforged SD client in cases where previously "glow" effects in particular were very smooth even on BLP release client textures.

So, regarding possible solutions, what comes to mind would be the following steps:

1. First verify that the uncompressed source assets for the "Classic" legacy artwork are preserved within the office. It is very likely that they are preserved. They may still be TGA files inside of the War3 version control system. Or, they might be .TIF now in which case please do verify that they were converted in a lossless manner. It is likely that the conversion from TGA to TIF would have been lossless, though, because both of these are generally lossless formats, so that is good. But if I had to guess I would figure that for the sake of time the old assets were probably kept as TGA. (Either way, whether these files are TGA or TIF in the office, obviously this is not how players see them in the compiled game release client, which was why some people were worrying the original source files might no longer exist.)
2. Once the uncompressed source assets are safely in the version control (they probably already are) we are hoping a programmer could re-evaluate the decision to compress "Classic" legacy artwork to .DDS using the same lossy compression algorithm as the Reforged artwork during the (jenkins?) build process.
3. Similar to how I mentioned BLP has multiple "codecs" for storing images, .DDS is an industry standard and has more codecs and possibilities than BLP. So, a possible straightforward solutions would be
   1. For the new branch during the game compile process for the "Classic" legacy artwork, use a different DDS codec that is less lossy so it looks equally as high quality as how the game looked prior to the forced Reforged patch OR
   2. For the new branch during the game compile process for the "Classic" legacy artwork, just compile these files to BLP as it was in the build process prior to Reforged because we cannot escape BLP format and Reforged engine still supports it anyway. (A strategic move to unify everything with DDS assets would be good for a new game system that does not support existing Warcraft III maps, but at the time of writing there are literally tens of thousands of Classic custom user maps and the majority of them include texture files in BLP format that we are not at liberty to change to DDS for various technical reasons)

 

[Spacebuns]

lol MORE fuckups. What a trainwreck.

 

[Retera]

(@Spacebuns)
What do you hope to achieve by posting an appeal to emotion to turn people against the game on a thread where users were discussing the intricate and legitimate details of very real and fixable specific technical problems? Do you feel that there has not been enough commentary on the state of Reforged, seriously?

I made similar posts to the one you just did for years leading up to Reforged. They trusted me. They implemented my suggestions into the game, when I was just ranting and posting appeals to emotion. Please try to understand the consequences of your actions. We collectively are the trainwreck. I am the trainwreck. I'm not interested in your opinion. You're either going to help fix technical problems, or you're not. Do you have something you needed to say/post @Spacebuns ? Or are you just posting in the hopes of getting validation to be upset, because you lost control and once you're not in control you surely aren't playing Warcraft III?

I certainly posted like that many times. It is a sickness upon this land. I should not be here on this forum. I should not post this reply to you.

In this world, there are technical problems with legitimate solutions. We either solve them, or we do not. Let us please solve them.