Issue parsing object data files

[Drake53]

This guide once taught me that the end of a modification structure ends with either 0, or the original object ID, or the new object ID, and that "when reading files you can use this to check if the format is correct".

However, I've recently found this map, made in v1.32.6, where this doesn't always seem to be the case. In its war3map.w3u file, the value is occasionally "dumy", which is different from the unit's original/new object ID. The ability data file has the same issue.

So I was wondering, since this map was made with the latest (non-PTR) reforged version, is this a deliberate change to the file format? If so, what does the value mean if it's neither the original nor the new object ID? Or is it simply a bug in reforged that basically corrupts your object data files for any tool that actually bothers to validate the file it parses?

 

[GhostWolf]

I don't see what that corrupts. If the file works, then it works. Just because you (or someone in the past) assumed this can be used for validation, it doesn't mean much without seeing Blizzard's code.

Either way, I personally don't know what that field is used for, it looks useless to me.

If there isn't some "dumy" object in the map, then it can be just that, a dummy value.

If you want to test this field, put random data in it and see if the game cares.

 

[Drake53]

I did some testing (in 1.31.1 world editor) and noticed that the value of this unknown field is always 0 when you create a new modification, so I had to create an object data file with external tools to test.
In the test file I have one new unit (copy of peasant) with one field modified (gold cost 0), and that modification's unknown field was set to be the same as the new object ID.
When I created a copy of this unit in the world editor, its modification had the same unknown value, and editing this modification (gold cost 0 -> 300) did not change it either.

So when you assume the value must be 0, the original object ID, or the new object ID, you can see that copying a new unit in the world editor can generate invalid data.
However, I checked my test maps and it seems that this value is never really the object's new ID, only 0 or the original ID. So when you change the assumption that only 0 and the original ID are valid, and the new object ID (or any other value) is not valid, the world editor behaves correctly again when copying an object.

With that in mind...

I don't see what that corrupts. If the file works, then it works.

When a magic number, checksum, or validation value is incorrect, the file is technically corrupt. Of course you can still read it, for example if you start a .mdx file with "ABCD" but all the other data is valid. All you have to do in that case is ignore the first 4 bytes and it'll 'work'.

Just because you (or someone in the past) assumed this can be used for validation, it doesn't mean much without seeing Blizzard's code.

I still think this assumption is correct. People make mistakes, Blizzard too. Just because they invented the file format doesn't mean their code to read it can't have mistakes.

If there isn't some "dumy" object in the map, then it can be just that, a dummy value.

The value being "dumy" in this case was probably a coincidence. In the map's ability object data, I encountered "FFDg" where the original and new ID both started with 'A'.

If you want to test this field, put random data in it and see if the game cares.

I don't need to test to know the game probably won't care. I also checked the source code for HiveWE and it also skips over the bytes, so I guess that's the solution. It still bothers me though.

 

[GhostWolf]

If the game reads it, it isn't corrupt.
That is always the truth, because these files belong to the game. I have so many issues with how the game handles models, making them "just work" in many cases where the data makes no sense whatsoever. It doesn't make them corrupt, it just means the game does whatever it wants, and we need to follow suit if we are using these files.

This isn't at all like adding "ABCD" to the start of an MDX model, because the game won't load it.

As for a checksum, that value could easily be something that was used in some version and later stopped being used. How would you know the difference?
Honestly I never understood that assumption also when I implemented the map code.
Why would you have a "validation" member for each modification? Do you see such data in any other part of any file in WC3?

The thing that bothers me more is that you take the word of someone who looked years ago at the format, reverse engineered it, which is great, but then added all sorts of baseless assumptions.
Many assumptions have been proven wrong since, in many of the different file formats.
The only ones that can know for sure are Blizzard, and I doubt even they know at this point why many things are there, especially with how they handled the new formats

 

[Drake53]

If the game reads it, it isn't corrupt.

https://www.hiveworkshop.com/attachments/wc3-blp-buffer-overflow-jpg.154918/

This isn't at all like adding "ABCD" to the start of an MDX model, because the game won't load it.

I used this as a simple example, if I were to write my own parser I could decide to skip the first four bytes which should contain "MDLX".
But if it's the game's parsers you want to talk about, how about a different example. I think somewhere around patch 1.29 there were some changes, causing old textures (.blp) and models (.mdx) to no longer load or even crash the game. So before these patches, the game would read these files, which according to you implies they are not corrupt. But then in later patches, the game can no longer read them, does that mean the files are now suddenly corrupted? But they weren't before, and nothing about the files itself changed. This is what I meant when I say Blizzard can make mistakes too.

Why would you have a "validation" member for each modification? Do you see such data in any other part of any file in WC3?

I agree I don't think the validation is really needed for every modification. But to be fair there are other WC3 files with something similar. For example in .w3s files with format version 2 or higher, the sound's name and path strings are repeated. And though I can't know for sure this can really be used for validation, I'm going to do it anyways, so if someone using my tools ever inputs a file that proves me wrong, they will get an exception and can tell me my assumption was incorrect.

The thing that bothers me more is that you take the word of someone who looked years ago at the format, reverse engineered it, which is great, but then added all sorts of baseless assumptions.

In all files I've seen so far this assumption holds, so I don't think it's baseless. Of course the map I linked in my first post is an exception, but I don't even know for sure if the author used external tools or if only the vanilla world editor was used. I only know it was made in 1.32.6 by reading the .w3i file. As long as I can't reproduce it in 1.31 editor, I'm gonna assume it's a bug either in Reforged world editor, or some external tool the author may or may not have used.

 

[GhostWolf]

Oh come on, don't be a smarty pants and link to clear bugs in the game xD
Same with the patches in recent years, considering they are filled with more bugs than fixes, up to maybe the actual Reforged release, which still has many bugs that weren't there prior to this new age of warcraft...

Regardless, I just can't figure why there would be a need to add these extra bytes for validation.
I can see e.g. having a checksum for a file, makes sense.
But having extra 4 bytes per modification?
And not only that, but these 4 bytes can by definition have multiple different values?
I don't see the connection between that and validation.
I also don't see why someone would assume this to be the case to begin with.
Since when is repeating data any sort of form of validation? It's just useless data.
And since you now have a map that invalidates this assumption and (supposedly?) works in Reforged, well, there's your answer - it's not validation.

That being said, if you want to be sure, I suppose you should make a map for older game versions with different values and see if the game cares.
Or reverse engineer the saving/loading code, that will certainly show the truth.

 

[Drake53]

You know you keep saying to test and see if the game cares, but I already know the game doesn't care, because that's exactly my problem.

Just take MPQ files for example. You may know that these files get malformed by map protectors, so the world editor and maybe some third party tools can't open them. But the game doesn't care, even if the archive offset is negative, it'll just read data from BEFORE the file header.
I also saw some maps where the MPQ format version was set to 1, which wasn't used until The Burning Crusade, so Warcraft III shouldn't be able to read those, right? Nope, it reads them anyways. I honestly can't even tell if those maps really were in format version 1, or the value was just changed for protection, since I only implemented support for format version 0 in my code.
It makes me wonder why they even bother adding a version to the file format if they don't seem to use it. Sure they may check if it's not the first version when they need to know if data introduced in a newer format version should be read or not (like jass/lua in 1.31's format for war3map.w3i), but then you only check if the version is older than the known latest version. But checking if it's newer? No no no, let's just try to parse it anyways and if it fails we'll just print "level data missing or invalid". Can't even bother to make two separate error messages for that, missing or invalid, so I guess a third message saying newer versions are not supported is too much to ask.
What's even worse is the .doo file format. It has not one, but TWO versions, both of which have two possible values that I know of and guess what, I've seen all four possible combinations. So you'll just have to guess if a change/addition or whatever is linked to the first version or the second. If that's not bad enough, with reforged they added the 'skin' value to the format WITHOUT incrementing the version number. Good luck figuring out if you need to parse these extra four bytes or not.

I guess what I'm trying to say is it sucks to deal with Blizzard's file formats, but you're right, "the game does whatever it wants, and we need to follow suit if we are using these files". Otherwise I just end up with a bunch of failing tests in my test project.

 

[GhostWolf]

I sense a lot of passive aggression towards Blizzard in your post. I agree with all of it

There are so many inconsistencies and useless data in all of the file formats, we just have to live with them.

You'd think the new versions/formats would at least be better, but 18 years later they can still only make them worse.

Personally from my experience with models, I can see just why for example the game has constant lags for people, why it has poor FPS, and why it is now 30GB, even though it looks like a game from 10 years ago.

I don't know what type of people work for Blizzard, or who they outsource all of their work to, but this is the quality we have, nothing to do about it...

 

[Dr Super Good]

https://www.hiveworkshop.com/attachments/wc3-blp-buffer-overflow-jpg.154918/

That was a buffer underflow like error when parsing BLP textures. It has since been fixed (I hope).

This is why one has to make sure that a file provides all the data is claims it does. Just because it says it has 65,536 bytes of data does not mean the file is even that big to begin with. The correct behaviour would be to fail to load the file due to an unexpected end of file encounter.

I also saw some maps where the MPQ format version was set to 1, which wasn't used until The Burning Crusade, so Warcraft III shouldn't be able to read those, right? Nope, it reads them anyways. I honestly can't even tell if those maps really were in format version 1, or the value was just changed for protection, since I only implemented support for format version 0 in my code.

Warcraft III does not process the version field as version 0 was the only version of MPQ that existed when it was made. As such for the purposes of Warcraft III the field can be considered a dummy field. However for the purposes of third party applications that are compatible with many MPQ versions the field can be considered the version identifier. This is how the "protection" works since it corrupts that field tricking third party MPQ editors into treating the header as the incorrect version while Warcraft III keeps treating it as version 0. Logically when treated with the incorrect version the archive will seem corrupt since garbage data will end up being parsed.

Another form of protection that works in a similar way is to stick a corrupt MPQ 1B header in front of the MPQ 1A header. Warcraft III is not aware of the MPQ 1B structure and so skips it when searching for the MPQ 1A header. Third party editors will detect and parse this MPQ 1B header which causes the archive to seem corrupt.

If Blizzard updated the MPQ code to support multiple versions of MPQ then all such "protected" maps would no longer be playable since the very protection would break Warcraft III.

It makes me wonder why they even bother adding a version to the file format if they don't seem to use it. Sure they may check if it's not the first version when they need to know if data introduced in a newer format version should be read or not (like jass/lua in 1.31's format for war3map.w3i), but then you only check if the version is older than the known latest version. But checking if it's newer? No no no, let's just try to parse it anyways and if it fails we'll just print "level data missing or invalid". Can't even bother to make two separate error messages for that, missing or invalid, so I guess a third message saying newer versions are not supported is too much to ask.

I thought it did warn that a map was made with a newer version of World Editor. Used to see this a lot when using JNGP which used an old World Editor version.

Good luck figuring out if you need to parse these extra four bytes or not.

Likely fed in as a parameter from parsing another file. In theory only 1 of the map data files needs to store what version of Warcraft III was used to make the map. One can then infer the format versions of all other files from that as they are well defined at build time. Given that these files are only intended to be used by Warcraft III, adding a version field to them might not have made much sense in the first place.

 

[Drake53]

I thought it did warn that a map was made with a newer version of World Editor. Used to see this a lot when using JNGP which used an old World Editor version.

This warning only applies to the world editor version (and I think the game version, that was introduced to the file format in v28 (1.31), can also trigger this warning, since I remember getting this warning multiple times when opening one of my test maps).
The warning does not appear when the map's war3map.w3i file format version is newer than the world editor can handle. I just tried opening a map with file format v31 (1.32 / reforged) in 1.31.1 PTR editor and got the "Level Info data missing or invalid" error.

Likely fed in as a parameter from parsing another file. In theory only 1 of the map data files needs to store what version of Warcraft III was used to make the map. One can then infer the format versions of all other files from that as they are well defined at build time. Given that these files are only intended to be used by Warcraft III, adding a version field to them might not have made much sense in the first place.

Even if this is true, it doesn't make sense to me since all file formats have their own version, and .doo files supposedly even have two versions.
Anyways, the way I solved this issue is to check the byte where the skin value gets inserted into the file format. That byte normally has a close to zero value, and the four bytes of the skin value should be alphanumeric (let's say spacebar is allowed too, then the lowest possible value is 32), so if the byte's value is >= 0x20, I read the skin value. I don't know if this is the correct approach though.