Hosted Project: WME
- Joined
- Apr 19, 2008
- Messages
- 2,188
I'm posting this message here instead of on the thread about a hobgoblin model where @Archian originally asked me, @ThompZon and @twilac about it:
Oh yea, this is probably the result of the new formatting that I started saving with on Retera Model Studio 4.5 ("experimental") version right now. The idea is that we were previously wrong in our understanding of the Reforged MDX format, and it's actually possible to save Reforged models with more than 256 bones as long as there are only 256 used per geoset. Previously it was thought that no Reforged model could have more than 256 bones used because it was assumed that the
By changing the Matrices block not to be a repeat of bone information, but actually to be geoset-unique indexing into the bones, you could for example have one Geoset that uses bones 0 to 255 and then another that uses bones 256 to 511. And it would all still totally work. This allows for much larger custom models with unique animations.
But once I understood that and tried to support it, I turned back on the (previously only for "classic") generator when saving that creates the minimal matrices block necessary to specify whatever is in that geoset, and so it started breaking in 3rd party fan tools because they make this incorrect assumption even though the game itself is fine with the models. More recently I updated Retera Model Studio 4.5 so that it has a menu option to default to the "old formatting" so that incorrectly written 3rd party tools don't have to be updated, because there were a lot of user complaints on Retera Model Studio Users Group discord server that my new code was breaking their stuff (even though the Warcraft III game accepts these models and has since 2020 on Reforged release, technically, although probably nobody was making any that were formatted this way back then).
The code fix is basically a matter of copying an SD thing into an HD thing. Each vertex in SD has one
the cut corners solution was conceptually like this:
And because the indices in the matrices block are 4 byte uints but the indices in the vertex are 1 byte uints, that meant models using the "cutting corners" design cant' express themselves and explode if they have more than 255 bones, even if those bones were used by separate geosets. So, thus we have this incentive to support the 'true' value from what WC3 is doing, instead of cutting corners as we had in the open source tools for years.
Edit:
So, if it's not clear, assuming that the render stuff is upgraded correctly, the new understanding of the format is a superset of the previous understanding. As such, all old models load fine in an updated 'fixed' program (in the same way that all old models load fine on Reforged game client). But all new models don't load in old 'unfixed' programs (even though they still load fine in the Reforged game and always have all along even though we probably didn't try it).
Edit 2:
Based on the pseudo code above, I put in the
If you wanted to "not cut corners" on this second issue, it might be a really hellish change because Ghostwolf was already corner cutting even in the SD case to try to render in more of an HD paradigm where each vertex had up to 4 bones that were equally weighted, even though we know that on the original Warcraft 3 client there appeared to be possibly no limit to the size of matrices in the Matrices chunk. Ingame models go as high as 8 bones in one matrix in some cases on classic, and on Warsmash I ran into problems with that and was actively breaking things and bouncing ideas off of Ghostwolf who went back and fixed his viewer if I recall so it probably supports the classic models with 8 but in a weird way that hardcodes a special extended 8 mode or something for models detected to be large, so that it's "usually performant" and assuming only 4. And if you were writing a CPU program that didn't run on the graphics card and didn't take performance into account, you probably wouldn't want any of that and would want to just allow matrices of infinite size. Retera Model Studio does that in several places. But that's not really a performant solution -- it's just a hack for rendering raw data in a CPU intensive way so that software is easy to write and simpler for the programmer rather than for the machine. And thus do some people say that Retera Model Studio lags for them, even though Hive Viewer and/or Warsmash would not.
Oh yea, this is probably the result of the new formatting that I started saving with on Retera Model Studio 4.5 ("experimental") version right now. The idea is that we were previously wrong in our understanding of the Reforged MDX format, and it's actually possible to save Reforged models with more than 256 bones as long as there are only 256 used per geoset. Previously it was thought that no Reforged model could have more than 256 bones used because it was assumed that the
SKIN block, when it specified bone indices, was using ObjectId directly. But it turns out they were actually indices into the Matrices block which until now we were always generating to be a repeat of the bone information since it seemed to be that way on Reforged.By changing the Matrices block not to be a repeat of bone information, but actually to be geoset-unique indexing into the bones, you could for example have one Geoset that uses bones 0 to 255 and then another that uses bones 256 to 511. And it would all still totally work. This allows for much larger custom models with unique animations.
But once I understood that and tried to support it, I turned back on the (previously only for "classic") generator when saving that creates the minimal matrices block necessary to specify whatever is in that geoset, and so it started breaking in 3rd party fan tools because they make this incorrect assumption even though the game itself is fine with the models. More recently I updated Retera Model Studio 4.5 so that it has a menu option to default to the "old formatting" so that incorrectly written 3rd party tools don't have to be updated, because there were a lot of user complaints on Retera Model Studio Users Group discord server that my new code was breaking their stuff (even though the Warcraft III game accepts these models and has since 2020 on Reforged release, technically, although probably nobody was making any that were formatted this way back then).
The code fix is basically a matter of copying an SD thing into an HD thing. Each vertex in SD has one
VertexGroup value which is a 0-255 uint8_t numerical value (1-byte), and then on SD what we always did was to basically do geoset.matrices[thisVertex.VertexGroup] conceptually (this is pseudocode) and then use that to find which matrix to animate from (i.e. skeletal bone data). So for the trivial case of a matrix constructed from only one bone, we could imagine the overall lookup to be model.bones[geoset.matrices[thisVertex.VertexGroup][0]]. But on Reforged because their export tools that made all the Reforged models were usually constructing a set of matrices exactly parallel to the set of all bones in the model, previewers like ghostwolf cut corners and so on HD instead of finding the 4 skin bone indices like this:
Code:
for i in range(4):
bone = model.bones[geoset.matrices[thisVertex.skinBoneIndex[i]][0]]the cut corners solution was conceptually like this:
Code:
for i in range(4):
bone = model.bones[thisVertex.skinBoneIndex[i]]And because the indices in the matrices block are 4 byte uints but the indices in the vertex are 1 byte uints, that meant models using the "cutting corners" design cant' express themselves and explode if they have more than 255 bones, even if those bones were used by separate geosets. So, thus we have this incentive to support the 'true' value from what WC3 is doing, instead of cutting corners as we had in the open source tools for years.
Edit:
So, if it's not clear, assuming that the render stuff is upgraded correctly, the new understanding of the format is a superset of the previous understanding. As such, all old models load fine in an updated 'fixed' program (in the same way that all old models load fine on Reforged game client). But all new models don't load in old 'unfixed' programs (even though they still load fine in the Reforged game and always have all along even though we probably didn't try it).
Edit 2:
Based on the pseudo code above, I put in the
[0] hardcoded value because on Retera Model Studio preview I am still cutting corners and assuming that on HD every "matrix" in the matrices chunk has only one bone. This quite possibly could mean that i am still cutting corners and someone could construct a model that the game accepts and renders correctly, but that I do not, in the future. Someone could do some tested with hexedited binaries or something in the World Editor if they wanted to try it and see if you can combine the "classic" matrices made from multiple bones with the "HD" weights system so that each of the 4 weighted indices of a vertex each linked to 4 other bones or something, so that the motion of one vertex would depend on like 16 bones or something crazy. I haven't tested it Warcraft 3 supports that. I can't really imagine a legitimate use case why someone would want to do that, and there are other parts of Retera Model Studio that would process everything incorrectly and lose user data if someone tried to do that.If you wanted to "not cut corners" on this second issue, it might be a really hellish change because Ghostwolf was already corner cutting even in the SD case to try to render in more of an HD paradigm where each vertex had up to 4 bones that were equally weighted, even though we know that on the original Warcraft 3 client there appeared to be possibly no limit to the size of matrices in the Matrices chunk. Ingame models go as high as 8 bones in one matrix in some cases on classic, and on Warsmash I ran into problems with that and was actively breaking things and bouncing ideas off of Ghostwolf who went back and fixed his viewer if I recall so it probably supports the classic models with 8 but in a weird way that hardcodes a special extended 8 mode or something for models detected to be large, so that it's "usually performant" and assuming only 4. And if you were writing a CPU program that didn't run on the graphics card and didn't take performance into account, you probably wouldn't want any of that and would want to just allow matrices of infinite size. Retera Model Studio does that in several places. But that's not really a performant solution -- it's just a hack for rendering raw data in a CPU intensive way so that software is easy to write and simpler for the programmer rather than for the machine. And thus do some people say that Retera Model Studio lags for them, even though Hive Viewer and/or Warsmash would not.
Last edited:
