Model Optimizer and Converter v1.2.1

[loktar]

Alright, new version is up. Haven't tested it that much, so hopefully everything works well.
Btw @Macadamia I update the to do list between updates here, if you're interested: misc | HIVE

Most of the groundwork has been laid now, so future updates should be relatively straightforward

Edit: oops, it just crashes with any optimization lol, brb

 

[Macadamia]

Well I couldn't approve your ToDoList more ^^

I really like the growing number of options, and can't wait to test linearization's first implementation.
The way you re-arrange options is a nice move indeed.

I also experienced the crash, but as always I stay tuned

 

[loktar]

Really dumb move to upload without doing any testing. I was about to go to sleep and wanted to release, so I just rushed it as soon as I thought it was done.

 

[Macadamia]

Nothing wrong there, being a coder does not mean being a machine.

What matters is that your work will help many people, the hiccups along the way are not relevant !

Get some rest, and take care

 

[Macadamia]

@Everyone : just tested the fixed version and it works wonders, so don't hesitate.
There is already partial support for Linearizations and other nice optimizations, people need to check it out

 

[loktar]

Thanks! =) The animation-related optimizations currently apply to layers, texture animations, particle emitters 2, and ribbon emitters.
The next update will include bones and geoset animations in that, as well as merging/deleting geosets and geoset animations.

 

[Macadamia]

Nice, also, have you seen @GhostWolf update on sanity checker Mdx sanity tester ?

Thats probably something that you could add to the To Do list

 

[loktar]

Yeah once all the optimization features are in, I'm going to look at the sanity tester code and implement fixes for things it reports as I can

 

[Macadamia]

That really sounds promising.
It seems like the tool will end up being a complete model converter / optimizer / fixer.
At least for all the aspects that can be automated. It is already incredibly useful !

 

[MaximBit]

bug=(
If you only disable these items, the texture of the head will not disappear. (ElvenEnversor.mdx)

Update: this doesn't just work on this model (ElvenSwordmaster.mdx)

Update x2: It is better not to use these at all (Tank_WW1.mdx + T_GroundVehicles.blp)

 

[loktar]

Hmm, I'll have a look at that, thanks!

 

[Macadamia]

Nice to see we start having more testers !!!

 

[loktar]

Nice to see we start having more testers !!!

Indeed!

@MaximBit I think I fixed it, but will do some more testing first to be sure. Sorry for releasing it buggy.

Also need to test geoset merging more, but I think it's okay. So next update will probably be released tomorrow =)

 

[Macadamia]

Nice.

geoset merging

Wow !

 

[loktar]

New version is up. Materials & Textures options are fixed, and geoset merging has been added, and some other smaller stuff.

Edit: there's still a bug with material merging

 

[Macadamia]

Amazing !!!

Tested it on my usual series of models plus a couple new ones.
Seems to work nicely, Linearizations seem pretty functional as far as I have seen too.

I wanted to start testing the geoset merging on the list of models to see what issues would appear, unfortunately, with the merging option, the program quickly crashed on a simple model exported from Blender with 2 geosets.

I have attached the model.

 

[loktar]

Ah, yes, fixed that, thanks! The material merging bug is still eluding me, the issue is that it doesn't seem to be checking for the texture path properly, but I don't see why.

I've also found that geoset extents are always being read as all 0 values from MDL files, which is really weird because it's being read and written in the same way as all other extents, which are fine, and I don't see anything wrong with the code.

The vertex stuff takes reaaaally long on that model though, lol, I should add a cancel button.

 

[Macadamia]

Yes the blender model has quite a lot of vertices, and as Reforge models can potentially have that many, a cancel button could be helpful - or maybe better than cancel, a skip button

I didn't notice the extents issue as I mostly did an ingame + animations test.

This strange problem you are having with reading values is indeed weird. If you have traced that the issue comes straight from the reading function, might be an issue with the parsing itself ?
Would have been easier if the values were properly read and kind of zero-ed later ^^

As for the texture paths, did you check the parser handled them properly ?

Hope you'll find out, I know how frustrating those kind of inexplicable (until we find out) issues can be.

 

[loktar]

Yeah the parser seems to be working correctly, otherwise I would be getting unknown token warnings, and in the case of the materials, the texture paths wouldn't be written back to the saved model. I really have no idea what's going on

 

[Macadamia]

Just in case, could your post your actual C++ function that is checking for the texture path ?

 

[MaximBit]

Destroys the caterpillar animation (Tank_WW1.mdx + T_GroundVehicles.blp)

Crashes without any errors (I know about the caution)

Destroys the view Animation (ForgottenOneBirth.mdx)

 

[loktar]

Destroys the caterpillar animation (Tank_WW1.mdx + T_GroundVehicles.blp)
View attachment 357426

Crashes without any errors (I know about the caution)
View attachment 357429
Destroys the view Animation (ForgottenOneBirth.mdx)
View attachment 357433
View attachment 357443View attachment 357445

Thanks, I will have a look at those. The crash should be fixed but I will test it again before updating.

Just in case, could your post your actual C++ function that is checking for the texture path ?

Sure:

Hidden

void Optimizer::mergeIdenticalMaterials()
{
    for (size_t i=0; i<model.edit.materials.size(); i++) {
        for (size_t j=i+1; j<model.edit.materials.size(); j++) {
            // Materials match
            if (*model.edit.materials[i] == *model.edit.materials[j]) {
                // Change geosets materialId
                for (auto &geoset : model.edit.geosets) {
                    if (geoset->material == model.edit.materials[j]) {
                        geoset->material = model.edit.materials[i];
                    }
                }

                // Change ribbon emitters materialId
                for (auto &ribbonEmitter : model.edit.ribbonEmitters) {
                    if (ribbonEmitter->material == model.edit.materials[j]) {
                        ribbonEmitter->material = model.edit.materials[i];
                    }
                }

                // Delete matching material
                if (model.edit.deleteMaterial(j, true)) {
                    diffCount_++;
                    j--;
                }
            }
        }
    }
}

model.edit.materials is a vector of shared_ptrs to the actual objects.

The equality operators are overloaded:

Hidden

bool operator==(const MaterialEdit &lhs, const MaterialEdit &rhs)
{
    if (lhs.priorityPlane == rhs.priorityPlane && lhs.flags() == rhs.flags() && lhs.shader() == rhs.shader()
            && lhs.layers.size() == rhs.layers.size())
    {
        for (size_t i=0; i<lhs.layers.size(); i++) {
            if (*lhs.layers[i] != *rhs.layers[i]) {
                return false;
            }

            return true;
        }
    }

    return false;
}

bool operator==(const LayerEdit &lhs, const LayerEdit &rhs)
{
    return lhs.filterMode() == rhs.filterMode() && lhs.shadingFlags() == rhs.shadingFlags()
            && ((!lhs.texture && !rhs.texture) || (lhs.texture && rhs.texture && *lhs.texture == *rhs.texture))
            && ((!lhs.textureAnimation && !rhs.textureAnimation)
                || (lhs.textureAnimation && rhs.textureAnimation && *lhs.textureAnimation == *rhs.textureAnimation))
            && lhs.coordId == rhs.coordId && compf(lhs.alpha, rhs.alpha)
            && compf(lhs.emissiveGain, rhs.emissiveGain) && compf(lhs.fresnelColor, rhs.fresnelColor)
            && compf(lhs.fresnelOpacity, rhs.fresnelOpacity) && compf(lhs.fresnelTeamColor, rhs.fresnelTeamColor)
            && lhs.animations == rhs.animations;
}

bool operator==(const TextureEdit &lhs, const TextureEdit &rhs)
{
    return lhs.replaceableId() == rhs.replaceableId() && lhs.path() == rhs.path() && lhs.flags() == rhs.flags();
}

Something goes wrong in either the layer comparison or the texture comparison. It only happens with materials that have the same amount of layers.

edit: OMG I'm such a dumbass, that return true statement has to be outside the loop, how did I not notice this until now *facepalm*
Ok, fixed xD

edit:

Destroys the caterpillar animation (Tank_WW1.mdx + T_GroundVehicles.blp)

Fixed. I forgot to take global sequence animations into account when deleting unused keyframes.

 

[Macadamia]

Oh well I knew asking you to post the code would help

Interesting syntax, I think I should indeed learn some C++, it does indeed remind me of C when I was using it ages ago !

It reminds me I haven't tried altering operators since I came back to development, I suppose it can also be done in Lua, and will check it out, thanks for posting the code again.

 

[loktar]

Oh man it's a major bug that's causing the ForgottenOneBirth animation to break. All implemented nodes lose their parent nodes if they aren't of an implemented type. *facepalm*

Edit: updated, all the reported bugs should be fixed. The MDL file geoset extent reading is still bugged though.

^ Yeah IMO C++ is a rewarding language to code with. Although I don't have much experience with anything else besides PHP and jQuery (and vJASS ).

Edit: updated again, found the issue with MDL geoset extent reading, something really dumb again ^^

Edit: uploaded v1.1.8, geoset animation deleting implemented =)

 

[Macadamia]

Sounds like 99% of the mistakes that made my map bug along the last 6 months

Thanks for your work, this is getting big. Will try the animation deletion a bit later.

 

[loktar]

My pleasure, thanks for all the help!

New version is up, linearizing etc should now work everywhere =)

BTW, is anyone else getting a windows error when closing the program, or something wrong with my computer? :x
Edit: fixed

 

[Macadamia]

Wow I was so busy that I nearly missed the update (although I'd need to stay away from Hive long to really miss that ^^)

thanks for this update, will tell you if I hit any bug.

 

[loktar]

Updated again, the basic optimizations are now all implemented. Next I'll be copying more of GhostWolf's work
Also changed the UI style, stole the Hive design xD hope that's ok

 

[Macadamia]

Awww nice !

Didn't encounter issues with 1.1.9 yet anyways

So yes, nice UI change, for some inexplicable reasons it feels like being on The Hive ;-)

I like it a lot, I hope everyone will agree on this.

Small note : where has the default precision display gone ? ^^

EDIT : Ok it is there, just not enough width to see it properly.

EDIT 2 :

Ok rather than testing the new version with my usual list of models, I extracted all the current models from my map, and tried the optimization alone for MDXs without conversion.
All options were left as default.

First, know that the map has lost 200Kb after the operation, when many models were already optimized with an early version of your tool.

Nearly all models that were reduced in size seem to be working so far, but I had issues with 2 models.

First my StormBell tower, a model made out of pieces from @Mythic's Starfall Empire series.

Here is how the model looks in RMSRH before optimization :

and then after :

The second problematic model is a missile I made by modifying the POTM's Classic Searing Arrows missile. Here it is before optimization :

and then after :

I have attached both models for you to see what went wrong there.

Thanks again for your hard work !

 

[MaximBit]

AfricanFoot.mdx (Animation)

HumanGalleon.mdx (Animation)

Good luck ( I hope you can do it fix without bugs )

 

[Macadamia]

Thanks for your extra test !

Fun enough you don't seem to have the width problem with the precision drop down.

Maybe it's because I have high DPI small screen :O

 

[loktar]

Thanks for the tests! Will fix asap.
I think the dropdown width may be because its padding is 0, didn't notice it on my screen either.

Edit: Could you maybe send the reforged textures too? Only the ones for the particles should be fine, so I can see if they're still blocky like in the screenshot.

 

[Macadamia]

Wait.

The missile is the Classic Reforged version, I just forgot to convert it to 800 when I modified it, but it should use the classic blp textures.

As for the StormBell edit, I just attached the only external texture it uses.

 

[loktar]

First my StormBell tower, a model made out of pieces from @Mythic's Starfall Empire series.

The second problematic model is a missile I made by modifying the POTM's Classic Searing Arrows missile.

Fixed! I will fix the UI problems later.

AfricanFoot.mdx (Animation)

This one is a bug in the model. It has two geoset animations for the same geoset, the "Delete Duplicate Geoset Animations" option deletes all except the first, there's no way to know which one is best.

HumanGalleon.mdx (Animation)

Fixed!

 

[Macadamia]

Fixed! I will fix the UI problems later.

Amazing, you really are a hard worker !!!

Thanks again for all your efforts to give this great tool to the community !

 

[loktar]

My pleasure Wouldn't have been possible without the extensive bug reports

 

[Macadamia]

True, but you did the hardest part.

And now the community has a complete optimizer / converter with maximum compatibility already.
And I can't imagine when more features are added and more bugs fixed.

I can't imagine this not being one of the main tools for any dedicated modeler (and not only modelers)

 

[IcemanBo]

Just came by to say it looks really good. Good job!

 

[loktar]

Thanks!

@Macadamia I tweaked the UI a bit, is the dropdown showing properly now?

 

[Macadamia]

Yes it is, wonderful !!!

Thanks for the fix.

EDIT :

Ok just had a glimpse at v1.2.0, the UI improvement indeed gives better visibility !

Haven't found any more visual glitch on the interface btw

Haven't yet checked the new sanity options yet, but will do some tests tomorrow !

 

[loktar]

Ok just had a glimpse at v1.2.0, the UI improvement indeed gives better visibility !

Haven't found any more visual glitch on the interface btw

Haven't yet checked the new sanity options yet, but will do some tests tomorrow !

Great, thanks!

Edit: I just ran the optimizer on all classic Blizzard models:
Total filesize reduction: 41.80MiB (36.04%).

And without Sanitation:
Total filesize reduction: 41.93MiB (37.21%).

 

[Macadamia]

Right.
tested the sanity options.
Took older versions of my map models (tested them with @GhostWolf 's sanity tester, had some incorrect light attenuation values, and missing (or too many) sequence extents) on the other hand I am not 100% I had models with missing Bind Pose Matrices as all the models worked fine ingame.

Anyways after optimizing the MDX models (no conversion for this test) with sanity options enabled, and then pasting them all in GW's tool, it seems that the only severe warnings I had left are the missing death sequences (but thats something I did on purpose) and missing opening and closing tracks on some sequences as normal warnings.

So I guess sanity options do work as far as I am concerned

EDIT : I still had a model with unused global sequences, maybe removing these wouldn't be too hard to implement ?

 

[loktar]

Yay! Thanks!

I wanted to add an option to add opening and closing tracks, but I'm not sure if there's a good way to do it. Maybe sometimes the best thing is to copy the nearest keyframe, but maybe sometimes it's best to copy the closing/opening frame respectively, if available.
Or sometimes maybe just 0 values is best.

Edit:

EDIT : I still had a model with unused global sequences, maybe removing these wouldn't be too hard to implement ?

Huh, now why didn't I think of that xD Will add it!

Edit: added

I did notice two big problems though: 1) each model processed adds to the tool's memory consumption and isn't freed until the programs closes. 2) processing continues in the background if you close the program while it's still running.

 

[Macadamia]

Nice, thanks about the addition

For your first issue, I just learned things about C++ even before having started to use it :
"
How to avoid Memory Leak?

• Instead of managing memory manually, try to use smart pointers where applicable.
• use std::string instead of char *. The std::string class handles all memory management internally, and it’s fast and well-optimized.
• Never use a raw pointer unless it’s to interface with an older lib.
• The best way to avoid memory leaks in C++ is to have as few new/delete calls at the program level as possible – ideally NONE. Anything that requires dynamic memory should be buried inside an RAII object that releases the memory when it goes out of scope. RAAI allocate memory in constructor and release it in destructor, so that memory is garanteed to be deallocated when the variable leave the current scope.
• Allocate memory by new keyword and deallocate memory by delete keyword and write all code between them."

I suppose you already know all that and that you'll easily find out. Fun enough feels like good old wc3 scripting ^^

As for the process not ending, found this interesting thread that taught me it could be related to your previously mentioned memory leak(s) :
How to end C++ code

I believe I will soon need to learn a lot again when getting to know C++ !


EDIT : back to sanity options.

There are issues that persist with the Ember Forge and Multishot tower models.
I didn't notice for Ember Forge because it does not show on stand animation, but it affects all other animations except spell one :

My Multishot tower model looked perfect in Retera's MSHRH, but its "stand victory" animation looks weird ingame :

Attached both models used before their optimization + sanitization.
The Ember Forge I used is an edit from @Mythic himself that fixed a blinking terrain bug in top down view.

 

[loktar]

Nice, thanks about the addition

For your first issue, I just learned things about C++ even before having started to use it :
"
How to avoid Memory Leak?

• Instead of managing memory manually, try to use smart pointers where applicable.
• use std::string instead of char *. The std::string class handles all memory management internally, and it’s fast and well-optimized.
• Never use a raw pointer unless it’s to interface with an older lib.
• The best way to avoid memory leaks in C++ is to have as few new/delete calls at the program level as possible – ideally NONE. Anything that requires dynamic memory should be buried inside an RAII object that releases the memory when it goes out of scope. RAAI allocate memory in constructor and release it in destructor, so that memory is garanteed to be deallocated when the variable leave the current scope.
• Allocate memory by new keyword and deallocate memory by delete keyword and write all code between them."

I suppose you already know all that and that you'll easily find out. Fun enough feels like good old wc3 scripting ^^

As for the process not ending, found this interesting thread that taught me it could be related to your previously mentioned memory leak(s) :
How to end C++ code

I believe I will soon need to learn a lot again when getting to know C++ !

As far as I can tell I'm doing everything right. I didn't have any virtual destructors for base classes though, I added them but to no avail
I guess it's not a big deal in most cases, but it does add up if you optimize a lot of models.
Process ending is related to threading, processing happens in a separate thread and I need to interrupt its event loop. Currently I just tell it to quit but then it waits until the event loop ends.

EDIT : back to sanity options.

There are issues that persist with the Ember Forge and Multishot tower models.
I didn't notice for Ember Forge because it does not show on stand animation, but it affects all other animations except spell one :

View attachment 357975

My Multishot tower model looked perfect in Retera's MSHRH, but its "stand victory" animation looks weird ingame :

View attachment 357976

Attached both models used before their optimization + sanitization.
The Ember Forge I used is an edit from @Mythic himself that fixed a blinking terrain bug in top down view.

Ah, will try to fix it asap. Unfortunately in the next few weeks I'll only have time in the weekends, so updates will slow down.

 

[Macadamia]

Currently I just tell it to quit but then it waits until the event loop ends.

Does "return" break out of entire functions in C++ too ? This could be a way to prevent them to finish the loops.

I'll only have time in the weekends

As "unfortunate" as it sounds, you are entitled to have a real life ;-)

Take care !!!

 

[Macadamia]

UPDATE :

OK I went through my Flying Balls and Rolling Ball models as I plan to do some changes on them.
I hadn't tested them with the latest version of MOP, so I thought it would be a good occasion to test them with it.

At first i put all the options to convert from MDX to MDL, format 800, and the resulting models had some issues:

- Both models had their first bone Pivot Point set to { 0, 0, 0 } instead of the initial { 0, 0, 120 } and { 0, 0, 60 }, messing up the corresponding geoset animations
- Deleted consecutive identical keyframes resulted in messed up visibility for the Flying Balls : all wings seemed to show at the same time
- The first geoset of the Flying Balls is a plane that drive the first wings type and their flapping is managed by the 2 first Bones. For this to work, all wings planes have these two bones using the same geoset, with a different Bone rotation animation, but the same Geoset Animation because that one manages the visibility.

For some reason, the first wings plane had its 2 bones messed up (all the others were properly handled even with their identical structure) :
The both had their Geoset Id and Geoset Anim Id reset in spite of the proper matrices in the Geoset :

Bone "WingsLeft" {
ObjectId 0,
GeosetId Multiple,
GeosetAnimId None,
.....
Bone "WingsRight" {
ObjectId 1,
GeosetId Multiple,
GeosetAnimId None,
.....
And I had to change them back to make the wings flap properly again :

Bone "WingsLeft" {
ObjectId 0,
GeosetId 0,
GeosetAnimId 0,
.....
Bone "WingsRight" {
ObjectId 1,
GeosetId 0,
GeosetAnimId 0,
.....

Once these values changed (having disabled the option to Delete Same-ish keyframes, the model worked perfectly again)
Note that the Rolling Ball model worked fine by just fixing the first bone Pivot point.

With the same options, I then tried to convert a special Jaina model from the tutorial explaining how to change textures ingame using material animations, but the converted MDL crashed Retera's straight away.

I have attached all 3 source MDX models, including both textures for the texture animation of the special Jaina model.

I hope you'll be able to fix these issues, when you have more time.
Take care !!!

EDIT : I hope you still have the Ball and the 3 Wings textures, if not let me know.

EDIT 2 :
@Retera s MSRH doesn't seem to be able to handle the Jaina model animated textures, and removes the first one.
I had to use Magos Model Editor to be able to save it to MDL and replicate the material structure to try to apply that to the geosphere geoset of my Flying and Rolling Balls models.

The idea is replacing the Billboarded planes for the victory animation (roasting balls) because they have transparency issues (can't render a plane with transparent parts to not be transparent in all angles), and also because if they are vertical planes, they won't show properly if the camera is in Top-down view.

The main reason I want to use texture change with material animations is to avoid having to add a new Geosphere geoset to the model, at it is quite large compared to the rest of the model.

EDIT 3 : Currently finishing my adaptation of Flying Balls.

It made me realize that @Retera's MSRH was not responsible for the Jaina texture issue :
The maker of the model did only include the second texture in his texture change, thus there were no reference of the first texture.

 

[loktar]

Hmm, I'll try fix this next weekend.
I think I may have made it so a frame iis deleted when it's the same as the previous OR the next one, while it should be both, I think.

- Both models had their first bone Pivot Point set to { 0, 0, 0 } instead of the initial { 0, 0, 0 } and { 0, 0, 0 }, messing up the corresponding geoset animations

Don't really understand this one?

 

[Macadamia]

Yes sorry about pivot points, I edited my post, wrong copy and paste ^^

Just in case you'd want to double check, I just attached my 3 last models, the Balls are now simplified thanks to the texture animations.
They use a new texture at path "Custom Textures\Star.blp".

I had to remove the other Roasting Ball texture called "sun.blp" as it was made for a billboarded plane, the Star one is made for geosphere with proper polar projections.

 

[FlameofChange]

Great tool for compressing models.
Even if i'm not a software professional nor a experienced modeler, i can tell this can have some more polishing (like any devices that's not surprising).
I would suggest a feature to optimize a map mpq archive directly, and for each models find the best options to make it squishy. Sometimes the optimized model is far heavier so making everything auto would be a fine addition to our collection. For example it would convert a model into 10-20 common options variants then picks the one which is the smallest. A bit long but of course you do would this only once per map.
And why not couple it with the BLP lab to make a very strong map ressource optimizer. Even if map size limit is 256Mb, lighter it is, faster it would be downloaded with potato wifi.