Model Optimizer and Converter v1.2.1

[Macadamia]

Well thats still great progress, happy I found the little bug, that was real luck to be honest !

I hope you'll end up finding the strange issue with the flapping wings, my Flying Balls will thank you eternally

And don't worry, being obsessive about a project is really a good sign, it shows how much you are dedicated, and that you love what you are doing, even when it is not easy !

 

[loktar]

Yesss, fixed it. Merging vertices should also be much faster now, there were a few really inefficient things in there (stuff that happened each loop which can just as well be done once after the loop).

I'm gonna double check all the flags, because recently I replaced all the flag numbers with enum variables for better readability etc, so I may have made more mistakes.

BTW, so far merging vertices also merged their vertex groups (so the merged vertex would be connected to all the bones of the matching vertices that were merged into it), but I'm gonna make this optional. I think for your ball models it would break your model otherwise.
Edit: actually, I'm not gonna make it optional, I'm just gonna always take vertex groups into account.

 

[Macadamia]

Nice !

Yes that would be nice to make it optional, like anything else that can alter model functionality.

Hehe now I am becoming impatient ^^

 

[loktar]

Updated!

 

[Macadamia]

OK.
Thats a lot of improvements !!!!

First series of test with Format 800 as a target :

I tried with MDL and MDX as sources, with or without conversion.
All models were processed !!!

First, the options for MDL are perfectly handled in the different situations... wonderful.
All my Ballwings models work perfectly... so great.

I have tried to test as many models as I could (my list has 125 models ^^), they all seemed to work.

I have found 2 exceptions.

First, the Classic Archer model from Reforged (that I attached 6 messages earlier) is empty after the Materials section, same as an Axe Missile (classic) also from Reforged.
I will attach the Axe so that you can have a second classic reforged model to compare.
Note that the version of this archer where I had manually removed Reforged specific fields was converted and optimized perfectly !

Now for the naming, its much better, but I think it should only add suffixes when there is actually a collision. Especially when there is a conversion. Also maybe it isn't necessary to add both suffixes in case of optimization and conversion (in fact conversion is obvious as the extension changes, although I understand that in some cases it could be interesting to differentiate optimized from not optimized, or converted mdx from original mdx, but then if you restrict these suffixes in the event of a collision, it should be just fine.

As a conclusion, the program is just about ready, I believe the problem with classic reforged models should be easy to fix (just copy the reforged fields over as you explained before).

Second series of tests with Format 1000 as target :

With MDL as source and no conversion, with Magos compatible unticked (it should be greyed !) and MDL option ReforgedData was greyed (thats correct because thats the principle of format 1000).

Note that I tried to enable the Magos option (hehe thats a dedicated beta tester !!) and as expected it crashed the program

Again all models but the two I mentioned before went through perfectly, with the exception of the two models mentioned before with the same symptoms.
Here are the errors displayed by the way :
for the Archer :
"Exception: stoi
Exception: Unknown token in Layer: static Emissive

Exception: Unknown token in Layer: static Emissive

Exception: Unknown token in Layer: static Emissive

Exception: Unknown token in Layer: static Emissive

Exception: Unknown token in Geoset: LevelOfDetailName"
for the Axe Missile, just this :
"Exception: stoi'"

I am confident it is just the fact that your program is not expecting these entries to be already there before adding them (although then it should not have caused issues with the Format 800 tests then... anyways I trust you know what's going in ^^

With MDX as source and with conversion, again the option for Magos was not greyed but this time neither was the option for Reforged Data (it should always be there for format 1000).

Now the list did not complete, the program had the infinite loop issue back with the first Flapping Wings model encountered - well you know where the problem is now ^^
Also same issue as other tests with classic archer and axe_missile models, but that was expected.

Most of the models converted to Format 1000 seem to crash Retera's MSRH though. Here is the error :
"
Unknown error occurred:
java.lang.ArrayIndexOutOfBoundsException: 1
at com.hiveworkshop.wc3.mdl.Geoset.parse8ByteSkin(Geoset.java:356)
at com.hiveworkshop.wc3.mdl.Geoset.read(Geoset.java:442)
"

Note :
The Classic Reforged Archer (so format 1000) had no Skin entries, (nor the version without the Shader and LevelOfDetail and static Emissive that I made manually) and your conversion added "SkinWeights" entries to the manually edited one.
The Axe_Missile classic Reforged model has no SkinWeights entries but Skin entries in the geosets.

This part is confusing for me, I am not yet used to these specific entries, although Retera explained this in some thread that I need to find back.


I am sorry there always seems to be new errors coming to the table, but at least they are caught now rather than later.
This said, you expected difficulties with format conversion from the start, so this is not 100% of a shock.

As far as my selfish case is concerned the tool is functional as the format conversion can be handled by Retera's MSRH (although I heard about some issues somewhere - but I haven't faced such problems yet). The only feature I am now missing is the Hermite and Bezier to Linear (wait did I mention this before ? ) and i'm good to go !!!

Now I trust that you can easily fix the remaining issues with the handling of Format 800, and I believe the headaches will come from Reforged (again, thank you Blizzard :/)

Anyways, I can appreciate the amount of work you have done so far, this is unreal !!!

I am staying tuned !!!

 

[loktar]

Ah, kinda disappointed there are still bugs and crashes But glad you're here to report them I should probably do more testing before releasing new versions

I'll go through your post in more detail tomorrow as I check out the issues. One thing I can already say though, the token names Retera's reads and and writes for reforged models are incorrect, he basically just guessed them. The ones this tool uses are the right ones, so that's why it can't read retera's reforged tokens and vice versa.
I'll also make a list of the correct tokens vs the ones retera's uses, I know it can be pretty confusing

I think the disabling of checkboxes is as intended atm though, but I'll check again tomorrow and explain my reasoning behind it.

Thanks for all your help, really appreciate it!

 

[Macadamia]

Yes you are right, I remember @Retera mentioning he kind of made his own "Reforged MDL".
This is annoying (although he didn't have much choice then) as his tool is probably the only one available for Reforged models.

Luckily this does not affect mdx version ingame as far as I have noticed.
Maybe there should be an option to keep the MDL "Retera compatible" like we have a "Magos compatible", what do you think ?

I will be happy to read your reasoning about options checkboxes !
Now I hope you will have allowed yourself a nice well deserved rest when you will read this reply ^^

Lastly, don't worry, I consider myself lucky to be able to participate to your project, because this will really be godsend for the community.

 

[loktar]

First, the Classic Archer model from Reforged (that I attached 6 messages earlier) is empty after the Materials section, same as an Axe Missile (classic) also from Reforged.
I will attach the Axe so that you can have a second classic reforged model to compare.

It seems I underestimated the difficulty in reading values for unknown tokens.
I've decided to add in support for reading RMS's Reforged MDL format. Don't think I will add the option to write it like that though, as it will just perpetuate an incorrect format. I just hope Retera updates his tool soon
I'm not sure I have all of his tokens added in, so let me know if there's any more errors (fresnel stuff for example).
So model loading will once again fail if there are unknown tokens, but it should be able to read Reforged MDL files written by RMS.

Now for the naming, its much better, but I think it should only add suffixes when there is actually a collision. Especially when there is a conversion. Also maybe it isn't necessary to add both suffixes in case of optimization and conversion (in fact conversion is obvious as the extension changes, although I understand that in some cases it could be interesting to differentiate optimized from not optimized, or converted mdx from original mdx, but then if you restrict these suffixes in the event of a collision, it should be just fine.

Hmm, good point. I will just move the suffix option to Optimize and Convert Options respectively as a checkbox instead.

with Magos compatible unticked (it should be greyed !) and MDL option ReforgedData was greyed (thats correct because thats the principle of format 1000).

Note that I tried to enable the Magos option (hehe thats a dedicated beta tester !!) and as expected it crashed the program

So, the logic is this:
- Magos compatible: enabled whenever MDL files are written (so when processing MDL files and not converting, or or when processing MDX files and converting)
- Reforged data: enabled whenever models are converted from MDX to MDL. If a MDL file is processed, it will always write Reforged data back if it's already present.

It's surprising that it crashed in this case though, will check it out.
Edit: it doesn't seem to be crashing for me, maybe it was caused by something else?

 

[Macadamia]

Need to give a lift to my daughter to the doctor and the chemist right now, I'll check the updated version as soon as I will be back

As for the Magos / Reforged Data in MDL, make sure people can not enable both at the same time

Back !

I did a few more tests and strangely enough this time it also crashed the BallWings from MDX to Format 800 (Magos Compatible).

Thats strange as the MDL to MDX worked perfectly again.
I hope you will find the issues for next version.

I will stay tuned, good luck on your side, and take care !!!

 

[loktar]

Need to give a lift to my daughter to the doctor and the chemist right now, I'll check the updated version as soon as I will be back

Oh, hope she's alright! But no rush, not even sure it will be ready tonight.

Maybe there should be an option to keep the MDL "Retera compatible" like we have a "Magos compatible", what do you think ?

I don't think I will do this, supporting reading of it is already cluttering up the code a bit. But you can always convert back to mdx and open that with Retera's.
I may change my mind later though, especially if it looks like Retera won't be updating his tool for a long time.

As for the Magos / Reforged Data in MDL, make sure people can not enable both at the same time

Hmmm, right, I didn't think about this. Although it's possible to input/output different files of different versions at the same time, but I will make sure the magos compat is only applied when writing v800 files and add a note to the tooltip. And I'll also disable it when Version is ticked and 900 or 1000 selected.

 

[Macadamia]

Yes she's ok, thanks

I also think the Retera compatibility is not a priority.

Take your time for the update, I am not in any rush - currently battling a countdown timer with a progress bar model whose animation refuses to synchronize with the spell duration for mysterious reasons, so I'm in the same boat as you at the moment

This said I will be there when you will update to help hunt the last bugs !!!

 

[loktar]

Ah, that's good to hear.

I think it's done now actually. There was a bug that caused some wrong data to be written to Reforged MDX files without any FaceFx, I think the rest of the bugs/crashes you reported may have been caused by this. At least I can't seem to reproduce it.

The errors with RMS when opening Reforged MDL files from my tool are probably because it can't read the Reforged tokens. This can only be fixed if I add an "RMS Compatible" option like you suggested.

Let's see... I hope I didn't forget anything, but I'm going to upload the new version in a minute. The only thing missing that I can remember right now is some of the Reforged MDL tokens from RMS, I don't know what some of them are (the ones you reported so far are included).
Edit: oh wait, still need to update the commandline exes.

 

[Macadamia]

Waiiit, no you're not done, I still need help removing thousands of InTan/OuTan Lines for Hermite/Bezier to Linear !!!!

Mmm if I keep on like that it will end up like harassment :O

 

[loktar]

Hahaha, don't worry, I won't report you
And Linearizing is definitely a priority once most of the kinks are worked out

Anyway, new version is up, let me know if there are any more problems!
Also a request, could you test if models converted to v1000 work ingame in reforged?

 

[Macadamia]

Oops fell asleep on a tricky trigger :O

But on it right now with a nice coffee attached to my left hand !!!!!

EDIT :

OK I did quite a lot of tests.

First, optimizing MDL seems to work great when saving to 800 or 1000 format.
Did not get any issue with the optimized MDLs in format 1000, except for the flapping wings models and the Classic Reforged Archer : their animations don't work in WE or ingame.
They also crash RMSH but thats something we expected

Also, the Axe Missile I posted earlier is still empty after the materials, with both formats as a target.

All the other Format 1000 models seem ok in WE or ingame (of course not in Retera for the Format 1000)

Next.
Converting from MDX to MDL.
Several models crashed your program. The flapping ball based model after the usual infinite loop - inflating memory usage, and others instantly.

Removing the MDXs that were problematic, all conversions went fine. Note these MDXs worked fine in Retera's

I attached a pack of the problematic models, because there are a few not related to the flapping wings.

I did not do any further test at this point, to avoid confusions.

Hope this will help,
Take care !!!

EDIT 2 :
Fulfilled with good will, I wanted to test your Importless Model Viewer to test all these Format 1000 models.
Well guess what, it seems to work fine in Reforged, but I feel a bit stupid : it seems I could not get the camera and animations controls to work as I needed (or maybe they do not 100% work after all).
Anyways, it's not time to distract you with these things (just wanted to point I like the model viewer idea, because it unfairly received divided appreciations).

EDIT 3 :
I hadn't noticed initially, but two other models have lost their animations. The models are invisible ingame, and that implies that Stand and other animations are bugged. (Fun enough one of the model's portrait works though.
I attached both the original working MDXs and the bugged converted format 1000 MDLs (model names = Colossus and Multiinfernal).

Lastly, @Mythic's Ember Forge has some issues with its animations too, hiding parts or showing them wrongly (also attached both versions).
Remember the MDX for all these 3 models are Format 800.

 

[loktar]

except for the flapping wings models and the Classic Reforged Archer : their animations don't work in WE or ingame.

I believe this is to be expected for now due to the simplistic way the version conversion happens, but I'm going to double check in case there's something I missed.

Also, the Axe Missile I posted earlier is still empty after the materials, with both formats as a target.

The flapping ball based model after the usual infinite loop - inflating memory usage

Huh, weird, it worked for me, maybe something I changed after testing messed it up again. Will check.

Fulfilled with good will, I wanted to test your Importless Model Viewer to test all these Format 1000 models.
Well guess what, it seems to work fine in Reforged, but I feel a bit stupid : it seems I could not get the camera and animations controls to work as I needed (or maybe they do not 100% work after all).

I only tested it in 1.31 patch, maybe that's why? The mouse controls were always a bit buggy (jumping around at times) though.

Anyway, it seems it's already working a lot better thanks to your help Just a few more bugfixes and it'll be ready to start adding features

 

[Macadamia]

OOps I added Edit 3 when you were posting your message

I am confident we found the last major bugs here !

 

[loktar]

Did not get any issue with the optimized MDLs in format 1000, except for the flapping wings models and the Classic Reforged Archer : their animations don't work in WE or ingame.

Hmm, I think it may be because Tangents and SkinWeights are written with all values set to 0. I'm not sure that's what the problem is, but either way it shouldn't be there, so I've made sure it isn't. Hopefully that fixes it.

Also, the Axe Missile I posted earlier is still empty after the materials, with both formats as a target.

This one was actually even crashing for me, very weird. Fixed now.
I believe there was an error in the original file's material alpha animation, it had 10 instead of 2, which I had to change for it to work.

I will test the models you attached later. I'm hoping the two fixes above have also fixed those

 

[Macadamia]

Nice, keep up, we're nearly there !!!

 

[loktar]

I tested some of the ProblematicMDXs. The first one actually went through without a problem after the updates I made
Then the next ones threw an exception. It seems like MdlVis can't open them either. Is there anything unusual about these models that you know of, like things that models usually have but these don't, or vice versa?
I'll check them out more closely soon, Magos and Retera's can open them so I'll save them as MDL from there too.

Edit: huh, it seems like they have negative values for vertex groups. Weird. Should be an easy fix.
These models work
Edit: Hmm, they don't work ingame, it seems they got corrupted somehow?

 

[Macadamia]

That is indeed weird. Most of these models have been worked on wit Retera's MSRH, including the Demoness with deleted vertices (could this be a cause?).

 

[loktar]

Hmm, the BallWings and BurningFlyBall models don't seem to work ingame either. It looks like the bone attachments got corrupted somehow.
This is good news for me, because it means it's not a problem with the tool, but bad news for you because your models don't work?

Edit: tested all the ones from the zip in the editor, none of them can be loaded, the GlowRed/Blue models even crash the editor.
Edit2: The other models you attached all had the 0-value tangents and skinweights, which aren't being written anymore, so they should be fixed now too.

I'll upload the new version in a few minutes, hopefully the models with problematic animations will work now.

Edit: alright, new version is up

 

[Macadamia]

Ok I made a new series of tests with the map (as folder) i adapter earlier for this.

First, as I needed to double check, I used all original MDL files, and the imultinfernal, then Jaina with Alternate were not working to start with, so once again it's not your tool.

Note that the axe missile (a reforged version of the classic axe) had no data past materials in all scenarios.

1) Optimization : MDL

After the normal optimization to Format 800 with Magos compatibility, all the other models (including flapping wings ones) worked fine.
Same satisfying result with Format 1000 as a target.

2) Conversion : MDX

I first tested all mdx files in game, and they all work including colossus and Jaina .

Format 800 as target = all models work except Jaina and cmultiinfernal (as expected) flapping wings models work. The modified reforged archer (reforged model with removed vertices) also works !!!!

Format 1000 as target : the option for reforged data is accessible, people should not be able to untick it for format 1000.
Now the good news : all models work except Jaina. even reforged archer and... multiinfernal
Amazing job there. Oh the flapping wings work too

I guess you made it, congratulations !!!

Now all that is needed is grey the Reforged Data option in Format 1000 so that people don't disable it, and add the Bezier/Hermite to Linear feature and the tool will be ready.

Hats off, this is awesome !

 

[loktar]

Awesome Now it's ready to start adding more features, most importantly linearizing animations
Thanks again for your help, the quality of the tool has really improved immensely thanks to you.

About the axe missile, could you send the original MDX file?

 

[Macadamia]

Don't bother with the Axe_Missile, I just realized I did my test with the first converted MDX that had lost everything after the materials.
I grabbed the original one from Reforged Casc and it worked perfectly !!!

 

[loktar]

Ah, great! Ok, time for a little break from this, started on the human WC1 lumber mill
So next update will take a little longer, but hopefully it will be worth it

 

[Macadamia]

Nice, I have no doubt you'll make great WC1 lumber mills, and of course I am curious to see that !!!

Take the well deserved break, you have already done an outstanding job !!!

 

[Macadamia]

Mmm you just updated the package, what has changed ? It seems to be the same version number

 

[loktar]

Yeah just added the "Results" thing

Edit: hmm, I think the big difference when optimizing a model saved by mdlvis may be due to normals, mdlvis changes it somehow. I may add an option to ignore normals when merging vertices. Will have to do some testing.

 

[Macadamia]

Oh ok !

I haven't used MDLVis so far, but I plan to do so to make some less basic bone animations.

 

[loktar]

In the new version merging vertices should be about as effective for models not saved with mdlvis, as long as "Preserve Normals" is unchecked =)

 

[Macadamia]

Nice update. Can't test the MDLVis yet, but I have no doubt ^^

Nice to see you adding more and more options. Also appreciate the 0 length sequences removal !!!

 

[loktar]

Removing 0-length sequences is probably pretty useless though, I don't expect many models will have this problem It doesn't include global sequences yet.

 

[Macadamia]

Still, if it is there, I don't believe it is removed with other tools.

As for global sequences, yes I am aware, I now read your To Do List everytime

 

[eejin]

You are starting to get quite the extensive list of buttons and options in your application . When you need some help with a UI redesign then I'd be willing to help you out. Keep up the good work! You too for testing so much Macadamia.

 

[Macadamia]

I suppose one day in the far future we could have an amazing third party World Editor with integrated tools beyond any map maker's wildest fantasies.
It could integrate so many things, even a model editor and optimizer and converter !!!!

Fun thing, It feels like that is what the community was expecting from Blizzard with Reforged...

 

[Macadamia]

UPDATE :

While trying to optimize a model I made very recently, I came up with an entry the tool did not accept.
I tried to optimize a modified combination of FlameStrike and a fire version of Frost Armor (I used one of these classic Reforged models). It was edited with RMSRH and exported to MDL with it.

Here is the error :
"Exception: Unknown token in Bone Cylinder01: BillboardLockZ"

All options were ticked except for "Preserve Normals" in Geoset tab. and the target format was 800.

Note : RMSRH seems to keep the tag even when HD to SD and assign Format 800 options are used.

 

[loktar]

Hmm, this seems to be a bug in RMS. Mdlvis, Magos and the Hive viewer all used "Billboarded" instead of Billboard. @Retera
But I'll add support for it.

 

[Macadamia]

Thanks, so I guess the proper syntax would be BilldoardedLockZ ?

I am asking because replacing BillboardLockZ with just Billboard messed up the related bones.

 

[loktar]

You are starting to get quite the extensive list of buttons and options in your application . When you need some help with a UI redesign then I'd be willing to help you out. Keep up the good work! You too for testing so much Macadamia.

That would be nice, but maybe best to wait until the list of options is more complete?

Thanks, so I guess the proper syntax would be BilldoardedLockZ ?

That is correct.
BillboardedLockX, BillboardedLockY, BillboardedLockZ and Billboarded respectively.

 

[Macadamia]

Thanks for this information !
I think bringing the official MDL tags back is really important.
And I believe the support you are adding for RMSRH's home made ones will help doing the transition.

I know this does not affect MDX directly, but it is when it comes to use models ingame as MDL or even for testing purposes, and even future compatibility with other potential tools, then it really matters !

Again thanks for your dedicated efforts to make this tool so helpful.

 

[loktar]

Yeah RMS is used by a lot of people, so makes sense to support its MDL syntax.

BTW, unfortunately animation linearizing will take a while. Because of the way I'm writing the parser I'm going to have to implement each animated object first before I can do it properly.
Next update will have material deleting/merging.

 

[Macadamia]

Ok thanks for letting me know.

Well take the time you need, I am sure many will appreciate this extra feature when it will be there, but no need to break your neck in the process ^^

Out of curiosity, what development language and environment do you use ?

 

[loktar]

C++, with Qt. The parser is written without the Qt framework (but in the Qt IDE), but for the tool I am using it.

 

[Macadamia]

Ok another great tool made with C++.
I suppose I should start learning this

 

[Retera]

Yeah RMS is used by a lot of people, so makes sense to support its MDL syntax.

Infinite facepalm.

Also, did I really mess up BillboardedLockZ and keep using BillboardLockZ for 8 years? I mean that would have to be a typo in my handling of SD models. But it seems to me equally possible that other fan tools made a typo since it is pretty obscure. Has anybody tested which one the game accepts?

We should also change future versions of Retera Model Studio to accept both if that's the route we are taking.

 

[loktar]

But it seems to me equally possible that other fan tools made a typo since it is pretty obscure. Has anybody tested which one the game accepts?

Yeah that's possible too I guess, haven't tested it. Possibly both work ingame? Though that seems less likely.

In my parser I'm trying to make it be able to read as many MDL syntax variations as possible, but write what seems to be the "real" one. So if we both take this approach, reading variations could be phased out later, except for any written by ubiquitous old tools such as mdlvis and magos.

I don't think it's that important anyway, as the MDL format is probably hardly ever used ingame, but it would still be nice to get as close to the native format as possible.

 

[Macadamia]

Yes I don't think MDL will be buried any time soon (but I wouldn't say never).

This said I tried both versions of the model with BillboardLockZ and BilldoardedLockZ :

They both opened and worked in RMSRH, but WE refused to load them, and of course the models are invisible ingame.
What are we missing here ?

 

[loktar]

Just tried it, it seems to work for me when I change it to "Billboarded" and "BillboardedLockZ", but fails to load in WE when it's "Billboard" and "BillboardLockZ".
So I guess that confirms that the proper tokens are the former.

 

[Macadamia]

Ok wells that worth knowing. I don't know what went wrong on my side, honestly.