New Optimizer

[Hotwer]

Things are going forward but we still use old tools (Vexorian's Optimizer, SLK and the Widgetizer).

This may be a hard task, but now that Vexorian's Optimizer doesn't works properly (with some wacky hacks it may work like before) maybe we should start on a new tool for map optimization, even something with command line, and maybe integration with WEX if needed.

I'm not sure what the resources we would need to start this, and if we have any foundation resources to start this from Vexorian's Optimizer or SLK source codes...

Anyway you guys see this demand too? If you do, do you have anything that might help with achieving a new tool with open source code like WEX is?

 

[Dr Super Good]

Problem is that optimization is a very complex subject, so much so that Vexorian's optimizer was terrible at it. It does not help that JASS has such strange execution order such as natives interrupting a thread for another event.

For example...

globals
    integer A
endglobals

function example takes nothing returns integer
    set A = 1
    set A = A + 7
    set A = A * 8
    return A
endfunction

Now one might think this could optimize to the following and be functionally the same...

globals
    integer A = 64
endglobals

function example takes nothing returns integer
    return 64
endfunction

Except what if A had an event bound to it? Suddenly the code is not functionally the same and the optimizer has broken it.

Now maybe one could inspect such events and label all involved variables as volatile so writes are not optimized away. However what if there is no direct representation of the variable name as it is generated by concatenation or some crazy algorithm?

So for an optimizer to be fully compatible with code one would have to have most optimizations default to disabled unless explicitly enabled via some kind of comment text macro.

 

[WaterKnight]

@Dr Super Good: Variables having events bound to them? I only know TriggerRegisterVariableEvent for reals, which is notorious for being bad practice. But yeah, here again, we should work towards a proper baseline of writing code and refrain from the hacky malpractices. Special things like the return bug or the fogstate workaround currently used, if it cannot be avoided, at least expose it only by centralized places that can be adjusted a posteriori and do it in a standardized fashion. If everyone does it in his/her own way, of course that weakens the assumption model and renders treatment impossible (and makes the development of the processing tools overly complex or requires them to have more configurations to cover for these exceptions).

@OP: Not sure what Vexorian's optimizer exactly does or what optimizations you are thinking of or what would be feasible and nice to have. Maybe collect ideas regarding that? As such venture may incorporate other code-transforming issues like debugging support or safety measures, maybe look at my (old) jassAid thread.

As for the Widgetizer part, I have worked on some objectMod-to-slk routines in my wc3libs. They are probably still not perfect (wc3 has a lot of eventualities...) but @Frotty started/tried to integrate it in his Escape Builder map some time ago, so he may have some experience about remaining weaknesses.

 

[Frotty]

Idk, in the end I really stopped caring about optimizing *maps*
Mapsize and loadtime simply aren't that important anymore imo, due to 256mb limit and hostbots.
Wurst does an excellent way of optimizing the map script, and that's the only thing I need since it increases the actual performance of the code.
Fast load times via slks are neat, but not neccessary.

I have worked actually on such a tool, might still publish it some day.

 

[Hotwer]

The things is, I’m in demand of such a tool. The map is just impossible to even select in the game without optimization, and wheb I say optimazation, it’s everything, Objetcs, compressions of blps and mdx, jass, remove of WE data that doesn’t need to exist in the map, etc.

@Frotty why don’t you share this tool with us?

 

[WaterKnight]

Converting to slk has drawbacks, too. You may want to keep the original data or part of it, which forces you to copy the information into your map when overwriting the files. Or some fields and especially string attributes of objects are stored in .txt files that are localized, overwriting them makes you lose internationalization.

There are a couple of things that can and are practicable to be done, no doubt, but there are different aspects to optimization like you can be opting for execution speed, low ingame memory consumption, map size, ... and it may not be so straightforward. For example, imported resources that are never going to be used should be removed from the map archive but there is functionality like the AddSpecialEffect jass native where you can access models by arbitrary strings. Not only may some dynamic concatenation of strings you put in there be hard to track down, it is fast to run into undecidable problems.

Media like images, models, sounds also most likely do not have to be perfect, so you could potentially compress them with more or less loss, an opportunity for parametrization.

As for the map script of course, identifiers should be reduced in length. I would delete the (initialization call to) blizzard.j but maybe you require it. The uses of bj functions could be substituted. This may make the map bigger and improve its runtime performance. Multiple simple variables could be merged to an array or vice versa. Remove functions that cannot be reached, which again depends on dynamic invocations, inlining too is not a yes/no decision, more control flow graph analysis problems as DSG hinted at above.

Hypothetically, you can do a lot. It becomes difficult and not easy to distinguish if this or that is better pretty fast. Considering that, it is not like everything could be shouldered by the build tool; it is important for a map to be properly designed and therefore have some understanding of how wc3 works.

But yeah, I would definitely convert to slk/txt as I want to fully customize and not have any original data in my maps, internationalization is often half-baked, anyway or you want everything in English/a specific language. Models can at least be squished without real loss by some rearrangements of values. BLPs maybe remove some mipmaps, jass script do what I said above and kick the WE-only files at least. Finally rebuild the mpq, the required hashtable may have shrunken down.

 

[Hotwer]

You see, I understand that it's not very clear how one would proceed with optimization, but there are a lot of things to do regarding those.

I wanted to do a more updated version, and more important, OPEN SOURCE, version of an optimizer. I know what I could do alone, but that would take much more time to finish. I know that here in the community we've a lot of works that may be helpful to this matter, so I wanted to see how much we can get togehter to see how much work a project like this would need.

 

[actboy168]

I think `W3x2Lni` is what you need.

sumneko/w3x2lni

 

[Hotwer]

I think `W3x2Lni` is what you need.

sumneko/w3x2lni

Well, it's very nice to have something of the sort. Sadly I don't know Chinese, so I tried to transalte your readme.md file.

I saw that one of your TODO items is to support multiple languages. Do you think you could prioritize something like this?

Also, I have no experience with C++, but I think that would not be an issue I guess. If you think it's possible for me to help your an get forward with this as a major project, please, send me a PM message.

 

[actboy168]

What do you want to do? `W3x2lni` is an open source project and it welcomes anyone to provide a pull request.

Supporting multiple languages has two meanings.

1. Correctly handle maps in different languages. To convert the map to slk, you need to provide the correct data file. W3x2Lni provides Chinese and English data files. Other languages you may need to export from war3. So currently W3x2Lni should be able to handle Chinese or English maps correctly. If not, please let me know.

2. Provides a GUI for different languages. This is not my current priority. I'm replacing the new GUI for W3x2Lni, it should be after this.

 

[Hotwer]

What do you want to do? `W3x2lni` is an open source project and it welcomes anyone to provide a pull request.

Supporting multiple languages has two meanings.

1. Correctly handle maps in different languages. To convert the map to slk, you need to provide the correct data file. W3x2Lni provides Chinese and English data files. Other languages you may need to export from war3. So currently W3x2Lni should be able to handle Chinese or English maps correctly. If not, please let me know.

2. Provides a GUI for different languages. This is not my current priority. I'm replacing the new GUI for W3x2Lni, it should be after this.

For now, having alternate documentation in english should be good if you want to have someone to help you with your project.

 

[actboy168]

Yes, good idea.

 

[Hotwer]

Yes, good idea.

If you want me to help you with that and with your optimizer, please send a PM.

 

[Dr Super Good]

What does your optimizer support?

 

[actboy168]

What does your optimizer support?

* slk
* remove unused objects data
* remove files used only in WE
* compressions of mdx
* jass removes comments and extra white space characters
* jass identifier confusion
* inline wts's string

 

[Dr Super Good]

jass identifier confusion

Obfuscation is not really a feature of an optimizer. Vexorian's does it as part of shortening variable names since shorter variable names is an optimization due to the run time name resolution of variables.

inline wts's string

How does it handle multiple localized wts files?

 

[actboy168]

Obfuscation is not really a feature of an optimizer. Vexorian's does it as part of shortening variable names since shorter variable names is an optimization due to the run time name resolution of variables.

Maybe.It's just an option, you can turn it off if you don't want it.Many people asked me for this feature, so I added it. Personally, I don't like encrypted maps.

How does it handle multiple localized wts files?

No handle.The slk map would need to have different versions for different languages.The significance of not inline wts is that you need users in different languages to play in the same game.I don't know how many people need this feature, at least I have not heard of it. If you need it, I can make it an option.

 

[WaterKnight]

How does it handle multiple localized wts files?

Localized mpq entries work?

@actboy168: Have you measured the benefit (initial loading time/mpq size/ingame memory/performance) of inlining wts strings? I mean maybe it's better to leave them space-wise. Also what object data exactly do you remove?

 

[actboy168]

Have you measured the benefit (initial loading time/mpq size/ingame memory/performance) of inlining wts strings? I mean maybe it's better to leave them space-wise.

Yes. Inline wts strings can greatly increase the speed of map loading. It may be the most important reason that affects the speed of map loading. Compared to this, map size, in-game performance and memory consumption are not as important.

Also what object data exactly do you remove?

Every object data should be referenced by something else. If not, it is very likely that your map will not actually use it, so it can be removed.

For example, you create an Archmage in Jass, and the Archmage has a Blizzard skill. This means that Archmage and Blizzard are marked as used and they will not be removed. However, if you have never used Blood Mage anywhere, then Blood Mage can be removed.

Of course, this method is not perfect. If you have a unit id that is calculated dynamically, it may not be statically analyzed. For this unit you can write a jass to let it be referenced, for example

function ReferencedObject takes nothing returns nothing
local integer id
set id = 'H001'
set id = 'H002'
set id = 'H003'
endfunction

W3x2Lni also supports a more advanced approach if you do not want to contaminate your jass code. You can implement a plugin for your map (using lua) and let it tell W3x2Lni which object data your map needs to reference. This technique is also used in my map actboy168/MoeHero

 

[Dr Super Good]

Localized mpq entries work?

MPQ supports file localization. Hash table entries have a field for locale. How this locale is loaded I am not sure but some people claim it works.

Of course, this method is not perfect. If you have a unit id that is calculated dynamically, it may not be statically analyzed. For this unit you can write a jass to let it be referenced, for example

Again this is forcing people to work around the optimizer, the problem I referenced above.

That said how much time do these unreferenced objects take to load? If it is ~1-2 seconds who cares?

 

[actboy168]

MPQ supports file localization. Hash table entries have a field for locale. How this locale is loaded I am not sure but some people claim it works.
Again this is forcing people to work around the optimizer, the problem I referenced above.

That said how much time do these unreferenced objects take to load? If it is ~1-2 seconds who cares?

The optimizer cannot know what you think. You want better optimization results, you need to understand the optimizer and help the optimizer to understand you think is very necessary.

For example, inline wts string has a huge impact on map loading speed, but not all wts strings are inline. You need to understand why it cannot be inline and solve it yourself. If you want your map to load faster.

The biggest difference between w3x2lni and other optimizers is that w3x2lni tells you how it should help it optimize your maps.

 

[actboy168]

W3x2Lni has been released.
W3x2Lni v2.1.1

 

[Hotwer]

W3x2Lni has been released.
W3x2Lni v2.1.1

Thanks, I'm going to check this out.

I see you created and English documentation. And I saw that you're working along with someonelse. It's is kind of unfriendly to join in as I know nothing about chinese, but I may hop in to help with some pull requests at some point, as I have some suggestions to give.

 

[actboy168]

Thanks, I'm going to check this out.

I see you created and English documentation. And I saw that you're working along with someonelse. It's is kind of unfriendly to join in as I know nothing about chinese, but I may hop in to help with some pull requests at some point, as I have some suggestions to give.

Thanks, welcome at any time.

 

[Spatulon]

Hey, I'm using this. Works great, does everything I need it to do!

I had one little issue where the jass optimization was breaking Pipedream's optimal save system. I debugged it and it turned out the jass optimization was messing with the log function in the BigNum library. Basically just gets stuck in an infinite loop.
exitwhen y < 1.0001 was the culprit.
So I assume it's turning it into 1.000 or something.
It works fine if I lower the precision down to 1.001

Also I can't use the GUI because of that font error or whatever. But using the command line is good enough.

But yeah besides that everything seems solid! This is awesome!

 

[DracoL1ch]

Obfuscation is not really a feature of an optimizer. Vexorian's does it as part of shortening variable names since shorter variable names is an optimization due to the run time name resolution of variables.?

it's a mistake to think that variable's name affect anything. Every string, every name is hashes at the start and it's irrelevant further

 

[actboy168]

Hey, I'm using this. Works great, does everything I need it to do!

I had one little issue where the jass optimization was breaking Pipedream's optimal save system. I debugged it and it turned out the jass optimization was messing with the log function in the BigNum library. Basically just gets stuck in an infinite loop.
exitwhen y < 1.0001 was the culprit.
So I assume it's turning it into 1.000 or something.
It works fine if I lower the precision down to 1.001

Also I can't use the GUI because of that font error or whatever. But using the command line is good enough.

But yeah besides that everything seems solid! This is awesome!

Thank you. The current jass's real will be converted to lua's number. It is indeed a problematic implementation. I will fix it.

 

[Dr Super Good]

it's a mistake to think that variable's name affect anything. Every string, every name is hashes at the start and it's irrelevant further

Any evidence to back this up? Since a lot of people around Vexorians time posted benchmark evidence showing that variable name length did impact run time performance.

Thank you. The current jass's real will be converted to lua's number. It is indeed a problematic implementation. I will fix it.

Possibly use some kind of big number type for them to retain precision? Either that or you would need your own IEEE float library so as to interpret them like Warcraft III does so one can perform compile time optimizations.

It is also worth noting that any global constants from the blizzard.j file cannot be optimized away, unless specifically permitted to (an exception list). This is because they might be subject to change, as history has shown.

Examples of constants that can be inlined away...

    constant real      bj_PI                            = 3.14159
    constant real      bj_E                             = 2.71828
    constant real      bj_RADTODEG                      = 180.0/bj_PI
    constant real      bj_DEGTORAD                      = bj_PI/180.0

All of these are mathematic constants. They will never change and hence can be inline optimized away.

Examples of constants that cannot be inlined away...

    constant integer   bj_MAX_INVENTORY                 =  6
    constant integer   bj_MAX_PLAYERS                   =  GetBJMaxPlayers()
    constant integer   bj_PLAYER_NEUTRAL_VICTIM         =  GetBJPlayerNeutralVictim()
    constant integer   bj_PLAYER_NEUTRAL_EXTRA          =  GetBJPlayerNeutralExtra()
    constant integer   bj_MAX_PLAYER_SLOTS              =  GetBJMaxPlayerSlots()

These are all game engine constants. Blizzard might change them in the future.

This is to stop a repeat of what happened with the recent patches breaking a lot of Vexorian Optimizer optimized maps.

 

[DracoL1ch]

Any evidence to back this up? Since a lot of people around Vexorians time posted benchmark evidence showing that variable name length did impact run time performance.

how much - thats the question. where are their bmarks?

 

[actboy168]

Possibly use some kind of big number type for them to retain precision? Either that or you would need your own IEEE float library so as to interpret them like Warcraft III does so one can perform compile time optimizations.

Actually I don't care about the actual value of real, so I converted jass's real to lua's string.

It is also worth noting that any global constants from the blizzard.j file cannot be optimized away, unless specifically permitted to (an exception list). This is because they might be subject to change, as history has shown.

Examples of constants that can be inlined away...

    constant real      bj_PI                            = 3.14159
    constant real      bj_E                             = 2.71828
    constant real      bj_RADTODEG                      = 180.0/bj_PI
    constant real      bj_DEGTORAD                      = bj_PI/180.0

All of these are mathematic constants. They will never change and hence can be inline optimized away.

Examples of constants that cannot be inlined away...

    constant integer   bj_MAX_INVENTORY                 =  6
    constant integer   bj_MAX_PLAYERS                   =  GetBJMaxPlayers()
    constant integer   bj_PLAYER_NEUTRAL_VICTIM         =  GetBJPlayerNeutralVictim()
    constant integer   bj_PLAYER_NEUTRAL_EXTRA          =  GetBJPlayerNeutralExtra()
    constant integer   bj_MAX_PLAYER_SLOTS              =  GetBJMaxPlayerSlots()

These are all game engine constants. Blizzard might change them in the future.

This is to stop a repeat of what happened with the recent patches breaking a lot of Vexorian Optimizer optimized maps.

I didn't let anything inline. So this tool does not have this problem.

 

[Spatulon]

Hey, I think I found another bug.

Basically my buffs that have a similar ID such as B@g& and B@G& are being merged or something, so one of the buffs looks like the other buff instead.

Example: I added 3 buffs to a unit.

2 of these buffs are:
ID B@g& - Piston Perfection
ID B@G& - Hatching

Before optimization, the different buffs appeared as normal.
After optimization, Hatching now looked like Piston Perfection.

Before:

After:

Here's part of the error log with some stuff that seems relevant:

Spoiler: Error log

==========================
No SLK data (454)
==========================
--------------------------------------------
Txt is conflicted between different objects
--------------------------------------------
Buff B@t& |cff00FF00Occult Shell|r - Buffubertip --> B@T&
Ability A@j& Mortality - Name --> A@J&
Ability A@s& Flesh Wound - Name --> A@S&
Ability A@G& Hatching - Name --> A@g&
Ability A@w& Mephetic Drip - Name --> A@W&
Buff B@w& Mephetic Drip - Buffubertip --> B@W&
Ability A@f& Overloaded Spasm Generator - Name --> A@F&
Ability A@t& Occult Shell - Name --> A@T&
Buff B@R& Laceration - Buffubertip --> B@r&
Buff B#M& Crystal Cage - Buffubertip --> B#m&
Ability A#M& Crystal Cage - Name --> A#m&
Ability A@E& Collusion - Name --> A@e&
Buff B@V& Seasick - Buffubertip --> B@v&
Ability A@V& Seasick - Name --> A@v&
Buff B@g& |cff00FF00Piston Perfection|r - Buffubertip --> B@G&
Buff B#K& |cff00FF00Cyclonic Separation|r - Buffubertip --> B#k&
Buff B#r& Gangrene - Buffubertip --> B#R&
Ability A#r& Gangrene - Name --> A#R&
Ability A@B& Sediment - Name --> A@b&
Ability A#C& Giant - Name --> A#c&
Buff B#S& |cff00FF00Reinvigorating Blessing|r - Buffubertip --> B#s&
--------------------------------------------

Edit: maybe i should be posting this in the actual tool thread

 

[actboy168]

This is a bug, but it may be very difficult to solve it.
The problem is that the object id ignores the case in the txt file, and the w3* file does not ignore the case. I cannot simply do the same thing because some data use this. The ideal result is that I do not optimize these problematic objects, so the best option is to you remove these object ids that are not slk-optimized. These ids will tell you in the log.

 

[Dr Super Good]

The problem is that the object id ignores the case in the txt file, and the w3* file does not ignore the case. I cannot simply do the same thing because some data use this. The ideal result is that I do not optimize these problematic objects, so the best option is to you remove these object ids that are not slk-optimized. These ids will tell you in the log.

This does not reflect well in the quality of your optimizer... You are basically telling users to change their maps to work with your tool. Is this even a rare occurrence seeing how the automatic object type ID generation algorithm uses both upper and lower case characters?

There should be 2 automatic solutions.

• Keep object types with case insensitive object type ID collisions in the w3* files. Rest are put into the "txt" (SLK?) file. This would handle the case while keeping most of the advantages of using a SLK file. A few w3* object type declarations will likely make trivial difference to load times.
• Try to automatically refractor an object type ID upon a case insensitive object type ID collision to one without such collision. This would still allow such object types to be optimized into SLK files. It is a reasonable assumption that every object type ID referenced by a trigger should be declared in script files as either a 4 ASCII type code or as an integer number. It is also reasonable to not support bad JASS code that performs arithmetic on object type ID numbers, as in such a case the correct programming approach would be that object type IDs should be mapped to an array and arithmetic applied to an array index number instead.

 

[actboy168]

This does not reflect well in the quality of your optimizer... You are basically telling users to change their maps to work with your tool. Is this even a rare occurrence seeing how the automatic object type ID generation algorithm uses both upper and lower case characters?

There should be 2 automatic solutions.

• Keep object types with case insensitive object type ID collisions in the w3* files. Rest are put into the "txt" (SLK?) file. This would handle the case while keeping most of the advantages of using a SLK file. A few w3* object type declarations will likely make trivial difference to load times.
• Try to automatically refractor an object type ID upon a case insensitive object type ID collision to one without such collision. This would still allow such object types to be optimized into SLK files. It is a reasonable assumption that every object type ID referenced by a trigger should be declared in script files as either a 4 ASCII type code or as an integer number. It is also reasonable to not support bad JASS code that performs arithmetic on object type ID numbers, as in such a case the correct programming approach would be that object type IDs should be mapped to an array and arithmetic applied to an array index number instead.

This is a bug. When the optimizer is correct, this object should be rolled back without slk optimization.It is similar to Solution 1 you said. Modifying the object id is not what I want to see because it does not guarantee that there are no errors.
To make the optimizer work 100%, you can only modify the map. Just as someone told you that a local variable must be set null at the end to make warcraft3 happy, the slk/txt format contains many similar problems that require you to understand and avoid it. As I said at the beginning, helping the optimizer to make it work better.