New Script Optimizer

[Frotty]

Froptimizer

Froptimizer is a small tool to optimize warcraft3 script- and mapfiles (.j,.w3x,.w3m)

Features

• Removing useless stuff like comments, whitespaces and empty lines
• Compressing variable and function names
• Replacing some JassVars to have the smallest possible filesize

Pros&Cons @ vex'
Pros:

• Compressing names generates with a more advanced charmap (a-z, A-Z, _, 0-9)
• Namecompression with custom natives works
• name compression with ExecuteFunction and TRVE uses a StringPattern to leave the code functional

Cons:

• Only optimizes script (yet) and doesn't provide any protection
• Should be used in combination with vex'

Coming Features:

• Some sort of BJ/GUI removal
• Removing unused variables & functions
• maybe numbershortening

I already have some more feature plans, but would like your guys feedback on this.

Requirements

As I'm doing this in Java, you need to have the latest Java Runtime Enviroment(1.6 or higher) installed on your system.


UI
Screenshot of the current GUI

Please post requests, comments and/or constructive criticism.

Download in Attachments.
(It's 10 mb because I had to import many dependent libs)

Also there is no real error-handling at the moment. When the progressbar doesnt complete/the labael doesn't say "Finished!" report the bug with the labelmsg and map.

 

[Bribe]

Is Java Version 7 different than JRE? I have the latest JRE on my system. Why not compile it to an Exe so that it doesn't need Java?

+12 rep when this thing is released and shown to be working right.

Advice for ExecuteFunc and TriggerRegisterVariable event:

if you have SCOPE_PRIVATE + "variableName" in vJass it compiles to "libraryName__" + "variableName", which breaks Vexorian's optimizer. If yours treated constant string concatenation better than his then we can ignore this problem. Obviously something like "FunctionName" + I2S(index) is not going to compile intuitively in which case it gets annoying like finding every function that could start with that function name and has an integer-type suffix, and then shortening both the string and then all compatible functions except for, of course, the integers appended to the end of the function name. Obviously then TRVE and EF relevent strings should compile first.

Vexorian currently uses all-capital variable names if there are cases where TRVE and EF are found, but you don't have to be so closed minded. All you have to do is make sure that any TRVE and EF strings do not have any case clashing, problem solved.

Also every call of Condition() should be turned into Filter() because it does the same but is a shorter function name.

Don't worry about things that already exist like compressing model files, but if you have a passion for that then fine.

 

[Frotty]

Is Java Version 7 different than JRE? I have the latest JRE on my system. Why not compile it to an Exe so that it doesn't need Java?

JRE 7 is what you need. http://www.oracle.com/technetwork/java/javase/downloads/java-se-jre-7-download-432155.html

Advice for ExecuteFunc and TriggerRegisterVariable event:

if you have SCOPE_PRIVATE + "variableName" in vJass it compiles to "libraryName__" + "variableName", which breaks Vexorian's optimizer. If yours treated constant string concatenation better than his then we can ignore this problem. Obviously something like "FunctionName" + I2S(index) is not going to compile intuitively in which case it gets annoying like finding every function that could start with that function name and has an integer-type suffix, and then shortening both the string and then all compatible functions except for, of course, the integers appended to the end of the function name. Obviously then TRVE and EF relevent strings should compile first.

Well I didnt think about that so much, but if you call EF terribly like that ("FunctionName" + I2S(index)) :/
Also how does:
if you have SCOPE_PRIVATE + "variableName" in vJass it compiles to "libraryName__" + "variableName", which breaks Vexorian's optimizer.
Break exactly?

Vexorian currently uses all-capital variable names if there are cases where TRVE and EF are found, but you don't have to be so closed minded. All you have to do is make sure that any TRVE and EF strings do not have any case clashing, problem solved.

Did I understand that right, that TRVE and EF can't take a string as argument that has case clashing?

Also every call of Condition() should be turned into Filter() because it does the same but is a shorter function name.

okay

Don't worry about things that already exist like compressing model files, but if you have a passion for that then fine.

Shadowdemon wanted to update blplap with a commandline function, so I thought about adding it. It's not yet implemented.

 

[Bribe]

Well if you pass to TRVE the variable "variable" it will be treated the same as if you had a variable named "VariAble". Same with ExecuteFunc.

Vexorian's optimizer doesn't recognize concatenation at all. If you have:

function HelloWorld takes nothing returns nothing
endfunction
...ExecuteFunc("Hello" + "World")

His optimizer will turn it into:

function A takes nothing returns nothing
endfunction
...ExecuteFunc("Hello" + "World")

 

[Frotty]

Okay, then I'll check for that.

 

[Magtheridon96]

OMG, I love you <3

Tip: To avoid mistakes that Vex made, you could look at the changelog of his optimizer.

Here's another optimization option you could add:
"Nes Mode" xD

It looks for boolean expressions like this: x==0
and changes them to: 0==x
x==null -> null==x

If a function is called only once, inline it.

If a local isn't used, delete it.
Same for globals.

Good luck with this Frotty :3
+3 rep

 

[Bribe]

If a function is called only once, inline it.

Definitely in support of this one.

The user would be pretty stupid to have globals and locals that aren't used anywhere though, so I wouldn't even waste energy coding that.

 

[WaterKnight]

Guess the only clean solution for dynamic arguments to ExecuteFunc would be to create a trigger for every function, assign them to their original names in a gamecache/hashtable and replace ExecuteFunc.

 

[Troll-Brain]

You could use some constants defined by the user, in a GUI or in a configuration text file (or just both which is better).
Like them :

SAFE_MODE = true/false

If it's true it doesn't try to shorten up real variables and all functions which could be called with an ExecuteFunc (string concatenation).

For Example :

call ExecuteFunc("MyF"+I2S(<integer variable>))
call ExecuteFunc("<string variable>" + "MyF"+I2S(<integer variable>))

function MyF1 ... // won't be shorten up

function ThisIsMyF3 ... // won't be shorten up

function MyF ... // will be shorten up 

function MyF1a // will be shorten up

The first function won't be shorten up because it match the pattern of the first ExecuteFunc call, and there is no safe way to be sure of the variable value part (not a constant).

The second function won't be shorten up because it match the pattern of the second ExecuteFunc call, and there is no safe way to be sure of the variable value part (not a constant).

The two other functions will be shorten up because they don't follow any ExecuteFunc call pattern (even if that are closed to it).

Ofc it's the same for real variable events (TriggerRegisterVariableEvent), but i think you would safely ignore this safety for real locals (why the hell would you link a such event on a local variable ?)

Or if you think it's just to much you can also support some prefix or postfix like "__protected" :

function ThisIsMyFunction__protected // won't be shorten because of the postfix "__protected".

These postfix/suffix could be included in a jass preprocessor with a special keyword, or directly used in jass by the user.

Again you are welcome to support both systems

About variables and functions shorten up, don't forget the character "_", even if this one can't be used in the first character, the first character has to be a letter, and the last one can be anything except "_" (letter or number).
Also don't forget you can user uppercases and lowercases.
In fact most of real maps will have at max 2 characters length name for variables and functions, using the full power of this special base.

You should reserve one character length for locals.

Remember that in newest patches the globals have the priority against locals, meaning variable shadowing doesn't work anymore, and if you declare a local variable which has the same name of a global, the global will be used.

Though, you should avoid this conflit, 2 character length seems enough fair.

And again with a jasspreprocessor (or not) and a postfix/prefix you could reserve the available one character for functions/variables defined by the user (speedfreak power, like the word "register" in C, even if of course isn't the same at all in the background)

Something really important, you should make your tool usable without a GUI in command line (this way you can include the optimisation directly in the JNGP or whatever, even if i've only sucessfully executed a .exe file, not a java one or whatever else, but maybe i did it wrong)
Also your tool should be able to work with a text file, not only with a map.

Care also that the user could inlude in his map a custom common.j (and maybe blizzard.j,common.ai) with new custom natives.

Good luck.

 

[Frotty]

It looks for boolean expressions like this: x==0
and changes them to: 0==x
x==null -> null==x

Will be easy to add

If a function is called only once, inline it.

planned that

If a local isn't used, delete it.
Same for globals.

Maybe this will be included with unused code removals

You could use some constants defined by the user, in a GUI or in a configuration text file (or just both which is better).
Like them :

SAFE_MODE = true/false

If it's true it doesn't try to shorten up real variables and all functions which could be called with an ExecuteFunc (string concatenation).

There is an .ini file for all options available in the GUI

Or if you think it's just to much you can also support some prefix or postfix like "__protected" :

function ThisIsMyFunction__protected // won't be shorten because of the postfix "__protected".

These postfix/suffix could be included in a jass preprocessor with a special keyword, or directly used in jass by the user.

Again you are welcome to support both systems

Possible Idea

About variables and functions shorten up, don't forget the character "_", even if this one can't be used in the first character, the first character has to be a letter, and the last one can be anything except "_" (letter or number).
Also don't forget you can user uppercases and lowercases.
In fact most of real maps will have at max 2 characters length name for variables and functions, using the full power of this special base.

I know, tho i'll just stay with a-z and A-Z. With 3 chars there are > 14000 possible combinations.

Something really important, you should make your tool usable without a GUI in command line (this way you can include the optimisation directly in the JNGP or whatever, even if i've only sucessfully executed a .exe file, not a java one or whatever else, but maybe i did it wrong)
Also your tool should be able to work with a text file, not only with a map.

Commandline-maybe
and y you can open .j files

Care also that the user could inlude in his map a custom common.j (and maybe blizzard.j,common.ai) with new custom natives.

Since rtc is broken there is no reason at all to use custom native files.

 

[Troll-Brain]

Ignoring the numbers and "_" is bad :

Only letters :

1 character = 52¹ = 52 combinations
2 characters = 52² = 2704 combinations
3 characters = 52³ = 140608 combinations

Jass grammar :

1 character = 52¹ = 52 combinations
2 characters = 52*62 = 3224 combinations
3 characters = 52*63*62 = 203112 combinations

I wouldn't say it's negligible, even if i'm agree that even only with letters, you would hardly need more than 3 characters, but in map with really huge scripts you could have to use 3 characters instead of 2, sooner than following the jass grammar.

And it's not that hard to use this special base, now i'm agree that in maps which are not heavily scripted that doesn't matter.

About RTC, it's just a general purpose i was talking, you can include some common.ai natives in a custom common.j, for example UnitAlive.
And nothing in programming world is dead for ever and ever.
But i suppose if you list the natives of common.j and common.ai, it will be enough for now, i'm just trying to make it work in every situations (suggestions)

Oh and btw you shoud'nt include speedfreak senseless optimizations if someone says it without a valid benchmark providen (Nestharus or not)

 

[Dr Super Good]

Does your optimizer reserve small length names for locals?

 

[Troll-Brain]

That reminds me by "locals" i meant also function arguments (takes ...)

Also inlining every constant globals is bad, you should inline only constants value (basically not a function call, except for the code type maybe)

constant timer Tim = CreateTimer() // fail it will create a new timer each time Tim is used ...

You could argue that constant handles are useless, but at least they are (i mean it's possible to use them)

 

[Dr Super Good]

Do not forget to check for unreachable code. An example of unreachable code is a function that is defined but never called or executed.

 

[jim7777]

well i suggest something like before compiling the map, check first for the listfile of the imports, if there are any unused imports, delete them. perhaps this maybe pain in the computer since it will look for both in the units and trigger strings but kinda useful especially unused icons

.slk optimization please

 

[Bribe]

Let's not put too much workload on the Froptimizer all at once.

Personally I feel the priorities should be:

Compress names/remove comments/whitespace and inline code, real, integer, string and boolean constants.

Then, optimization things, like inlining functions which are only called once, for example.

Afterward, then it's the non-JASS related things.

 

[jim7777]

A thing to add in the priorities list
make it a stand alone executable file D:

well first version may be java one o.o

 

[Switch33]

well i suggest something like before compiling the map, check first for the listfile of the imports, if there are any unused imports, delete them. perhaps this maybe pain in the computer since it will look for both in the units and trigger strings but kinda useful especially unused icons

If your gonna do this. Make checks for files imported with TerrainArt\ . . . (which are tiles). And LoadingScreen.mdx, and FullScreen.blp (for the loading screen ofc). As these aren't used in either object editor or trigger editor. Also remember that trigger strings always have two \ 's for paths.

 

[Frotty]

well i suggest something like before compiling the map, check first for the listfile of the imports, if there are any unused imports, delete them. perhaps this maybe pain in the computer since it will look for both in the units and trigger strings but kinda useful especially unused icons

.slk optimization please

if you got unused imported stuff youre doing it wrong.
Use Vex for .slk opt since i have no idea what he is doing with them.

A thing to add in the priorities list
make it a stand alone executable file D:

well first version may be java one o.o

Whats the problem with a .jar?

Let's not put too much workload on the Froptimizer all at once.

Personally I feel the priorities should be:

Compress names/remove comments/whitespace and inline code, real, integer, string and boolean constants.

Then, optimization things, like inlining functions which are only called once, for example.

Afterward, then it's the non-JASS related things.

Yup, I'll focus on Script Optimization right now.

EF and TRVE are kinda nasty, like why would you use concatenation if not with a variable part. Also this ("abc" + variable "cde") would be ass to use with compress names.

 

[Bribe]

Just because a .exe is easier to spread around (doesn't require
users to download JRE on top).

 

[Dr Super Good]

Just because a .exe is easier to spread around (doesn't require
users to download JRE on top).

Only for windows machines. Mac machines should come with java and they can not run .exe files. Remember that a .exe is a windows executable only.

Thus he has eithor 2 choices.
1. Use java (like he is) as that is cross platform and expect all windows users to have an appropiate JRE installed.
2. Use C/C++ and create different executables for different opperating systems.

2. Would provide optimum performance due to its native level execution on the appropiate platform. However as a jass script optimizer is hardly computationally challenging nor is it a realtime system, such performance gain is pointless and does not justify the inconvenience of cross platform C/C++ development.

 

[Bribe]

Can't Java compile to .app just as easily?

So then the betas could be .jar and the final version could be available as
.exe and .app.

 

[Frotty]

Can't Java compile to .app just as easily?

So then the betas could be .jar and the final version could be available as
.exe and .app.

Applets are for online/browser use

 

[Bribe]

.app's are for any Mac applications.

 

[Frotty]

So? It'll be a .jar
Get over it.

 

[Bribe]

Just a recommendation dude ?

 

[Frotty]

Just a recommendation dude ?

.app s cant be just run on windows afaik.
And why would I wanna convert something cross platform to an .exe and limiting it again?
Java should be on like every machine and also installing it just takes a few minutes.
Really I'm rather concerned getting the features done.

 

[Bribe]

What I meant is to compile it twice: once as a .exe, for windows people, and once as a .app, for Mac people.

Anyway it's of course fine to keep it as a .jar file. You can let someone else do the work I mentioned, for example

 

[Luorax]

I should have simply pressed the subscribe button, but I felt like telling you that I'm excited to see this new optimizer may be a good idea. So hell yea, here it is.

Finally the most annoying bugs of Vexrorian's optimizer are getting fixed. Thought this day will never come.

 

[Bribe]

I can write you a "shorten names" code if you want to take some of the workload off the top.

I get the feeling like this project is as incomplete as mine.

 

[Frotty]

Nah it's going pretty well,
but it got implemented into WurstScript, a project peq and me are working on.

 

[Bribe]

Well if you want some help at any point send me a PM and I could do my best.

 

[Magtheridon96]

Honestly, the current features are already way more than enough.
You don't need any more optimization-features

 

[Troll-Brain]

I don't know if you're still on it, but maybe fix lame jass things would be good, such as the handle local leak :

function Test takes ... returns unit
    local unit u = ...
    // stuff with u
    return u // awesome we will got an handle id leak when the unit will be removed
endfunction

How to fix it :

globals
    unit Temp_unit = null
endglobals

function Test takes ... returns unit
    local unit u = ...
    // stuff with u
    set Temp_unit = u
    set u = null
    return Temp_unit
    // no more handle id leak, it could still make a tiny memory one in some
    // cases, but seriously we can't handle that
endfunction

But yes it's in fact quite the opposite of a script optimisation, so it may not fit with a script optimizer.

 

[Frotty]

If this would be considered, the wurstcompiler would do that.
But it is really not necessary.
And yes, we are still on Wurst

 

[Troll-Brain]

Explain why it's not necessary, i mean you think that shouldn't be a part from your optimizer or basically you're saying that i'm lying ?

 

[Frotty]

What does "not being necessary" has to to with you lying?

 

[Troll-Brain]

Because handle id leaks can cause a performance issue later, not because how many memory it uses, but because it slows down the jass virtual machine.
Ofc it will happen only if there is a massive handle id leak.

For me, that's something that should be fixed by some tool, coz it's way too anoying for a manual process, and senseless anyway.

 

[Magtheridon96]

Here's another optimization:

After you remove whitespace, search for "==true" and erase it
"==true" is useless in all cases.


"==false" is ok.

 

[Troll-Brain]

Are you sure, the IsUnitType bug is fixed ?
http://www.wc3jass.com/viewtopic.php?t=2370 from http://www.wc3c.net/showthread.php?t=80693 (outdated, return bug and variable shadowing at least, but still good to have on mind)

 

[Bribe]

The IsUnitType bug is only a problem if it's the direct return value in a filter, in which case an optimizer shouldn't risk removing any "==true" type things.

 

[Troll-Brain]

Ofc (i believe that we can all read and understand correctly what tell the links that i've given), but testing if the bug is fixed can't be bad.
Any volonteer ?

 

[BBQ]

People, people, people. The bug with IsUnitType() occurs only when UNIT_TYPE_TOWNHALL is used. It's one of the many "superstitions" that have been present in the JASS world since forever.

library Test
{
    function B2I(boolean whichBoolean) -> integer
    {
        return whichBoolean;
        return 0;
    }
    
    function onInit()
    {
        unit townHall = CreateUnit(Player(0xF), 'htow', 0., 0., 0.);
        
        BJDebugMsg("Testing for UNIT_TYPE_TOWNHALL:\n"
        + I2S(B2I(IsUnitType(townHall, UNIT_TYPE_TOWNHALL))) + "\n" /* Prints 64. The bug is present ONLY here. */
        + I2S(B2I(IsUnitType(townHall, UNIT_TYPE_TOWNHALL) == true))); /* Prints 1. No bug. */
        
        KillUnit(townHall);
        
        BJDebugMsg("\nTesting for UNIT_TYPE_DEAD:\n"
        + I2S(B2I(IsUnitType(townHall, UNIT_TYPE_DEAD))) + "\n" /* Prints 1. No bug. */
        + I2S(B2I(IsUnitType(townHall, UNIT_TYPE_DEAD) == true))); /* Prints 1. No bug. */
        
        townHall = null;
    }
}

 

[Bribe]

I have heard it reported for UNIT_TYPE_HERO as well.

To test it, we can use this (I will do some tests in some hours when I am home)

struct Tester extends array
    static method check takes nothing returns boolean
        call SelectUnit(GetTriggerUnit(), false)
        return IsUnitType(GetTriggerUnit(), UNIT_TYPE_HERO)
    endmethod
    static method run takes nothing returns nothing
        call BJDebugMsg("True")
    endmethod
    static method onInit takes nothing returns nothing
        local unit u = CreateUnit(Player(0), 'Hpal', 0, 0, 0)
        local trigger t = CreateTrigger()
        call TriggerAddCondition(t, Filter(function thistype.check))
        call TriggerAddAction(t, function thistype.run)
        call TriggerRegisterUnitEvent(t, u, EVENT_UNIT_SELECTED)
        set t = null
        set u = null
    endmethod
endstruct

 

[BBQ]

I remember testing all of them, and only UNIT_TYPE_TOWNHALL was buggy. I just tested UNIT_TYPE_HERO (as per your suggestion), and it worked as expected.

 

[Bribe]

Ok, I have updated it. Can you test it again?

struct Tester extends array
    static method check takes nothing returns boolean
        call SelectUnit(GetTriggerUnit(), false)
        return IsUnitType(GetTriggerUnit(), UNIT_TYPE_TOWNHALL)
    endmethod
    static method run takes nothing returns nothing
        call BJDebugMsg("True")
    endmethod
    static method onInit takes nothing returns nothing
        local unit u = CreateUnit(Player(0), 'htow', 0, 0, 0)
        local trigger t = CreateTrigger()
        call TriggerAddCondition(t, Filter(function thistype.check))
        call TriggerAddAction(t, function thistype.run)
        call TriggerRegisterUnitEvent(t, u, EVENT_UNIT_SELECTED)
        set t = null
        set u = null
    endmethod
endstruct

 

[watermelon_1234]

^Works fine for me.

 

[Dr Super Good]

Hmm I guess Blizzard must have fixed that in one of the more recent patches. I remember using it a long time ago when I was learning the language with UNIT_TYPE_HERO and it was always returning false (at the time I did not know of the problem).

 

[Frotty]

Here's another optimization:

After you remove whitespace, search for "==true" and erase it
"==true" is useless in all cases.


"==false" is ok.

Well, this isn't generally adaptive and maybe happens 1/5000(not a necessary opt).
Also I don't remove the white-spaces, they just aren't printed.

 

[Bribe]

I think at least an option to remove all "==true" would be nice for a future edition, and "== false" would only be replaceable with the "not" keyword if you were somehow able to detect the complete compared value on the left, which could get ugly with things like "if blahandblah == false and blahnotblah == false and blahorblah == false" where you have to decypher the keywords as well, I just think the removal of "==false" is not worth the effort though removing ==true is pretty much find & delete.