[Benchmarks] The truth is (out) there.

[Troll-Brain]

I was bored of statements on the fly without any benchmarks provided, and because jass is enough fun for making false assumptions which beat the logic, i decided to make benchmarks.

I use cJass, just because it's the best tool i know for making benchmarks.
You can get it there : http://cjass.xgm.ru/

Because variable and function name length really do matter i will just assume that the code will be correctly optimized (one letter for locals, and two for globals/functions).

1 for locals because 52 combinations is more than enough (a-Z), and two for globals/functions because it seems also fair (52*62 = 3224) , (62 -> a-Z ; 0-9 but not _ )

Also don't take me wrong, i'm not a speed-freak, i'm perfectly aware that most of times script optimization is not needed and even that some (many ?) benchmarks here are quite or more pointless in real cases.

Fps drop test could give different results depending the operating system and the hardware configuration, but i will presume that it is reliable, it's not like we have decent other ways.
But you're welcome to test these codes by yourself and give your results.
Or submit your own tests, but plz give the code and the results together, just giving the result is bullshit.

Oh and btw the title could be pretentious but it's really not, it's just a (bad) pun (X-Files)

So first, here is a cJass template for benchmarks :

benchmark template

library Benchmark initializer init

    #define private TEST_TO_DO = 1
    #define private PERIOD = 0.01
    #define private CHECK_LIMITOP = true
    
/*

TEST_TO_DO -> Attribute unique integers for your tests, then the user has just to edit its value,
to compare your test codes.

PERIOD -> it's the timout of the timer, depends mostly your hardware configuration.
If it lags to much try an higher value, if there isn't enough fps drop, try a lower one.
Just be aware that the min timeout possible for a timer is 0.0001

CHECK_LIMITOP -> Set it to true if you want to ensure that your script doesn't reach the limitop.
When you have checked all your codes (by editing the value of TEST_TO_DO and compiling/running the map),
set it to false to perform the benchmarks.

*/

    globals
        // put your globals here
    endglobals
    
    #if (CHECK_LIMITOP)
        private boolean limit_op_reached = true
    #endif
            
    nothing T() { // Test function
    
        #if  (TEST_TO_DO == 1)
           
           // put your code here
           
        #elseif  (TEST_TO_DO == 2)
        
            // put your code here
            
        #else
            PauseTimer(GetExpiredTimer()) ; DisplayTimedTextFromPlayer(GetLocalPlayer(),0,0,666,"invalid TEST_TO_DO value, define it with a valid one")
        #endif
        
        #if (CHECK_LIMITOP)
            limit_op_reached = false
        #endif
        
    }
    
    private nothing init() {
    
        #if (CHECK_LIMITOP)
            TimerStart(CreateTimer(),0,false,function T) ; TriggerSleepAction(0)
            if (limit_op_reached)
                DisplayTimedTextFromPlayer(GetLocalPlayer(),0,0,666,"WARNING : limitop reached for your test " + I2S(TEST_TO_DO))
            else
                DisplayTimedTextFromPlayer(GetLocalPlayer(),0,0,666,"limitop not reached for your test " + I2S(TEST_TO_DO))
            endif
        #else
            TimerStart(CreateTimer(),PERIOD,true,function T)
        #endif
    }
    
endlibrary

locals vs globals in a theoretical scenario (1 declaration/setting + 2 read)

I just realized that this test is a fail because of what i've said above, i keep it because theoretically globals are faster than locals but in a practical case i don't know yet, just because of the number of globals and their name length.
I will do a better one when i will have the appropriate tool for it.

library Benchmark initializer init {

    #define private TEST_TO_DO = 2  // 1 for locals test and 2 for globals test
    #define private PERIOD = 0.007 // depends your hardware configuration, if it lags to much try an higher value, if there is not enough fps drop try a lower one, btw the min is 0.0001
    
    globals
        #for i (1,5000)
            integer J##i
        #endfor
    endglobals
            
    private nothing Test() {
    
        integer x
    
        #if  (TEST_TO_DO == 1)
        
            #for i (1,5000)
                integer j##i = 1
                x = j##i
                x = j##i
            #endfor
        
        #elseif  (TEST_TO_DO == 2)
            
            #for i (1,5000)
                J##i = 1
                x = J##i
                x = J##i
            #endfor
            
        #else
        
            PauseTimer(GetExpiredTimer()) ; DisplayTextToPlayer(GetLocalPlayer(),0,0,"invalid TEST_TO_DO")
            
        #endif
        
        // DisplayTextToPlayer(GetLocalPlayer(),0,0,"limit op not reached")
        
    }
    
    private nothing init() {
        TimerStart(CreateTimer(),PERIOD,true,function Test)
    }
    
}

Result : 30 fps for locals and 58 fps for globals (just starting to lost fps, my max is 60 fps)

Conclusion : Since it's only a theoretical use i can't conclude yet

PS : I'm open to all critics, comments and suggestions as long they are constructive.

[Troll-Brain]

locals vs globals with the same name length (1) and heavy usage (write)

library Benchmark initializer init {

    #define private TEST_TO_DO = 2  // 1 for locals test and 2 for globals test
    #define private PERIOD = 0.003 // depends your hardware configuration, if it lags to much try an higher value, if there is not enough fps drop try a lower one, but be aware that the min is 0.0001
    
    globals
        #for i (1,5000)
            integer J##i
        #endfor
        integer X
    endglobals
            
    private nothing Test() {
    
        #if  (TEST_TO_DO == 1)
        
            integer x
            #for i (1,20000)
                x = 1
            #endfor
        
        #elseif  (TEST_TO_DO == 2)
            
            #for i (1,20000)
                X = 1
            #endfor
            
        #else
        
            PauseTimer(GetExpiredTimer()) ; DisplayTextToPlayer(GetLocalPlayer(),0,0,"invalid TEST_TO_DO")
            
        #endif
        
        // DisplayTextToPlayer(GetLocalPlayer(),0,0,"limit op not reached")
        
    }
    
    private nothing init() {
        TimerStart(CreateTimer(),PERIOD,true,function Test)
    }
    
}

Result : 25 fps for locals and 45 fps for globals.
Note that i have the same results with 10 000 globals declared :

globals
    #for i (1,10000)
        integer J##i
    #endfor
    integer X
endglobals

In other words, the number of global variables doesn't matter.

locals vs globals in a practical usage (local length =1 , global length = 2) with heavy usage (write)

library Benchmark initializer init {

    #define private TEST_TO_DO = 1  // 1 for locals test and 2 for globals test
    #define private PERIOD = 0.003 // depends your hardware configuration, if it lags to much try an higher value, if there is not enough fps drop try a lower one, but be aware that the min is 0.0001
    
    globals
        #for i (2,5000)
            integer J##i
        #endfor
        integer J1
    endglobals
            
    nothing T() {
    
        #if  (TEST_TO_DO == 1)
        
            integer j
            #for i (1,20000)
                j = 1
            #endfor
        
        #elseif  (TEST_TO_DO == 2)
            
            #for i (1,20000)
                J1 = 1
            #endfor
            
        #else
        
            PauseTimer(GetExpiredTimer()) ; DisplayTextToPlayer(GetLocalPlayer(),0,0,"invalid TEST_TO_DO")
            
        #endif
        
        // DisplayTextToPlayer(GetLocalPlayer(),0,0,"limit op not reached")
        
    }
    
    private nothing init() {
        TimerStart(CreateTimer(),PERIOD,true,function T)
    }
    
}

Result : 34 fps for locals and 26 for globals.

Conclusion : Globals are theoretically faster than locals in an heavy usage (like an huge loop), but not in a practical use, see the main post of this thread for more details.
Note that i'm talking about heavy usage of the variables not the usual loops or classic usage in general, i will test them later.

[Troll-Brain]

testing the matter of the variable length name (write)

library Benchmark initializer init {

    #define private TEST_TO_DO = 2  // 1 for short variable name and 2 for the longer name
    #define private PERIOD = 0.003 // depends your hardware configuration, if it lags to much try an higher value, if there is not enough fps drop try a lower one
    
    globals
        #for i (1,20000)
            integer J##i
        #endfor
        integer J
    endglobals
            
    private nothing Test() {
    
        integer x
    
        #if  (TEST_TO_DO == 1)
        
            #for i (1,20000)
                J = 1
            #endfor
        
        #elseif  (TEST_TO_DO == 2)
            
            #for i (1,20000)
                J20000 = 1
            #endfor
            
        #else
        
            PauseTimer(GetExpiredTimer()) ; DisplayTextToPlayer(GetLocalPlayer(),0,0,"invalid TEST_TO_DO")
            
        #endif
        
        // DisplayTextToPlayer(GetLocalPlayer(),0,0,"limit op not reached")
        
    }
    
    private nothing init() {
        TimerStart(CreateTimer(),PERIOD,true,function Test)
    }
    
}

Result : 46 fps for the short name and 18 fps for the longer.
Note that i have the same results with only 2 globals declared :

globals
    integer J
    integer J20000
endglobals

In other words, the number of global variables doesn't matter (again).

testing the matter of the function length name (write)

library Benchmark initializer init {

    #define private TEST_TO_DO = 2  // 1 for short name test and 2 for long name test
    #define private PERIOD = 0.006 // depends your hardware configuration, if it lags to much try an higher value, if there is not enough fps drop try a lower one, but be aware that the min is 0.0001
            
    nothing S() {
    }
    
    nothing SuperLongName() {
    }
    
    nothing T() {
    
        #if  (TEST_TO_DO == 1)
        
            #for i (1,20000)
                S()
            #endfor
        
        #elseif  (TEST_TO_DO == 2)
            
            #for i (1,20000)
                SuperLongName()
            #endfor
            
        #else
        
            PauseTimer(GetExpiredTimer()) ; DisplayTextToPlayer(GetLocalPlayer(),0,0,"invalid TEST_TO_DO")
            
        #endif
        
       //  DisplayTextToPlayer(GetLocalPlayer(),0,0,"limit op not reached")
        
    }
    
    private nothing init() {
        TimerStart(CreateTimer(),PERIOD,true,function T)
    }
    
}

Result : 44 fps for the short name and 21 fps for the longer one

Conclusion : The length name of variables and functions indeed does matter about the script efficiency.

[Troll-Brain]

Originally Posted by Bribe

Interesting that globals have a better write speed, what I recommend is doing the first
test again with more like 3-4 uses per local. The reason being is because declaring a local
for just 1 repeat use is almost always a complete waste of time and is not a common
scenario. In fact I won't declare a local for things like GetTriggerUnit() unless it is called
more than 2-3 times.

Actually that's 1 write and 2 reads, but i can do more if you want, even if i bet that won't change that much.

What do you suggest ? (1 write, 4 reads ? )

Anyway i will do more tests , like hashtable vs array, GetHandleId + constant handle vs constant integer, but at very least 11 hours later from now.

Also for all, feel free to post your own.

[geries]

Nice job :) That I would see is Hashtables vs Global arrays, Hashtables vs Game Caches( due it is a table too ), BJ things vs non BJ things like

function A takes integer a , integer b returns integer
     return a + b
endfunction

function B takes nothing returns nothing
     call DisplayTextToPlayer( Player( 0 ) , 0 , 0 , I2S( A( 1 , 1 ) ) )
endfunction

// VS

function C takes nothing returns nothing
     call DisplayTextToPlayer( Player( 0 ) , 0 , 0 , I2S( 1 + 1 ) )
endfunction

and Reals vs Integers :)

EDIT: and the ForGroup vs unit group looping( with FirstOfGroup )

[Bribe]

I can't imagine why unit group looping would be slower than ForGroup - ForGroup opens up
a new thread for every unit.

[Magtheridon96]

I can't imagine why unit group looping would be slower than ForGroup - ForGroup opens up
a new thread for every unit.

You don't need ForGroup if you're going to use the filter of the GroupEnumUnitsInRange function to manipulate the units :P

Btw, are you sure ForGroup creates a new thread per unit?

and Reals vs Integers :)

It's likely that integers are marginally faster :P (Meaning it wont matter much ^^)
Reals need more memory than integers, so logically, they would be slower than integers :D

[geries]

Originally Posted by Magtheridon96

It's likely that integers are marginally faster :P (Meaning it wont matter much ^^)
Reals need more memory than integers, so logically, they would be slower than integers :D

I know. Just cause this is a benchmark with FPS test then I am curious that how much slower.

[Bribe]

Originally Posted by Magtheridon96

Btw, are you sure ForGroup creates a new thread per unit?

All code-based stuff opens up a new thread. In fact, there is a common.ai native (regrettably
doesn't work because Blizzard cheated us on this one) which is called StartThread which
takes a code argument.

If you take a look at Nestharus' BigInt library, he evaluates a boolexpr and even that
boolexpr has its own op limit which does not crash the thread where it was called from -
indicating that it has a thread of its own. It wouldn't surprise me.

The reason "executes" are so slow is becuase they allow waits.

[Magtheridon96]

I know. Just cause this is a benchmark with FPS test then I am curious that how much slower

Well, since:

Integers -> 32 bits (4 bytes)
Reals -> (They're floats in WC3) ->
32 bits + 3 bits per decimal digit

I think that should be about right :)
And since floats hold 7 decimal digits
->

32 bits + 3 * 7 bits = 53 bits

So, integers should be about 1.65625 times faster than reals :D

[geries]

All right I made some benchmarks too:

All value was tested in 10000 OP / sec for 10 seconds
I was monitoring the CPU process too

Reals VS Integers
There were no difference in the FPS, real's benchmark took 1-3% more CPU

BJs VS non BJs
There were NO DIFFERENCE

Hashtables VS Game Caches
Well this benchmark was the most succesful. Game caches took 2-3x more CPU than hashtables. However there were no difference in the FPS

Okey I made the same with 10x operations

Reals VS Integers
No difference again...

BJs vs non BJs
Finally. Difference. BJs FPS ~50FPS non BJs ~60FPS

Hash VS GameCache
Game caches became unhandleabe laggy( 2 FPS or less ) so It was not really took 10 seconds... was
Hashtables better here, but while they freezed the computer while processing gamecaches wasn't.

Conclusion: Reals not dangerous, however they can be unnaccure. Do not use game caches too much just if it needs. BJs does less harm than we tought. I say everyone use BJs as much as he want. The 10 FPS difference shown up at 100000 BJ operations

[Magtheridon96]

All right I made some benchmarks too:

All value was tested in 10000 OP / sec for 10 seconds
I was monitoring the CPU process too

Reals VS Integers
There were no difference in the FPS, real's benchmark took 1-3% more CPU

BJs VS non BJs
There were NO DIFFERENCE

It depends on what BJs you were using :P

Are one-line BJs like GetLastCreatedUnit() inlined by JNGP? :S

I'd say those results are impossible.
It's like saying a function call has no overhead..

Oh and when I said reals were slower than integers, it's quite marginal (obviously)

[Nestharus]

Hm... that's very different from what I had when deving Timer Tools.


The 6 global sets in it completely crippled it (many global sets + global reads). Also, the 3 local declarations crippled it too (1 local read + 1 local write was a bit faster than declaring that extra local).



Anyways... here's the next thing I'm wondering.. global lookups require a search, that's just normal to all programming languages. The bigger the scope, the longer the search. On TH when it was benched, more globals made global reads/writes slower than locals.

What you should try is declaring half above the global var in question and half below to see if that makes a different. Do like x, y, z and test y.

[Magtheridon96]

That makes sense Nestharus >:P

[geries]

Originally Posted by Magtheridon96

It depends on what BJs you were using :P

Are one-line BJs like GetLastCreatedUnit() inlined by JNGP? :S

I'd say those results are impossible.
It's like saying a function call has no overhead..

Oh and when I said reals were slower than integers, it's quite marginal (obviously)

Believe me that I told :) Tell me one BJ I'll test it out. I used the example function BJ takes integer a , integer b returns integer

About the reals: maths for computer like + - is easy for it. Also I have already noticed that reals after a time become unaccure they show different value than they has to.

The function calls for the BJs are that game has to search that function and thats the only pluss in it.

EDIT:

Okey. You wanted. I choosed a very much used BJ TriggerRegisterAnyUnitEventBJ . The results now better for the BJs. Test made with 1200 op/sec.

RESULT FOR NON BJ=48~50 FPS
RESULT FOR BJ = ~45 FPS

Thats the sad truth

The test contained operation counting and trigger creating( local ) and destroy

Alsooo If you don't null variable it will return epic fast memory usage increment