JASS Benchmarking Results

[PurgeandFire]

People are biting at straws in this thread. So what if one method is 30% slower than the other? In most maps there are bigger fish to fry.

Has anyone actually benchmarked how much wasted time all those trigger evaluations cause when a custom ability is cast? The common way of doing a fully MUI custom ability is a any unit begins effect event followed by some kind of test to kill the thread if the wrong ability is cast. This is fine for 1, 2 and even 5 abilities but some maps have dozens of custom abilities doing this. The overhead of evaluating custom abilities is thus O(n) per cast where n is the number in the map. This technically means a map with hundreds of custom abilities would have a pretty huge overhead per ability cast just to check if it was the correct ability being cast. Worse is these evaluations apply to even non-triggered abilities as they still are tested if they are triggered.

So the solution? Well two spring to mind...
1. Use a single ability cast system that registers functions and the ability to trigger them. When an ability is cast by a unit it then performs a hashtable lookup for the ability function. If none exists then it was not a triggered ability and nothing happens. If one exists then it evaluates it. This results in a O(1) overhead no mater how many triggered abilities are registered.
2. Only register units which have the ability to the ability trigger using a specific unit event. This still results in O(n) tests but n is only the triggered abilities that the specific unit has so is considerably smaller and a non-issue. The system needs to manage removed units to be MUI so that adds some additional overhead. The advantage of this approach is no tests for units that do not have triggered abilities at all.

The potential savings is small but I imagine it will add up over time. Especially seeing how each evaluation creates a separate trigger thread.

Yep. That led to the creation of systems like RegisterPlayerUnitEvent, SpellEffectEvent, OrderEvent, GTrigger, etc.

Although, I don't think it needs a benchmark. Complexity wise it is clearly superior.

P.S. I like those idioms in your post (biting at straws, bigger fish to fry).

 

[Zwiebelchen]

People are biting at straws in this thread. So what if one method is 30% slower than the other? In most maps there are bigger fish to fry.

Has anyone actually benchmarked how much wasted time all those trigger evaluations cause when a custom ability is cast? The common way of doing a fully MUI custom ability is a any unit begins effect event followed by some kind of test to kill the thread if the wrong ability is cast. This is fine for 1, 2 and even 5 abilities but some maps have dozens of custom abilities doing this. The overhead of evaluating custom abilities is thus O(n) per cast where n is the number in the map. This technically means a map with hundreds of custom abilities would have a pretty huge overhead per ability cast just to check if it was the correct ability being cast. Worse is these evaluations apply to even non-triggered abilities as they still are tested if they are triggered.

So the solution? Well two spring to mind...
1. Use a single ability cast system that registers functions and the ability to trigger them. When an ability is cast by a unit it then performs a hashtable lookup for the ability function. If none exists then it was not a triggered ability and nothing happens. If one exists then it evaluates it. This results in a O(1) overhead no mater how many triggered abilities are registered.
2. Only register units which have the ability to the ability trigger using a specific unit event. This still results in O(n) tests but n is only the triggered abilities that the specific unit has so is considerably smaller and a non-issue. The system needs to manage removed units to be MUI so that adds some additional overhead. The advantage of this approach is no tests for units that do not have triggered abilities at all.

The potential savings is small but I imagine it will add up over time. Especially seeing how each evaluation creates a separate trigger thread.

A lot of people go way too far to over-optimize code when simple if-nesting would actually be faster. For the weight of a single trigger-evaluation, you can do dozens of if-statements checking ability IDs on an any-unit-spellcast event.

And even if you have enough abilities to make a dynamic indexing of spells worth it: if you create just one single damn dummy unit in your spell script, the speed increase goes into diminishing returns vs the nuclear bomb that is CreateUnit().


I mean... look at this, for example:
http://www.hiveworkshop.com/forums/jass-resources-412/system-destructablehider-219569

Compared to the bullshit that WC3 does in the background (that you can not prevent), coding efficiency on such a microscopic scale just doesn't matter.

 

[WaterKnight]

@Dr Super Good: But, but, we have already done that >.< Sad enough, if you do not find anything anymore to significantly optimize. Though I do think like Zwiebelchen that passive checking is very cheap and the dynamic invocation may become even more expensive because of systems like a priority queue you may want to blow in. However, it boosts the comfort and as a mapmaker you want to neatly realize all the stuff.

 

[Waffle]

if you have ever seen a hundred semi-identical triggers in a map you wont whine about "overcomplicating" stuff again..ever

i have literally had days when i write 3 pages of code and the map shrinks by 0.2 MB in the end. this stuff does matter!
(and i won't even go in to the performance implications of getting rid of a hundred triggers..)

 

[Malhorne]

We're not talking about memory but efficiency....
And if you're using effiency in every point you'll win some performances....
If you're using each time the wrong choice it can impact !

 

[Zwiebelchen]

if you have ever seen a hundred semi-identical triggers in a map you wont whine about "overcomplicating" stuff again..ever

i have literally had days when i write 3 pages of code and the map shrinks by 0.2 MB in the end. this stuff does matter!
(and i won't even go in to the performance implications of getting rid of a hundred triggers..)

It's not about overcomplicating stuff. Surely copies of an identical trigger are an extreme that only bad GUIers do.

What we are talking about is the difference between using a dynamic system and for example hardcoding some if-nesting to select which function to run.


The average spellcasting system will dynamically run TriggerEvaluation on a registered function indexed by the spell ID that you can retrieve via the spellcast event.

The noob approach will just do if spellID == 'A000' then, elseif spellID == 'A001' then, etc.

It doesn't actually make any difference in terms of the size of the code. The specific spell's function still needs to be fired, no matter what you want to do.
In fact, the noob approach has exactly the same amount of lines of code than the professional approach (minus the additional registry periphery).
In terms of speed, the noob approach is actually faster for low amounts of spells (I expect the break even somewhere at around 30 spells).

And even for 100s of spells, the noob approach will not suffer any noticable loss of speed. That's because simple integer comparisons are super-fast compared to TriggerEvaluations, which will always fire a new thread.


Having multiple copies of the same function is just bad programming. That's not what we are talking about. However, using a case selection structure with if-nesting over any dynamic approach is not. It's just convenient.

 

[Ceday]

It is not noob approach it is classic approach

 

[Cokemonkey11]

"naive approach" is probably what you're looking for http://stackoverflow.com/questions/...-algorithm-and-what-is-a-closed-form-solution

 

[Zwiebelchen]

Okay, forget about the noob approach and call it naive approach.

The fact remains that in this particular case, the naive approach isn't even that bad.


There's other cases where the simple naive approach actually beats heavily dynamic systems:

For example, custom procc effects on weapons. I could use a dynamic system that stores a function to the specific unit when the specific item is picked up and then dynamically call that stored function upon hit.
Or I could just use if-cases and check for the items that have a procc effect manually.

The first is way more elegant. However, the naive approach is faster in most cases (unless you got like 50 items with custom procc effects). And it gets rid of a few lines in the item scripts that apply the procc function to the unit.


What I'm trying to say is: depending on the scope of your map, over-optimizing stuff can actually be counter-productive.


I'm currently doing all on-damage stuff related to buffs by if-casing and checking for the BuffId, instead of using dynamic triggers for the buffs in my map.
Sure, the on-damage trigger bloats up a lot by that. But I simply know I will never have enough on-damage effects in my map to make a dynamic approach actually worth the trouble.

 

[Cokemonkey11]

Can someone benchmark IssueTargetOrderById() vs IssueTargetOrder() ?

 

[Malhorne]

Could you give the id and the order and I can do it if you want.

 

[Cokemonkey11]

"harvest" and 852018

 

[Malhorne]

Hum sorry I have something to do with family you'll have to wait ^^'

 

[WaterKnight]

via string is marginally slower, takes ~2,6% more time, both suffer from init inflation though

 

[BPower]

Peanuts, still good to know. Thank you.

 

[Cokemonkey11]

Someone should sticky this thread.

 

[Ceday]

360000 iterations of A() took 270 ms
360000 iterations of TriggerEvaluate(T) took 2295 ms

A() is 87.691% faster than TriggerEvaluate(T)

A() is 750% faster than TriggerEvaluate(T)
TriggerEvaluate(T) is 88% slower than A()

 

[Cokemonkey11]

Can someone benchmark

GetUnitGoldCost(u)

vs

set q = 5 // global
...
local integer s = q

 

[Malhorne]

How am I supposed to benchmark with local declaration ?
I can't declare a local xxx times :/

 

[Cokemonkey11]

just set s = q then

 

[Malhorne]

Ok I'll do it
But I think the set s = q will be faster

EDIT : My sharpcraft don't want to run it :O !!

 

[BPower]

Can someone run a test, which compares SetUnitX/Y vs SetUnitPosition while the unit
has collision set to 0 and is paused? I read a very interesting theory about SetUnitPosition
performance with paused units and I would like to see a benchmark test.

 

[Bribe]

If I can find my old benchmarking tool I will cook that up

Edit: here's the one I was using (I think tooltiperror made it originally): FarSightBJ

There's also this old thread: http://www.hiveworkshop.com/forums/triggers-scripts-269/benchmarks-truth-out-there-201810/

 

[pusha1999]

in my tests if group have too much units, firstofgroup is slowest method, i dont need the tests with benchs, i just look on monitor and see the lag. forgroup and filters are more faster and never lags for me...mb im wrong, but its true. i see what i see...try to use it for 200-300 units, and firstofgroup will be lag instead of forgroup or filterfunctions.

 

[Bribe]

You were doing something else differently.

 

[pusha1999]

You were doing something else differently.

whats different could be in your code and this code?

function blablabla takes nothing returns boolean
set u = GetFilterUnit()
if UnitAlive(u) then
some code....
//if need add unit in group then
return true
endif
return false
endfunction

1. group g = CreateGroup()
call groupenumunits...(g,player(0),condition(funciton blablabla))
2. group g = CreateGroup()
call forgroup(g,function....) do actions in function..
3. group g = CreateGroup()
unit u
loop
set u = firstofgroup(g)
exitwhen u == null
some code
call groupremoveunit(g,u)
endloop



your loops got smth special?
if too much units, then loop starts to lag on ms. i test all my spells specially on pentium II 533 MHz how many lags they will have and maximum optimize script for minimize lags. i know thats forgroup its not very good way for using qoz leaking and need everytime destroy group, and then no point to create one global group and etc.. but, for me its works faster, ofc on core i5 for example like mine second main pc, there is no different to use loops with 1000 units and for group, no lags, but not all ppl have good PCs, and most of them still have smth like my test comp.

its smth like sometimes ive seen thats structs works faster than hash, but in some situations hash works faster. idk why, i havent source code of libs functions for WE and dont know whats there algoritm, but its just mine expirience. ive tested lot of systems, your systems also like spellevents and look whats happening and i there is also sometimes its good way, sometimes not (i has a desync problem in map after this)) after i switch off this lib, desync disapear) magic. depends of map and how many spells there and code of map qoz could be some conflicts. so for now i have only 1 trigger wich detects spell event and there is check whats spell and starts needed function there wich in library. so ive got over 200 spells, so i dont wanna create 200 triggers for evaluate them, just 1 trigger and 1 library with functions also i tested grouputils and timerutils and etc...in true i made descision dont use them qoz best way its create ur own system for smth and u will understand everything what u doing.
anyway im always opened for discuss and for get knowledges.

and btw also mb you know how to avoid pause periodic timers bug ( i mean after pause, and resume, timer will not periodic anymore )...so in my map when happens duel, all units wich not in duel, are paused, so i need pause spells...i didnt find other ways, only one, i create timer or periodic trigger, and when duel starts, after some checks i pause triggers and after resume them...not best way, but have no choice now. :\ tryed to use non periodic timers and creating triggers with timerevent finish...but this way too strange. also i tryed to save in hash triggeractions wich used in timers codefunctions and then restart timer...but sometimes works, sometimes fatal, sometimes just dont works... so mb if u know how to make this better, il very appretiate this.

 

[PurgeandFire]

Whoa, a Pentium II.

Anyway, it isn't clear which is better. As far as performance programming goes, "it runs well on my computer" or "it runs poorly on my computer" doesn't really say much (although the latter is usually more interesting). Could you post your code you used to benchmark it? If you just tested it in your actual playable map, then it might not be as convincing of a test because there are too many variables to consider. Just to name a few: it can easily be confounded with GPU performance if your camera is not in the same location between two tests (e.g. one test might be rendering 20 units in the scene whereas the other one might be in fog of war); different events may be going on that slow down performance; it is harder to isolate the issue, etc.

You probably won't notice speed differences unless you're doing stress testing by spamming the same instruction every 0.001 seconds or so (even then, it isn't really that great of a performance-measure). I'm happy you brought it up if those were your results though. But we have to make sure it is a correct test.

As for hashing vs. arrays, it really depends on the context. Again, that is something that is difficult to measure. Without stress testing, the results won't be prominent enough to notice. With stress testing, CPU caching comes into play (and usually array access will win out), but that might not be the results we're curious about. Use whichever one makes sense for the situation. Hashtables are fine to use--a lot of people prefer not to use them because they're verbose, the performance difference is negligible.

As for the periodic timers bug, you just have to restart the timer with TimerStart(). I don't see why you would have issues with it unless there is something wrong with your code.

 

[pusha1999]

Whoa, a Pentium II.

Anyway, it isn't clear which is better. As far as performance programming goes, "it runs well on my computer" or "it runs poorly on my computer" doesn't really say much (although the latter is usually more interesting). Could you post your code you used to benchmark it? If you just tested it in your actual playable map, then it might not be as convincing of a test because there are too many variables to consider. Just to name a few: it can easily be confounded with GPU performance if your camera is not in the same location between two tests (e.g. one test might be rendering 20 units in the scene whereas the other one might be in fog of war); different events may be going on that slow down performance; it is harder to isolate the issue, etc.

You probably won't notice speed differences unless you're doing stress testing by spamming the same instruction every 0.001 seconds or so (even then, it isn't really that great of a performance-measure). I'm happy you brought it up if those were your results though. But we have to make sure it is a correct test.

As for hashing vs. arrays, it really depends on the context. Again, that is something that is difficult to measure. Without stress testing, the results won't be prominent enough to notice. With stress testing, CPU caching comes into play (and usually array access will win out), but that might not be the results we're curious about. Use whichever one makes sense for the situation. Hashtables are fine to use--a lot of people prefer not to use them because they're verbose, the performance difference is negligible.

As for the periodic timers bug, you just have to restart the timer with TimerStart(). I don't see why you would have issues with it unless there is something wrong with your code.

about timers, ive got a lot of spells were need a periodic timer, like i said, map got duel system, and when duel starts, all mobs, all creeps and everything wich not in duel are paused, so i need a pause timers too, qoz some of those spells, got a buffs or effects wich not paused if use only PauseAllUnits. they disapear when spell is finished with periodic timer.
so for example i use
call Timerstart(timer,.03,true, function ...)
if il pause this timer and after il resume him, then he will not be a periodic anymore, qoz in codefunction i use for example
local timer t = GetExpiredTimer()
so how can i use there call TimerStart(same timer)? ) i need then same function wich be upper of this and again and again...ofc i can use execute funtion method, but sometimes they are not works good after optimize the map.
so for now i just create a periodic trigger instead of timer, and after spell finished, i destroy him, and when need to pause spell, i stop the trigger. and after duel i resume him. its not the best way, ofc i can made vars with function names for every timers, but then need made over 200 vars for that qoz function codes cannot be in arrays, its not good point. about stress tests. well, i didnt use fog of war, i just putted on map over 200-300 units and used the spell and my cam position was on point where those units. i didnt tell that use forgroup or filter is faster everytime, but in my map this works faster, ofc its depends of coding, but loops with 200-300 units everytime worked slower for me(there need use fog group for add back units into main group again with 200-300 units loop). its could be not for everyone, but for me loops are slower for big groups. if units in group over 10-15, then ofc i use loop method, qoz no point to made anothrer one leak.

example code in periodic timer with timeperiod 0.03-0.001:
function bbbb takes noting returns nothing
local timer t = GetExpiredTimer()
local unit target
local unit caster = lets it be some loaded unit
loop
set target = Firstofgroup(group1)
exitwhen target == null
some code....
call GroupAddUnit(group2,target)
call GroupRemoveUnit(group1,target)
endfunction
//and after this need add back those units in group1 (fog group method)
loop
set target = FirstOfgroup(group2)
exitwhen target == null
call GroupAddUnit(group1,target)
call GroupRemoveUnit(group2,target)
endloop
null locals
endfunction

so this method will be 100% for slower than if i just use ForGroup method if in group more than 15-20 units...if there is 100 or more then will starts lag spikes, but forgroup will not start those spikes. (only for me,cant be sure 100% for other ppl) this tested on pentium 2

 

[Bribe]

Why go through the group a second time instead of swapping the group pointers at the end of the loop? An FoG only needs 1 loop.

 

[pusha1999]

Why go through the group a second time instead of swapping the group pointers at the end of the loop? An FoG only needs 1 loop.

you mean that?
loop
set u = firstofgroup(g1)
exitwhen u == null
//some actions
call groupaddunit(g2,u)
call groupremoveunit(g1,u)
set u = firstofgroup(g2)
call groupaddunit(g1,u)
call Groupremoveunit(g2,u)
endloop
if yes, then its a little strange...will not be like one unit will be always firstofgroup and loop will never end?

 

[edo494]

the second set with groupadd and groupremove are most likely actually bug and should not be there if you want single pass over group, and yes, it would create infinite loop

 

[pusha1999]

the second set with groupadd and groupremove are most likely actually bug and should not be there if you want single pass over group, and yes, it would create infinite loop

thats why i never do that
if use groupaddgroup, then there is forgroup function. so then dont see the point use loops instead a forgroup( lets imagine they everytime works with same speed)...

 

[PurgeandFire]

Bribe meant like this:

loop
    set u = FirstOfGroup(g1)
    exitwhen u == null
    /* Do Actions */
    call GroupRemoveUnit(g1, u)
    call GroupAddUnit(g2, u)
endloop
call DestroyGroup(g1)
set g1 = g2

Of course, this is destructive (keep that in mind if you're writing a function that accepts the group as an argument). However, it works just fine if you use it as it should be used.

 

[pusha1999]

Bribe meant like this:

loop
    set u = FirstOfGroup(g1)
    exitwhen u == null
    /* Do Actions */
    call GroupRemoveUnit(g1, u)
    call GroupAddUnit(g2, u)
endloop
call DestroyGroup(g1)
set g1 = g2

Of course, this is destructive (keep that in mind if you're writing a function that accepts the group as an argument). However, it works just fine if you use it as it should be used.

so this good for 2 executes, and as i think need in executing function of timer everytime create groups, globals they or not, doesnt metter here. well, i think this method anyway can live but anyway i think its depends of situations, what spell and what there need to do, but if group will be destroyed, so whats the point made loops instead forgroup calls if this leak will be removed also DestroyGroup? only if loops really works faster

 

[PurgeandFire]

You shouldn't use FirstOfGroup() loops just for speed. That sort of min-maxing isn't going to help performance (not significantly, anyway).

The best thing about FirstOfGroup() loops is that you don't have to set a bunch of temporary variables. You can refer to your locals directly and the code usually ends up less complex because of that. You should use FirstOfGroup() loops for convenience--the performance is just a side bonus.

If you already implemented all your spells using ForGroup() or GroupEnum...(...) filters, just leave it as it is. It is just convenient to have FirstOfGroup() as an option when coding. And when you don't need to retain unit group members (i.e. you are just using the groups for enumeration), then FirstOfGroup() really shines (you don't even need a second group!).

 

[Dr Super Good]

ForGroup can also cope with null units (if unit was removed while inside the group). This makes it more robust for use with persistent groups. FirstOfGroup loops break in that situation as the returned value of null might not mean the end of group has been reached.

Swap group FirstOfGroup loops are just not that useful. They only work well if there is a single reference to the group so swapping is easy. Any speed benefits are small so only noticeable on performance critical code. Maintainability of the code is reduced as it makes loops more complex. Unless the function is recursive, any advantage of using local variables can be pushed to a ForGroup loop by using global variables as common registers. There are also potential performance implications for very large groups since the time taken to add or remove a unit from a group increases with group size.

If your map is suffering performance problems then chances are there are "bigger fish to fry" than ForGroup loops. For example instead of checking every few seconds every unit on the map for a specific item to do some actions, something I have seen people suggest on this Forum, you could detect when a unit acquires or drops the item and add or remove them from a global group which has the actions applied to it periodically. Such a change would save iterating potentially hundreds of units every few seconds for a tiny overhead on every item manipulation, an event which will only happen at most a few hundred times in most maps.

 

[pusha1999]

nono, most of my spells with firstofgroup where is sigle or double actions, qoz there is no more than 15-20 units and they are not lags. i use only forgroup or filter where need make multiactions in periodic timer or in big groups where is 50-100 units or more and where need multiactions.

 

[Dr Super Good]

nono, most of my spells with firstofgroup where is sigle or double actions, qoz there is no more than 15-20 units and they are not lags. i use only forgroup or filter where need make multiactions in periodic timer or in big groups where is 50-100 units or more and where need multiactions.

What do you mean by actions? Function calls? Jobs? Tasks?

 

[pusha1999]

What do you mean by actions? Function calls? Jobs? Tasks?

i mean function calls by periodic timer for example. where is getexpiredtimer. if there i need to do actions in groups, then i use forgroup, if group is not big and need only 1-2 actions with, then i use loops with firstofgroup.

 

[Xonok]

I'd very much appreciate if someone tested...

1. Array reads and writes.
VS
2. Sized array reads and writes.
VS
3. Hashtable reads and writes.

I'm fairly sure that the first is the fastest and the last is the slowest, but I'm interested in how much exactly their speeds differ.

 

[edo494]

there are no sized arrays, so they are the same. Hashtable vs array has been tested already

 

[Xonok]

By sized array I mean the vJASS construct that essentially uses ifs to combine several usual arrays. Link.

Could you link me the array vs hashtable one? I looked through the whole thread and didn't find it.

 

[edo494]

ah ok you mean sized above 8192, I thought maybe you mean below sorry for that

Apparently I was wrong, but there is second thread about performance I think in the lab, this one only has hash vs cache

 

[pusha1999]

By sized array I mean the vJASS construct that essentially uses ifs to combine several usual arrays. Link.

Could you link me the array vs hashtable one? I looked through the whole thread and didn't find it.

for me hash works slower than arrays and structs, but more stable and safety than structs and 100% avoid vars stacks. but i think dont need to look on benchs now when lot of ppls have multicores PCs with minimum 1600mhz .

 

[Xonok]

Warcraft III doesn't get much or any benefit from multicore CPUs. Heck, the graphics and logic are even in the same thread.

 

[Malhorne]

Ahah yeah!

 

[pusha1999]

Warcraft III doesn't get much or any benefit from multicore CPUs. Heck, the graphics and logic are even in the same thread.

if you dont get it, it was sarcasm

 

[pusha1999]

i did reread this topic, and mb im wrong...but, how its possible FirstOfGroup fastest method if for firstofgroup or forgroup need first enumunits in rect\range and etc..? enumunits its the loop too, function take every unit in area and check, there she also can do some actions in this filter...so, how its possible that 2 loops or 3 loops are faster than 1 loop?)

 

[Xonok]

It depends on the situation. If you need to do something to units in an area, use GroupEnumUnitsInRange/InRect/etc and do your actions in the filter.

If you already have a group and need to iterate on it, then FoG is faster because ForGroup seems to have a large per-unit overhead. Although FoG initially does slower(2-3 units), it quickly catches up for larger numbers of units and thus works out better when it really matters.

 

[Bribe]

FoG loops are only meant to be used when the group is done with the units it contains, as the group is empty afterward. The next thing to consider is that having a "null" filter requires the least work from the game engine as it doesn't need to open up a new thread per unit it evaluates.

99% of the time what I see is a group being used for quick "is that unit in range, and if so do actions" and then the group is obsolete. A single unit group can be used for each of these tasks, unless you need to nest a FoG loop within another one. Then you use a second permanent unit group for that second enumeration.

If you are part of the minority who will do things with that unit group and not just use it for those quick enums, then ForGroup or a unit array should be used for iterating. A unit array will be faster if more than 1 or 2 units need to be iterated over.

FirstOfGroup (and unit arrays) allows you to access local variables; ForGroup and the filterfunc passed to GroupEnum/ requires they all be globals.