[vJASS] Universal Unit Recycling System

[Robbepop]

Hiho,

in order to solve some critical virtual memory based fatal errors with my line tower wars I tried to search for several solutions and tested the map for hidden memory leaks.

Since I haven't found any fixable memory leak within my whole codes I came up with something I have read ages ago about units which says that every instanciated unit will leave a leak after it gets destroyed or even removed from the game. This leak is not preventable for us, mappers.

That's why I have started to think about a solution for this. I have to minimize the amount of unit creations - I had to code a unit recycling system. This would solve all my unit allocation problems ... perhaps. However, a problem was that I had to allocate tousand of units because there is no functionality for a unit type swap ... wrong. With chaos ability and a good unit database I was able to fix this problem, so every recyclable unit type in my map can be converted dynamically to any other recyclable unit type with the chaos ability, this worked fine.

However, tests provided me with some strange results. I have tested a minor dummy recycling system on a map where I spawn about 500 units per second and remove them again (with and without recycling and with and without unit type swap.)

The results where very weird ...
As you might think without recycling these massive amounts of units created a big memory leak after some time while the units which got recycled (basically hidden and unhidden etc.) created (nearly!!!) no memory leak.
However, with the chaos ability based unit type swap the memory leak was the same as without unit recycling!

I couldn't believe my own tests and started to implement the whole system in my map - Line Tower Wars Evolution ...

And here is the code (but keep in mind that I have not followed the awesome syntax and code writing rules because when I started the map I was a hardcore fanbody for C# syntax style. xDDD - I won't change it now as this system is not intended to be publicly available since it needs ugly unit databases which isn't very handy imo.)

library CreepRecycling requires GroupUtils

struct CreepRecycling extends array
    private static constant real MIN_LIFE = .405

    private static constant integer CREEP_AURA_DEFENSE = 'B000'
    private static constant integer CREEP_AURA_OFFENSE = 'B001'
    private static constant integer CREEP_AURA_SPEED   = 'B002'
    private static constant integer TOWER_AURA_SLOW    = 'B003'
    private static constant integer TOWER_AURA_CURSED  = 'B00A'
    private static constant integer BUILDER_AURA_HEADWIND = 'B008'

    private static constant integer DUMMY_CREEP_ID = 'n007'

    private static constant real BOUNTY_PERCENTAGE = 0.1

    private static group recycled
    private static integer countRecycled
    private static trigger detectDamageTrigger

    private static unit currentCreep

    private static key UNIT_TYPE_ID_KEY

    private static method RemoveEnum takes nothing returns nothing
        call RemoveUnit(GetEnumUnit())
    endmethod

    public static method Clear takes nothing returns nothing
        set thistype.countRecycled = 0
        call ForGroup(thistype.recycled, function thistype.RemoveEnum)
        call GroupClear(thistype.recycled)
        call DestroyGroup(thistype.recycled)
        call DisableTrigger(thistype.detectDamageTrigger)
        call TriggerClearActions(thistype.detectDamageTrigger)
        call TriggerClearConditions(thistype.detectDamageTrigger)
    endmethod

    private static method CreateBountyTexttag takes unit whichUnit, integer bounty, player killer returns nothing
        // Var Init
        local texttag tt = CreateTextTag()
        local string text = "+" + I2S(bounty)

        // Create Visuals
        call SetTextTagText(tt, text, 0.024)
        call SetTextTagPos(tt, GetUnitX(whichUnit) - 16.0, GetUnitY(whichUnit), 0.0)
        call SetTextTagColor(tt, 255, 220, 0, 255)
        call SetTextTagVelocity(tt, 0.0, 0.03)
        call SetTextTagVisibility(tt, GetLocalPlayer() == killer)
        call SetTextTagFadepoint(tt, 2.0)
        call SetTextTagLifespan(tt, 3.0)
        call SetTextTagPermanent(tt, false)

        // Clear
        set text = null
        set tt = null
    endmethod

    private static method RemoveBuffs takes unit creep returns nothing
        call UnitRemoveAbility(creep, thistype.CREEP_AURA_DEFENSE)
        call UnitRemoveAbility(creep, thistype.CREEP_AURA_OFFENSE)
        call UnitRemoveAbility(creep, thistype.CREEP_AURA_SPEED)
        call UnitRemoveAbility(creep, thistype.TOWER_AURA_SLOW)
        call UnitRemoveAbility(creep, thistype.TOWER_AURA_CURSED)
        call UnitRemoveAbility(creep, thistype.BUILDER_AURA_HEADWIND)
    endmethod

    private static method GetTypeSwapAbility takes integer creepId returns integer
        //call BJDebugMsg("GetTypeSwapAbility - name = " + GetAbilityName(LoadInteger(GameGlobals.Hashtable, thistype.UNIT_TYPE_ID_KEY, creepId)))
        return LoadInteger(GameGlobals.Hashtable, thistype.UNIT_TYPE_ID_KEY, creepId)
    endmethod

    private static method RemoveTypeSwapAbility takes unit creep returns nothing
        //call BJDebugMsg("RemoveTypeSwapAbility - name = " + GetUnitName(creep))
        call UnitRemoveAbility(creep, thistype.GetTypeSwapAbility(GetUnitTypeId(creep)))
    endmethod

    private static method SetUnitTypeId takes unit creep, integer creepId returns nothing
        call UnitAddAbility(creep, thistype.GetTypeSwapAbility(creepId))
        //call BJDebugMsg("SetUnitTypeId(" + GetUnitName(creep) + ", " + I2S(creepId))
    endmethod

    public static method Get takes player owner, integer creepId, real posX, real posY, real facingAngle returns unit
        if thistype.countRecycled <= 0 then
            //set thistype.currentCreep = CreateUnit(owner, creepId, posX, posY, facingAngle)
            set thistype.currentCreep = CreateUnit(owner, Creep.DummyCreep.Id, posX, posY, facingAngle)
            call thistype.AddCreep(thistype.currentCreep)
            call thistype.SetUnitTypeId(thistype.currentCreep, creepId)
            //call BJDebugMsg("CreepRecycling - Get - created new")
        else
            set thistype.countRecycled = thistype.countRecycled - 1
            set thistype.currentCreep = FirstOfGroup(thistype.recycled)
            call GroupRemoveUnit(thistype.recycled, thistype.currentCreep)
            call SetUnitOwner(thistype.currentCreep, owner, true)
            call thistype.SetUnitTypeId(thistype.currentCreep, creepId)
            call SetUnitX(thistype.currentCreep, posX)
            call SetUnitY(thistype.currentCreep, posY)
            call SetUnitFacing(thistype.currentCreep, facingAngle)
            call PauseUnit(thistype.currentCreep, false)
            call ShowUnit(thistype.currentCreep, true)
            call SetUnitInvulnerable(thistype.currentCreep, false)
            //call BJDebugMsg("CreepRecycling - Get - recycled" + " - count = " + I2S(thistype.countRecycled))
        endif
        return thistype.currentCreep
    endmethod

    public static method Recycle takes unit creep returns nothing
        call SetWidgetLife(creep, 9999999.)
        call SetUnitInvulnerable(creep, true)
        call thistype.RemoveBuffs(creep)
        call thistype.RemoveTypeSwapAbility(creep)
        call PauseUnit(creep, true)
        call ShowUnit(creep, false)
        call SetUnitOwner(creep, Player(PLAYER_NEUTRAL_AGGRESSIVE), false)
        call GroupAddUnit(thistype.recycled, creep)
        call SetUnitX(creep, 0.)
        call SetUnitY(creep, 0.)
        set thistype.countRecycled = thistype.countRecycled + 1
        //call BJDebugMsg("CreepRecycling - Recycle - " + GetUnitName(creep) + " - count = " + I2S(thistype.countRecycled))
    endmethod

    private static method AddCreep takes unit creep returns nothing
        call TriggerRegisterUnitEvent(thistype.detectDamageTrigger, creep, EVENT_UNIT_DAMAGED)
        //call BJDebugMsg("CreepRecycling - AddCreep - " + GetUnitName(creep))
    endmethod

    private static method onDamageCondition takes nothing returns boolean
        // Var Init
        local unit killed
        local real damage = GetEventDamage()
        local Gamer owner
        local Creep creep

        // Initial Check
        if damage > 0. then
            return false
        endif

        // Check & Execute
        set killed = GetTriggerUnit()
        if GetWidgetLife(killed) - damage <= thistype.MIN_LIFE then
            set owner = Gamer.GetOwner(GetEventDamageSource())
            set creep = Creep.GetInstance(killed)
            call thistype.CreateBountyTexttag(killed, creep.Bounty, owner.Controller)
            call thistype.Recycle(killed)
            call owner.AddGold(creep.Bounty)
        endif

        // Clear
        set killed = null

        // Default Return
        return false

        //return GetEventDamage() > 0. and GetWidgetLife(GetTriggerUnit()) - GetEventDamage() <= thistype.MIN_LIFE
    endmethod

    //private static method onDamageAction takes nothing returns nothing
        //call thistype.Recycle(GetTriggerUnit())
        //call thistype.CreateBountyTexttag(GetTriggerUnit(), GetOwningPlayer(GetEventDamageSource()))
    //endmethod

    private static method InitDummyAbility takes integer creepId, integer abilityId returns nothing
        call SaveInteger(GameGlobals.Hashtable, thistype.UNIT_TYPE_ID_KEY, creepId, abilityId)
    endmethod

    private static method InitDummyAbilities takes nothing returns nothing
        // Tier 1
        call thistype.InitDummyAbility('n00E', 'S000') // Sheep
        call thistype.InitDummyAbility('n00C', 'S001') // Wolf
        call thistype.InitDummyAbility('u003', 'S002') // Skeleton
        call thistype.InitDummyAbility('u000', 'S003') // Acolyte
        call thistype.InitDummyAbility('n005', 'S004') // Corruped Treant
        call thistype.InitDummyAbility('h00U', 'S005') // Swordsman
        call thistype.InitDummyAbility('n000', 'S006') // Chaos Grunt
        call thistype.InitDummyAbility('n00A', 'S007') // Vile Temptress
        call thistype.InitDummyAbility('u001', 'S008') // Shade
        call thistype.InitDummyAbility('u00A', 'S009') // Shade (-na)
        call thistype.InitDummyAbility('n00F', 'S00A') // Mud Golem
        call thistype.InitDummyAbility('h01B', 'S00B') // Demolition Machine
        call thistype.InitDummyAbility('h00W', 'S00C') // Draenei (-ns)
        call thistype.InitDummyAbility('n008', 'S00D') // Rot Golem

        // Tier 2
        call thistype.InitDummyAbility('h01H', 'S00E') // Knight
        call thistype.InitDummyAbility('o000', 'S00F') // Tauren
        call thistype.InitDummyAbility('n002', 'S00G') // Troll
        call thistype.InitDummyAbility('n003', 'S00H') // Harpy
        call thistype.InitDummyAbility('n001', 'S00I') // Gnoll (-na)
        call thistype.InitDummyAbility('h01I', 'S00J') // Siege Engine
        call thistype.InitDummyAbility('h00X', 'S00K') // Siege Engine (-ns)
        call thistype.InitDummyAbility('n006', 'S00L') // Faceless Void
        call thistype.InitDummyAbility('n009', 'S00M') // Magnataur
        call thistype.InitDummyAbility('n004', 'S00N') // Dragonspawn
        call thistype.InitDummyAbility('u002', 'S00O') // Banshee
        call thistype.InitDummyAbility('u00B', 'S00P') // Banshee (-na)
        call thistype.InitDummyAbility('n00B', 'S00Q') // Wendigo
        call thistype.InitDummyAbility('n00D', 'S00R') // Hellbeast
        call thistype.InitDummyAbility('h01M', 'S00S') // Phoenix
        call thistype.InitDummyAbility('h00Y', 'S00T') // Felhound (-na + -ns)

        // Tier 3
        call thistype.InitDummyAbility('u008', 'S00U') // Brutal Ghoul
        call thistype.InitDummyAbility('u00C', 'S00V') // Armored Ghoul (-ns)
        call thistype.InitDummyAbility('n00H', 'S00W') // Satyr
        call thistype.InitDummyAbility('u006', 'S00X') // Cryptfiend
        call thistype.InitDummyAbility('o002', 'S00Y') // Orcish Shaman
        call thistype.InitDummyAbility('o001', 'S00Z') // Void Walker
        call thistype.InitDummyAbility('h02E', 'S010') // Spellbreaker
        call thistype.InitDummyAbility('u007', 'S011') // Necromancer
        call thistype.InitDummyAbility('h021', 'S012') // Gryphon Rider
        call thistype.InitDummyAbility('h00Z', 'S013') // Owlbear (-na)
        call thistype.InitDummyAbility('u004', 'S014') // Abomination
        call thistype.InitDummyAbility('e000', 'S015') // Mountain Giant
        call thistype.InitDummyAbility('e002', 'S016') // Mountain Giant (-ns)
        call thistype.InitDummyAbility('u009', 'S017') // Hell Skeleton
        call thistype.InitDummyAbility('u005', 'S018') // Frost-Wyrm
        call thistype.InitDummyAbility('u00D', 'S019') // Mathog (-na)
    endmethod

    public static method Init takes nothing returns nothing
        call thistype.InitDummyAbilities()

        set thistype.recycled = NewGroup()
        set thistype.countRecycled = 0
        set thistype.detectDamageTrigger = CreateTrigger()

        call TriggerAddCondition(thistype.detectDamageTrigger, Condition(function thistype.onDamageCondition))
        //call TriggerAddAction(thistype.detectDamageTrigger, function thistype.onDamageAction)
    endmethod
endstruct

endlibrary

Maybe I have some critical memory leaks within my system which cause these weird memory leak results in my many many tests. I am looking forward for bug reports or memory leaks within this system or performance improvement ideas of you guys. Another thing I haven't mentioned is that this system is awefully slow with an enabled onDamageCondition trigger with massive amounts of units at the same time - it is so slow that I sometimes had about 0.5 fps during the tests ... without it everything performed even better than without the recycling system as I have not to create units all the time which is a very performance critical procedure.

Robbepop

 

[Malhorne]

If I remember well the unit leak doesn't exist anymore.
CreateUnit is a costly function that's why some people created unit recycler but not for the lag.

 

[Nestharus]

With chaos ability and a good unit database I was able to fix this problem, so every recyclable unit type in my map can be converted dynamically to any other recyclable unit type with the chaos ability, this worked fine.

Yes, chaos has the exact same leak. This was discovered a long time ago.

If I remember well the unit leak doesn't exist anymore.
CreateUnit is a costly function that's why some people created unit recycler but not for the lag.

Ehm, the last patch introduced the leak, or the patch before it, so it's still there.


The only way to resolve the leak is to use dummy units that have no model. You attach an effect to them and you're done. You would also need to use a Bonus system to set the unit's stats. You can also use SharpCraft .

 

[Malhorne]

The unit leak is still there ?
Are you serious ?
Did you test it ?

 

[DysfunctionaI]

Another thing I haven't mentioned is that this system is awefully slow with an enabled onDamageCondition trigger with massive amounts of units at the same time - it is so slow that I sometimes had about 0.5 fps during the tests ...

Wait, so is this really worth using?

The unit leak is still there ?
Are you serious ?
Did you test it ?

I think people that play LTW are testing that every time they play.

 

[deathismyfriend]

The unit leak is still there ?
Are you serious ?
Did you test it ?

Yes the unit leak is still there. It is a small amount of memory though. About 4kb per unit. You can easily reproduce this leak just create and remove units and watch the memory rise in the task manager.

 

[Malhorne]

Ok I'll test it.

 

[Nestharus]

The unit leak is still there ?
Are you serious ?
Did you test it ?

they haven't released a new patch since they introduced the leak, so the real question is, why wouldn't it be there?

 

[Cokemonkey11]

About 4kb per unit.

So, 0.5KB? I'll Assume you meant 4KB

 

[deathismyfriend]

So, 0.5KB? I'll Assume you meant 4KB

Yes i meant 4KB.

 

[Robbepop]

Hiho,

thanks for the answers so far! =)

4KB per unit is about the value which resulted in my own tests and it is in my opinion quite a lot per single unit instance. can't understand how one can code something so badly ...

Is the dummyunit without a model + effect attached a real solution for games like line tower wars? I mean, is it possible to make the attached special effects animate with the unit or would it look like a frozen or standing unit all the time?

It is really important to me that this serious memory leak is getting fixed because it makes LTW an nearly unplayable experience which is quite sad ...

Wait, so is this really worth using?

The code which I have posted above is definitely not worth using because of the massive laggs. However, I am looking forward to be able to improve its performance. I think the worst part of it is that I always allocte several local variables for every damage (damage is dealt very very very often in LTW games especially with splash towers and massive amounts of units!). So one solution could be to split the function again up in two parts - a condition and an action where the condition acts like a normal filter so that only the action allocates local stack variables. But this is just a theoretical solution, nobody knows how the old wc3 engine works behind the scene ...

Robbepop

 

[DysfunctionaI]

This is a wild idea, no clue if it would work but, what if every unit had the ability Resurrection?

You would create the units normally as needed, and any unit that dies would be hidden and sent to a pool. From there, whenever someone wants to send that type of unit, it's revived, and pulled from the pool.

So at some point during the game, you would basically stop calling CreateUnit() and just be pulling units from a massive pool.

I'd honestly love to see this fixed because I enjoy playing LTW

So one solution could be to split the function again up in two parts - a condition and an action where the condition acts like a normal filter so that only the action allocates local stack variables.

This is true, but just call the "action" function from your condition, instead of using TriggerAddAction().

 

[Garfield1337]

Have you tried the Bear Form transformation? (If you aren't familiar with the method, you basically create a Bear Form ability with initial unit type different from the unit to which the ability is given, which makes the target unit transform instantly.) Maybe it doesn't leak.
Although it has a huge drawback of having 2 parameters for transformation: initial unit and new unit (iirc, chaos only has new unit).

 

[DysfunctionaI]

Edit: Meh, this is the same exact thing as Reincarnation (that's what I meant to say instead of Resurrection) except Reincarnation works much better for this method. This method requires a copy of every unit that will be using this, plus additional timers.

Got another idea, similar to my last one, and I tested this one.

Every spawned unit has the "Pheonix Morphing (Egg Related)" ability! This means, whenever they die, they just transform into something else, and shortly after return to their normal form.

Example:

Sheep is sent (create new sheep)

Sheep dies -> Pheonix Morphing is casted -> play Sheep death animation -> hide sheep

Another sheep is sent (use the last sheep that died)


So basically every sheep that "dies" is hidden and added to a pool. They will transform into their new form which is just a copy of the sheep, that will make it transform back into its normal form. It takes like 10 seconds to transform back into the normal form (idk how to change this, but that's ok, each unit won't be declared 'ready' in the pool until 10 seconds has expired then).

 

[PurgeandFire]

Unit recycling is always a neat topic. Good luck.

they haven't released a new patch since they introduced the leak, so the real question is, why wouldn't it be there?

The leak has existed for a while:
http://www.thehelper.net/threads/be-warned-theres-still-a-memory-leak-in-frozen-throne.132644/
(that isn't the earliest time, but regardless they had a couple of patches since then). iirc, it was a lot worse back then, but they hotfixed it without documentation (AFAIK). I haven't tested it recently on the current patch though (I tested it a while back but I always forget the results).

 

[Nestharus]

Well, it was when they fixed the return bug/introduced hashtables that the leak was introduced. Before that, 0 problems. When that patch came out, hell was unleashed. It was present in the latest patch as well.

 

[Robbepop]

Hiho,

again thank you for all your comments on this topic.

I am currently thinking about the ressurection idea and what problem it could solve, however, I can't seem to find any problem it could solve as you still have to use chaos in order to swap the unit type and you even still have to check if that unit's life is lower than 0.405 to "recycle -> hide" it properly.

The bear transform is truely a cool idea and I didn't know about it to be honest. I don't know how much work it would be to try this with the bear form exchange but it has to behave very similar to the chaos ability. So it must be possible to change the unit type with it and it must be possible to do it afterwards again with another unit type. But beforehand we should test if bearform also leaks like chaos. A major problem could be that 10 seconds duration before changing the unit type again as the units may be recycled very quickly in LTW.

Is anybody here free to test if bearform leaks memory?^^

Robbepop

 

[DysfunctionaI]

I am currently thinking about the ressurection idea and what problem it could solve, however, I can't seem to find any problem it could solve as you still have to use chaos in order to swap the unit type

The problem it tries to solve is limiting the number of times you have to create new units. Units will be created, and recycled whenever they can.

and you even still have to check if that unit's life is lower than 0.405 to "recycle -> hide" it properly.

You can give the units the Defend ability, and catch their death with "unit was issued an order with no target, order = undefend"

I've made a little test map to show how it could work. There is still some stuff to work out, like the timing of a death animation, and whether or not you can hide/pause/move a unit while it's going through the Reincarnation delay time. Because of that, I just temporary moved them off to the right while they're finishing Reincarnation, but I'm sure something can be figured out to hide them right on the spot while having their death animation play. Also, I wrote it in GUI cuz it's faster, and this is just proof of concept.

 

[Robbepop]

Hiho,

Thank you for your reponse.

The problem it tries to solve is limiting the number of times you have to create new units. Units will be created, and recycled whenever they can.

that's exactly what I have meant. I can absolutely see no point in using this approach as the current unit recycling system already prevents the creation of new units which is the whole idea behind the system. You can look into the get method of the system. Then you can see that I am checking if there is any recycled unit left and only then will the system create a new unit instance. So you have a total amount of units on the map recycled or activ equal to the highest number of units at the same time on the game until that time.

In a normal LTW there can be about 300-500 creeps at the same time trying to steal lives of the oponents and so a normal amount of units created in total is about that value per game.

To be precise about the recycling of dying units. At the moment the system checks whenever a unit is about to die and instantly prevents its death, hides and deactivate it which is the actual recycling process. Once the system is asked to output a new unit and it still has some recycled units backed-up it just has to reactivate and unhide the unit as well as changing its unit type to that of the required one. ;-)

As far as I know checking for orders is as performance critical as checking for incoming damage but for that I am not entirely sure.

Robbepop

 

[DysfunctionaI]

As far as I know checking for orders is as performance critical as checking for incoming damage but for that I am not entirely sure.

That was basically what I was trying to achieve; a way to detect a units death without damage detection.

As for the event that catches the 'undefend' unit order, I think that's how Nestharus's Unit Indexer system works, he may have input on it.

 

[Nestharus]

undefend will detect deaths

but so will the death event : P

UnitEvent used to have a death event using undefend

 

[Xonok]

Chaos is a pretty buggy ability in general, as it messes up temporary stat boosts(buffs) with permanent ones.

I suggest trying if Hero Passive Transformation works (google it if you don't know what it is)

 

[Robbepop]

Hiho,

thank you for your comments.

I will definitely try out the bear form transformation as soon as possible and will inform you if I could at least fix the terrible memory leak with it. =)

Robbepop

 

[DysfunctionaI]

You're going to pee yourself when you see this.

DYS'S GHETTO UNIT RECYCLER!

So I'm playing around with Pheonix (Egg Related) and I found something extremely useful. If you set the Alternate Form to nothing, then when a unit goes to change form, it just gets 1/1 Life, and the expiration timer just loops over and over again.

Now, if you set "Morphing Flag = Permanent" then the unit gets 1/1 life and the timer only expires once, then the ability is removed from the unit.

What does this mean?! To recycle the unit, just restore the unit's original max life (with the max life bonus glitch*) and set the unit's current life to 100%, then re-add the Pheonix (Egg Related) ability so you can do it all over again!


Some notes:

1.) The duration for the timer is set with "Duration - Hero" even for normal units. So I just set that to 0.01 and it's almost instantaneous.

2.) There is always a 0.5 second delay after a unit casts Pheonix (Egg Related). This is actually perfect, because it allows you to play the death animation before the unit is recycled. It looks like this:

Unit dies -> Begins casting ability Pheonix (Egg Related)
0.5 second delay, invulnerability
Unit gets 1/1 life, vulnerable, Stops casting ability Pheonix (Egg Related)

So that's completely perfect for catching the start and end of the dying sequence.


Final note: Right now, when a unit dies, you can see the HP bar which isn't normal. That can be fixed by adding locust when the unit dies, and removing it when spawning the unit. In my test map, I have the lines to add/remove locust, but I disabled and added a comment explaining why.

Map attached~

* The max life glitch is used to permanently set a unit's max life. You add the item abiliy "Item - Life Bonus", then increase the level of it, then remove it. The max life will stay to whatever you set it to, and isn't removed upon removing the ability. It accepts negative values to decrease max life as well.

 

[Robbepop]

Hiho,

I have tested your dys's ghetto unit recycler and it totally owns.
Its pros are inarguable:
- playing the death animation (healthbar waynes)
- giving bounty to the killing player if settings are correctly (not true, I just broke the system when this was correct. >.<)
- the unit death events are extremely cool and I hope that they are not as performance impacting as the damage event.

The downsides are:
- I have found a bug where I press ESC in your testmap and sometimes I spawn dead sheeps which instantly dies after spawning.

Another downside of this system is that it is not yet known if it leaks memory just like the chaos ability does. So this has to be tested before I want to implement it in LTW.

Robbepop

 

[Dr Super Good]

Crashes generally are not caused by memory leaks unless you deplete some sort of critical finite structure.

More likely is some strange interaction doing something WC3 was never intended to do. An example would be a Damage Detection System destroying a trigger as the wrong time, or you removing a unit when something is still interacting with it (and does not support unit removal at that point) or something exiting the map bounds (specifically art outside or near the map bounds will cause a crash).

 

[Robbepop]

Hiho,

the crashes especially caused by LTW games in generally (not only my own version with more or less advanced codes compared to other LTW maps) is very well known and tests showed at least to me that the permanent memory leaks which got introduced since some versions are a serious problem to maps which behave kind of different from the normal warcraft III playstyle where you for example send massive amounts of units and have huge armies of towers defending a base over a time of sometimes half an hour.

The fatal error crash message is always the same and states that warcraft III ran out of virtual memory which is caused by the permanent memory consumption and I am also thinking that there is much external fragmentation because of this process so we are effectively left with even less memory than we can test.

I am not destroying triggers in general, nor do I cause other sorts of generally known memory leak intense things in this map as far as I know. The test maps I have made for example just create 500 units per second and remove them a second later. With this testmap one can easily notice that the memory consumption is heavily influenced by the amount of units created in total. The initial memory consumption of the testmap was about 130 kB and after about 15 to 20 minutes it was over 300 kB without anything besides the said functionality and without anything that could cause leaks besides it (e.g. of course I used coordinates instead of locations to create units randomly).

The fact that nestharus stated that this bug was introduced some time ago states for me that blizzard is the only one who could really fix this annoying thing for us which makes so many games unplayable for many people.

LTW generally crashes first for those players who got the least virtual memory space or RAM and crashes the last for those who have more of it. When a player has already played LTW a game before he will eventually drop out earlier as the memory isn't even cleaned up correctly after the game - what a shame ...

Robbepop

 

[Dr Super Good]

The fatal error crash message is always the same and states that warcraft III ran out of virtual memory which is caused by the permanent memory consumption and I am also thinking that there is much external fragmentation because of this process so we are effectively left with even less memory than we can test.

Do not jump to conclusions about such errors, especially if you have little understanding how computers work. You should read up about virtual memory systems and how they deal with memory fragmentation at the OS level.

That error actually means that WC3 has run out of virtual address space. Since WC3 is a x86 compiled process this means that its last page allocation request could not be fitted inside the 4 GB (or less if OS limits) virtual address space the process can have. This error is very different from an Out of Memory error which occurs when your OS no longer has enough page space to back new virtual memory pages. Since modern computers have in excess of 4 GB of memory + about double that in page file it is virtually impossible to get WC3 to throw an Out of Memory error because it just cannot allocate enough memory in its virtual address space.

Even very leaky WC3 maps will become unplayable long before the process exceeds 1GB of virtual address space allocation and will likely crash due to finite resource depletion (which usually result in an access violation as it tries to modify locked pages).

So what does this mean? Something in your map is either trying to allocate a huge segment of memory that cannot sit un-fragmented inside the process virtual address space or that you have an infinite loop occurring which continuously allocates pages until the process runs out of free virtual memory address space and crashes.

With this testmap one can easily notice that the memory consumption is heavily influenced by the amount of units created in total.

You can easily get through >6k units a session and not have any problems with crashing or degraded performance. Yes it does leak memory (although some people argue a garbage cycle eventually cleans it up) but the amount leaked is not really a concern on modern computer systems.

LTW generally crashes first for those players who got the least virtual memory space or RAM and crashes the last for those who have more of it.

That is because if the person lacks enough memory it has to use the page file (very slow) to back up the 4 GB of pages that WC3 is demanding. This further points to my infinite loop theory since it is not trying to allocate something big, but instead makes an infinite amount of small allocations instantly (game wise) and does something with them (so the OS has to page them). The paging slows down the loop period so the person takes a while to crash compared to people like myself with 6GB of memory (most free/immediately free-able) where the loop can literally run at the speed of memory I/O (in my case that's around 4 GB/second so it will take under a second to crash).

When a player has already played LTW a game before he will eventually drop out earlier as the memory isn't even cleaned up correctly after the game - what a shame ...

So he crashes at a different time or same time but with more immediate delay?

In any case log the commit size of wc3 as you play your map. If it starts to become stupid (1GB, >2GB etc) then you have a serious leak. If not then you do not. Commit size represents the total amount of allocated virtual memory pages for a process and so reflects how much of the process virtual memory address space has been used.

 

[Nestharus]

In any case log the commit size of wc3 as you play your map. If it starts to become stupid (1GB, >2GB etc) then you have a serious leak. If not then you do not. Commit size represents the total amount of allocated virtual memory pages for a process and so reflects how much of the process virtual memory address space has been used.

I find that maps that have a lot of unit creation/destruction will crash wc3 after a few games (maybe even 1). On my laptop, I can only do 1 run of LTW before I have to restart. This is true of any game with lots of unit creation/destruction. On my desktop, I have 16 gigs of RAM . I can go all day w/o having it crash.

When a map uses unit recycling (drops creation/destruction), I can go all day with no crashes on my laptop.

 

[DysfunctionaI]

- playing the death animation (healthbar waynes)

It can be removed by adding locust, however, the only way to completely remove locust is to give the unit Bear Form and have it morph into a copy of itself.

While this is do-able, it's resource costly, because you'll need a copy of Bear Form for every unit you have, and it's more for the triggers to do when a unit 'dies'. Also, still not sure if that leaks.

- giving bounty to the killing player if settings are correctly (not true, I just broke the system when this was correct. >.<)

Damn, that's a huge problem. I couldn't find a way to get the killing player, but for LTW, you could write a quick function with GetUnitX() and compare it to each lane's X and then return the player that's in that lane.

- the unit death events are extremely cool and I hope that they are not as performance impacting as the damage event.

I hammered this test map with a 0.01 spawn timer and set WC3 speed to 10x, recycling a little over 20k units (with only a max of ~120 units created). It seemed fine.

- I have found a bug where I press ESC in your testmap and sometimes I spawn dead sheeps which instantly dies after spawning.

My bad, I was adding the sheeps to the pool when they STARTED dying, and I should have added them when they FINISHED dying. So what you saw, was sheeps that started to die, and were pulled from the pool before they finished dying. I fixed it and re-attached with a multiboard that counts some stuff. Also, I had to add one more ability to the system "Ghost (Visible)" to remove the dying unit's collision. It's removed when the unit is respawned.

Another downside of this system is that it is not yet known if it leaks memory just like the chaos ability does. So this has to be tested before I want to implement it in LTW.

I responded earlier about recycling over 20k units. My memory went up, but it wasn't significant at all. It seems like the usage goes up whether I'm recycling units or not. But there will definitely be a spike where units are in need to be created (the pool is too small for the demand of players).

 

[Malhorne]

function RemoveLocust takes unit u returns nothing
    local boolean b = not IsUnitHidden(u)
    call ShowUnit(u, false)
    call UnitRemoveAbility(u, 'Aloc')
    call ShowUnit(u, b)
endfunction

 

[Robbepop]

Hiho,

thanks again for all of your comments!

@Malhorne: great workaround to get this working. =)

Originally Posted by Robbepop View Post
- giving bounty to the killing player if settings are correctly (not true, I just broke the system when this was correct. >.<)

Damn, that's a huge problem. I couldn't find a way to get the killing player, but for LTW, you could write a quick function with GetUnitX() and compare it to each lane's X and then return the player that's in that lane.

That's not a major problem. With the current system which triggers on damaging event I also have to manually add bounty for the killing player - however it is a huge performance overhead I think. The creep database with all important information already exists.

===

I have made a testmap in order to "benchmark" the bearform ability based unit type swap. Generally it was much slower than just remove and recreate units or to recycle them with the chaos ability. The memory usage was also more or less the same as the chaos ability based unit type swap - at least for my test map. I ran the map in local area network so that I can follow the memory usage while recycling 150 units every 0.25 seconds (450/sec). This however, led to about 20 fps ingame while I normally have constant 60 fps.
So again, it has a massive performance impact to swap unit types via bearform - even more than with chaos ability.

At least in object editor the phoenix ability seemed to be based of the bearform ability - however I can be wrong here. This was the main reason why I haven't "benchmarked" it as well.

Robbepop

 

[Malhorne]

My RemoveLocust function works ^^'

 

[Dr Super Good]

On my desktop, I have 16 gigs of RAM . I can go all day w/o having it crash.

Someone who is as knowledgable about computers as you should know that WC3 can only allocate 4 GB of memory as it is only x86 built (not x86-64) so has a maximum virtual memory address size of 4 GB. Seeing how all modern laptops have > 4 GB memory and at least twice that in page file I do not see how you having 16 GB of memory will make the slightest difference compared to a laptop.

Sure if you use an old laptop that might be the case but you could always up the page file. Equally well if you are using a 32 bit os (seriously in 2014?!) you will be limited to 4 GB of memory total so will probably crash before a 64 bit OS.

 

[DysfunctionaI]

At least in object editor the phoenix ability seemed to be based of the bearform ability - however I can be wrong here. This was the main reason why I haven't "benchmarked" it as well.

That's true, they're both morphing abilities, but that's where the trick comes in. Since the "Alternate Form" is never set, when the unit goes to morph, it just wacks out and gives the unit 1/1 HP, and it never morphs at all (then add max hp, reset current hp, etc).

My RemoveLocust function works ^^'

I haven't tested it myself, but try using that function, and then targeting the unit. People say that they can't be targeted with spells (idk about attack and move and stuff). Even if it was just a targeting problem, idk how major that is for LTW cuz I think there's only 1 spell that targets a creep.

Also, would have to test if you can add locust again, after it's been removed. Then consequently removed again.

 

[Robbepop]

Hiho,

I am again thankful for all comments so far.

So I can conclude until now that we have some advancements in the following points:
- We now know that chaos as well as bearform unit type swap leaks an equal amount of memory. However, Nestharus stated that one will not run into a crash as long as you use unit recycling without one of these instead standart unit recreation. The main problem - the memory leak itself - however, isn't fixed at this point.
- We also have got a night way to simulate a creep's death with the phoenix morph ability in addition to the locust removal work around.
- And in the end Dr. Super Good pointed out that nothing will help unless blizzard decides to finally fix this "newly" introduced bug as the bug is definitely not fixable by mapper like we are.
- While chaos is better decision performance wise, bearform comes with less built in bugs and thus one has to cleanly decide what mechanism to take in order to swap the unit types.

I think this is a nice conclusion to far.
Correct me please if something is missing or wrong.

In my opinion conclusions are important for a longer discussion and this discussion is already long enough. =)

Thanks in the name of all LTW players out there who also want this map (and many more) to be playable again!

Robbepop

 

[Nestharus]

- We now know that chaos as well as bearform unit type swap leaks an equal amount of memory. However, Nestharus stated that one will not run into a crash as long as you use unit recycling with one of these instead of standart unit recreation. The main problem - the memory leak itself - however, isn't fixed at this point.

No, unit recycling without changing types will not crash*****

Create a pool of units for each type. Whenever you create a unit, first check the pool. If the pool is empty, then create the unit. Whenever a unit dies, resurrect it after so and so amount of time and throw it back into the pool .

You might need to check and see if resurrect leaks.


Alternatively, you could just use dummies, effects, abilities, and bonuses ;O, but the name of the unit will be all messed up

 

[Robbepop]

Hiho,

hm, ... okay I got that and fixed it in the conclusion.

It would be awesome if I could just implement such a system with static unit types.
However, in LTW there are about 40 different unit types which a player can send and every unit type can be theoretically about 2400 times on the map at the same time which requires to store about 40 * 2400 = 96000 units at the same time as recycled versions in worst case. xD

And I really don't want to test this one ...

Compare 96k units to the 2.4k units it would take in worst case with unit type change.

Robbepop

 

[Malhorne]

In my function they can zzz
The trick is to hide before ^^'

 

[Dr Super Good]

can be theoretically about 2400 times on the map at the same time which requires to store about 40 * 2400 = 96000 units at the same time as recycled versions in worst case. xD

Not our fault your map design is flawed. WC3 is not meant to have that many units at once so it is obvious you will hit problems.

I advise making the units tougher or scale better so fewer are needed. Instead of having 2400 units have 480 units. This number is much more reasonable and more what WC3 was designed for. SWAT Aftermath only gets through about 6k-7k units a session and that is overly long sessions.

Now for something more helpful. Use a unit cache instead of a full recycling system. Maps can easily support going through >6k units and still not crash. As long as most units that are sent are recycled you should be fine. If you run out of storage space (I would advise less than 8000 units on the map as that causes problems of its own) then remove some of the least used (victim algorithm?) and replace them with the new units. You might for some reason send 2400 of the same unit but most of the time you will be working with a far smaller set (as they are being killed constantly) so you may find only 200-400 are ever made.

 

[Nestharus]

I advise making the units tougher or scale better so fewer are needed. Instead of having 2400 units have 480 units. This number is much more reasonable and more what WC3 was designed for. SWAT Aftermath only gets through about 6k-7k units a session and that is overly long sessions.

ehm, any tower wars map is going to have those counts. Clearly, you've never played tower wars. I'm not even just talking ltw. wmw counts can get that high too.

 

[Xonok]

Someone has to invent a map where creeps can be upgraded mid-lane...
That would make it much easier to reduce unit counts.

 

[DysfunctionaI]

You could remove recycled units from the map that aren't going to be used anymore. Like, if no one sends a sheep for the last 3 minutes then start removing them.

 

[Dr Super Good]

Clearly, you've never played tower wars.

I have but instead of crashes I was getting units with armor type "unknown" which took 0 damage from every source. No I looked into the constants and no armor type was modified to that, this was units being created without any armor type.

Instead of sending lots of units, make them more expensive so you send less. To stop people spamming weaker units you bar them that option.

 

[Robbepop]

Hiho,

@Dr. Super Good: that's an idea which I also had in my mind - this would be the easiest possible solution to decrease to amounts of crashes (not fix them at all!). However, the negative side effects are that players would be confused and finally think that they are actually not playing "LTW" when they notice that they aren't spawning three units at any time but only one - players in B.Net are kind of that type after my years long experience. Many people wouldn't like to see this change happen and would play this version at all - this is a major problem and that's why we are trying to think about other solutions.

Robbepop

 

[Dr Super Good]

B.Net are kind of that type after my years long experience.

It still has players? Last time I looked it was reaching sub 10k (far from the 100k odd when I started).

Frankly most battlenet players (like most users) have little idea what they want. Look at the Diablo III forums on BattleNet to see the type of rubbish modern gamers post. To ease the resistance you could combine methods. Instead of spawning 3 units, you spawn 2 units which are then recycled. Fewer maximum units improves efficiency of unit cacheing while still 2/3 of the unit count remain. On the plus side it might improve performance so players have overall better experience.

A unit cache sounds like the best approach at the moment. As the game progresses you can remove basic unit types from it since players will not use them and replace them with units they are using. A limit of 2000 units should work with an appropriate unit victim algorithm (like page victim algorithms in virtual memory OSs) so that uncommonly used units are removed. As long as the cache works >90% of the time for spawns you will have reduced number of unit creations by at least a factor of 10. This means that your 90,000 units becomes more like 9,000 units which is a much more reasonable number. However you will need benchmarking to get the actual results and effectiveness of the unit cache but I suspect they will still be very good.

If you do not plan to reduce spawn amounts you could try playing with the unit cache size. Larger caches will mean fewer unit removals and you could possibly find a sweet spot value which eliminates this as a serious problem. You may also advise/remind players to restart WC3 process after every few games but many do already due to widgitizer messing up the data caches.

 

[Robbepop]

Hiho,

I really really like the idea with the unit recycling system based on a page replacement algorithm functionality.
That sounds even funny to implement.
Maybe I will give it a try when I got enough free time for it. ;-)

The major problem with changing the unit spawn count in general is that the superior majority of players (as far as I know them, and I think I know them very well) won't accept it.

Robbepop