1. Updated Resource Submission Rules: All model & skin resource submissions must now include an in-game screenshot. This is to help speed up the moderation process and to show how the model and/or texture looks like from the in-game camera.
    Dismiss Notice
  2. DID YOU KNOW - That you can unlock new rank icons by posting on the forums or winning contests? Click here to customize your rank or read our User Rank Policy to see a list of ranks that you can unlock. Have you won a contest and still havn't received your rank award? Then please contact the administration.
    Dismiss Notice
  3. The Lich King demands your service! We've reached the 19th edition of the Icon Contest. Come along and make some chilling servants for the one true king.
    Dismiss Notice
  4. The 4th SFX Contest has started. Be sure to participate and have a fun factor in it.
    Dismiss Notice
  5. The poll for the 21st Terraining Contest is LIVE. Be sure to check out the entries and vote for one.
    Dismiss Notice
  6. The results are out! Check them out.
    Dismiss Notice
  7. Don’t forget to sign up for the Hive Cup. There’s a 555 EUR prize pool. Sign up now!
    Dismiss Notice
  8. The Hive Workshop Cup contest results have been announced! See the maps that'll be featured in the Hive Workshop Cup tournament!
    Dismiss Notice
  9. Check out the Staff job openings thread.
    Dismiss Notice
Dismiss Notice
60,000 passwords have been reset on July 8, 2019. If you cannot login, read this.

Things That Leak

Discussion in 'Triggers & Scripts' started by Ralle, Jun 10, 2007.

  1. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,033
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    Images, Texttags are not agents. They have their custom id ranges, therefore do not pose a problem. My point was that you can create units and co. in an async manner IF you avoid all false transmissions. For example it should be possible to create a special effect for one player and fill the missing id with a location for the other player. Then when releasing the effect, you should do so too for the location.
     
  2. Magtheridon96

    Magtheridon96

    Joined:
    Dec 12, 2008
    Messages:
    6,006
    Resources:
    26
    Maps:
    1
    Spells:
    8
    Tutorials:
    7
    JASS:
    10
    Resources:
    26
    I'd like texttags more if the limit was 128 or 256 or I don't know, non-existent?
    Is there a handle limit? No? Why a texttag limit then? ;_;
    WHY? ;____;
     
  3. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,033
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    Well, why is the counter backwards? Maybe the 100 texttags are static for whatever reason and just stacked at map init.
     
  4. Zwiebelchen

    Zwiebelchen

    Joined:
    Sep 17, 2009
    Messages:
    6,789
    Resources:
    12
    Models:
    5
    Maps:
    1
    Spells:
    1
    Tutorials:
    1
    JASS:
    4
    Resources:
    12
    This would only work if we assume that the handle stack simply takes the next handle ID from the stack. But that isn't the case, for some reason.
    It seems that all types of handles have a certain range of IDs inside the handle ID stack.

    Creating handles locally is never a good idea. But I don't see how a global variable pointing towards a handle should ever cause a local handle?
     
  5. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,033
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    Agents are in the same range. This can easily be tested.

    References decide when an object's id is released after it has been destroyed. To keep the id pool in sync (else the next independent snippet of code could make it go haywire), variables should not be added to/removed from the object asyncly unless you have a substitute.

    Ex:
    Code (Text):
    if (GetLocalPlayer() == Player(0)) then
        call DestroyEffect(e)

        set e = null
    else
        call RemoveLocation(l)
    endif
    Assuming effect and location had the same id, nulling e would free the id for player 0 but the l reference remains, therefore its still blocked for the other players.

    The reference-counting of objects is most probably for security reasons. References allow you to access an object. If the ids were released before all pointers are gone, you could falsely get the next object with the same id, which is practically abused for getting rid of unit shadows for example (which works since images are not agents, not ref-counted).

    There are types that have really no need to be in the upper id pool though like effects, multiboards, multiboarditems, quests. They do not cause net-traffic nor do they have read functions. Or functions like
    IsQuestCompleted
    are actually superfluous. You set this attribute yourself, so you should avoid determining a sync decision from it. For all I care, such functions could be removed but this would break backwards compatibility. Having those types in the upper id pool makes them async-unfriendly although it would make sense to have them for specific clients. If you could create them locally, this would also benefit performance.
     
  6. Zwiebelchen

    Zwiebelchen

    Joined:
    Sep 17, 2009
    Messages:
    6,789
    Resources:
    12
    Models:
    5
    Maps:
    1
    Spells:
    1
    Tutorials:
    1
    JASS:
    4
    Resources:
    12
    That agents should not be destroyed or created inside local code is nothing new and only logical. I don't really see a problem with that and your example doesn't prove anything here because the desync is not caused by a global variable leaking a reference, but actual differences in IDs due to local creation of agents.

    I'll shamelessly edit your example because I'm a lazy bitch:
    Code (vJASS):

    local effect e = AddSpecialEffect(bla)
    call DestroyEffect(e)
    if (GetLocalPlayer() == Player(0)) then
        set e = null
    endif

    This would be a much better example of what you are trying to prove.

    If what you said about reference count is correct, then this should cause a desync as soon as a new effect is created. Why?
    Because the reference count of the special effect is zero for Player(0) but 1 for Player(1), which should prevent the ID from being put back to the stack.

    However, I did a quick test and this did not desync for me.
    The reason for that is maybe that:
    1) effects leak memory even when destroyed properly, which means that there HAS to be a reference to that effect somewhere in the WC3 code, no matter what the reference count is. Which basicly renders the whole argumentation of IDs not getting freed when there is a reference somewhere pointless
    2) References to removed agents do not get freed in the game memory (hence they leak memory), but still reduce the reference count
    3) As soon as an ID is allocated again, the pointer to that removed agent becomes a nullpointer

    I'm not sure about those, it's just my assumptions.
     
  7. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,033
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    The same id via before async-creation was assumed. Your example is more evident.

    *As soon as the id becomes relevant to net-traffic.

    That is the case.

    Neither does it for me, although the unit I create after comes with a local-dependent id.

    While I cannot make any statements about wc3s internals, we can at least say it influences the jass control.

    I can verify that the ids are blocked. Since it did not cause a desync (game kill) in my current test, which surprises me too at the moment, I wonder if objects/agents have another id they use for net-traffic that is not exposed to jass (which of course may alter the life cycle).
     
  8. kkots

    kkots

    Joined:
    Jun 21, 2007
    Messages:
    534
    Resources:
    0
    Resources:
    0
    Zwiebelchen, how can you test desynchronizations in single player?
    Don't you need two computers connected over LAN network to play the map?
     
  9. Tirlititi

    Tirlititi

    Joined:
    Jul 11, 2010
    Messages:
    396
    Resources:
    12
    Models:
    6
    Maps:
    2
    Spells:
    3
    JASS:
    1
    Resources:
    12
    Having a local-dependant ID may not desync instantatly, but it will desync as soon as an hashtable or such is involved.
     
  10. edo494

    edo494

    Joined:
    Apr 16, 2012
    Messages:
    3,846
    Resources:
    5
    Spells:
    1
    JASS:
    4
    Resources:
    5
    there is finite number of handles you can have in handle table before crashing the game, it is however somewhere around several milions of handles
     
  11. Zwiebelchen

    Zwiebelchen

    Joined:
    Sep 17, 2009
    Messages:
    6,789
    Resources:
    12
    Models:
    5
    Maps:
    1
    Spells:
    1
    Tutorials:
    1
    JASS:
    4
    Resources:
    12
    Which basicly brings us back to the point admitting that some things in WC3 are a deadly mixture of magic and insanity.
    This or WC3 is just plain spaghetti coding at its finest. :/
     
  12. Magtheridon96

    Magtheridon96

    Joined:
    Dec 12, 2008
    Messages:
    6,006
    Resources:
    26
    Maps:
    1
    Spells:
    8
    Tutorials:
    7
    JASS:
    10
    Resources:
    26
    Of course, but I'm sure it's not hardcoded (unless someone tried maxing it out before =o)

    I don't have enough RAM to test for such a limit.
    It will be no more than 2.146 billion given infinite RAM (it can be ~4.293 billion, but who knows what type modifiers they're using in their actual source?), but who's to say they didn't hardcode a limit less than that? ;p
     
  13. edo494

    edo494

    Joined:
    Apr 16, 2012
    Messages:
    3,846
    Resources:
    5
    Spells:
    1
    JASS:
    4
    Resources:
    5
    when I was just like: I will make wc3 take all my ram(4 giga because of 32 bit OS) it went only up to 1,6 giga aprox and crashed

    I was generating 1 point leak and 1 unit without killing it at random position on the map(GetRandomReal(..., ...) with every 0.00 seconds timer
     
  14. Magtheridon96

    Magtheridon96

    Joined:
    Dec 12, 2008
    Messages:
    6,006
    Resources:
    26
    Maps:
    1
    Spells:
    8
    Tutorials:
    7
    JASS:
    10
    Resources:
    26
    That's approximately 3.9 million handles =o (Not sure - Depends on the size of a unit in memory. Some sources tell me 3KB, others tell me 44 bytes >.<)

    I guess the limit might be close to 1 million if it's present? =o

    My record for filling up my RAM with Wc3 is different
    I generated 1GB of leaks with locations. Knowing that a location is 361 bytes give or take 10 bytes, that's about 2.974 million handles.

    There might not actually be a limit for the handles hardcoded internally.
    The crash might be caused by some other factor.
     
  15. Zwiebelchen

    Zwiebelchen

    Joined:
    Sep 17, 2009
    Messages:
    6,789
    Resources:
    12
    Models:
    5
    Maps:
    1
    Spells:
    1
    Tutorials:
    1
    JASS:
    4
    Resources:
    12
    I'm curious, how did you do that? WC3 does only allow creating a certain number of units (either 4k or 8k, i don't remember) until some units must be removed/killed first.
     
  16. edo494

    edo494

    Joined:
    Apr 16, 2012
    Messages:
    3,846
    Resources:
    5
    Spells:
    1
    JASS:
    4
    Resources:
    5
    well, the crash maybe happened becuase I had no RAM left or something like that

    I did simple GUI trigger with 1 action - Create 1 unit of type footman at position(Random position at(Playable map area))

    with event every 0.00 second

    and I tested this again with only doing(very similar):
    Code (vJASS):

    scope myscope initializer i
         private function run takes nothing returns nothing
             call Location(GetRandomReal(-1000, 1000), GetRandomReal(-1000, 1000))
         endfunction

        private function i takes nothing returns nothing
            call TimerStart(CreateTimer(), 0.00, true, function run)
        endfunction
    endscope
     


    and funny enough, it seemed like Wc3 actually tried to force remove the leaks because at 1 GB it started dropping a bit and after 1.2 GB it popped error: no CD disk detected in driver or w/e and crashed :D
     
  17. deathismyfriend

    deathismyfriend

    Joined:
    Oct 24, 2012
    Messages:
    6,530
    Resources:
    14
    Spells:
    12
    Tutorials:
    2
    Resources:
    14
    mine crashes around the same number. i believe its the max ram that wc3 can handle my pc has 16 gigs of ram and can use all of it. i have windows 7 64 bit ultimate and i have maxed out the ram b4 using rendering software. wc3 crashed around 1.6 to 1.8gb. i havent noticed a limit on the units but i havent sat there and counted them. i added a unit to map and increased an integer to count units on map. making 100 units every .01 seconds i got around 20k for the integer until it finally crashes. i did clean up the point leak w that test
     
  18. themerion

    themerion

    Joined:
    Jan 27, 2010
    Messages:
    132
    Resources:
    0
    Resources:
    0
    Yeah, so, I tested nulling globals vs. not nulling globals using Maker's test code. Conclusion: null globals nulling globals might prevent a very, very insignificant leak. As Zwiebelchen rightly points out below, it should never matter (the test created and destroyed 60 MILLION locations, and memory usage only went up a few megabytes).

    Not nulling


    Timers created Memory Consumption
    2 702 500 88.1M
    4 585 000 88.3M
    6 407 000 88.6M
    8 240 000 87.7M
    11 746 500 83.3M
    13 539 500 89.7M
    33 932 000 91.0M
    40 913 500 90.4M
    58 903 000 92.2M


    The trend is clearly rising in the long term. Just like Maker said, it's a really minor leak (considering the code is creating 500x100=50000 locations every second).

    Code (vJASS):
    library leakTest initializer onInit
        globals
            private timer t = CreateTimer()
            private constant integer LOCS = 500
            private integer c = 0
        endglobals

        private function leaks takes nothing returns nothing
            local integer i = LOCS
           
            loop
                exitwhen i == 0
                set udg_loc = Location(0,0)
                call RemoveLocation(udg_loc)
                // set udg_loc = null
                set c = c + 1
                set i = i - 1
            endloop
        endfunction
       
        private function pauseResume takes nothing returns nothing
            if ModuloInteger(GetTriggerExecCount(GetTriggeringTrigger()) , 2) == 1 then
                call BJDebugMsg("started")
                call TimerStart(t, 0.01, true, function leaks)
            else
                call BJDebugMsg("paused, " + "locs created: " + I2S(c))
                call PauseTimer(t)
            endif
        endfunction

        private function onInit takes nothing returns nothing
            local trigger trg = CreateTrigger()
            call TriggerRegisterPlayerEvent(trg, Player(0), EVENT_PLAYER_END_CINEMATIC)
            call TriggerAddAction(trg, function pauseResume)
        endfunction
    endlibrary


    Nulling


    Timers created Memory Consumption
    4 010 500 85.5M
    7 815 000 84.8M
    12 700 000 84.9M
    25 000 000 84.9M
    34 800 000 85.1M
    42 000 000 85.5M
    48 600 000 85.9M
    55 000 000 83.3M
    59 400 000 82.7M


    There is no rising trend here...

    Code (vJASS):
    library leakTest initializer onInit
        globals
            private timer t = CreateTimer()
            private constant integer LOCS = 500
            private integer c = 0
        endglobals

        private function leaks takes nothing returns nothing
            local integer i = LOCS
           
            loop
                exitwhen i == 0
                set udg_loc = Location(0,0)
                call RemoveLocation(udg_loc)
                set udg_loc = null
                set c = c + 1
                set i = i - 1
            endloop
        endfunction
       
        private function pauseResume takes nothing returns nothing
            if ModuloInteger(GetTriggerExecCount(GetTriggeringTrigger()) , 2) == 1 then
                call BJDebugMsg("started")
                call TimerStart(t, 0.01, true, function leaks)
            else
                call BJDebugMsg("paused, " + "locs created: " + I2S(c))
                call PauseTimer(t)
            endif
        endfunction

        private function onInit takes nothing returns nothing
            local trigger trg = CreateTrigger()
            call TriggerRegisterPlayerEvent(trg, Player(0), EVENT_PLAYER_END_CINEMATIC)
            call TriggerAddAction(trg, function pauseResume)
        endfunction
    endlibrary

    Nulling only at end of function

    13 800 000 82.5M
    37 000 000 82.8M
    59 000 000 82.5M


    And this is horribly confusing! Is WC3-functions somehow making extra memory for globals, which is only released upon nulling? I really expected this to be 499/500 as "leaky" as the not nulling version...
    Code (vJASS):
    library leakTest initializer onInit
        globals
            private timer t = CreateTimer()
            private constant integer LOCS = 500
            private integer c = 0
        endglobals

        private function leaks takes nothing returns nothing
            local integer i = LOCS
           
            loop
                exitwhen i == 0
                set udg_loc = Location(0,0)
                call RemoveLocation(udg_loc)
                set c = c + 1
                set i = i - 1
            endloop
            set udg_loc = null
        endfunction
       
        private function pauseResume takes nothing returns nothing
            if ModuloInteger(GetTriggerExecCount(GetTriggeringTrigger()) , 2) == 1 then
                call BJDebugMsg("started")
                call TimerStart(t, 0.01, true, function leaks)
            else
                call BJDebugMsg("paused, " + "locs created: " + I2S(c))
                call PauseTimer(t)
            endif
        endfunction

        private function onInit takes nothing returns nothing
            local trigger trg = CreateTrigger()
            call TriggerRegisterPlayerEvent(trg, Player(0), EVENT_PLAYER_END_CINEMATIC)
            call TriggerAddAction(trg, function pauseResume)
        endfunction
    endlibrary
     
    Last edited: Apr 23, 2013
  19. Zwiebelchen

    Zwiebelchen

    Joined:
    Sep 17, 2009
    Messages:
    6,789
    Resources:
    12
    Models:
    5
    Maps:
    1
    Spells:
    1
    Tutorials:
    1
    JASS:
    4
    Resources:
    12
    If we compare this to the unfixable leaks of properly removed units and special effects, this goes into diminishing returns.

    I definitely won't go through all the hassle to null my globals just for a megabyte of memory saved after a 3 hour game...

    The conclusion should more be:
    If you are paranoid, null globals. If you live in the real world, save the time.

    And to be honest, I don't even think it's a real leak. More like increasing memory allocation of WC3.
    Which means, WC3 probably doesn't always use the same memory adress for a global variable, but changes its adress (possibly due to internal garbage collection) from time to time.
    This would mean that WC3 allocates more memory that is just not used yet, but might be used later on.

    This is like making a parking lot bigger. You don't have more cars on it, but you could have more cars in it if you wish. That doesn't mean that new cars can not get on the empty spots if the area of the parking lot is enlarged (which would be the case with 'real' leaks). I hope you get the idea...

    I'd take that guess a step further by saying, that this is probably caused by your operating system! Operating systems use optimizing strategies so that to reallocate memory for faster reading that is used more frequently. Remember that your nulling operation basicly increases the amount of memory operations by a half. This might affect your OS aswell or the memory allocation of WC3!
    The reason I'm saying this is that the rise of memory is so goddamn low, that it's hard to say wether this is an actual memory leak or caused by overhead of WC3 or your OS we don't have any control over (and isn't harmful in any way).
    Actually, the random fluctuation of memory usage of WC3 is much larger than the increase seen... (look for example at the sudden drop to 83 megs at 11.000.000 ...)


    And looking at the numbers, I feel we should stop making all the less experienced users out there, especially the GUIers worried about nulling globals.
    You need literally BILLIONS of global reassignements for a few megabytes of memory. Really, there are far worse things than that you don't even have control over...
     
    Last edited: Apr 23, 2013
  20. themerion

    themerion

    Joined:
    Jan 27, 2010
    Messages:
    132
    Resources:
    0
    Resources:
    0
    Quite possible! I just find it fascinating that it would do this when not nulling, but not do this when nulling (even if nulling only at end of function). I mean, the values fluctuated a bit over time, but the general trend of the not nulling one... WC3 is such a magic box sometimes...


    Definitely.