[JASS] Nulling leaks...

[HINDYhat]

I posted this on wc3campaigns, but due to the inactivity, I haven't gotten many clear replies... so here goes:

I'm not so sure about this, but I suppose I'm trying to figure things out on my own.

I'm guessing that non-nulled handles only link when associated with gamecache. Explanation: not nulling the handle increases the reference counter, and the I2S(H2I()) call in gamecache storage leaks a string for each new handle, since the ID constantly grows. In that case, if your map does not use gamecache at all, you wouldn't need to null local handles.

Is that correct?

 

[PurplePoot]

1) It also fails to release the memory the pointer took (until the end of the game, presumably)

2) Wc3 crashes after it hits a certain handle ref. counter, apparently.

 

[HINDYhat]

1) Aha I see there was noticeable lag even without string leakage.

2) The game became unplayable (yet didn't crash) when I reached about 2000000 locations (FPS dropped under 1).

 

[darkwulfv]

Your game will lag if you do not destroy and/or null variables or created handles. Unless, of course, you're using some complex struct/ABC system, or some shit like that. But even then.

It's generally very good practice to null handles... And I don't quite understand where you get the idea that not using gamecache means you don't have to null handles.

 

[PurplePoot]

It's just that the problem everyone mentions with not nulling is that it screws up H2I, darkwulf.

 

[darkwulfv]

Why would people not be nulling handles they don't need anyways? Isn't it common practice by now to do that? So why should people have to worry about breaking H2I because they aren't nulling handles?

 

[HINDYhat]

And I don't quite understand where you get the idea that not using gamecache means you don't have to null handles.

I thought that handle nulling only prevented the string leakage caused by I2S(H2I(handle)), but I guess there are other stupid reasons for the leak.

 

[darkwulfv]

Uhh...

function leak takes nothing returns nothing
local unit u = GetTriggerUnit()
  call Location(GetUnitX(u), GetUnitY(u))
endfunction

That leaks twice: The variable u, and the location you've made.

 

[HINDYhat]

Yes I know now that the non-null leaks because...

Ghost reference take memory and slow down the allocator down, no matter what attachment system you use, don't go around leaking things you don't have to.

which is really lame by the way...

And the location just leaks like handles should : don't remove the handle == leak.

 

[PurplePoot]

Why would people not be nulling handles they don't need anyways? Isn't it common practice by now to do that? So why should people have to worry about breaking H2I because they aren't nulling handles?

Exactly. He was questioning whether without H2I it was actually a problem, since, as I said, that's the problem everyone mentions...

 

[PandaMine]

Honestly its only a problem in the most extreme circumstances, often you can get away with not nulling handles and not notice much difference in performance for gameplay

You only start getting problems when you use it with systems that do the H2I - 0x100000 method for storing data attach to handles (like my HSAS) or the old CSData.

Apparently the curve for performance vs handles index leak is more accuratley percieved as a log graph then a diagonal line, which means you only really start noticing a difference when your leaks go above somethime like a million which really shouldn't happen in the first place

 

[Jacek]

oh no pandamine... u have to null handles or u lag![/sarcasm]

 

[Drone]

Uhh...

function leak takes nothing returns nothing
local unit u = GetTriggerUnit()
  call Location(GetUnitX(u), GetUnitY(u))
endfunction

That leaks twice: The variable u, and the location you've made.

Please explain why, technically please. Why do those local handles leak if they're not used for caches to provide global data references. I seriously doubt that, and I created a post asking some clarification regarding the subject.

If nulling would prevent increase of reference that would mean that the code is continually monitored for the instructions that nulls the local pointer. More logical to me would seem that there is an active reference throughout the function and not anymore when the stackframe the function uses is released.

ps. the location part I get. An object needs to be destroyed if not used further.

 

[HINDYhat]

Well it seems that the handle allocator is not O(1), as shown here:

Well after playing with your test a little I just realized that handle allocator is actually not O(1)

How do I know?
Well fps drops only while test is working (a.k.a while new handles are requested for things to leak)
when you pause test fps jumps back to normal.

Well I guess blizz won lamer of the year award again....

PS: my estimate is that it is O(log n) in case anyone cares

That would be the leaks caused by not nulling local handles (but string leakage would also happen if you use gamecache).

 

[PurplePoot]

Please explain why, technically please. Why do those local handles leak if they're not used for caches to provide global data references. I seriously doubt that, and I created a post asking some clarification regarding the subject.

If nulling would prevent increase of reference that would mean that the code is continually monitored for the instructions that nulls the local pointer. More logical to me would seem that there is an active reference throughout the function and not anymore when the stackframe the function uses is released.

ps. the location part I get. An object needs to be destroyed if not used further.

A glitch with this apparently causes the handle index to not be released when the temporary object is destroyed (eg unit dies, location is removed, etc), and thus taking longer for the index allocator to allocate a new index.

 

[PandaMine]

Its the pointer (or reference handle) that leaks. You can test it by doing H2I on the handle, and you will see for that code that it will slowly increase (Blizzard uses the outdated tombstone system for dealing with handles)

You can test it for yourself if you don't believe me

 

[Drone]

Its the pointer (or reference handle) that leaks. You can test it by doing H2I on the handle, and you will see for that code that it will slowly increase (Blizzard uses the outdated tombstone system for dealing with handles)

You can test it for yourself if you don't believe me

Its the pointer (or reference handle) that leaks. You can test it by doing H2I on the handle, and you will see for that code that it will slowly increase (Blizzard uses the outdated tombstone system for dealing with handles)

You can test it for yourself if you don't believe me

It's not I don't want to believe you
I just find the thing to be a little vague.

I did do some experimenting. First to see if this so called reference count by pointers works like it should. If this is implemented like the way it should then there wouldn't be a need to destroy objects we create using Jass/GUI. Why? Well isn't the main point of reference count to deallocate dynamic heap objects when that counter is 0? To me it seems. Because there would always be a tombstone pointer that holds a reference to the object in the heap and therefore it should be destroyed if not done so locally. Since Jass has no underlying garbage collection why is pointer reference count being mentioned here at all?

I used the H2I return bug to get a tombstone pointer to a hero unit object. Before removing that hero unit, I got its integer pointer by using H2I. Then I removed the hero by using RemoveUnit(hero) After doing that I tried accessing the unit object through my dangling integer pointer to the tombstone pointer of the hero unit object. I tried to revive the already removed hero by using the native ReviveHero(I2H(temp_unit_pointer), x, y, true) If WC3 has pointer error detection something not nice should happen. Reason being it should set the tombstone pointer to null, therefore an illegal reference to a non existing object. But guess what, nothing happened. That seems to me that it doesn't seem to care when random pointers are pointing at non existing objects, hence the dirty tombstone solution.

Normally a local pointer shouldn't cause leaking of any sort. Non referred objects do. The point I'm trying to make is that if a local function is responsible for releasing allocated heap memory for an object, (locations, effects, sounds, etc) it should do so. If not it would cause a leak if using a local handle to that object. If the function is NOT responsible for the release of allocated heap memory of an object, (for example a unit that is still used after the function is done) then the local handle in stack memory will be released when the function ends. It would mean that somehow it does allocate space for pointers in the heap (why?) but doesn't release them properly unless you null local handles, which strikes me as being very odd. What I do think is that the wrapping pointer is not released and that causes leaks. Those tombstone pointers eat up space over time and aren't ever freed during runtime. That means it's not the locals but the wrapper pointer that causes leaks. But only when you create an object. It would still leak even after destroying the object and nulling the local.

 

[darkwulfv]

I think you're taking this wayyyyy too logically. Blizzard's coding is actually pretty bad. There's a lot of flaws that really should've been fixed.

In fact, I hardly understood a word of what you yammered on about up there. From what I did gather, you're basically saying that nothing leaks unless you make an object and it will then leak indefinitely because of the way Blizzard made their coding work? If that is true, then can you explain how systems that create, destroy, and null objects and their locals hundreds of times a minute do not leak (and thus do not lag), but maps made in GUI that leak locations (due to GUI's shittyness) will lag?

 

[Drone]

I think you're taking this wayyyyy too logically. Blizzard's coding is actually pretty bad. There's a lot of flaws that really should've been fixed.

In fact, I hardly understood a word of what you yammered on about up there. From what I did gather, you're basically saying that nothing leaks unless you make an object and it will then leak indefinitely because of the way Blizzard made their coding work? If that is true, then can you explain how systems that create, destroy, and null objects and their locals hundreds of times a minute do not leak (and thus do not lag), but maps made in GUI that leak locations (due to GUI's shittyness) will lag?

Yeah well sorry for the confusing bit. I've programmed quite a bit in assembly and therefore like to know what's going on and why. Nasty habit perhaps.

I think you misunderstood something. I'm not saying that. If the underlying memory management is working like it should, then nothing should leak, not even when you don't release objects in memory. In a less ideal situation (most of the times) the programmer does things like not removing objects when done with them or losing reference to a created object. That will cause leaks. In an even lesser ideal situation (WC3 probably ) you have to destroy created objects and null their local handles. Because somehow it reserves space for something it doesn't release (no clue why). Furthermore even if you do cleanup like you should, it will still leak a pointer. The wrapping pointer (tombstone pointer like someone mentioned earlier.) leaks, since its memory is never released. It's just an easy, cheap solution to fix dangling pointers. I'm not saying that this is a big issue or something, but still a pointer is always leaked when creating and then destroying an object and nulling its local handle.

 

[darkwulfv]

But that pointer leak obviously is not significant or every map ever would die pretty fast.

For future refrence, a 'leak' that we are referring to is a Handle/Local pointer leak. Eg:

AddSpecialEffect(x, y, "effectstringhere")

set sfx = AddSpecialEffect(x, y, "effectstringhere")

set sfx = AddSpecialEffect(x, y, "effectstringhere")
call DestroyEffect(sfx)

These 3 leak. 1st one leaks a handle, 2nd leaks a handle and a local, 3rd leaks a local. To fix the second, destroy the handle and null the local. To fix the 3rd, null the local. To fix the first, set it to a local, destroy the local's handle, then null the local.

Make sense? Oh, and don't forget: JASS is quite flawed, don't assume that the way things work is the way this will, because JASS is a lot different and a lot of easy flaws were left un-fixed.

 

[Earth-Fury]

All variables in WC3 are 4 bytes big. Handles are pointers, but not first class pointers. Strings are also pointers. I have no idea about the "code" type, but its weird. All other types are stored within the variable itself.

Handles are allocated in a table. Handles have a reference count, but it is broken. WC3 has a Garbage Cleaner, but because of the broken reference counter, its rather useless. Assignment increments the reference counter. Reassignment (Changing the value to something else) decrements the reference counter as it should. Going out of scope (a problem only found in locals, of course) does not decrement the reference counter. Thus why you need to change the value of local handle-type variables, most commonly changing the value to null. According to vex, removing a handle which still has active references (i.e, the reference counter is not 0) will leak a 4 byte placeholder. (of course, it will only actually leak if you've lost all references to it. Thats what a leak is: an object, unremoved in memory, which has nothing valid pointing to it.)

Strings are held in a different table, and are never destroyed. (except, possibly by a Garbage Cleaner, but who knows) Creating new unique strings is bad for memory consumption. String leaks with GameCache are from I2S, where you are creating a large number of integer-representational strings.

I2H is evil because of all the errors it can cause. Don't use it. (You don't need it anyway, especially with vJASS around.) H2I is not evil, as it can't cause as many errors if (when?... its WC3 after all) something goes wrong.

 

[Drone]

All variables in WC3 are 4 bytes big. Handles are pointers, but not first class pointers. Strings are also pointers. I have no idea about the "code" type, but its weird. All other types are stored within the variable itself.

You can have a pointer to any sort of data type. Everything that's stored in memory can have a pointer to it. So code functions can be referenced by pointers as well.

Handles are allocated in a table. Handles have a reference count, but it is broken. WC3 has a Garbage Cleaner, but because of the broken reference counter, its rather useless. Assignment increments the reference counter. Reassignment (Changing the value to something else) decrements the reference counter as it should. Going out of scope (a problem only found in locals, of course) does not decrement the reference counter. Thus why you need to change the value of local handle-type variables, most commonly changing the value to null. According to vex, removing a handle which still has active references (i.e, the reference counter is not 0) will leak a 4 byte placeholder. (of course, it will only actually leak if you've lost all references to it. Thats what a leak is: an object, unremoved in memory, which has nothing valid pointing to it.)

Makes sense and explains why locals need to be nulled.

I2H is evil because of all the errors it can cause. Don't use it. (You don't need it anyway, especially with vJASS around.) H2I is not evil, as it can't cause as many errors if (when?... its WC3 after all) something goes wrong.

I'm not sure about that. What errors are you referring to? I did some testing with it and didn't come accross any wierd errors, where its supposed to do exactly that. I'm referring to invalid references and stuff. I abused the I2H to get a pointer to a unit object and then remove it. Using H2I to convert it back to a unit handle and try to access an already removed object. Now it should cause some error (because of an invalid reference) but it doesn't. Because all dangling pointers will point to a null pointer after an object is destroyed. Anyway I don't see why it should cause an error because exceptions can be caught silently without the user knowing it. And dangling pointers won't cause data corruption because they're always pointing to the null (the first class pointer) pointer of the destroyed object.

 

[Earth-Fury]

You can have a pointer to any sort of data type. Everything that's stored in memory can have a pointer to it. So code functions can be referenced by pointers as well.

Pointers in warcraft 3 are not 'real' pointers. they are simply indexes in the handle or string table.

I'm not sure about that. What errors are you referring to? I did some testing with it and didn't come accross any wierd errors, where its supposed to do exactly that. I'm referring to invalid references and stuff. I abused the I2H to get a pointer to a unit object and then remove it. Using H2I to convert it back to a unit handle and try to access an already removed object. Now it should cause some error (because of an invalid reference) but it doesn't. Because all dangling pointers will point to a null pointer after an object is destroyed. Anyway I don't see why it should cause an error because exceptions can be caught silently without the user knowing it. And dangling pointers won't cause data corruption because they're always pointing to the null (the first class pointer) pointer of the destroyed object.

Warcraft 3 is poorly coded. Seemingly random errors can (and in all likely hood, will) occur which will fuck things up. When that happens, if you are casting from integers to handles, things will get worse. Casting from handles to integers is a much safer operation, as you aren't then trying to perform native operations on the handles which have lost type safety.