[Lua] Callback functions and garbage collection

[Antares]

Hello,

I'm again and again running into situations where I'm unsure how the Lua garbage collector would handle things and how I should write my code to prevent unexpected behavior.

I've just had ChatGPT explain to me how the Lua garbage collector determines whether a closure where objects are used in is still reachable or not, and therefore determine whether objects referenced within those closures should be garbage collected.

"In Lua, the garbage collector uses a form of reachability analysis to determine whether an object (including closures and their upvalues) is reachable or not. Objects are considered reachable if they are part of the root set or if they can be reached through a chain of references starting from the root set.

The root set typically includes global variables, local variables in currently executing functions, and other objects that the collector knows are reachable. During the garbage collection process, Lua's collector traces the objects starting from the root set and follows references from one object to another."

Which leads me to the question... is the Lua garbage collector smart enough to take triggers and timers into account when determining which code is still reachable?

function CreateNumbers()
    local numbers = {1, 3, 5}
    local newTimer = CreateTimer()
    TimerStart( newTimer, 10, false, function()
        print(numbers[1] + numbers[2] + numbers[3])
    end)
end

Here, we create a table with some numbers in it. If we disregard the timer, the table should quickly be garbage collected because it is defined as a local variable and is out of scope as the code exits the function CreateNumbers. The anonymous callback function is also only defined within CreateNumbers and becomes out of scope. This should make the table be cleaned unless the garbage collector is smart enough to realize that it is still referenced by a function used by the currently running timer, but I'm doubtful that it understands what the Warcraft III natives are doing.

Please correct me if I'm wrong and if I'm not, how would one write the above code to prevent premature garbage collection?

 

[HerlySQR]

In this case, the anonymus function should also be collected since it doesn't have any reference to it apart of the timer call thus the table that it uses, unless behind the scene it still references the function.

 

[Antares]

I guess that's true, but clearly that doesn't happen. So, the question is, how does it know to not garbage collect it? Does it understand what the timer does? And is destroying the timer enough to make the garbage collector realize that the anonymous function is now unreachable?

 

[HerlySQR]

but clearly that doesn't happen.

What do you mean?

 

[Antares]

I did some tests regarding when callback functions are cleaned up.

Hidden

--This will cause the function to be garbage collected eventually.
TestTable = setmetatable( {}, {__mode = 'v'} )

OnInit.main(function()
    local myFunc = function()
        DestroyTimer(GetExpiredTimer())
    end
    TimerStart(CreateTimer(), 1.0, false, myFunc)

    TestTable[1] = myFunc
end)

OnInit.main(function()
    TimerStart(CreateTimer(), 1.0, true, function()
        print(TestTable[1])
    end)
end)

Hidden

--This will not.
TestTable = setmetatable( {}, {__mode = 'v'} )

OnInit.main(function()
    local myFunc = function()
        --DestroyTimer(GetExpiredTimer())
    end
    TimerStart(CreateTimer(), 1.0, false, myFunc)

    TestTable[1] = myFunc
end)

OnInit.main(function()
    TimerStart(CreateTimer(), 1.0, true, function()
        print(TestTable[1])
    end)
end)

Hidden

--This will also not garbage collect myFunc.
TestTable = setmetatable( {}, {__mode = 'v'} )

OnInit.main(function()
    local myFunc = function()
        DestroyTrigger(GetTriggeringTrigger())
    end
  
    local trig = CreateTrigger()
    TriggerAddAction(trig, myFunc)
    TriggerExecute(trig)

    TestTable[1] = myFunc
end)

OnInit.main(function()
    TimerStart(CreateTimer(), 1.0, true, function()
        print(TestTable[1])
    end)
end)

It seems like, from my testing, timer functions do get cleaned up if the associated timer is destroyed, but trigger functions never get cleaned up.

 

[HerlySQR]

In that case I think you should nil all the possible references when you no longer need them.

But try using the function TriggerClearActions before destroying the trigger to be sure, because I heard that the triggeractions leak even if the trigger is destroyed.

 

[Antares]

Unfortunately, that didn't work either. Also tried using trigger conditions instead of actions, also no luck. I guess you'd have to manually nil everything.

 

[Wrda]

If there's at least one reference to an object, it won't be garbage collected. If there are none, it will sooner or later collect it.
That's how I understood it.
On the last example, TestTable[1] (a global) is referencing the local function. It will never be cleaned.

 

[Antares]

With the

setmetatable( {}, {__mode = 'v'} )

line, the values of TestTable are set to be ignored by the garbage collector. TestTable no longer counts as a reference and myFunc should be garbage collected if all other references are removed.

 

[Eikonium]

That's indeed unexpected. The non-referenced callback of the trigger action should have been garbage collected in an ideal world after the trigger was destroyed.

As you said, we are lacking knowledge on about Wc3 handles Lua references behind the scenes. Wc3 objects seem to hold a reference to the Lua objects they utilize, thus preventing garbage collection of callbacks.

According to your own tests (callbacks from trigger actions not cleaned up properly after destroying the trigger), the other way around doesn't necessarily hold. @HerlySQR might be on the right train of thought, when he said that trigger actions are known to leak in Wc3 (thus stay in memory), so the referenced callbacks do remain referenced and consequently stay as well. This again is just an assumption.

Triggers and timers also show different behaviour when it comes to automatic collection of their Wc3 handle:

--The following trigger is not referenced from root (at least not by us, maybe by the game), but it stays in place and keeps working forever.
do
    local t = CreateTrigger()
    TriggerRegisterPlayerChatEvent( t, Player(0), "Hello", true )
    TriggerAddAction(t, function() print("World") end)
end

--The following timer is not referenced from root either, but will be garbage collected after some time (at least due to testing done in some earlier Wc3 patch, which could be out of date).
--The time of collection can differ for players in the same game, so manipulating game state inside anonymous timers might lead to desyncs.

TimerStart(CreateTimer(), 1., true, function() print("Test") end)

It seems like, from my testing, timer functions do get cleaned up if the associated timer is destroyed, but trigger functions never get cleaned up.

I would expect that the timer function is garbage collected when the timer stops using it, with destroying the timer just being one such scenario.

 

[Antares]

As you said, we are lacking knowledge on about Wc3 handles Lua references behind the scenes. I've always assumed that all Wc3 objects hold a reference to the Lua objects they utilize, thus preventing garbage collection of callbacks. So far, this assumption has not led to any bugs in my personal development, but that doesn't mean anything. It's still good practice to keep your own references to your callbacks, until they aren't needed anymore, just to be sure. Timer Queue for instance is doing that, so you don't need to worry about garbage collection ahead of time.

I don't quite see the benefit of keeping references to callbacks right now. As I see it, they either get cleared when the associated timer/trigger is destroyed or they don't, in which case there's no way to garbage collect them. In my example above, I don't know how to make Lua garbage collect the callback function. This isn't a big deal if it's just a function, but in the system I wrote, the callback function holds various tables that will all stay in memory forever as a consequence.

--The following timer is not referenced from root either, but will be garbage collected after some time (at least due to testing done in some earlier Wc3 patch, which could be out of date).
--The time of collection can differ for players in the same game, so manipulating game state inside anonymous timers might lead to desyncs.

TimerStart(CreateTimer(), 1., true, function() print("Test") end)

This function keeps printing "Test" for me indefinitely. Is this maybe based on outdated info?

 

[Eikonium]

Is this maybe based on outdated info?

It's based on old info and might be outdated, yes. I wrote that in the first line of the comment you quoted
It would be beneficial to know the exact circumstances it has been tested in, but I don't. Even if it's a problem of the past, the Wc3 garbage collection process might change again tomorrow with any upcoming patch, so from my perspective it's safer to just keep a reference to the timer, until it's not needed anymore.

I don't quite see the benefit of keeping references to callbacks right now. As I see it, they either get cleared when the associated timer/trigger is destroyed or they don't, in which case there's no way to garbage collect them. In my example above, I don't know how to make Lua garbage collect the callback function. This isn't a big deal if it's just a function, but in the system I wrote, the callback function holds various tables that will all stay in memory forever as a consequence.

Absolutely right. I probably intended to write that I keep references to my timers to prevent their potential collection, not to the callbacks, but I was too tired yesterday when I wrote this I deleted that part from my post to prevent confusion for future readers. Yes, you can use anonymous functions in timers. I do that, too. Jesus.

Regarding the collection issue in your system, you could try using only one single function in every of its trigger actions. The function would need to fetch the button-specific data during execution instead of using upvalues to make it work (using event responses for the triggering frame and the triggering player). It's not a real solution, but it would effectively reduce memory leaks.

 

[Antares]

It's based on old info and might be outdated, yes. I wrote that in the first line of the comment you quoted

In a trigger comment that I have to scroll right to read!

Regarding the collection issue in your system, you could try using only one single function in every of its trigger actions. The function would need to fetch the button-specific data during execution instead of using upvalues to make it work (using event responses for the triggering frame and the triggering player). It's not a real solution, but it would effectively reduce memory leaks.

I will keep it in mind for stuff I write in the future. Probably not worth it to rewrite my system to fix minor memory leaks when it's something that will be called at most a handful of times per map.