1.32 The LUA GC is seemingly disabled.

[ScrewTheTrees]

This is just a heads up for whoever uses Lua / TypescriptToLua or similar frameworks for their warcraft 3 maps.
The Lua GC is seemingly disabled.

Over in the hiveworkshop discord we did quite a bit of experimentation on this issue after i realised my map was leaking 0.6MB/s, but the leaks made absolutely no sense.
After 3 days of debugging we noticed that the GC is never called, memory is never cleaned up.
The memory simply keeps building up forever and __gc is never called.

Courtesy of @MindWorX for helping me test this.

(My image).

(Williams image).

If you want to test this for yourself, the test map is attached.


1.31 does not have this issue as it still uses the standard Lua incremental GC, but it also desyncs so.

 

[Boar]

Welp, Typescript and Lua were the reason I was semi-excited for wc3r modding. So this not working basically kills any motivation I had to finish my map.

fun.

 

[LeP]

Welp, Typescript and Lua were the reason I was semi-excited for wc3r modding. So this not working basically kills any motivation I had to finish my map.

fun.

return to monke jass

 

[TriggerHappy]

As we found out in Discord, it seems if you put too much pressure on the GC (like creating tables every 0.01 seconds) the garbage never gets collected. In the example map if you increase the timer from 0.01 to 1.00 the data gets GC'd properly.

I still don't fully understand why the garbage collector does what it does, but I posted something about it yesterday: Local multiplayer checking is bugged on Lua

 

[Kam]

This information will get passed on. Hopefully it can be fixed soon.

 

[Ralle]

Fred can you try to push it and see how often your timer can update while still collecting garbage? Would be interesting to see if it's the same threshold on all computers due to the lockstep way it all runs. I bet it is.

 

[ScrewTheTrees]

Fred can you try to push it and see how often your timer can update while still collecting garbage? Would be interesting to see if it's the same threshold on all computers due to the lockstep way it all runs. I bet it is.

Yeah, i also want to see if its only timer events or if its other events like unit takes damage as well, my preliminiary thesis is that all events causes the delays but more testing is needed.

 

[Ralle]

it will be hard to make a unit die every 0.01 second without using a timer

 

[ScrewTheTrees]

it will be hard to make a unit die every 0.01 second without using a timer

You underestimate my power!

(Unit takes damage is easy thou, as i can just set the damage taken to 0).



Edit:
After in-depth testing the results are very indecisive, garbage collection can occur anywhere between 6-300 seconds depending on how it feels like and how much data is handled, and its not dependent the quantity entirely, some higher numbers GC earlier than lower numbers.

Timers suffer from the same issue
Switching between different triggers/timers to try and offload has minimal impact.
Once the GC hits its not guaranteed to clear the actual ram.

 

[Dr Super Good]

Oh just noticed this topic.

Lua normally uses an incremental garbage collector. It marks at such a rate that by the time the Lua heap doubles (both time and size configurable) it will have mostly finished marking and so can almost immediately sweep. However this is not deterministic between clients due to subtle differences in heap sizes. Since there may be clean-up functions called by the garbage collector, such as my group/force/location garbage collection system, the result can easily cause a desync.

To try and get around this they removed user control over the garbage collection and tried to make it more deterministic so it should just work. The simplest way to do this would be to periodically run a full garbage collection cycle every some number of frames. However this could manifest as a significant stutter, especially for larger heap sizes as both mark and sweep must be completed instantly. They likely tried to implement some more intelligent solution such that marking still occurs incrementally but sweeping happens in a synchronized way.

They must have made it so that periodically some exact amount of mark work gets done on all clients. Once marking is complete then it runs a full sweep garbage collection cycle on the same tick for all clients. However this now has imposed an artificial limit on how fast you can allocate to the heap with Lua, since if you allocate faster than the rate of marking then the mark cycle will never complete and so the sweep stage can never occur, effectively preventing the garbage collector from ever being run. Or at least that is my theory.

Once the GC hits its not guaranteed to clear the actual ram.

This is the case for most garbage collectors. It might decide to keep the virtual memory allocated for re-use as part of the heap in future as allocating memory is quite resource intensive due to involving OS kernal calls.

 

[ScrewTheTrees]

This is the case for most garbage collectors. It might decide to keep the virtual memory allocated for re-use as part of the heap in future as allocating memory is quite resource intensive due to involving OS kernal calls.

Thats a lot of interesting information.
Ive noticed that __gc actually does trigger and some things do run, but what i mean with clear the actual ram (bad wording) is that it doesnt seem to reuse the ram, both me and Boar have problems with our maps that the Ram usage keeps climbing forever which makes having a map run for a long time impossible unless you have 16gb of ram or higher.

This happens even if the __gc metamethod is called and memory is "seemingly" cleared.

 

[Dr Super Good]

Ive noticed that __gc actually does trigger and some things do run, but what i mean with clear the actual ram (bad wording) is that it doesnt seem to reuse the ram, both me and Boar have problems with our maps that the Ram usage keeps climbing forever which makes having a map run for a long time impossible unless you have 16gb of ram or higher.

This would certainly be a leak of sorts. Possibly with the memory allocated for the heaps.

This certainly did not used to happen. When manual garbage collection was permitted I tested it and even after leaking several gigabytes of data it would drop the WC3 virtual address size down to something reasonable as would be expected.