1. Join other hivers in a friendly concept-art contest. The contestants have to create a genie coming out of its container. We wish you the best of luck!
    Dismiss Notice
  2. The Melee Mapping Contest #4: 2v2 - Results are out! Step by to congratulate the winners!
    Dismiss Notice
  3. We're hosting the 15th Mini-Mapping Contest with YouTuber Abelhawk! The contestants are to create a custom map that uses the hidden content within Warcraft 3 or is inspired by any of the many secrets within the game.
    Dismiss Notice
  4. The 20th iteration of the Terraining Contest is upon us! Join and create exquisite Water Structures for it.
    Dismiss Notice
  5. Check out the Staff job openings thread.
    Dismiss Notice

[Lua]Obliterate all GUI leaks with 1 trigger!

Discussion in 'The Lab' started by Dr Super Good, Jun 5, 2019.

  1. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,271
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    Imagine a world where GUI triggers do not need a mess of temporary variables to hold locations. Or a world where the annoying custom script lines like "RemoveLocation" do not exist. A world where we could leak as much as we want without any performance consequences.

    Well imagine no more GUI users. Thanks to the power of patch 1.31 with Lua that world is real!

    Behold a single Lua trigger solution which will fix all force, group and location leaks. No custom script calls required, it works fully automatically as if by witchcraft!
    Code (Text):
    GC = {}
    GC.Table = {}
    setmetatable(GC.Table, {__mode = "k"})
    GC.Native = {}
    GC.Type = {}
    GC.Type.__index = GC.Type
    GC.Debug = true

    function GC.Type:new(name, remove, constructors)
        local type = {}
        setmetatable(type, self)

        type.name = name
        type.remove = remove
        type.constructors = constructors
        type:register()
        return type
    end

    function GC.Type:printstats()
        if self.stats then
            print("Statistics for " .. self.name .. ":")
            print("Automatically freed: " .. self.stats.autofree)
            print("Explicitly freed: " .. self.stats.explicitfree)
            print("Currently alive: " .. self.stats.alive)
            print("Bad free calls: " .. self.stats.badfree)
            print("Free calls on untracked: " .. self.stats.unknownfree)
        else
            print("Can only print statistics when GC is running in debug mode.")
        end
    end

    function GC.Type:register()
        local debug = GC.Debug

        if not GC.Native[self.remove] then
            GC.Native[self.remove] = _ENV[self.remove]
        end

        local removefunc = GC.Native[self.remove]

        if debug then
            if not GC.FreedTable then
                GC.FreedTable = {}
                setmetatable(GC.FreedTable, {__mode = "k"})
            end

            self.stats = {
                autofree = 0,
                explicitfree = 0,
                alive = 0,
                badfree = 0,
                unknownfree = 0
            }
        end

        local collectfunc
        if not debug then
            collectfunc = function(obj)
                removefunc(obj[1])
            end
        else
            collectfunc = function(obj)
                removefunc(obj[1])
                self.stats.autofree = self.stats.autofree + 1
                self.stats.alive = self.stats.alive - 1
            end
        end

        local collectmetatable = {
            __gc = collectfunc
        }

        if not debug then
            _ENV[self.remove] = function(obj)
                local table = GC.Table[obj]
                if table then
                    removefunc(obj)
                    setmetatable(table, nil)
                    GC.Table[obj] = nil
                end
            end
        else
            _ENV[self.remove] = function(obj)
                local table = GC.Table[obj]
                if table then
                    removefunc(obj)
                    setmetatable(table, nil)
                    GC.Table[obj] = nil
                    GC.FreedTable[obj] = true
                    self.stats.explicitfree = self.stats.explicitfree + 1
                    self.stats.alive = self.stats.alive - 1
                else
                    if not obj or GC.FreedTable[obj] then
                        if obj then
                            print("Tried to free an already freed object: " .. obj)
                        else
                            print("Tried to free nil of type: " .. self.name)
                        end
                     
                        self.stats.badfree = self.stats.badfree + 1
                    else
                        self.stats.unknownfree = self.stats.unknownfree + 1
                    end

                end
            end
        end

        local registerfunc
        if not debug then
            registerfunc = function(obj)
                local table = {obj}
                setmetatable(table, collectmetatable)
                GC.Table[obj] = table
                return obj
            end
        else
            registerfunc = function(obj)
                local table = {obj}
                setmetatable(table, collectmetatable)
                GC.Table[obj] = table
                self.stats.alive = self.stats.alive + 1
                return obj
            end
        end

        for i, v in ipairs(self.constructors) do
            if not GC.Native[v] then
                GC.Native[v] = _ENV[v]
            end

            local constructorfunc = GC.Native[v]

            _ENV[v] = function(...)        
                return registerfunc(constructorfunc(...))
            end
        end
    end

    GC.Location = GC.Type:new("location", "RemoveLocation",
        {"BlzGetTriggerPlayerMousePosition",
            "CameraSetupGetDestPositionLoc",
            "GetCameraEyePositionLoc",
            "GetCameraTargetPositionLoc",
            "GetOrderPointLoc",
            "GetSpellTargetLoc",
            "GetStartLocationLoc",
            "GetUnitLoc",
            "GetUnitRallyPoint",
            "Location"
        })

    GC.Group = GC.Type:new("group", "DestroyGroup",
        {"CreateGroup"
        })

    GC.Force = GC.Type:new("force", "DestroyForce",
        {"CreateForce"
        })
    Attached is a demo map to show its usage, and allow one to experiment with it. Press Esc to get technical information such as how many leaks were caught.

    Proof that it works is easy. Duplicate the actions of the Leaky Demo Trigger 20 or 40 times and test the map. Thanks to the Lua based garbage collector the Warcraft III application memory usage will not really increase. Now if one disables the Garbage Collector trigger and tests then the Warcraft III application memory usage will increase at the rate of 10s of megabytes per second.

    From a technical point of view it works by replacing all references to constructor and destructor functions for force, group and location within the Lua virtual machine to proxies. These proxies register or deregister an object for automatic garbage collection. Automatic garbage collection is implemented using the properties of a weak keyed map to map an object to a table holding the object which uses the garbage collected metamethod to destroy the object it wraps. When all references to a location are lost, the properties of the weak keyed map allow the wrapping table to be garbage collected which then runs the clean up function.

    It is worth noting that this approach only works if the Lua garbage collector ever runs. Since the object memory itself is not factored in as part of the Lua heap size it is possible that the default Lua garbage collector settings do not run the collector frequently enough to prevent leak related performance issues. To prevent this one might have to fine tune the Lua garbage collector settings to encourage it to run more frequently. This is done in the demo map in the Lua Test trigger. In theory this could be tuned on a per map basis as required.

    WARNING: This has not been tested in multiplayer. It might potentially be a cause of out of sync errors. It will work for single player. If people could test it in multiplayer to see if it causes out of sync or not it would be very helpful.
     

    Attached Files:

    Last edited: Jun 5, 2019
  2. Devalut

    Devalut

    Joined:
    Feb 9, 2009
    Messages:
    737
    Resources:
    2
    Spells:
    2
    Resources:
    2
    Sweet Merciful...
     
  3. pOke

    pOke

    Joined:
    Mar 24, 2013
    Messages:
    1,079
    Resources:
    1
    Maps:
    1
    Resources:
    1
    I don't understand lua or what is happening in the code.

    BUT..the concept sounds nice, and hopefully with the likely new crop of GUI coders coming soon with reforged this will be a nice tool.
     
  4. JFA

    JFA

    Joined:
    Jun 2, 2009
    Messages:
    400
    Resources:
    3
    Maps:
    3
    Resources:
    3
    Sounds amazing. But is someone tested in multiplayer?
     
  5. Fasolace

    Fasolace

    Joined:
    Sep 14, 2009
    Messages:
    240
    Resources:
    6
    Skins:
    1
    Template:
    5
    Resources:
    6
    Is the TypeId2Integer function necessary? Can you not access UnitIds with single quotes in Lua like you do in JASS?

    Code (Text):
    CreateUnit(Player(0), TypeId2Integer("Hamg"), 0, 0, 0)
    vs
    Code (Text):
    CreateUnit(Player(0), 'Hamg', 0, 0, 0
     
  6. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,271
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    No Lua does not support that. Read every Lua tutorial every written and you will see that such syntax does not exist in Lua. JASS only supports it because it was specifically made for Warcraft III rather than Lua which is a general programming/scripting language.

    Also that is part of the test script so not required for the system to work.
     
  7. Fasolace

    Fasolace

    Joined:
    Sep 14, 2009
    Messages:
    240
    Resources:
    6
    Skins:
    1
    Template:
    5
    Resources:
    6
    Interesting. So to access any object by it's Id value with Lua, we would need to create a function like the one you wrote?

    Is there any easier way to do it without using shifts and other bit operations?
     
  8. Tasyen

    Tasyen

    Joined:
    Jul 18, 2010
    Messages:
    1,144
    Resources:
    13
    Tools:
    2
    Maps:
    2
    Spells:
    7
    Tutorials:
    1
    JASS:
    1
    Resources:
    13
    GUI Lua code uses FourCC("hfoo") to convert unitIds.
     
  9. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,271
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    It is only needed for object types to go from human readable raw code to integer. All object types returned by functions are already integers so can be passed around directly just like with JASS.

    There is also a way to decode directly using built in Lua functions (not shown in the map). Additionally I think with 1.31.1 there are now natives to perform the conversion from string as well.
    Yes you can use the Lua standard library encode and decode mechanisim. This may or may not be faster. There might also be natives to do such a conversion.
    This was added after the PTR, hence why the code above did not use it as it was test code left in from the PTR in the map I developed this system.
     
  10. Yousef

    Yousef

    Joined:
    Jul 19, 2011
    Messages:
    505
    Resources:
    0
    Resources:
    0
    Hey, thanks a lot for your work, but I'm having a problem, when I try to save it gives me a syntax error with a message saying "missing endblock", any idea how to fix this?
    Untitled.jpg
     
    Last edited: Jun 14, 2019 at 8:18 PM
  11. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,271
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    Try disabling JASSHelper. Apparently there are compatibility issues with it and maps running in Lua mode. For some reason...

    If importing into your own map. Make sure the map is operating in Lua mode. This is an Lua only fix for using the Lua virtual machine. The JASS2 virtual machine does not allow one to perform such a fix.

    One can still use GUI and JASS (JASS2) with Lua thanks to a transpiler on save. Just for some reason it does not work well with vJASS.
     
    Last edited: Jun 14, 2019 at 9:00 PM
  12. Yousef

    Yousef

    Joined:
    Jul 19, 2011
    Messages:
    505
    Resources:
    0
    Resources:
    0
    Well now I got a whole bunch of other errors about the lua code after I disabled vJass.
    Untitled.jpg
    With vJass it works on other maps without missing endblock error.
     
    Last edited: Jun 14, 2019 at 11:25 PM
  13. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,271
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    Did you set the map to use Lua? One must explicitly specify the virtual machine used and all older maps will default to JASS2.
     
  14. Yousef

    Yousef

    Joined:
    Jul 19, 2011
    Messages:
    505
    Resources:
    0
    Resources:
    0
    I think it's already activated, can you tell me how to activate it in case it's not activated?
     
  15. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,271
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    Scenario -> Map Options... -> Script Language:

    From the pulldown select Lua.
     
  16. Marshmalo

    Marshmalo

    Joined:
    Jul 1, 2008
    Messages:
    970
    Resources:
    4
    Maps:
    4
    Resources:
    4
    Hi Dr Super good,

    If it works in multiplayer this looks like a great tool! Me and Yousef are working on a popular Bnet map: Lordaeron the Foremath (LTF), however since latest Blizzard patches the game becomes very unstable after 1 hour game play. As this is an epic RTS game sometimes the battles can rage on for up to 2 hours so stability is paramount to the map's enjoyment, we think this Lua trash collector could really help us.

    However, we already use a lot of Jass custom scripts in the game, this is usually for stuff like cleaning up location variables and other simple functions. If I understand correctly we cna remove these if we are using your clean script as they will become redundant?

    Unfortunately we do also use jass custom scripts for other functions, such as making sounds only play to single player, or setting filters for a specific player. Examples below, can you advise on how we can replace these jass scripts with Lua versions? Neither me nor Yousef are trigger experts.

    We should then be able to convert the map script to Lua and try out your script. Annoyingly there are quite a lot of triggers on this map, its huge so it will be no easy task for us!

    Custom Jass script:
    if GetLocalPlayer() == Player(4) then
     
  17. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,271
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    You could. However there is also no harm keeping them in as the system will work around them to stop leaks that might otherwise have been missed.

    That said in order for the system to work efficiently one might have to tune how often the garbage collector is run, as I did in the demo map. When the Lua garbage collector runs it will remove all leaks, but the rate at which that happens might not be optimal with the default garbage collector settings. In the demo map I changed the settings, but one could also explicitly run the garbage collector, eg every 15 or 60 seconds.
    The built in JASS2toLua transpiler should handle all that. GUI still compiles to JASS on map save but when the map is in Lua mode it will convert the resulting JASS script to Lua using the built in transpiler.
     
  18. Marshmalo

    Marshmalo

    Joined:
    Jul 1, 2008
    Messages:
    970
    Resources:
    4
    Maps:
    4
    Resources:
    4
    I think once very 60 seconds would be perfect, the map is huge so it really depends how much extra stress the cleanup script causes on the game when it runs.

    If the build in transpiler can do all the converting that would make it so much easier, there's ALOT of triggers. However, the problem I'm finding is that I don't even get the option to convert the script. Its greyed out:

    Lua Disabled.jpg
     
  19. Tasyen

    Tasyen

    Joined:
    Jul 18, 2010
    Messages:
    1,144
    Resources:
    13
    Tools:
    2
    Maps:
    2
    Spells:
    7
    Tutorials:
    1
    JASS:
    1
    Resources:
    13
    if your map contains one line of hand written code you are not allowed to swap language mode. From offical patch-notes.
    • Maps save in the selected language
    • Maps that are written only in standard libraries can switch to Lua
    • Maps that use custom script will have to disable all custom triggers before conversion
    Disable all triggers with custom script then you should be able to swap to Lua.