1. Updated Resource Submission Rules: All model & skin resource submissions must now include an in-game screenshot. This is to help speed up the moderation process and to show how the model and/or texture looks like from the in-game camera.
    Dismiss Notice
  2. DID YOU KNOW - That you can unlock new rank icons by posting on the forums or winning contests? Click here to customize your rank or read our User Rank Policy to see a list of ranks that you can unlock. Have you won a contest and still haven't received your rank award? Then please contact the administration.
    Dismiss Notice
  3. Rubbed the right way, the genie is out of its lamp! The 12th Concept Art Contest Results have been announced.
    Dismiss Notice
  4. Race against the odds and Reforge, Don't Refund. The 14th Techtree Contest has begun!
    Dismiss Notice
  5. Check out the Staff job openings thread.
    Dismiss Notice
Dismiss Notice
60,000 passwords have been reset on July 8, 2019. If you cannot login, read this.

[Lua]Obliterate all GUI leaks with 1 trigger!

Discussion in 'The Lab' started by Dr Super Good, Jun 5, 2019.

  1. Bribe

    Bribe

    Joined:
    Sep 26, 2009
    Messages:
    8,154
    Resources:
    25
    Maps:
    3
    Spells:
    10
    Tutorials:
    3
    JASS:
    9
    Resources:
    25
    Export your triggers and re-import them. Also make sure there is still no custom script in your code prior to saving.

    Edit: @Dr Super Good is this also including all the Blizzard.j functions that create these things?
     
  2. Pyrogasm

    Pyrogasm

    Joined:
    Feb 27, 2007
    Messages:
    3,788
    Resources:
    1
    Spells:
    1
    Resources:
    1
    Yes as long as they use something on the list of functions this Lua method is watching for, which they all do.
     
  3. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,926
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    Yes it does. Whenever Lua tries to call those functions it uses the proxies which register the object.

    If someone could confirm it is network safe then it is the ultimate solution for GUI coders. The infrequency of the garbage collector being run can be solved by a trigger that explicitly executes the garbage collector every 10 or 60 seconds.
     
  4. Tasyen

    Tasyen

    Joined:
    Jul 18, 2010
    Messages:
    1,482
    Resources:
    18
    Tools:
    2
    Maps:
    3
    Spells:
    8
    Tutorials:
    4
    JASS:
    1
    Resources:
    18
    The demo map disconnects for me. I tested it with 2 applications on the same machine playing over lan. After something like 8k where catched (currently alive), it start to free some. When the free count reached 2 the not host got a dc. 2 was the number shown for the not host when he dc. I spammed pressing esc with the host.

    I added another ESC event for player(1) and changed blue to a user player.
     
  5. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,926
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    This would mean that the garbage collector running is not synchronized. For some reason Warcraft III must be check summing the existence of locations to determine if clients go out of sync.

    All hope might not be lost yet. Here is another test which disables the standard garbage collection mechanics and instead explicitly executes the garbage collector every 10 game seconds. This execution of garbage collection should be network safe and due to the usage of all objects being the same between clients the objects collected should also be the same between all clients.

    However the order the objects are collected might not be the same. I would hope this does not make a difference since the same object set should exist between all clients however only testing could tell.

    Also I am aware of the negative aspects of this solution. Specifically it will perform a full garbage collection sweep instantly every 10 seconds. With very complex or busy maps this could potentially result in stutter every time the garbage collector runs. This normally does not happen because Lua uses an incremental garbage collector which will run over a number of frames.

    EDIT: It appears that this approach might work for multiplayer.
     

    Attached Files:

    Last edited: Jun 30, 2019
  6. Bribe

    Bribe

    Joined:
    Sep 26, 2009
    Messages:
    8,154
    Resources:
    25
    Maps:
    3
    Spells:
    10
    Tutorials:
    3
    JASS:
    9
    Resources:
    25
    Dr SG would you be interested in please sharing the Lua code you're using in that map?
     
  7. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,926
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    Same code as before except with this garbage collector added...
    Code (Text):

    collectgarbage("stop")

    nsgc = {
    trigger = CreateTrigger(),
    func = collectgarbage}

    nsgc.event = TriggerRegisterTimerEvent(nsgc.trigger, 10, true)
    nsgc.action = TriggerAddAction(nsgc.trigger, nsgc.func)
     
    Basically it forces the garbage collector to run in lock step. The issue with this is that it disables incremental garbage collection so each collection cycle may freeze LUA for many frames with very complex maps.

    The reason I assign the event and action to variables is due to a bug in Warcraft III. If one does not do this the event and action are automatically garbage collected and so the system breaks and no further garbage collection occurs. This only applies to actions and events created in the Lua initialization thread, standard JASS style creation of events and actions work fine in Lua and do not need to be assigned to a variable.
     
  8. Bribe

    Bribe

    Joined:
    Sep 26, 2009
    Messages:
    8,154
    Resources:
    25
    Maps:
    3
    Spells:
    10
    Tutorials:
    3
    JASS:
    9
    Resources:
    25
    What I'm wondering in terms of reliability is to manually check when the variable loses its assignment. I read the garbage collector is able to track references to a variable, and that users have access to some kind of debug script to find the same, but I'm unsure what that script is.

    Another possibility is to just assume all locations, groups and forces should be removed and maybe require the user to set an inverse of "wantDestroy" like "wantKeepPosition = true" and just remove everything else after a 0.00 second timer. Obviously still be hooking (replacing) the problem triggers.
     
  9. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,926
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    As far as I am aware Lua does not use a reference count based system for garbage collection. Instead it uses the traditional approach of a mark and sweep garbage collector with support for an incremental operating mode. Fundamentally this works by iterating all nodes in the state graph that are reachable from root and marking them. After the entire state graph has been iterated any objects not marked are garbage (not referenced) and so are eligible for garbage collection. The advantage of this approach is minimal overhead with assignments and strong robustness against cyclic references.

    As such it is not possible to know when an object in Lua is no longer referenced other than applying a __gc metamethod to it and run a full garbage collection cycle.
    This kind of defeats the purpose of automatic garbage collection. If a programmer is doing that, they might as well just explicitly destroy the objects they create.

    The point of this garbage collector is to provide 2 services.
    • Easily fix up a very leaky map with literally 1 imported trigger. Very newbie friendly and might turn a map that quickly became unplayable due to leaks to being a good experience. The user does not even need to know anything about leaks for this to work as it is all automatic.
    • Provide instrumentation to detect leaks. Many people think their map is leak free but is it actually so? This is a very common question one sees on the forum or in chat rooms. When running in debug mode the garbage collector will count all leaks and incorrect destruction calls. If those numbers are very small then yes their map is leak free and any performance issues they have are caused by trigger complexity. If not then their map was not leak free and likely the leaks were causing the performance issues.
     
  10. Bribe

    Bribe

    Joined:
    Sep 26, 2009
    Messages:
    8,154
    Resources:
    25
    Maps:
    3
    Spells:
    10
    Tutorials:
    3
    JASS:
    9
    Resources:
    25
    I agree with all those points. I've made a few one-script Lua drop-ins that all serve to make life simpler for GUI users without them having to know Custom script. I look forward to building more, and hopefully someone can test your latest iteration and be sure it doesn't desync so as to ensure users have access to it safely.
     
  11. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,926
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    Someone did and they said it worked. The potential problem with it is how resource intensive a full garbage collection cycle could potentially be.
     
  12. Tasyen

    Tasyen

    Joined:
    Jul 18, 2010
    Messages:
    1,482
    Resources:
    18
    Tools:
    2
    Maps:
    3
    Spells:
    8
    Tutorials:
    4
    JASS:
    1
    Resources:
    18
    Recently, I tested the demo maps in local lan with 2 applications on 1 machine. The first version desync when the non host did try to garbage collect.
    The second version worked in lan. It reached over 178k freed leaks without a desync which takes more than 15 mins for the demo trigger.
    DSG GC Test cutted.jpg
     
    Last edited: Jul 10, 2019
  13. Bennoloth

    Bennoloth

    Joined:
    Apr 29, 2019
    Messages:
    18
    Resources:
    0
    Resources:
    0
    I tried this on my map, with the "Network Safe" script alongside it, and it stopped my orc waves from spawning. Technically, I suppose I had no unit leaks =)

    Uploaded the map in case you wanted to look at it. The scripts are there but disabled.
     

    Attached Files:

  14. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,926
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    The issue is caused by trigger nonsense. For example this trigger...
    • first wave
      • Events
        • Time - Elapsed game time is 45.00 seconds
      • Conditions
      • Actions
        • Unit - Create 10 Goblin for Player 12 (Brown) at (Center of Region 000 <gen>) facing (Position of (Triggering unit))
        • Unit Group - Pick every unit in (Units in Region 000 <gen> owned by Player 12 (Brown)) and do (Unit - Order (Picked unit) to Attack-Move To (Center of Region 001 <gen>))

    There is no Triggering unit in that context. Hence you are attempting to get the position of nil.

    Warcraft III normally handles this gracefully however a bug in my garbage collection script causes a Lua thread crash instead. Hence the units never are created as code never advances enough to create them. The thread crash was because I tried to wrap nil for garbage collection which makes no sense.

    I have attached a fixed version to the main post, GC Demo Network Safe 1.2.w3x. This version will correctly propagate nil returns from constructors and log them as bad constructor calls for debugging purposes. Such errors will now print a message in debug mode rather than crash the thread.
     
    Last edited: Aug 10, 2019
  15. Bennoloth

    Bennoloth

    Joined:
    Apr 29, 2019
    Messages:
    18
    Resources:
    0
    Resources:
    0
    Oh. Thanks! I didn't even realize I made that one a nonsense facing. Do you think, for any reason, it would be better to make it face "default building degrees" or whatever instead? Then have that leak cleaned instead.

    Since I'm going to be using this map as a published version - and I don't know code - I could guess you just remove the "print("Constructor call returned nil: " .. v)" line to prevent it from giving the error message, but I don't want to mess things up further. Is that accurate?
     
    Last edited: Aug 11, 2019
  16. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,926
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    It should be made default facing degrees as currently the trigger does not make any sense and is relying on fall back/default behaviour.

    As mentioned above the new version of the system, available on the first post, has fixed the thread crash issue. When operating in debug mode it will instead print a warning telling you that some nonsense has happened. These warnings are not produced in non-debug mode.
    You can disable debug mode to turn the warning messages off. The reason warning messages are produced is to notify map developers that they are doing something silly/nonsensical by mistake. For example they tried to create a location with invalid parameters resulting in nil being returned or that they tried to call RemoveLocation twice on the same location or on a nil location. These are technically harmless, but do not really make sense to allow to happen as it basically results in wasted cycles.

    In your case you would want the units to face some other location, or a default facing angle.
     
  17. Pasto

    Pasto

    Joined:
    Feb 10, 2016
    Messages:
    4
    Resources:
    0
    Resources:
    0
    There are other stuff that leak as well, for example Floating Text (called TextTag in Blizzard functions). As far as I understand how this works, simply adding:

    Code (Text):
    GC.TextTags = GC.Type:new("textTag", "DestroyTextTag",
        {"CreateTextTag"
        })
    should clean those as well. I'm not sure what else is leaking as well but few things that comes to mind is Destructible object groups, special effects and ground deformations.
     
  18. Dr Super Good

    Dr Super Good

    Spell Reviewer

    Joined:
    Jan 18, 2005
    Messages:
    25,926
    Resources:
    3
    Maps:
    1
    Spells:
    2
    Resources:
    3
    There is a limit to at most 100 of them, after which the last allocated one is recycled. Additionally they come with GUI accessible self destruct mechanics and they are easy to notice when leaking as they have a visual appearance. The self destruct mechanics are not compatible with this implementation.

    Floating text leaks have not and will never be a problem. At worst displaying them will break, which developers can easily notice and fix. They will not cause performance issues like group, force and location leaks do.
     
  19. Pasto

    Pasto

    Joined:
    Feb 10, 2016
    Messages:
    4
    Resources:
    0
    Resources:
    0
    That explains some weird behavior I had while testing a lot of texts :D For destructable groups, it seems that they are never created in the way as unit groups are, since all destructable functions that do something for multiple destructables return nothing.
    Special Effects and other stuff have GUI destructors, so thats why you don't need them in this system, got it :)
     
  20. Bribe

    Bribe

    Joined:
    Sep 26, 2009
    Messages:
    8,154
    Resources:
    25
    Maps:
    3
    Spells:
    10
    Tutorials:
    3
    JASS:
    9
    Resources:
    25
    There are no destructable groups, item groups, widget groups. There are item pools, unit groups and player forces.

    Other things which can be grouped are boolexprs and anything added to a single trigger.