• 🏆 Texturing Contest #33 is OPEN! Contestants must re-texture a SD unit model found in-game (Warcraft 3 Classic), recreating the unit into a peaceful NPC version. 🔗Click here to enter!
  • 🏆 Hive's 6th HD Modeling Contest: Mechanical is now open! Design and model a mechanical creature, mechanized animal, a futuristic robotic being, or anything else your imagination can tinker with! 📅 Submissions close on June 30, 2024. Don't miss this opportunity to let your creativity shine! Enter now and show us your mechanical masterpiece! 🔗 Click here to enter!

SyncedTable

This bundle is marked as high quality. It exceeds standards and is highly desirable.
  • Like
Reactions: deepstrasz

Overview

Code

Restrictions

Changelog

Overview

A Lua-table with multiplayer-safe pairs()-iteration.

Lua:
-- Create a new SyncedTable. After creation, use it like any other Lua table.
PlayerColors = SyncedTable.create()

-- In this example, we add color codes for Player(0) to Player(2) to the table.
PlayerColors[Player(0)] = "FF0303"
PlayerColors[Player(1)] = "0042FF"
PlayerColors[Player(2)] = "1CE6B9"

-- Loop order will be the same for all clients in a multiplayer game (although it wouldn't matter in this particular example)
for player, color in pairs(PlayerColors) do
    print("|cff" .. color .. GetPlayerName(player) .. "|r")
end

--Check, if a given table is a SyncedTable or not
print( SyncedTable.isSyncedTable( PlayerColors) ) --> prints "true"

Description

In general, iterating through a Lua-table via pairs() is not multiplayer-safe, because iteration order is not guaranteed to be the same for different clients. Thus, manipulating game state inside a pairs()-loop can lead to a desync.

The SyncedTable-library allows for creating tables, where the pairs()-iteration is guaranteed to have the same order for every client in a multiplayer game, no matter which keys you use.
Apart from the synchronized pairs-iteration, SyncedTables behave just like normal tables. They can contain any (key, value)-pair, you can write table[key] = nil to remove an element, table[key] = value to add or change an element, and you can use it in any function that takes a table, such as table.sort() or ipairs().

You can find the pseudo-solution orderedPairs on lua-users.org. It basically sorts all keys in the given table before every loop to make it synchronous. That "solution" doesn't only add much overhead to every loop (because it sorts before every loop), but it also doesn't work on tables using a keyset that can't be entirely sorted via the "<"-relation.
That means, it doesn't work for tables with mixed keys (e.g. {[1] = true, x = true}) and it also doesn't work, if you use keys other than strings and numbers. Unfortunately, this happens quite quickly in Wc3, when we use units, players, etc. as keys for a table.
Yes, the same website offers another multitype-compatible orderedKeys option, but that only works in singleplayer (where it guarantees to have the same iteration order in every iteration), but not in multiplayer (where it still desyncs, because the tostring()-function yields different results for different players).

Note that iterating over a SyncedTable as well as adding and removing elements is pretty fast, but the implementation adds overhead to table creation instead. Thus, only use a SyncedTable, if you actually intend to iterate over it.

When using a SyncedTable, the restrictions listed in the restrictions tab apply.

Alternative Constructors

Lua:
--Creates a new empty SyncedTable
local t = SyncedTable.create()

--Same as SyncedTable.create()
local t2 = SyncedTable()

--Creates a new SyncedTable and adds all elements of the specified existing table into it.
local t3 = SyncedTable.create({'a', 'b', 'c', a = true, b = true, c = false})

Note that passing an existing table requires the SyncedTable to sort all table keys once to ensure initial synchronicity, using the standard <-relation. Hence, all existing keys must be <-comparable, which is only true for strings and numbers (and in theory any object having the __lt-metamethod defined).
This restriction only holds for the table passed upon SyncedTable construction.
You can add any key to a SyncedTable after creation and it will work just fine.

Installation

Just copy the code from the "Code" tab into your map and you are ready to go.

Lua:
if Debug and Debug.beginFile then Debug.beginFile("SyncedTable") end
--[[

---------------------------
-- | SyncedTable v1.1a | --
---------------------------

 by Eikonium

 --> https://www.hiveworkshop.com/threads/syncedtable.353715/

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| A Lua table with a multiplayer-safe pairs()-iteration.
| Syncedtables achieve this safe pairs() by looping in the same order that elements got inserted into it (can change upon element removal), which is deterministic and identical across clients.
| You can do everything with a SyncedTable that you could do with a normal Lua table, with the few restrictions listed below.
| SyncedTables are pretty quick on adding and removing elements as well as looping over them, but they add some overhead to table creation. Thus, only create a SyncedTable instead of a normal one, if you intend to loop over it via pairs() in multiplayer.
|
| -------
| | API |
| -------
|    SyncedTable.create([existingTable]) --> SyncedTable
|        - Creates a new SyncedTable, i.e. a Lua-table with a multiplayer-synchronized pairs()-iteration.
|        - Specifying an existing table will add all of its (key,value)-pairs to the new SyncedTable. In this case, SyncedTable will derive its initial loop order from it by sorting all keys by datatype (primarily) and by normal <-relation (secondarily).
|          Hence, creation will throw an error, if existingTable contains keys of the same type that can not be <-sorted.
|        - Example:
|           local PlayerColors = SyncedTable.create()
|           PlayerColors[Player(0)] = "FF0303"
|           PlayerColors[Player(1)] = "0042FF"
|           PlayerColors[Player(2)] = "1CE6B9"
|           for player, color in pairs(PlayerColors) do -- loop order will be the same for all clients in a multiplayer game (not that it matters in this particular example)
|               print("|cff" .. color .. GetPlayerName(player) .. "|r")
|           end
|    SyncedTable.isSyncedTable(table) --> boolean
|        - Returns true, if the specified table is a SyncedTable, and false otherwise.
| ----------------
| | Restrictions |
| ----------------
|        - Detaching a SyncedTable's metatable or overwriting it's __index, __newindex or __pairs metamethods will stop it from working. You can however add any other metamethod to it's existing metatable.
|        - You can't use next(S)==nil on a SyncedTable S to check, whether it is empty or not. You can however use pairs(S)(S)==nil to check the same (although this isn't super performant).
--]]-----------------------------------------------------------------------------------------------------------------------------------------------------------------------

-- disable sumneko extension warnings for imported resource
---@diagnostic disable

do
    --comparison function that allows sorting a set of objects that already have a natural order.
    local function comparisonFunc(a,b)
        local t1,t2 = type(a), type(b)
        if t1 == t2 then
            return a<b
        end
        return t1 < t2
    end

    --Help data structures for SyncedTable loops that memorize the current table key (and it's array position) that is looped over.
    local loopKeys = {} ---@type table<integer,any> memorizes the current loop key for a given loopId.
    local loopCounters = {} ---@type table<integer,integer> memorizes the array position of the current loop key for a given loopId.
    local loopId = 0 -- an Id unique per loop currently running on any SyncedTable. Recycled after a loop has finished running. Counts upwards.

    ---@class SyncedTable : table
    SyncedTable = {}

    ---Creates a table with a multiplayer-synchronized pairs-function, i.e. you can iterate over it via pairs(table) without fearing desyncs.
    ---After creation, you can use it like any other table.
    ---The implementation adds overhead to creating the table, adding and removing elements, but keeps the loop itself very performant. So you should only used syncedTables, if you plan to iterate over it.
    ---You are both allowed to add and remove elements during a pairs()-loop.
    ---Specifying an existing table as input parameter will add its elements to the new SyncedTable. This only works for input tables, where all keys are sortable via the "<"-relation, i.e. numbers, strings and objects listening to some __lt-metamethod.
    ---@param existingTable? table any lua table, whose elements you want to add to the new SyncedTable. The table is required to only contain keys that can be sorted via the '<'-relation. E.g. you might write SyncedTable.create{x = 10, y = 3}.
    ---@return SyncedTable
    function SyncedTable.create(existingTable)
        local new = {}
        local metatable = {class = SyncedTable}
        local data = {}
        --orderedKeys and keyToIndex don't need to be weak tables. They reference keys if and only if those keys are used in data.
        local orderedKeys = {} --array of all keys, defining loop order.
        local keyToIndex = {} --mirrored orderedKeys, i.e. keyToIndex[key] = int <=> orderedKeys[int] = key. This is used to speed up the process of removing (key, value)-pairs from the syncedTable (to prevent array search in orderendKeys).
        local numKeys = 0

        --If existingTable was provided, register all keys from the existing table to the keyToIndex and orderedKeys help tables.
        if existingTable then
            --prepare orderedKeys array by sorting all existing keys
            for k,v in pairs(existingTable) do
                numKeys = numKeys + 1
                orderedKeys[numKeys] = k --> the resulting orderedKeys is asynchronous at this point
                data[k] = v
            end
            table.sort(orderedKeys, comparisonFunc) --result is synchronous for all players
            --fill keyToIndex accordingly
            for i = 1, numKeys do
                keyToIndex[orderedKeys[i]] = i
            end
        end

        --Catch read action
        metatable.__index = function(t, key)
            return data[key]
        end

        --Catch write action
        metatable.__newindex = function(t, key, value)
            --Case 1: User tries to remove an existing (key,value) pair by writing table[key] = nil.
            if data[key]~=nil and value == nil then
                --swap last element to the slot being removed (in the iteration order array)
                local i = keyToIndex[key] --slot of the key, which is getting removed
                keyToIndex[orderedKeys[numKeys]] = i --first set last slot to i
                keyToIndex[key] = nil --afterwards nil current key (has to be afterwards, when i == numKeys)
                orderedKeys[i] = orderedKeys[numKeys] --i refers to the old keyToIndex[key]
                orderedKeys[numKeys] = nil
                numKeys = numKeys - 1
            --Case 2: User tries to add a new key to the table (i.e. table[key] doesn't yet exist and both key and value are not nil)
            elseif data[key]==nil and key ~= nil and value ~= nil then
                numKeys = numKeys + 1
                keyToIndex[key] = numKeys
                orderedKeys[numKeys] = key
            end
            --Case 3: User tries to change an existing key to a different non-nil value (i.e. table[existingKey] = value ~= nil)
            -- -> no action necessary apart from the all cases line
            --Case 4: User tries to set table[nil]=value or table[key]=nil for a non-existent key (would be case 1 for an existent key)
            -- -> don't do anything.
            --In all cases, do the following:
            data[key] = value --doesn't have any effect for case 4.
        end

        --- Iterator function that is used the retreive the next loop element within a SyncedTable loop.
        --- This iterator is identical for every loop on the same SyncedTable. Different loops are distinguished by different loopIds.
        --- The iteration loops through orderedKeys[] in ascending order, saving the current position and key externally.
        ---@param state integer loopId to identify the loop to fetch the next element for.
        ---@return any key, any value
        local function iterator(state)
            if loopKeys[state] == orderedKeys[loopCounters[state]] then --check, if the last iterated key is still in place. If not, it has been removed in the last part of the iteration.
                loopCounters[state] = loopCounters[state] + 1 --only increase i, when the last iterated key is still part of the table. Otherwise use the same i again. This allows the removal of (key,value)-pairs inside the pairs()-iteration.
            end
            local i = loopCounters[state]
            loopKeys[state] = orderedKeys[i]
            return orderedKeys[i], data[orderedKeys[i]] -- (key,value)
        end

        --- Metamethod to define the pairs-loop for a SyncedTable. Runs every time a new loop is initiated.
        --- Fetches a unique loopId and returns the above iterator together with this id.
        ---@param t SyncedTable
        ---@return function iterator
        ---@return integer state loopId of the new loop
        metatable.__pairs = function(t)
            --if the current maximum loopId is still in use (the loopKey of that loop not being nil implies that the loop is still running), increase that loopId by one.
            while loopKeys[loopId] ~= nil do --it's near to impossible that this while-loop runs more than once, because the next higher loopId is entirely new, if not reached again after arithmetic overflow, which requires around 4 billion loops to run. Theoretically achievable, if current loop is set on hold via coroutine.
                loopId = loopId + 1 -- resistent against arithmetic overflow, because math.maxinteger + 1 yields math.mininteger, from where loopId would simply continue to count upwards.
            end
            local state = loopId
            loopCounters[state] = 1
            return iterator, state, nil
        end

        setmetatable(new, metatable)
        return new
    end

    ---Returns true, if the input argument is a SyncedTable, and false otherwise.
    ---@param anyObject any
    ---@return boolean isSyncedTable
    SyncedTable.isSyncedTable = function(anyObject)
        local metatable = getmetatable(anyObject)
        return metatable and metatable['class'] == SyncedTable
    end

    --Allows writing SyncedTable() instead of SyncedTable.create().
    setmetatable(SyncedTable, {__call = function(func, t)
        return SyncedTable.create(t)
    end})
end

if Debug and Debug.endFile then Debug.endFile() end

The following minor limitations apply, when using SyncedTables:
  1. Setting a new metatable for any SyncedTable will stop it from working. You are however allowed to edit any SyncedTable's existing metatable, as long as you don't change __index, __newindex and __pairs.
  2. Removing elements from a SyncedTable inside a pairs()-loop via table[key] = nil is allowed as long as you only remove the current loopElement.
    Lua:
    --Let t be a SyncedTable with any elements.
    --Allowed:
    for k,v in pairs(t) do
        t[k] = nil
    end
    
    --Not Allowed:
    for k,v in pairs(t) do
        t[anyKey] = nil --where anyKey ~= k
    end
  3. Adding elements to a SyncedTable inside a pairs()-loop is allowed, but new elements will also be looped over. This might lead to an endless loop, if done poorly. Remember that the same rule holds for normal tables, so this is not really a restriction.
    Lua:
    --The following code will lead to an endless loop (well, it will maybe end at max integer range)
    t = SyncedTable{1}
    for k,v in pairs(t) do
        t[k+1] = true
    end
  4. Creating a SyncedTable from an existing table via <varName> = SyncedTable.create(existingTable) only works, if all keys in existingTable are either numbers or strings (or have the __lt-metamethod defined in their metatable). Keys of any other type have to be added after creation.
  5. You can't use next(S) on a SyncedTable S to check, whether or not it is empty. You can however use the iterator returned by pairs to achieve the same result. A SyncedTable is empty, if pairs(S)(S) == nil (yes, the double brackets look confusing, but it's absolutely fine this way. Note that for a normal table, pairs(T) just returns next, so the check next(T) == nil is equivalent to pairs(T)(T) == nil anyway).

30.05.2021, v1.0:
  • Initial Release
11.05.2024, v1.1:
  • No longer creates an iterator function upon every pairs()-loop, but instead creates a fixed iterator upon SyncedTable creation that uses global indexing to distinguish different loops running at the same time. This puts less pressure on the garbage collector.
14.05.2024, v1.1a:
  • Fixed a bug introduced with version 1.1, where using false as a key in a SyncedTable would break a loop early, if another loop was nested upon iterating this exact key.
Contents

SyncedTable (Binary)

Reviews
Wrda
I believe there could be a small improvement on the pairs metamethod: --Define, how the pairs iteration works metatable.__pairs = function(t) local i = 0 local latestKey return function() if latestKey == orderedKeys[i] then...

Wrda

Spell Reviewer
Level 26
Joined
Nov 18, 2012
Messages
1,906
I believe there could be a small improvement on the pairs metamethod:

Lua:
--Define, how the pairs iteration works
metatable.__pairs = function(t)
    local i = 0
    local latestKey
    return function()
        if latestKey == orderedKeys[i] then
            i = i+1 --only increase i, when the last iterated key is still part of the table. Otherwise use the same i again. This allows the removal of (key,value)-pairs inside the pairs()-iteration.
        end
        latestKey = orderedKeys[i]
        return orderedKeys[i], data[orderedKeys[i]]
    end, t, nil
end

By taking a derived form of maddeem's advice. Since it gets rid of the creation of anonymous function per loop. NewTable and ReleaseTable belongs to a table recycler script he uses. Perhaps that could be integrated too.
If you ever decide to recycle tables here is a stateless elements function that doesn't create a function per loop:

Lua:
    local function setIterator(state)
        local i = state.i
        if state.lastKey == state.orderedKeys[i] then
            i = i + 1
            state.i = i
        end
        state.lastKey = state.orderedKeys[i]
        if not state.lastKey then
            state.orderedKeys = nil
            state.i = nil
            ReleaseTable(state)
        end
        return state.lastKey
    end

    function Set:elements()
        local state = NewTable()
        state.i = 0
        state.orderedKeys = self.orderedKeys
        return setIterator, state
    end
Creating such objects are only a problem when there's a lot of demand going on, and the garbage collector probably can't keep up, so this isn't a big deal in most cases.
Nonetheless, this resource is beautiful, and extremely useful for all Lua users, avoiding a lot of headaches.

Approved
 
I believe there could be a small improvement on the pairs metamethod:
That's true. Although creating an iterator function per loop is standard in Lua, object creation should be spared where not necessary, especially in Wc3. I think I'd go for an approach that doesn't ever create tables or functions, but just writes necessary loop-data into a pre-existing array. This would even spare table recycling and should be the most efficient way.

Untested example code:
Lua:
--Idea: allocate one unique loopId to each started SyncedTable-loop. Use that id to save / retreive all necessary data in each step of every loop.
local loopCounters, loopKeys = {}, {}
local loopId = 0 --unique to each loop

local function iterator(state)
    if loopKeys[state] == orderedKeys[loopCounters[state]] then
        loopCounters[state] = loopCounters[state] + 1 --only increase i, when the last iterated key is still part of the table. Otherwise use the same i again. This allows the removal of (key,value)-pairs inside the pairs()-iteration.
    end
    local i = loopCounters[state]
    loopKeys[state] = orderedKeys[i]
    return orderedKeys[i], data[orderedKeys[i]]
end

metatable.__pairs = function(t)
    while loopKeys[loopId] do --loopKeys[loopId] will be nil for finished loops. we could theoretically spare this check, because loopId would realistically never reach an existing loop anyway.
        loopId = loopId + 1
    end
    local state = loopId
    loopCounters[state] = 0
    return iterator, state, nil
end

I will probably find time to incorporate this next weekend.

Nonetheless, this resource is beautiful, and extremely useful for all Lua users, avoiding a lot of headaches.
❤️
 
Update to version 1.1

  • No longer creates an iterator function upon every pairs()-loop, but instead creates a fixed iterator upon SyncedTable creation that uses global indexing to distinguish different loops running at the same time. This puts less pressure on the garbage collector.

Thanks to @maddeem and @Wrda for suggesting improvements on garbage collection workload.

I also improved the resource description. It will hopefully provide a more intuitive explanation to new Lua-mappers as compared to before.
 
Last edited:
Hotfix 1.1a:
  • Fixed a bug introduced with version 1.1, where using false as a key in a SyncedTable would break a loop early, if another loop was nested upon iterating this exact key (very rare but possible).
(In other words, I did while x do, while I should have done while x ~= nil do, which makes a difference for x = false).
 
Top