• Check out the results of the Techtree Contest #19!
  • Listen to a special audio message from Bill Roper to the Hive Workshop community (Bill is a former Vice President of Blizzard Entertainment, Producer, Designer, Musician, Voice Actor) 🔗Click here to hear his message!
  • Read Evilhog's interview with Gregory Alper, the original composer of the music for WarCraft: Orcs & Humans 🔗Click here to read the full interview.
  • Create a void inspired texture for Warcraft 3 and enter Hive's 34th Texturing Contest: Void! Click here to enter!
  • The Hive's 22nd Icon Contest: Creep Abilities is now concluded, time to vote for your favourite set of icons! Click here to vote!

Optimized SyncStream and StringEscape

Author(s)
Tomotz, Trokkin
Tags
Lua, System
Size
191.42 KB
Rating

(0 ratings)

Downloads
56
Created
Nov 20, 2025
Updated
Nov 30, 2025
Resources
1
State
Pending
This bundle is marked as pending. It has not been reviewed by a staff member yet.
View usage Statistics Legacy download
  • Like
Reactions: Trokkin
Optimized SyncStream v1.0.0
StringEscape v1.0.0
by Tomotz
(Based on Trokkin's version [Lua] - SyncStream)

SyncStream

StringEscape

Features:
- Sync strings between users to allow accessing data only one user has (like data saved in files on his computer for save/load systems)
- Comfortable blocking interface where the function only returns after the data was synced (must be called where you can use TriggerSleepAction).
Optimized SyncStream is a more performant version of Trokkin's SyncStream system. It should have half or less the sync traffic (amount of calls to
BlzSendSyncData) than the original version. It is using StringEscape to escape the input, so it can send null terminators (which appears to be the only character unsupported by BlzSendSyncData).

Advantages over the original version are mostly performance related (This library needs about half the packets on any input size):
1. No header packet - this means that for small data sizes (up to 254 character strings) this sends half the amount of packets
2. Smaller headers for data packets - each header is only 1 character instead of 6 in the original version I think (so more room for data)
3. Packet size increased to the max possible (I think) - 254 characters instead of 200
4. Encoder is more efficient - only encodes null terminators since they are the only unsupported character for syncs.

Interface:
--- Adds data to the queue to be synced.
--- Note that SyncStream.sync must be called from all clients, even the ones that don't have the data. getLocalData can be different between clients.
---@param whichPlayer player -- the player who's data is used as the sync data
---@param getLocalData string | fun():string -- the data to sync, or a callback that returns the data to sync
---@param callback fun(syncedData:string, ...) -- the callback to call once the sync is done.
---@param ... any -- additional arguments to pass to the callback. Note that the args must be the same for all clients.
function SyncStream.sync(whichPlayer, getLocalData, callback, ...)
--- Same as SyncStream.sync, but waiting until the sync was done and only then returns.
--- Must be called from a context where you can call TriggerSleepAction().
---@param whichPlayer player
---@param getLocalData string | fun():string
---@return string
function SyncStream.blockingSync(whichPlayer, getLocalData)

Requires:
Total Initialization by Bribe @ [Lua] - Total Initialization
StringEscape by Tomotz

Optionally Requires:
LogUtils by Tomotz @ LogUtils
DebugUtils by Eikonium @ [Lua] - Debug Utils (Ingame Console etc.)

Lua: 
if Debug then Debug.beginFile("SyncStream") end
do
--[[
Optimized SyncStream v1.0.0 by Tomotz
Original version By Trokkin https://www.hiveworkshop.com/threads/syncstream.349055/
Provides functionality to designed to safely sync arbitrary amounts of data.
Uses timers to spread BlzSendSyncData calls over time.
API:
--- Adds data to the queue to be synced.
--- Note that SyncStream.sync must be called from all clients, even the ones that don't have the data. getLocalData can be different between clients.
---@param whichPlayer player -- the player who's data is used as the sync data
---@param getLocalData string | fun():string -- the data to sync, or a callback that returns the data to sync
---@param callback fun(syncedData:string, ...) -- the callback to call once the sync is done.
---@param ... any -- additional arguments to pass to the callback. Note that the args must be the same for all clients.
function SyncStream.sync(whichPlayer, getLocalData, callback, ...)
--- Same as SyncStream.sync, but waiting until the sync was done and only then returns.
--- Must be called from a context where you can call TriggerSleepAction().
---@param whichPlayer player
---@param getLocalData string | fun():string
---@return string
function SyncStream.blockingSync(whichPlayer, getLocalData)
Patch by Tomotz Nov 2025:
Advantages over the original version are mostly performance related (This library needs about half the packets on any input size):
1. No header packet - this means that for small data sizes (up to 254 character strings) this sends half the amount of packets
2. Smaller headers for data packets - each header is only 1 character instead of 6 in the original version I think (so more room for data)
3. Packet size increased to the max possible (I think) - 254 characters instead of 200
4. Encoder is more efficient - only encodes null terminators since they are the only unsupported character for syncs.
Requirements:
    Total Initialization by Bribe                   @ https://www.hiveworkshop.com/threads/317099/
    StringEscape by Tomotz
Optionaly requires:
    DebugUtils by Eikonium                          @ https://www.hiveworkshop.com/threads/330758/
    LogUtils by Tomotz                              @ https://www.hiveworkshop.com/threads/logutils.357625/
]]
--CONFIGURATION
local PREFIX = "Sync"
--- Setting the next locals to 1, 8 seems to not be noticable when syncing during the game. Up to 32, 32 should be desync safe, but will cause lag spikes when syncing large data
local PACKAGE_PER_TICK = 1
local PACKAGE_TICK_PER_SECOND = 8
local IS_DEBUG = false -- enable debug prints
local LAST_HUMAN_SLOT = bj_MAX_PLAYER_SLOTS - 1 -- the last slot id that might belongs to a human player
--END CONFIGURATION
--- Calculated values from configuration
local MAX_PAYLOAD = 255 -- Maximum payload (not including the null terminator) that can be sent with BlzSendSyncData
FLIT_DATA_SIZE = MAX_PAYLOAD -- max data length in a single flit that is sent with BlzSendSyncData
TRANSFER_RATE = FLIT_DATA_SIZE * PACKAGE_PER_TICK * PACKAGE_TICK_PER_SECOND -- bytes per second. Do not change this.
--internal
---@type table<integer, SyncStream>
local streams = {}
local syncTimer
local localPlayer ---@type player
--- A SyncStream callback and a table of arguments for it
---@class StreamFuncAndArgs
---@field func fun(fullData:string, whichPlayer:player, ...:any)
---@field args any[]
--- Sends or receives player's data assymentrically
---@class SyncStream
---@field owner player
---@field is_local boolean
---@field outPackets string[] -- list of outPackets to send
---@field callbacks StreamFuncAndArgs[] -- when a SyncStream.sync ends the syncing, we will call the first function in this list, and remove it
---@field inData string[] -- aggregated data received in this stream so far. zeroed when a full sync is done
SyncStream = {}
SyncStream.__index = SyncStream
---@param owner player The player owning the data from the stream
---@return SyncStream
local function CreateSyncStream(owner)
    return setmetatable({
        owner = owner,
        is_local = owner == localPlayer,
        outPackets = {},
        callbacks = {},
        inData = {},
    }, SyncStream)
end
--- Only print error messages unless we're in debug mode
---@param isError boolean
---@param ... any
local function debugPrint(isError, ...)
    if isError or IS_DEBUG then
        if LogWriteNoFlush == nil then
            print(...)
        else
            LogWriteNoFlush(...)
        end
    end
end
--- Sets the speed of the syncing. The higher the speed is, the faster the data will be sent, but the more it interfere with the game.
--- The default is 1, 8 which doesn't feel like the game is slowed down at all. Original was 32, 32 where you couldn't really do anything until the sync was done.
--- In my map I start the rates as 16, 8 for fast syncing of the initial data I need, but then reduce it to 1, 8 for the rest of the game for smoother syncing of the small dynamic data I need.
---@param packetsPerTick integer
---@param ticksPerSecond integer
function SetSyncRate(packetsPerTick, ticksPerSecond)
    PACKAGE_PER_TICK = packetsPerTick
    PACKAGE_TICK_PER_SECOND = ticksPerSecond
end
--- Same as SyncStream.sync, but waiting until the sync was done and only then returns.
--- Must be called from a context where you can call TriggerSleepAction().
---@param whichPlayer player
---@param getLocalData string | fun():string
---@return string
function SyncStream.blockingSync(whichPlayer, getLocalData)
    local blockSyncedData = nil
    SyncStream.sync(whichPlayer, getLocalData, function(syncedData)
        blockSyncedData = syncedData
    end)
    while blockSyncedData == nil do
        TriggerSleepAction(0.1)
    end
    return blockSyncedData
end

---@param stream SyncStream
---@param inData string
local function parsePackage(stream, inData)
    -- sync doesn't work with null terminators.
    local data = AddEscaping(inData, {0})
    for i = 1, #data, FLIT_DATA_SIZE do
        local curData = data:sub(i, i + FLIT_DATA_SIZE - 1)
        debugPrint(false, "parsePackage created flit", #curData)
        table.insert(stream.outPackets, curData)
    end
    if math.fmod(#data, FLIT_DATA_SIZE) == 0 then
        --- The receiver must get a flit shorter than FLIT_DATA_SIZE to know the package was completed.
        --- The last flit was exactly FLIT_DATA_SIZE bytes, so we must send an empty flit
        debugPrint(false, "parsePackage created empty flit")
        table.insert(stream.outPackets, "")
    end
end
--- Adds data to the queue to be synced.
--- Note that SyncStream.sync must be called from all clients, even the ones that don't have the data. getLocalData can be different between clients.
---@param whichPlayer player -- the player who's data is used as the sync data
---@param getLocalData string | fun():string -- the data to sync, or a callback that returns the data to sync
---@param callback fun(syncedData:string, ...) -- the callback to call once the sync is done.
---@param ... any -- additional arguments to pass to the callback. Note that the args must be the same for all clients.
function SyncStream.sync(whichPlayer, getLocalData, callback, ...)
    local pid = GetPlayerId(whichPlayer)
    local stream = streams[pid]
    debugPrint(false, "Sending sync request from player", pid, Debug.traceback())
    table.insert(stream.callbacks, {callback = callback, args = {...}})
    if stream.is_local then
        if type(getLocalData) == "function" then
            getLocalData = getLocalData()
        end
        if type(getLocalData) ~= "string" then
            getLocalData = "sync error: bad data type provided " .. type(getLocalData)
        end
        parsePackage(stream, getLocalData)
    end
end
function startSyncTimer()
    --- Setup sender timer
    local stream = streams[GetPlayerId(GetLocalPlayer())]
    if not stream.is_local then
        debugPrint(true, "SyncStream panic: local stream is not local")
        return
    end
    TimerStart(syncTimer, 1 / PACKAGE_TICK_PER_SECOND, true, function()
        for _ = 1, PACKAGE_PER_TICK do
            --- no more packets to send
            if next(stream.outPackets) == nil then
                break
            end
            local package = stream.outPackets[1]
            debugPrint(false, "Sending package", #package, package)
            if BlzSendSyncData(PREFIX, package) then
                table.remove(stream.outPackets, 1)
            else
                debugPrint(true, "BlzSendSyncData FAILED for package of length", #package)
            end
        end
    end)
end
---@param owner player
---@param package string
function handleData(owner, package)
    local stream = streams[GetPlayerId(owner)]
    if stream == nil then
        debugPrint(true, "SyncStream panic: no stream found for player: " .. GetPlayerName(owner))
        return
    end
    if package == nil then
        debugPrint(true, "SyncStream panic: bad package received from player: " .. GetPlayerName(owner))
        return
    end
    debugPrint(false, "Got sync package from player", GetPlayerId(owner), #package, package)
    if next(stream.callbacks) == nil then
        debugPrint(true, "SyncStream panic: sync packet received but no function set to handle it")
        return
    end
    table.insert(stream.inData, package)
    if #package < FLIT_DATA_SIZE then
        --- got a packet that is not full. This means it's the last packet
        local callbackData = table.remove(stream.callbacks, 1)
        local rawData = RemoveEscaping(table.concat(stream.inData), {0})
        stream.inData = {}
        debugPrint(false, "Last flit received for player", GetPlayerId(owner), "calling callback", #package)
        callbackData.callback(rawData or "", owner, table.unpack(callbackData.args))
    end
end
OnInit.global(function()
    syncTimer = CreateTimer()
    localPlayer = GetLocalPlayer()
    for i = 0, LAST_HUMAN_SLOT do
        streams[i] = CreateSyncStream(Player(i))
    end
    --- Setup receiver trigger
    local syncTrigger = CreateTrigger()
    for i = 0, LAST_HUMAN_SLOT do
        BlzTriggerRegisterPlayerSyncEvent(syncTrigger, Player(i), PREFIX, false)
    end
    TriggerAddAction(syncTrigger, function()
        local owner = GetTriggerPlayer()
        local package = BlzGetTriggerSyncData()
        handleData(owner, package)
    end)
    startSyncTimer()
end)
end
if Debug then Debug.endFile() end
Features:
- Escape special characters in strings to create a new string without any unsupported characters.
- For maximal efficiency, it replaces the unsupported characters which are usually used a lot in streams (null terminator, line feed, etc.) with unprintable characters, and escapes those characters as well. That means that for none random streams, the escaped string length should be very close to the original string length.

Interface:
--- Add escaping to specific chars in a string.
---@param str string -- original string we want to escape
---@param unsupportedChars integer[] -- the characters that needs to be replaced
---@return string -- the escaped string
function AddEscaping(str, unsupportedChars)
--- Opens a string previosly escaped with AddEscaping
---@param str string -- the escaped string
---@param unsupportedChars integer[] -- the characters that were replaced
---@return string? -- the original string or nil on input error
function RemoveEscaping(str, unsupportedChars)

Requires:
DebugUtils by Eikonium @ [Lua] - Debug Utils (Ingame Console etc.)

Lua: 
if Debug then Debug.beginFile("StringEscape") end
do
--[[
    StringEscape v1.0.0 by Tomotz
    Allows escaping unsupported characters in strings.
    Since most characters we want to escape are pretty useful, (textual characters or null terminator which are very common in strings)
    we rather replace those characters with some unprintable characters, and escape the unprintable characters.
    This allows us to avoid bloating up non-random strings, while keeping packing ratio the same for random strings.
    API:
    --- Add escaping to specific chars in a string.
    ---@param str string -- original string we want to escape
    ---@param unsupportedChars integer[] -- the characters that needs to be replaced
    ---@return string -- the escaped string
    function AddEscaping(str, unsupportedChars)
    --- Opens a string previosly escaped with AddEscaping
    ---@param str string -- the escaped string
    ---@param unsupportedChars integer[] -- the characters that were replaced
    ---@return string? -- the original string or nil on input error
    function RemoveEscaping(str, unsupportedChars)
    Requirements:
        DebugUtils by Eikonium                          @ https://www.hiveworkshop.com/threads/330758/
--]]
--- CONFIGURATION
-- ESCAPE_CHAR, and REPLACE_CHARS are charachters we can write to the result string, but are not very
-- useful, and so we take the more useful characters and replace them with these. These characters will have to be escaped if
-- we ever want to use them in the result string.
local ESCAPE_CHAR = 247
-- the characters that will replace the unsupported characters. I recommand not extending this set to 254/255 as these are pretty usefull characters.
-- If you want to extand, change escape_char to be lower, and add more smaller replace chars
local REPLACE_CHARS = {248, 249, 250, 251, 252, 253}
--- CONFIGURATION END
---@param t table -- table of key, value. Note that values must be unique
---@return table -- returns a table with the keys and values swapped
local function getReversedTable(t)
    local out = {}
    for k, v in pairs(t) do
        out[v] = k
    end
    return out
end
--- Add escaping to specific chars in a string.
--- We assume most chars that needs escaping are very useful chars, so to avoid wasting space, we replace them with less useful
--- chars, and then escape the less useful chars (since escaped chars takes 2 bytes instead of 1).
---@param str string -- original string we want to escape
---@param unsupportedChars integer[] -- the characters that needs to be replaced
---@return string -- the escaped string
function AddEscaping(str, unsupportedChars)
    Debug.assert(#unsupportedChars <= #REPLACE_CHARS, "too many unsupported chars to replace. You can extend REPLACE_CHARS to avoid the issue")
    -- note that move function copies elements, it doens't remove them from the original function
    local replaceChars = {}
    table.move(REPLACE_CHARS, 1, #unsupportedChars, 1, replaceChars)
    local newStr = {}
    for i=1, #str do
        local char = str:byte(i)
        if char == ESCAPE_CHAR then
            table.insert(newStr, string.rep(string.char(ESCAPE_CHAR), 2))
            goto continue
        end
        for j, v in ipairs(unsupportedChars) do
            if char == v then
                table.insert(newStr, string.char(replaceChars[j]))
                goto continue
            end
        end
        for j, v in ipairs(replaceChars) do
            if char == v then
                table.insert(newStr, string.char(ESCAPE_CHAR) .. string.char(v))
                goto continue
            end
        end
        table.insert(newStr, string.char(char))
        ::continue::
    end
    return table.concat(newStr)
end
--- Opens a string previosly escaped with AddEscaping
---@param str string -- the escaped string
---@param unsupportedChars integer[] -- the characters that were replaced
---@return string? -- the escaped string or nil on input error
function RemoveEscaping(str, unsupportedChars)
    Debug.assert(#unsupportedChars <= #REPLACE_CHARS, "too many unsupported chars to replace. You can extend REPLACE_CHARS to avoid the issue")
    local replaceChars = {}
    table.move(REPLACE_CHARS, 1, #unsupportedChars, 1, replaceChars)
    local reversedReplaceableChars = getReversedTable(replaceChars)
    local newStr = {}
    local i = 1
    while i <= #str do
        local char = str:byte(i)
        i = i + 1
        if char == ESCAPE_CHAR then
            if i <= #str then
                char = str:byte(i)
            else
                Debug.throwError("escaped character at the end of the string")
                return nil
            end
            i = i + 1
            -- either we have 2 escape chars, which we should merge to one,
            -- or we have an escaped char and then a replaceable char which should turn to the
            -- replaceable char
            table.insert(newStr, string.char(char))
        elseif reversedReplaceableChars[char] then
            table.insert(newStr, string.char(unsupportedChars[reversedReplaceableChars[char]]))
        else
            table.insert(newStr, string.char(char))
        end
    end
    return table.concat(newStr)
end
end
if Debug then Debug.endFile() end
Contents

testMap_004 unprotected (Map)

Author
Tomotz
Filename
testMap_004_unprotected.w3x
Suggested players
Any
Players
3
Required game
Warcraft 3
Recommended version
2.00
Dimensions
20x20 (32x32)
Size
191.42 KB
Uploaded
Nov 20, 2025
Updated
Never
Map Info View Resources In Use Preview Triggers Download
You must log in or register to reply here.

Similar threads

Trokkin
[Lua] SyncStream
Replies
0
Views
2K
Trokkin
Trokkin
Chief-Oblivion
Boss Autocast System Optimized
Replies
13
Views
7K
Arena_Master
Arena_Master
hooman
Monsson and bombs
Replies
5
Views
3K
Zeatherann
Zeatherann
Antares
Click-And-Hold Movement
Replies
17
Views
8K
Antares
Antares
War3I4i
Grab and Pull [Jass]
Replies
25
Views
6K
deathismyfriend
deathismyfriend
Back
Top