• 🏆 Texturing Contest #33 is OPEN! Contestants must re-texture a SD unit model found in-game (Warcraft 3 Classic), recreating the unit into a peaceful NPC version. 🔗Click here to enter!
  • It's time for the first HD Modeling Contest of 2024. Join the theme discussion for Hive's HD Modeling Contest #6! Click here to post your idea!

[vJASS] Sync (Game Cache)

Documentation

Uses:
  1. SyncInteger (Required)
  2. PlayerUtils (Optional)

Demo Map: Codeless Save and Load (Multiplayer)

Core System
JASS:
library Sync requires SyncInteger, optional PlayerUtils
/***************************************************************
*
*   v1.3.0, by TriggerHappy
*   ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
*
*   This library allows you to quickly synchronize async data such as
*   the contents of a local file to all players in the map by using the game cache.
*
*   Full Documentation: -http://www.hiveworkshop.com/forums/pastebin.php?id=p4f84s
*
*   _________________________________________________________________________
*   1. Installation
*   ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
*   Copy the script to your map and save it (requires JassHelper *or* JNGP)
*
*       SyncInteger: https://www.hiveworkshop.com/threads/syncinteger.278674/
*       PlayerUtils: https://www.hiveworkshop.com/threads/playerutils.278559/
*   _________________________________________________________________________
*   2. API
*   ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
*       struct SyncData
*
*           method start takes nothing returns nothing
*           method startChunk takes integer i, integer end returns nothing
*           method refresh takes nothing returns nothing
*           method destroy takes nothing returns nothing
*
*           method addInt takes integer i returns nothing
*           method addReal takes integer i returns nothing
*           method addString takes string s, integer len returns nothing
*           method addBool takes booleanflag returns nothing
*
*           method readInt takes integer index returns integer
*           method readReal takes integer index returns integer
*           method readString takes integer index returns string
*           method readBool takes integer index returns boolean
*
*           method hasInt takes integer index returns boolean
*           method hasReal takes integer index returns boolean
*           method hasString takes integer index returns boolean
*           method hasBool takes integer index returns boolean
*
*           method isPlayerDone takes player p returns boolean
*           method isPlayerIdDone takes integer pid returns boolean
*
*           method addEventListener takes filterfunc func returns nothing
*
*           ---------
*
*           filterfunc onComplete
*           filterfunc onError
*           filterfunc onUpdate
*           trigger trigger
*
*           readonly player from
*
*           readonly real timeStarted
*           readonly real timeFinished
*           readonly real timeElapsed
*
*           readonly integer intCount
*           readonly integer boolCount
*           readonly integer strCount
*           readonly integer realCount
*           readonly integer playersDone
*
*           readonly boolean buffering
*
*           readonly static integer last
*           readonly static player LocalPlayer
*           readonly static boolean Initialized
*
*           static method create takes player from returns SyncData
*           static method destroy takes nothing returns nothing
*           static method gameTime takes nothing returns real
*
*       function GetSyncedData takes nothing returns SyncData
*
***************************************************************/

    globals
        // characters that can be synced (ascii)
        private constant string ALPHABET                    = " !#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\"\t\n"

        // safe characters for use in game cache keys
        // (case sensitive)
        private constant string SAFE_KEYS                   = " !#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`{|}~\"\t\n"

        // stop reading the string buffer when reaching this char (this char cannot be synced)
        private constant string TERM_CHAR                   = "~"
       
        // how fast the buffer updates
        private constant real UPDATE_PERIOD                 = 0.03125

        // automatically recycle indices when the syncing player leaves
        private constant boolean AUTO_DESTROY_ON_LEAVE      = true

        // automatically stop buffering when an error occurs
        private constant boolean STOP_BUFFERING_ON_ERROR    = true

        // preload game cache key strings on init
        private constant boolean PRELOAD_STR_CACHE          = true

        // size of the alphabet
        private constant integer ALPHABET_BASE              = StringLength(ALPHABET)

        // maximum number of strings *per instance*
        private constant integer MAX_STRINGS                = 8192 // or any arbitrary high number

        // filenames for gc (short names have faster sync time)
        private constant string CACHE_FILE                  = "i"
        private constant string CACHE_FILE_STR              = "s"

        // don't edit below this line
        constant integer EVENT_SYNC_CACHE       = 3
       
        constant integer SYNC_ERROR_TIMEOUT     = 1
        constant integer SYNC_ERROR_PLAYERLEFT  = 2
       
        private SelectionSync Synchronizer
    endglobals

    //**************************************************************

    function GetSyncedData takes nothing returns SyncData
        return SyncData(SyncData.Last)
    endfunction

    public function I2Char takes string alphabet, integer i returns string
        return SubString(alphabet, i, i + 1)
    endfunction

    public function Char2I takes string alphabet, string c returns integer
        local integer i = 0
        local string s
        local integer l = StringLength(alphabet)
        loop
            set s = I2Char(alphabet, i)
            exitwhen i == l
            if (s == c) then
                return i
            endif
            set i = i + 1
        endloop
        return 0
    endfunction

    public function ConvertBase takes string alphabet, integer i returns string
        local integer b
        local string s = ""
        local integer l = StringLength(alphabet)
        if i < l then
            return I2Char(alphabet, i)
        endif
        loop
            exitwhen i <= 0
            set b = i - ( i / l ) * l
            set s = I2Char(alphabet, b) + s
            set i = i / l
        endloop
        return s
    endfunction

    public function PopulateString takes string s, integer makeLen returns string
        local integer i = 0
        local integer l = StringLength(s)
        if (l == makeLen) then
            return s
        endif
        set l = makeLen-l
        loop
            exitwhen i > l
            set s = s + TERM_CHAR
            set i = i + 1
        endloop
        return s
    endfunction

    //**************************************************************

    globals
        // string table keys
        private constant integer KEY_STR_POS = (0*MAX_STRINGS)
        private constant integer KEY_STR_LEN = (1*MAX_STRINGS)

        // pending data storage space
        private constant integer KEY_STR_CACHE = (2*MAX_STRINGS)
    endglobals

    struct SyncData

        real timeout
        filterfunc onComplete
        filterfunc onError
        filterfunc onUpdate
        trigger trigger

        readonly integer lastError

        readonly player from

        readonly real timeStarted
        readonly real timeFinished
        readonly real timeElapsed

        readonly integer intCount
        readonly integer boolCount
        readonly integer strCount
        readonly integer realCount
        readonly integer playersDone

        readonly boolean buffering

        readonly static boolean Initialized = false
        readonly static integer Last        = 0
        readonly static player LocalPlayer
        readonly static integer LocalPlayerID

        private static integer Running   = 0
        private static real timeCounter  = 0.00
        private static trigger EventTrig = CreateTrigger()

        private static hashtable Table
        private static hashtable CharTable
        private static gamecache array Cache
        private static integer array PendingCount
        private static timer Elapsed
        private static timer BufferTimer
        private static integer AlphaHash
     
        private integer strBufferLen
        private trigger eventTrig
        private string mkey
        private boolean localFinished

        private thistype next
        private thistype prev
     
        static method bool2I takes boolean b returns integer
            if (b) then
                return 1
            endif
            return 0
        endmethod
     
        private static method hashString takes string c returns integer
            return StringHash(I2S(bool2I(StringCase(c, true) == c)) + c)
        endmethod

        static method char2I takes string alphabet, string c returns integer // requires preloading table with data
            return LoadInteger(SyncData.CharTable, .AlphaHash, .hashString(c))
        endmethod

        private method resetVars takes nothing returns nothing
            set this.intCount       = 0
            set this.strCount       = 0
            set this.boolCount      = 0
            set this.realCount      = 0
            set this.playersDone    = 0
            set this.strBufferLen   = 0
            set this.timeStarted    = 0
            set this.timeFinished   = 0
            set this.lastError      = 0
            set this.onComplete     = null
            set this.onError        = null
            set this.onUpdate       = null
            set this.timeout        = 0.00
            set this.buffering      = false
            set this.localFinished  = false
            set this.trigger        = null
        endmethod

        private static method getKey takes integer pos returns string
            local string position=""
   
            if (HaveSavedString(Table, KEY_STR_CACHE, pos)) then
                return LoadStr(Table, KEY_STR_CACHE, pos)
            endif
   
            set position = ConvertBase(SAFE_KEYS, pos)
            call SaveStr(Table, KEY_STR_CACHE, pos, position)
   
            return position
        endmethod

        static method create takes player from returns thistype
            local thistype this

            // Player has to be playing because of GetLocalPlayer use.
            if (GetPlayerController(from) != MAP_CONTROL_USER or GetPlayerSlotState(from) != PLAYER_SLOT_STATE_PLAYING) then
                return 0
            endif

            set this = thistype.allocate()

            set this.from   = from
            set this.mkey   = getKey(this-1)

            call this.resetVars()

            set thistype(0).next.prev = this
            set this.next = thistype(0).next
            set thistype(0).next = this

            set this.prev = 0

            return this
        endmethod

        method refresh takes nothing returns nothing
            local integer i = 0
            local integer p = 0
       
            loop
                static if (LIBRARY_PlayerUtils) then
                    exitwhen i == User.AmountPlaying
                    set p = User.fromPlaying(i).id
                else
                    exitwhen i == bj_MAX_PLAYER_SLOTS
                    set p = i
                endif

                call RemoveSavedInteger(Table, this, KEY_STR_POS + p)
                call RemoveSavedInteger(Table, this, KEY_STR_LEN + p)
                call RemoveSavedBoolean(Table, p, this) // playerdone

                set i = i + 1
            endloop

            call FlushStoredMission(Cache[0], this.mkey)
            call FlushStoredMission(Cache[1], this.mkey)

            call this.resetVars()
        endmethod

        method destroy takes nothing returns nothing
            if (this.eventTrig != null) then
                call DestroyTrigger(this.eventTrig)
                set this.eventTrig=null
            endif

            call this.refresh()

            set this.next.prev = this.prev
            set this.prev.next = this.next

            call this.deallocate()
        endmethod

        method hasInt takes integer index returns boolean
            return HaveStoredInteger(Cache[0], this.mkey, getKey(index))
        endmethod

        method hasReal takes integer index returns boolean
            return HaveStoredReal(Cache[0], this.mkey, getKey(index))
        endmethod

        method hasBool takes integer index returns boolean
            return HaveStoredBoolean(Cache[0], this.mkey, getKey(index))
        endmethod

        method hasString takes integer index returns boolean
            local integer i = LoadInteger(Table, this, KEY_STR_POS+index)
            if (index > 0 and i == 0) then
                return false
            endif
            return HaveStoredInteger(Cache[1], this.mkey, getKey(i + LoadInteger(Table, this, KEY_STR_LEN+index)))
        endmethod

        method addInt takes integer i returns nothing
            local string position=getKey(intCount)
   
            if (LocalPlayer == this.from) then
                call StoreInteger(Cache[0], this.mkey, position, i)
            endif
   
            set intCount=intCount+1
        endmethod

        method addReal takes real i returns nothing
            local string position=getKey(realCount)
   
            if (LocalPlayer == this.from) then
                call StoreReal(Cache[0], this.mkey, position, i)
            endif
   
            set realCount=realCount+1
        endmethod

        method addBool takes boolean flag returns nothing
            local string position=getKey(boolCount)
   
            if (LocalPlayer == this.from) then
                call StoreBoolean(Cache[0], this.mkey, position, flag)
            endif
   
            set boolCount=boolCount+1
        endmethod

        // SyncStoredString doesn't work
        method addStringEx takes string s, integer maxLen, boolean doSync returns nothing
            local string position
            local integer i = 0
            local integer strPos = 0
            local integer strLen = 0

            if (StringLength(s) < maxLen) then
                set s = PopulateString(s, maxLen)
            endif

            // store the string position in the table
            if (strCount == 0) then
                call SaveInteger(Table, this, KEY_STR_POS, 0)
            else
                set strLen = LoadInteger(Table, this, KEY_STR_LEN + (strCount-1)) + 1
                set strPos = LoadInteger(Table, this, KEY_STR_POS + (strCount-1)) + strLen

                call SaveInteger(Table, this, KEY_STR_POS + strCount, strPos)
            endif

            // convert each character in the string to an integer
            loop
                exitwhen i > maxLen

                set position = getKey(strPos + i)

                if (LocalPlayer == this.from) then
                    call StoreInteger(Cache[1], this.mkey, position, .char2I(ALPHABET, SubString(s, i, i + 1)))
                    if (doSync and LocalPlayer == this.from) then
                        call SyncStoredInteger(Cache[1], this.mkey, position)
                    endif
                endif

                set i = i + 1
            endloop

            set strBufferLen = strBufferLen + maxLen
            call SaveInteger(Table, this, KEY_STR_LEN+strCount, maxLen) // store the length as well
            set strCount=strCount+1
        endmethod
       
        method addString takes string s, integer maxLen returns nothing
            call addStringEx(s, maxLen, false)
        endmethod

        method readInt takes integer index returns integer
            return GetStoredInteger(Cache[0], this.mkey, getKey(index))
        endmethod

        method readReal takes integer index returns real
            return GetStoredReal(Cache[0], this.mkey, getKey(index))
        endmethod

        method readBool takes integer index returns boolean
            return GetStoredBoolean(Cache[0], this.mkey, getKey(index))
        endmethod

        method readString takes integer index returns string
            local string s = ""
            local string c
            local integer i = 0
            local integer strLen = LoadInteger(Table, this, KEY_STR_LEN+index)
            local integer strPos
   
            if (not hasString(index)) then
                return null
            endif

            set strLen = LoadInteger(Table, this, KEY_STR_LEN+index)
            set strPos = LoadInteger(Table, this, KEY_STR_POS+index)
   
            loop
                exitwhen i > strLen
       
                set c = I2Char(ALPHABET, GetStoredInteger(Cache[1], this.mkey, getKey(strPos + i)))

                if (c == TERM_CHAR) then
                    return s
                endif

                set s = s + c
                set i = i + 1
            endloop

            return s
        endmethod

        private method fireListeners takes nothing returns nothing
            set Last = this

            if (this.eventTrig != null) then
                call TriggerEvaluate(this.eventTrig)
            endif

            if (this.trigger != null and TriggerEvaluate(this.trigger)) then
                call TriggerExecute(this.trigger)
            endif
        endmethod

        private method fireEvent takes filterfunc func returns nothing
            set Last = this

            call TriggerAddCondition(EventTrig, func)
            call TriggerEvaluate(EventTrig)
            call TriggerClearConditions(EventTrig)
        endmethod

        method addEventListener takes filterfunc func returns nothing
            if (this.eventTrig == null) then
                set this.eventTrig = CreateTrigger()
            endif
            call TriggerAddCondition(this.eventTrig, func)
        endmethod

        public static method gameTime takes nothing returns real
            return timeCounter + TimerGetElapsed(Elapsed)
        endmethod

        private method error takes integer errorId returns nothing
            set this.lastError = errorId

            if (this.onError != null) then
                call this.fireEvent(this.onError)
            endif

            call this.fireListeners()

            static if (STOP_BUFFERING_ON_ERROR) then
                set this.buffering = false
            endif
        endmethod

        private static method readBuffer takes nothing returns nothing
            local boolean b = true
            local integer i = 0
            local thistype data = thistype(0).next

            loop
                exitwhen data == 0

                // find the nearest instance that is still buffering
                loop
                    exitwhen data.buffering or data == 0
                    set data=data.next
                endloop

                // if none are found, exit
                if (not data.buffering) then
                    return
                endif
               
                set data.timeElapsed = data.timeElapsed + UPDATE_PERIOD

                if (data.onUpdate != null) then
                    call data.fireEvent(data.onUpdate)
                endif

                if (data.timeout > 0 and data.timeElapsed > data.timeout) then
                    call data.error(SYNC_ERROR_TIMEOUT)
                endif

                // if the player has left, destroy the instance
                if (GetPlayerSlotState(data.from) != PLAYER_SLOT_STATE_PLAYING) then
                    call data.error(SYNC_ERROR_PLAYERLEFT)
                    static if (AUTO_DESTROY_ON_LEAVE) then
                        call data.destroy()
                    endif
                endif

                set b = true

                // make sure all integers have been synced
                if (data.intCount > 0 and  not data.hasInt(data.intCount-1)) then
                    set b = false
                endif

                // make sure all reals have been synced
                if (data.realCount > 0 and not data.hasReal(data.realCount-1)) then
                    set b = false
                endif

                // check strings too
                if (data.strCount > 0 and not data.hasString(data.strCount-1)) then
                    set b = false
                endif

                // and booleans
                if (data.boolCount > 0 and not data.hasBool(data.boolCount-1)) then
                    set b = false
                endif

                // if everything has been synced
                if (b) then

                    if (not data.localFinished) then // async
                        set data.localFinished = true

                        // notify everyone that the local player has recieved all of the data
                        call Synchronizer.syncValue(LocalPlayer, data)
                    endif
               
                endif

                set data = data.next
            endloop
        endmethod
       
        public method initInstance takes nothing returns nothing
            if (this.timeStarted != 0.00) then
                return
            endif

            set this.timeStarted = gameTime()
            set this.playersDone = 0
            set this.buffering   = true
            set this.timeElapsed = (UPDATE_PERIOD - TimerGetElapsed(BufferTimer)) * -1
   
            if (Running==0) then
                call TimerStart(BufferTimer, UPDATE_PERIOD, true, function thistype.readBuffer)
                call thistype.readBuffer()
            endif

            set Running=Running+1
        endmethod
       
        method syncInt takes integer i returns nothing
            local string position = getKey(intCount)
            call this.addInt(i)
            if (LocalPlayer == this.from) then
                call SyncStoredInteger(Cache[0], this.mkey, position)
            endif
            call this.initInstance()
        endmethod
       
        method syncReal takes real r returns nothing
            local string position = getKey(realCount)
            call this.addReal(r)
            if (LocalPlayer == this.from) then
                call SyncStoredReal(Cache[0], this.mkey, position)
            endif
            call this.initInstance()
        endmethod
       
        method syncBoolean takes boolean b returns nothing
            local string position = getKey(boolCount)
            call this.addBool(b)
            if (LocalPlayer == this.from) then
                call SyncStoredReal(Cache[0], this.mkey, position)
            endif
            call this.initInstance()
        endmethod
       
        method syncString takes string s, integer maxLen returns nothing
            local string position = getKey(strCount)
            call this.addStringEx(s, maxLen, true)
            call this.initInstance()
        endmethod

        method startChunk takes integer i, integer end returns boolean
            local integer n = 0
            local integer j = 0
            local integer p = 0
            local string position

            if (this.timeStarted != 0.00) then
                return false
            endif
           
            // Begin syncing
            loop
                exitwhen i > end

                set position = LoadStr(Table, KEY_STR_CACHE, i)
     
                if (i < intCount and LocalPlayer == this.from) then
                    call SyncStoredInteger(Cache[0], this.mkey, position)
                endif
                if (i < realCount and LocalPlayer == this.from) then
                    call SyncStoredReal(Cache[0], this.mkey, position)
                endif
                if (i < boolCount and LocalPlayer == this.from) then
                    call SyncStoredBoolean(Cache[0], this.mkey, position)
                endif
     
                if (i < strCount and LocalPlayer == this.from) then
                    set n = LoadInteger(Table, this, KEY_STR_LEN + i)
                    set p = LoadInteger(Table, this, KEY_STR_POS + i)
         
                    set j = 0
         
                    loop
                        exitwhen j > n
             
                        set position = LoadStr(Table, KEY_STR_CACHE, p + j)

                        if (LocalPlayer == this.from) then
                            call SyncStoredInteger(Cache[1], this.mkey, position)
                        endif

                        set j = j + 1
                    endloop
                endif
     
                set i = i + 1
            endloop
   
            call this.initInstance()
           
            return true
        endmethod

        method start takes nothing returns boolean
            local integer l = intCount

            // Find the highest count
            if (l < realCount) then
                set l = realCount
            endif
            if (l < strCount) then
                set l = strCount
            endif
            if (l < boolCount) then
                set l = boolCount
            endif

            return startChunk(0, l)
        endmethod

        method isPlayerIdDone takes integer pid returns boolean
            return LoadBoolean(Table, pid, this)
        endmethod

        method isPlayerDone takes player p returns boolean
            return isPlayerIdDone(GetPlayerId(p))
        endmethod

        private static method updateStatus takes nothing returns boolean
            local integer i = 0
            local integer p = GetSyncedPlayerId()
            local boolean b = true
            local boolean c = true
            local thistype data = GetSyncedInteger()
            local triggercondition tc
           
            if (GetSyncedInstance() != Synchronizer or not data.buffering) then
                return false
            endif
     
            set data.playersDone = data.playersDone + 1
            call SaveBoolean(Table, p, data, true) // set playerdone

            // check if everyone has received the data
            loop
                static if (LIBRARY_PlayerUtils) then
                    exitwhen i == User.AmountPlaying
                    set p = User.fromPlaying(i).id
                    set c = User.fromPlaying(i).isPlaying
                else
                    exitwhen i == bj_MAX_PLAYER_SLOTS
                    set p = i
                    set c = (GetPlayerController(Player(p)) == MAP_CONTROL_USER and GetPlayerSlotState(Player(p)) == PLAYER_SLOT_STATE_PLAYING)
                endif
       
                if (c and not data.isPlayerIdDone(p)) then
                    set b = false // someone hasn't
                endif

                set i = i + 1
            endloop

            // if everyone has recieved the data
            if (b) then
                set Running = Running-1

                if (Running == 0) then
                    call PauseTimer(BufferTimer)
                endif
     
                set data.buffering    = false
                set data.timeFinished = gameTime()
                set data.timeElapsed  = data.timeFinished - data.timeStarted
         
                // fire events
                if (data.onComplete != null) then
                    call data.fireEvent(data.onComplete)
                endif

                call data.fireListeners()
                call SyncInteger_FireEvents(EVENT_SYNC_CACHE)
            endif

            return false
        endmethod

        private static method trackTime takes nothing returns nothing
            set timeCounter = timeCounter + 10
        endmethod

        private static method preloadChar2I takes nothing returns nothing
            local integer i = 0
            local string c
         
            set .AlphaHash = .hashString(ALPHABET)
         
            loop
                exitwhen i >= ALPHABET_BASE

                set c = I2Char(ALPHABET, i)

                call SaveInteger(SyncData.CharTable, .AlphaHash, .hashString(c), Char2I(ALPHABET, c))

                set i = i + 1
            endloop
        endmethod

        private static method onInit takes nothing returns nothing
            static if (SyncInteger_DEFAULT_INSTANCE) then
                set Synchronizer = SyncInteger_DefaultInstance
            else
                set Synchronizer = SelectionSync.create()
            endif
           
            set Table = InitHashtable()
            set CharTable = InitHashtable()
         
            set Cache[0] = InitGameCache(CACHE_FILE)
            set Cache[1] = InitGameCache(CACHE_FILE_STR)

            set Elapsed     = CreateTimer()
            set BufferTimer = CreateTimer()

            static if (LIBRARY_PlayerUtils) then
                set LocalPlayer   = User.Local
                set LocalPlayerID = User.fromLocal().id
            else
                set LocalPlayer   = GetLocalPlayer()
                set LocalPlayerID = GetPlayerId(LocalPlayer)
            endif

            call OnSyncInteger(Filter(function thistype.updateStatus))
            call TimerStart(Elapsed, 10., true, function thistype.trackTime)
   
            static if (PRELOAD_STR_CACHE) then
                loop
                    exitwhen Last == ALPHABET_BASE
                    call getKey(Last)
                    set Last = Last + 1
                endloop
                set Last = 0
            endif
   
            call preloadChar2I()

            set Initialized = true
        endmethod

    endstruct

endlibrary
 

Attachments

  • Sync v1.3.0.w3x
    1.2 MB · Views: 275
Last edited:
LAN test results: here's 7 player, 2 PCs

I'm wondering if I should implement ahashtablefor unlimited instances.

I don't think people ill be needing many instances of this though.

Also, currently the system loops through the entire game cache and makes sure all values are there. I wonder if it's safe to just check the first/last index in the cache.

It's been safe in my tests but I don't know if it's 100%.


EDIT:

Updated.

I might remove units unless I can find a way to create them locally, or to use it some other way.

Code:
v1.0.1

- Added support for units and booleans

EDIT #2:

Updated.

Code:
v1.0.2

- Added a readonly boolean Initialized to the struct
- Added hasInt, hasString, hasBool
- Elapsed timer now uses a pre-loaded one
- Now only checks the first and last index of the cache
- Shortened key and mission key values to increase performance in some cases
- Fixed a bug that would occur due to cache keys being case in-sensitive
- Improved variable names
- Removed unit support
 
Last edited:
Can I ask when is this used? Sorry I have no idea in Async Data.

http://www.hiveworkshop.com/forums/spells-569/codeless-save-load-multiplayer-278664/?prev=status=p
Syncing camera position
Syncing GetLocationZ
Sync return values of custom natives

EDIT:

Updated.

Code:
v1.0.4

- Reworked the string table to function properly
- Can now auto-destroy instance when the syncing player leaves

v1.0.3

- Instance count increased from 512 > 8192
- Fixed TimerUtils double free
- Added refresh method to allow streaming of the same instance
- Added support for real numbers
- Minor efficiency improvements

Here's an example on how to stream data without having to wait for all players to have received it.

JASS:
// this code will sync Player 2's camera location to Player 1's

scope CameraTest initializer Init
    
    // i figured ints would be faster
    
    globals
    	private real LastX = 0
    	private real LastY = 0
    	private integer LastIndex = 0
    endglobals

 	// apply the camera
    private function ForceCam takes nothing returns nothing
    	local SyncData d = SyncData(1)

    	local real x
    	local real y

        if (not d.hasInt(LastIndex) or not d.hasInt(LastIndex+1)) then
        	return
        endif

        set x = d.readInt(LastIndex)
        set y = d.readInt(LastIndex+1)

        if (User.LocalId != 0 and (x != LastX or y != LastY)) then
        	call PanCameraToTimed(x, y, 0.2)
    	endif

        call ClearTextMessages()
        call BJDebugMsg(R2S(x) + ", " + R2S(y))

		set LastX=x
		set LastY=y
    	set LastIndex=LastIndex+2
    endfunction

    // refresh once all players have recieved the current buffer
    private function onComplete takes nothing returns boolean
        local SyncData d = GetSyncedData()

        set LastIndex = 0

        call ForceCam()
        call d.refresh()
        call d.start()

        return false
    endfunction

    // update stream
    private function UpdateCam takes nothing returns nothing
    	local SyncData d = SyncData(1)

        call d.addInt(R2I(GetCameraTargetPositionX()))
        call d.addInt(R2I(GetCameraTargetPositionY()))
    endfunction

    // begin stream
    private function StartTest takes nothing returns nothing
        local SyncData d = SyncData.create(Player(0))

        call d.addEventListener(function onComplete)
        call d.start()
    endfunction

    private function Init takes nothing returns nothing
        call TimerStart(CreateTimer(), 0, false, function StartTest)
        call TimerStart(CreateTimer(), 0.03, true, function UpdateCam)
        call TimerStart(CreateTimer(), 0.01, true, function ForceCam)
    endfunction

endscope
 
Last edited:
Can you make PlayerUtils optional? Nobody uses that shit.

Then fucking use it :thumbs_up:

I guess I can make it optional but it's going to bloat the code.

The library simply caches player data and keeps track of players. This way I can easily loop through only the playing players. It also use array lookups instead of native calls.

Of course nobody uses it. I submitted it what, 2 weeks ago?

EDIT: Overlooking the code I guess it could just be removed as a requirement. I mainly used it to speed up development time and for debugging, and since I recently just wrote it. I guess the performance increase is negligible. I also don't want to conflict with maps that use SetPlayerName and who do not want to convert their calls to the struct api, or have hooks enabled on that native.

Same goes for TimerUtils. Is there a specific reason that's in there?

I have been considering removing TU. Originally I had large loops inside the callback function and looping through all indices could have reached OP limit easily.
 

Deleted member 219079

D

Deleted member 219079

Hey is the speed enough for this:

There's dozens of good lossless compression types for pictures; a library that utilizes Sync, FileIO and one such algorithm to give out a 2D representation of a map is my dream.

Maps could then be loaded from user's hard drive.

Edit: Already something as simple as pattern inspection saves lots of space. But if someone here is actually experienced in lossless data compression, I would love to hear his solution.
 
Last edited by a moderator:
Update.

v1.0.9
  • SyncStoredX functions are now called from within the start method as opposed to from the addX methods.
  • The start method is ran in a new thread to prevent hitting the OP limit.
Hey is the speed enough for this:

I don't know it seems to throttle around 1kb/s. I don't really understand your idea though.

Anyway, can this get approved?
 
Last edited:
Update.

v.1.2.0
  • Added onComplete, onError, onUpdate properties
  • Error handling and timeout implemented
  • .lastError property will return the last error ID (SYNC_ERROR_TIMEOUT, SYNC_ERROR_PLAYERLEFT)
  • .start method is no longer ran in a new thread
  • .startChunk method added which allows you to only sync part of the data
  • Elapsed game time is now properly tracked (less inaccuracy over time)
  • gameTime method returns elapsed game time
  • STOP_BUFFERING_ON_ERROR constant added which will stop buffering but not destroy if there's an error
  • A few ConvertBase functions replaced with getKey (for preloading)

Example of the new features:

JASS:
scope SyncBenchmark initializer Init
   
    globals
        private constant integer SYNC_AMOUNT = 2500 // 10kb
    endglobals

    globals
        private integer chunkPart = 0
        private SyncData chunkSync = 0
    endglobals

    private function GetTotalSyncedIntegers takes SyncData d returns integer
        local integer i = 0
        local integer c = 0
        loop
            exitwhen i > d.intCount
            if (d.hasInt(i)) then
                set c = c + 1
            else
                return i
            endif
            set i = i + 1
        endloop
        return c
    endfunction

    private function ShowProgress takes nothing returns boolean
        local SyncData d = GetSyncedData()
        local integer done = GetTotalSyncedIntegers(d)
        local real percent = (I2R(done) / I2R(d.intCount)) * 100.

        if (percent == 100.00) then
            //return false
        endif

        call ClearTextMessages()
        call DisplayTimedTextToPlayer(GetLocalPlayer(), 0, 0, 60, "Progress: " + " %" + R2S(percent) + " " + I2S(done) + "/" + I2S(d.intCount))
        call DisplayTimedTextToPlayer(GetLocalPlayer(), 0, 0, 60, "Players Done: " + " " + I2S(d.playersDone))

        return false
    endfunction

    private function CallbackComplete takes nothing returns boolean
        local SyncData d = GetSyncedData()

        call BJDebugMsg("==============")
        call BJDebugMsg("Synced data #" + I2S(d) + " from " + GetPlayerName(d.from) + " in " + R2S(d.timeElapsed) + " seconds\n")
        call BJDebugMsg("Started    : " + R2S(d.timeStarted))
        call BJDebugMsg("Finished   : " + R2S(d.timeFinished))
        call BJDebugMsg("Speed      : " + R2S( (d.intCount * 4) / d.timeElapsed) + " bytes a second\n")

        call BJDebugMsg("Data = " + I2S(d.readInt(0)) + "-" + I2S(d.readInt(d.intCount-1))) // Hello world!

        // clean up
        call d.destroy()

        set chunkSync=0

        return false
    endfunction

    private function CallbackError takes nothing returns boolean
        local SyncData d = GetSyncedData()
       
        call BJDebugMsg("Error  : " + I2S(d.lastError))

        if (d.lastError == SYNC_ERROR_TIMEOUT) then
            call BJDebugMsg("Reason : The request has timed out.")
        elseif (d.lastError == SYNC_ERROR_PLAYERLEFT) then
            call BJDebugMsg("Reason : The syncing player has left.")
        else
            call BJDebugMsg("Reason : Unknown error.")
        endif
        // clean up
        call d.destroy()

        return false
    endfunction

    private function StartSyncingInChunks takes nothing returns nothing
        local SyncData sync = chunkSync
        local integer i = chunkPart
        local integer c = 0

        call sync.startChunk(i, i+49)
        set i = i + 50

        if (i >= SYNC_AMOUNT) then
            call PauseTimer(GetExpiredTimer())
        endif

        set chunkPart=i
    endfunction

    private function StartTest takes nothing returns nothing
        local SyncData sync
        local string s = ""
        local integer i = 0
        local integer c = 0 

        if (chunkSync != 0) then
            call BJDebugMsg("Please wait...")
            return
        endif

        set sync = SyncData.create(Player(0))
        set sync.timeout = 300 // 5 minutes

        call BJDebugMsg("Preloading sync data...")

        loop
            exitwhen i >= SYNC_AMOUNT
            call sync.addInt(i+1)

            set c = c + 1
            if (c == 500) then
                call TriggerSleepAction(0.01) // prevent OP limit
                set c = 0
            endif
            set i = i + 1
        endloop

        call BJDebugMsg("Ready!")

        set sync.onComplete = Filter(function CallbackComplete)
        set sync.onError    = Filter(function CallbackError)
        set sync.onUpdate   = Filter(function ShowProgress)

        // begin syncing
        call TriggerSleepAction(0.01) 
        set chunkSync=sync
        set chunkPart=0
        call TimerStart(CreateTimer(), 0.01, true, function StartSyncingInChunks) // in chunks to prevent OP limit
    endfunction

    private function Init takes nothing returns nothing
        local trigger t = CreateTrigger()
        call TriggerRegisterPlayerEvent( t, Player(0),  EVENT_PLAYER_END_CINEMATIC)
        call TriggerAddAction(t, function StartTest)
    endfunction

endscope
 
Last edited:

Deleted member 219079

D

Deleted member 219079

.start method is no longer ran in a new thread
Yeah imo, blocking operations are easier for the end user. Good change.

---

Your codeless save & load system's fileIO reads "modifications by TH", what are these modifications?
 
Level 6
Joined
Jul 30, 2013
Messages
282
Hmm.. blocking may be easier but, i am a bit concerned about how this might affect responsiveness.

eg a full house game and you sync a large save/load code between everybody, if this is implemented badly it can lead to an actual stall lasting minutes.
 
Update.

v1.2.1
  • Added isPlayerDone and isPlayerIdDone
  • Replaced the use of a hashtable to store if the local player has finished syncing, with a global
  • Local player Id is now cached and other minor efficiency improvements

I also added a benchmark demo map for you to run in multiplayer (or LAN), and report your stats here.

Hmm.. blocking may be easier but, i am a bit concerned about how this might affect responsiveness.

I don't understand what you're saying. I basically just removed a TriggerExecute call, so I don't see how that's going to affect "responsiveness"?

eg a full house game and you sync a large save/load code between everybody, if this is implemented badly it can lead to an actual stall lasting minutes.

In my tests this system has performed very fast.

I don't have specific numbers right now, but it should be able to handle save / load codes from a full house game within in a couple seconds.

Of course if someone is lagging it may take longer, but then the lag window will show if there's an issue there.
 
Last edited:
Level 6
Joined
Jul 30, 2013
Messages
282
just had a stray thought..

if you use this to sync strings.. you dont usually use most characters available..
so you could narrow the accepted range of values to pack more meaningful data in.

save/load codes for example are usually quite restricted in the letters used.
 
Level 10
Joined
Oct 5, 2008
Messages
355
Well, as far as i read the documentation, it hangs the orders of the local player whenever a data is synched.
For me it reads like when i need to synch like 2 integers per player every 0,1 - 0,05 seconds, it would make controlling units almost impossible.
But on the other hand, im currently unable to test this
So im just asking if it would still run smoothly.
Overall, a great system you got there
 
Well, as far as i read the documentation, it hangs the orders of the local player whenever a data is synched.
For me it reads like when i need to synch like 2 integers per player every 0,1 - 0,05 seconds, it would make controlling units almost impossible.
But on the other hand, im currently unable to test this
So im just asking if it would still run smoothly.
Overall, a great system you got there

I think I wrote that when I using pure selection events to sync values (with SyncInteger alone). I can't really reproduce it, probably because the selection event syncs so quick, and I'm not abusing them anymore.

I attached an example map which shows syncing of mouse position (can easily replace with camera position). You will want to use some interpolation so make it smoother, but in LAN it would sync the X,Y of mouse about 3-5 times a second. But you can see in the demo map that the unit orders aren't delayed at all, at least noticeably, even though data is constantly syncing.

JASS:
library SyncMouse initializer Init requires Sync
   
    private function SyncComplete takes nothing returns boolean
        local SyncData sync = SyncData.Last

        local real x = sync.readReal(0)
        local real y = sync.readReal(1)

        call BJDebugMsg("[" + R2S(SyncData.gameTime()) + "] " + GetPlayerName(sync.from) + " " + R2S(x) + ", " + R2S(y))
        call DestroyEffect(AddSpecialEffect("Abilities\\Weapons\\WaterElementalMissile\\WaterElementalMissile.mdl", x, y))

        call sync.refresh()
       
        call sync.addReal(GetMouseX())
        call sync.addReal(GetMouseY())

        set sync.onComplete = Filter(function SyncComplete)

        call sync.start()

        return false
    endfunction

    private function MapStart takes nothing returns nothing
        local SyncData sync = SyncData.create(Player(0))

        call sync.addReal(GetMouseX())
        call sync.addReal(GetMouseY())

        set sync.onComplete = Filter(function SyncComplete)

        call sync.start()
    endfunction

    private function Init takes nothing returns nothing
        call TimerStart(CreateTimer(), 0, false, function MapStart)
    endfunction

endlibrary
 

Attachments

  • Sync.w3x
    110.3 KB · Views: 158
Level 23
Joined
Jan 1, 2009
Messages
1,608
One issue I encountered is that the eventlistener wasn't called when a player left/desynced during thesyncing process. I guess it only calls onError.
I'm not sure why you split this up into eventlisteners and filterfuncs. What's the difference between an eventListener and onComplete?
In your example you call your function onComplete, yet it is not an onComplete callback, but eventlistener instead.
I would prefer the eventlistener(s) being called on any event so I can then check the status with lastError, instead of having to add the separate callback.

I didn't try your benchmark, but nevertheless this library causes problems when many syncs are running in parallel, especially with low spec-pcs and when other stuff is happening on the map at the same time. I have had players desync due to that, even tho my actual code doesn't cause a desync. E.g. we had no problems with up to 8-10 players, but with 12 players and therefore 12 syncs running in parallel at mapstart caused 1-2 players to desync pretty much every time (with the lag window appearing).
In my map I fixed this by running the syncs sequentially, but this bloats the code which should only contain sync logic. Here I would also prefer the library to optionally provide the feature to queue sync-tasks and execute them after each other instead of parallel.

Anyways thanks a lot for your research into doing this without the sync natives.
 
Last edited:
Level 6
Joined
Jul 30, 2013
Messages
282
i second that, have had desyncs when many players sync under heave load (game start, load codes).

it was an older version of the library tho. if you have changed the internals it may not be relevant anymore. i'll get back on you once i've done some more testing with the latest version.

also .. how does this lib interact with unit indexing libs.. need i do sth special to avoid conflicts? might have sth to do with issues i have been having.
 
i second that, have had desyncs when many players sync under heave load (game start, load codes).

it was an older version of the library tho. if you have changed the internals it may not be relevant anymore. i'll get back on you once i've done some more testing with the latest version.

also .. how does this lib interact with unit indexing libs.. need i do sth special to avoid conflicts? might have sth to do with issues i have been having.

I think old versions would be much more likely to cause issues under load, because SyncStoreX used to be called in the .addX methods.

And this lib shouldn't interfere with indexing libs at all. SyncInteger (the requirement) has some special code in that if UnitDex is found, it will store the dummy id's in an array rather than using unit user data. However if it's not found, it will set the units index manually, which can be overwritten by another unit indexer.

Anyway I have been personally testing this system in multiple maps on battle.net and I've ran into zero issues. But I haven't tested with 10-12 players (more like 4-8)

I'm not sure why you split this up into eventlisteners and filterfuncs. What's the difference between an eventListener and onComplete?

event listeners allow you to add multiple callbacks, while there can only be one onComplete.

In your example you call your function onComplete, yet it is not an onComplete callback, but eventlistener instead.

It was an old example that I just named the callback OnComplete. I will change it.

One issue I encountered is that the eventlistener wasn't called when a player left/desynced during thesyncing process. I guess it only calls onError.

I would prefer the eventlistener(s) being called on any event so I can then check the status with lastError, instead of having to add the separate callback.

You're right eventListeners should be called on all events, but what about onUpdate? That can cause lag without raising the Sync interval. Probably best to exclude that one.

I didn't try your benchmark, but nevertheless this library causes problems when many syncs are running in parallel, especially with low spec-pcs and when other stuff is happening on the map at the same time. I have had players desync due to that, even tho my actual code doesn't cause a desync. E.g. we had no problems with up to 8-10 players, but with 12 players and therefore 12 syncs running in parallel at mapstart caused 1-2 players to desync pretty much every time (with the lag window appearing).
In my map I fixed this by running the syncs sequentially, but this bloats the code which should only contain sync logic. Here I would also prefer the library to optionally provide the feature to queue sync-tasks and execute them after each other instead of parallel.

Right well thanks for testing this under such heavy load. I haven't tested with 12 players in anything other than LAN.

I'll make a way to limit how much data is sent (and keep it in queue).

Do you mind sharing what map you're working on, and the size of each players save code?

Another issue that might be causing a desync under heavy load, is the OP limit. Because of GetLocalPlayer, the current operation count may be different for other players, thus hitting OP limit at different times, which will run more/less code for some players (possible desync).
 
Last edited:
Level 6
Joined
Jul 30, 2013
Messages
282
anyone can comment on how much serializyng data sync can help with desyncs?

i got a culprit for what causes my desyncs but now that i will fix that i wonder if i should stride the syncs over time just in case...
or maybe outright serialize??
 
Level 23
Joined
Jan 1, 2009
Messages
1,608
I am just syncing 1 bool for all players (whether local files are enabled or not). With 12 players at start of the game this caused desyncs.

e: oh and why do you use code as listener, but then convert it to filterfunc ?
accepting filterfunc should also be an option since you can have filterfunc arrays.
 
Last edited:
Update.

v1.2.2
  • Removed an extra ` from the alphabet constant (this could cause data to sync improperly).
  • addEventListener now takes filterfunc instead of code.
  • event listeners now fire for errors.
  • Property .trigger added, which will run the same as event listeners.

@Frotty Does your code desync in LAN with 12 players (can use w3loader to check.) My code below doesn't, is your similar? Of course it could be different on battle.net.

JASS:
scope AllPlayers initializer Init

    private function OnSync takes nothing returns boolean
        local SyncData sync = SyncData.Last

        if (sync.readBool(0)) then
            call BJDebugMsg(User[sync.from].nameColored + " synced.")
        endif
     
        return false
    endfunction

    private function OnMapStart takes nothing returns nothing
        local SyncData array sync
        local integer i = 0
     
        loop
            exitwhen i == User.AmountPlaying
            set sync[i] = SyncData.create(User(i).handle)
            call sync[i].addBool(true)
            call sync[i].addEventListener(Filter(function OnSync))
            set i = i + 1
        endloop

        set i = 0

        loop
            exitwhen i == User.AmountPlaying
            call sync[i].start()
            set i = i + 1
        endloop

    endfunction

    private function Init takes nothing returns nothing
        call TimerStart(CreateTimer(), 0., false, function OnMapStart)
    endfunction

endscope
 
Level 23
Joined
Jan 1, 2009
Messages
1,608
Hi, my major problem now is that
Code:
.addString()
hits the oplimit at a length of about 170 characters, because Char2I is too slow.
It would be nice if it could add at least 209 characters as this is the limit of a FileIO line.
I was able to improve the performance by replacing Char2I with a faster alternative using this [vJASS] - StringHashEx guaranteed no-collision string hash:
JASS:
    public function Char2IFast takes string c returns integer
        return LoadInteger(ht, 0, StringHashEx(c))
    endfunction
Where ht contains the precomputed positions for all chars of the ALPHABET.
It is enough to increase it above 200 chars, but still not a significant improvement. Additionally a FileIO file might contain up to 15 of these lines.

I think the most convenient way to fix this would be to cache the strings added via .addString and convert them to integers later when .start is called. (Via timers to avoid hitting the oplimit)

What do you say? Will you address this or do I need to keep my hacks?
 

LeP

LeP

Level 13
Joined
Feb 13, 2008
Messages
537
If you go for optimizing Char2I here are my 2c.

You could use Ascii and then something like Char2Ascii(c)-' '.

Or use the same technique but optimized for only the printable characters which could save both space and time. Or only use the optimized method when LIBRARY_Ascii wasnt found.

JASS:
library FastPrintableIndex initializer init

    globals
        private integer array idx
        private string array chr
    endglobals

    private function hash takes string s returns integer
        return StringHash(s) / 3183177 + 641
    endfunction

    function char2index takes string s returns integer
        local integer h = hash(s)
        if chr[h] != s then
            return idx[h+100]
        else
            return idx[h] -1
        endif
    endfunction

    private function init takes nothing returns nothing
        local string charmap = " !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~"
        local integer i = 0
        local integer len = StringLength(charmap)
        local integer h
        local string c

        loop
        exitwhen i == len
            set c = SubString(charmap, i, i+1)
            set h = hash(c)
           
            if idx[h] == 0 then
                set chr[h] = c
                set idx[h] = i+1
            else
                set idx[h+100] = i
            endif

            set i = i +1
        endloop
    endfunction

endlibrary
 
Level 13
Joined
Nov 7, 2014
Messages
571
If you go for optimizing Char2I here are my 2c ...

There's another intresting way of implementing "Char2I"/ord.

Because all strings in Jass are stored in a table (strings are really integers/offsets in that table) one can implicitly initialize it so that the 48th entry is "0", 65th entry is "A", 97th entry is "a", etc. and use type casting to get the integer from the string.
You can see it being done here (search for chrord2). If the game is saved then loaded the string table needs to be reinitialized using the EVENT_GAME_LOADED for example.
 

LeP

LeP

Level 13
Joined
Feb 13, 2008
Messages
537
There's another intresting way of implementing "Char2I"/ord.

Because all strings in Jass are stored in a table (strings are really integers/offsets in that table) one can implicitly initialize it so that the 48th entry is "0", 65th entry is "A", 97th entry is "a", etc. and use type casting to get the integer from the string.
You can see it being done here (search for chrord2). If the game is saved then loaded the string table needs to be reinitialized using the EVENT_GAME_LOADED for example.
Yes both ways are possible too and especially the GetLocalizedHotkey method is cute. But using Ascii is probably the least complicated.
 
Level 6
Joined
Jul 30, 2013
Messages
282
hmm w3 is utf8 tho.. at least in theory it should be possible to transmit the whole set of representable values.

in practise perhaps it would not be useful.., or you could reduce utf-8 to an ascii representation before transmission.. but it is a tiny wart in many jass2 libraries that they can only handle a miniscule fragment of all the text usable by war3.

perhaps there is room for a separate library that can escape utf8 text for transmission in pure ascii and then decode it later?
 
Level 23
Joined
Jan 1, 2009
Messages
1,608
Any news?
Btw I get this message when using this lib and hosting with ENT bots:
fBU5CC8.png
 
Level 6
Joined
Jul 30, 2013
Messages
282
i dont think that actually has anything to do with this lib tho..

i think thats some ghost++ warning.
 
Top