• 🏆 Texturing Contest #33 is OPEN! Contestants must re-texture a SD unit model found in-game (Warcraft 3 Classic), recreating the unit into a peaceful NPC version. 🔗Click here to enter!
  • It's time for the first HD Modeling Contest of 2024. Join the theme discussion for Hive's HD Modeling Contest #6! Click here to post your idea!

[System] Data Sync

Level 31
Joined
Jul 10, 2007
Messages
6,306
JASS:
library IntegerSync /* v1.0.1.0
*************************************************************************************
*
*   Used to sync integers between players.
*
*           Special Thanks to Halo7568
*
*************************************************************************************
*
*   */uses/*
*
*       */ Alloc        /*  hiveworkshop.com/forums/jass-resources-412/snippet-alloc-192348/
*       */ Thread       /*  hiveworkshop.com/forums/submissions-414/snippet-thread-218269/
*
************************************************************************************
*
*   constant function IsPositionError takes integer position returns boolean
*       -   Checks if position is error. Position is retrieved from getErrorPosition in the
*       -   IntegerSyncStream struct.
*
*   struct IntegerSync extends array
*       -   Used to sync single integers
*
*       static method create takes nothing returns IntegerSync
*       method destroy takes nothing returns nothing
*
*       method sync takes integer data, boolean source returns integer
*
*   struct IntegerSyncStream extends array
*       -   Used to sync up many integers
*
*       readonly integer dataSize
*           -   The number of integers in the stream.
*           -   This starts at 1, not 0
*       readonly integer readIndex
*           -   The current integer being read from the stream. A stream can only ever be read
*           -   once. It should be read after syncing process is complete.
*           -   This starts at 1, not 0
*
*       static method create takes nothing returns IntegerSyncStream
*       method destroy takes nothing returns nothing
*
*       method startBurst takes nothing returns nothing
*           -   Begins a data burst. Used to prevent broadcasting player from desyncing
*           -   due to large operations. Furthermore, prevents op limit.
*
*       method operator burstOver takes nothing returns boolean
*           -   Will be true when the current burst is finished.
*
*       method broadcast takes integer data returns nothing
*           -   Broadcasts an integer to other players in the game
*           -   Always broadcast inside of a burst.
*
*               call startBurst()
*               loop
*                   call broadcast(data)
*                   exitwhen burstOver
*               endloop
*
*       method operator synced takes nothing returns boolean
*           -   This will be true when syncing is complete
*
*       method startSync takes nothing returns nothing
*           -   Call from broadcasting player when the broadcasting is complete (broadcasting, not bursts)
*           -   This will send a request to make the synced boolean true. The synced boolean will become
*           -   true after all of the broadcasted data is finished syncing.
*
*       method finishSync takes nothing returns nothing
*           -   Call after syncing is complete
*           -   This gets the data 
*
*       method getErrorPosition takes nothing returns integer
*           -   This searches for an error in the stream. Use IsPositionError to determine whether position was error or not.
*           -   This is properly sync'd for all players.
*
*       method read takes nothing returns integer
*           -   This reads from the stream in the same order the data was broadcasted.
*
*   Syncing Example
*
*       local integer iterator = DATA_SIZE //some arbitrary number just for this example
*       local IntegerSyncStream stream = IntegerSyncStream.create()
*
*       //only the broadcasting player should have the iterator
*       //the iterator can be used to see if the current player is broadcasting or not
*       //this works well because the iterator will be DATA_SIZE when the broadcasting
*       //is finished, meaning that the broadcaster will move from a broadcasting
*       //state to a waiting state after broadcasting is finished w/o any special vars
*       if (GetLocalPlayer() == BROADCASTER) then
*           set iterator = 0
*       endif
*
*       //this outer loop will continue until the syncing is completed
*       //the burst loop is located within this
*       loop
*           //this is where bursting is done. Always put TriggerSyncStart() and TriggerSyncReady()
*           //around the burst loop
*           call TriggerSyncStart()
*           if (iterator != DATA_SIZE) then
*               call stream.startBurst()
*               loop
*                   //simply broadcasting the iterator for the hell of it. Can broadcast anything.
*                   call stream.broadcast(iterator)
*
*                   //this is the get next integer step. In this case, next integer is retrieved by increasing iterator
*                   //in File I/O, the next integer would be retrieved by reading a file
*                   set iterator = iterator + 1
*
*                   //this exits when either the broadcast is complete or the burst is complete
*                   //in this case, broadcast is complete when iterator is DATA_SIZE.
*                   //In File I/O, broadcast would be complete after all files are
*                   //read
*                   exitwhen iterator == DATA_SIZE or stream.burstOver
*               endloop
*
*               //if the broadcasting is complete, begin the sync
*               //after this point, the broadcaster will be like the rest of the players waiting for stream.synced to be true
*               if (iterator == DATA_SIZE) then
*                   call stream.startSync()
*               endif
*           endif
*           call TriggerSyncReady()
*
*           exitwhen stream.synced or GetPlayerController(BROADCASTER) != MAP_CONTROL_USER
*       endloop
*
*   Error Check Example
*
*       //this should only be used in debug
*       //getErrorPosition will only search for errors when debug mode is enabled
*       //outside of debug mode, it never returns an error
*       set errorPosition = stream.getErrorPosition()
*       if (IsPositionError(errorPosition)) then
*           //there was an error
*
*           call stream.destroy()
*           return
*       endif
*
*   Read Example
*
*       local integer array data
*       loop
*           exitwhen stream.readIndex == stream.dataSize
*           set data[stream.readIndex + 1] = stream.read()
*       endloop
*
*************************************************************************************/
    globals
        private constant string BASE = "!#$%&()*+'-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{}~"
        private constant integer BASE_LENGTH = StringLength(BASE)
        private constant integer BASE_LENGTH2 = BASE_LENGTH*BASE_LENGTH
        private gamecache syncC
        private string array kthis
    endglobals
    
    private struct IdBuilder extends array
        private static method onInit takes nothing returns nothing
            local integer i = BASE_LENGTH
            loop
                set i = i - 1
                set kthis[i] = SubString(BASE, i, i + 1)
                exitwhen 0 == i
            endloop
            
            set i = (BASE_LENGTH - 1)*BASE_LENGTH + BASE_LENGTH
            loop
                set i = i - 1
                set kthis[i] = kthis[i/BASE_LENGTH] + kthis[i - i/BASE_LENGTH*BASE_LENGTH]
                exitwhen BASE_LENGTH == i
            endloop
            
            call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,9000,"Done Initializing")
        endmethod
    endstruct
    
    constant function IsPositionError takes integer position returns boolean
        return 0 != position
    endfunction
    
    private function BuildId takes integer v returns string
        local string s = ""
        
        loop
            set s = kthis[v - v/BASE_LENGTH2*BASE_LENGTH2] + s
            set v = v/BASE_LENGTH2
            exitwhen 0 == v
        endloop
        
        return s
    endfunction
    
    struct IntegerSync extends array
        implement Alloc
        
        static method create takes nothing returns thistype
            return allocate()
        endmethod
    
        method sync takes integer data, boolean source returns integer
            local Thread thread = Thread.create()
            
            call TriggerSyncStart()
            if (source) then
                call StoreInteger(syncC, kthis[this], "!!", data)
                call SyncStoredInteger(syncC, kthis[this], "!!")
            endif
            call TriggerSyncReady()
            
            call thread.sync()
            
            loop
                call TriggerSyncStart()
                exitwhen thread.synced
                call TriggerSyncReady()
            endloop
            
            if (HaveStoredInteger(syncC, kthis[this], "!!")) then
                set data = GetStoredInteger(syncC, kthis[this], "!!")
                call FlushStoredInteger(syncC, kthis[this], "!!")
            endif
            
            call thread.destroy()
            
            return data
        endmethod
        
        method destroy takes nothing returns nothing
            call deallocate()
        endmethod
    endstruct
    
    struct IntegerSyncStream extends array
        private Thread thread
        
        readonly integer dataSize
        readonly integer readIndex
        
        private static constant integer BURST_SIZE = 1000
        private integer burst
        
        private static constant integer READ_SIZE = 1000
        private integer readBurst
        
        static method create takes nothing returns thistype
            local thistype this = IntegerSync.create()
            
            set thread = Thread.create()
            set burst = BURST_SIZE
            
            return this
        endmethod
        
        method destroy takes nothing returns nothing
            call FlushStoredMission(syncC, kthis[this])
            
            set dataSize = 0
            set readIndex = 0
            
            set readBurst = 0
            
            call thread.destroy()
            
            call IntegerSync(this).destroy()
        endmethod
        
        method broadcast takes integer data returns nothing
            local string dc = BuildId(dataSize + 1)
            set dataSize = dataSize + 1
            call StoreInteger(syncC, kthis[this], dc, data)
            call SyncStoredInteger(syncC, kthis[this], dc)
            set burst = burst - 1
        endmethod
        
        method startBurst takes nothing returns nothing
            set burst = BURST_SIZE
        endmethod
        
        method operator burstOver takes nothing returns boolean
            return 0 == burst
        endmethod
        
        method read takes nothing returns integer
            if (0 == readBurst) then
                call TriggerSyncStart()
                set readBurst = READ_SIZE
                call TriggerSyncReady()
            endif
            set readBurst = readBurst - 1
            
            set readIndex = readIndex + 1
            return GetStoredInteger(syncC, kthis[this], BuildId(readIndex))
        endmethod
        
        method startSync takes nothing returns nothing
            call StoreInteger(syncC, kthis[this], "!", dataSize)
            call SyncStoredInteger(syncC, kthis[this], "!")
            call thread.sync()
        endmethod
        
        method finishSync takes nothing returns nothing
            set dataSize = GetStoredInteger(syncC, kthis[this], "!")
        endmethod
        
        method operator synced takes nothing returns boolean
            return thread.synced
        endmethod
        
        method getErrorPosition takes nothing returns integer
            static if DEBUG_MODE then
                local integer burst
                local integer dataCheck = dataSize
                
                loop
                    call TriggerSyncStart()
                    set burst = BURST_SIZE
                    loop
                        exitwhen 0 == dataCheck or 0 == burst or not HaveStoredInteger(syncC, kthis[this], BuildId(dataCheck))
                        set burst = burst - 1
                        set dataCheck = dataCheck - 1
                    endloop
                    call TriggerSyncReady()
                    
                    exitwhen 0 == dataCheck or not HaveStoredInteger(syncC, kthis[this], BuildId(dataCheck))
                endloop
                
                return IntegerSync(this).sync(dataCheck, 0 != dataCheck)
            else
                return 0
            endif
        endmethod
        
        private static method onInit takes nothing returns nothing
            set syncC = InitGameCache("!")
        endmethod
    endstruct
endlibrary

Sync Integer Excerpt
JASS:
private static method syncData takes integer count, player source, IntegerSyncStream stream returns nothing
    local integer i = StartBroadcast(count, source)
    
    loop
        call TriggerSyncStart()
        if (IsBroadcasting(i)) then
            call stream.startBurst()
            loop
                call stream.broadcast(i)
                set i = GetNextBroadcast(i)
                exitwhen IsBroadcastComplete(i) or stream.burstOver
            endloop
            
            if (IsBroadcastComplete(i)) then
                call stream.startSync()
            endif
        endif
        call TriggerSyncReady()
        
        exitwhen stream.synced or GetPlayerController(source) != MAP_CONTROL_USER
    endloop
    
    call stream.finishSync()
endmethod

Error Check Excerpt
JASS:
private static method errorCheck takes IntegerSyncStream stream returns nothing
    local integer errorPosition = stream.getErrorPosition()
    
    if (IsPositionError(errorPosition)) then
        call stream.destroy()
        set errorPosition = 1/0
    endif
endmethod

Read Excerpt
JASS:
private static method read takes IntegerSyncStream stream returns nothing
    loop
        call stream.read()
        exitwhen stream.readIndex == stream.dataSize
    endloop
endmethod

Fully Working Example
JASS:
globals
    timer downloadTimer
endglobals
function StartDownload takes nothing returns timer
    set downloadTimer = CreateTimer()
    call TimerStart(downloadTimer, 600000, false, null)
    return downloadTimer
endfunction

function GetDownloadTime takes timer downloadTime returns string
    local real time
    local integer timeI
    local integer hours
    local integer minutes
    local integer seconds
    local real decimal
    
    local string hoursS
    local string minutesS
    local string secondsS
    
    set time = TimerGetElapsed(downloadTime)
    set timeI = R2I(time)
    
    set decimal = time - timeI
    set decimal = R2I(decimal*100)/100.
    
    set seconds = timeI - timeI/60*60
    set timeI = timeI/60
    set minutes = timeI - timeI/60*60
    set timeI = timeI/60
    set hours = timeI
    
    set hoursS = I2S(hours)
    set minutesS = I2S(minutes)
    set secondsS = I2S(seconds)
    
    if (hours < 10) then
        set hoursS = "0"+hoursS
    endif
    if (minutes < 10) then
        set minutesS = "0"+minutesS
    endif
    if (seconds < 10) then
        set secondsS = "0"+secondsS
    endif
    
    call DestroyTimer(downloadTime)
    
    return "Downloaded Data in "+hoursS+":"+minutesS+":"+secondsS+SubString(R2S(decimal),1,4)
endfunction

function StartBroadcast takes integer size, player source returns integer
    if (GetLocalPlayer() == source) then
        return size
    endif
    return 0
endfunction
function IsBroadcasting takes integer i returns boolean
    return 0 != i
endfunction
function GetNextBroadcast takes integer i returns integer
    return i - 1
endfunction
function IsBroadcastComplete takes integer i returns boolean
    return 0 == i
endfunction

struct tester extends array
    private static constant integer COUNT = 10000
    private static constant player PLAYER = Player(2)
    
    private static method syncData takes integer count, player source, IntegerSyncStream stream returns nothing
        local integer i
        
        call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,600000,"Stream Sync Start")
        set i = StartBroadcast(count, source)
        loop
            call TriggerSyncStart()
            if (IsBroadcasting(i)) then
                call stream.startBurst()
                loop
                    call stream.broadcast(i)
                    set i = GetNextBroadcast(i)
                    exitwhen IsBroadcastComplete(i) or stream.burstOver
                endloop
                
                if (IsBroadcastComplete(i)) then
                    call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,600000,"Broadcasting Finished, Waiting For Sync Completion")
                    call stream.startSync()
                endif
            endif
            call TriggerSyncReady()
            
            exitwhen stream.synced or GetPlayerController(source) != MAP_CONTROL_USER
        endloop
        call stream.finishSync()
        call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,600000,"Stream Sync End")
    endmethod
    
    private static method errorCheck takes IntegerSyncStream stream returns nothing
        local integer errorPosition
        
        call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,600000,"Error Check Start")
        set errorPosition = stream.getErrorPosition()
        if (IsPositionError(errorPosition)) then
            call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,600000,"Position: "+I2S(errorPosition)+" Failed to Sync")
            
            call stream.destroy()
            
            set errorPosition = 1/0
        endif
        call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,600000,"Error Check End")
    endmethod
    
    private static method read takes IntegerSyncStream stream returns nothing
        local integer i
    
        call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,600000,"Read Start")
        set i = stream.dataSize
        loop
            call stream.read()
            set i = i - 1
            exitwhen 0 == i
        endloop
        call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,600000,"Read End")
    endmethod
    
    private static method sync takes integer count, player source, IntegerSyncStream stream returns nothing
        local timer downloadTime
        
        set downloadTime = StartDownload()
        call syncData(count, source, stream)
        call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,600000,"Downloaded Time: "+GetDownloadTime(downloadTime))
        set downloadTime = null
    endmethod

    private static method exec takes nothing returns nothing
        local IntegerSyncStream stream = IntegerSyncStream.create()
        
        call sync(COUNT, PLAYER, stream)
        call errorCheck(stream)
        call read(stream)
        
        call stream.destroy()
    endmethod
    
    private static method init takes nothing returns nothing
        call DestroyTimer(GetExpiredTimer())
        
        call ExecuteFunc(exec.name)
    endmethod

    private static method onInit takes nothing returns nothing
        call TimerStart(CreateTimer(),0,false,function thistype.init)
    endmethod
endstruct

Cool Download Progress Excerpt
JASS:
loop
    call TriggerSyncStart()
    //burst
    call TriggerSyncReady()
    exitwhen thread.synced

    if (GetLocalPlayer() != source) then
        loop
            exitwhen not HaveStoredInteger(sync, I2S(stream), I2S(syncCount))
            set syncCount = syncCount + 1
        endloop
        
        if (syncCount != oldSyncCount) then
            call DisplayTimedTextToPlayer(GetLocalPlayer(),0,0,10,R2S(syncCount/I2R(COUNT)*100)+"% Downloaded")
            set oldSyncCount = syncCount
        endif
    endif
endloop
 
Last edited:
Level 5
Joined
Jun 16, 2004
Messages
108
If you are concerned about sync speed, why is it that you use such a gamecache name as long as "sync" ? Did you know that the gamecache name is sent along with the packet with every working sync native?
 
Level 31
Joined
Jul 10, 2007
Messages
6,306
string size doesn't matter since the packet sizes aren't dynamic >.>. A string takes up 4096 chars, and that's that =\.


Also, regardless, sync max is 60 per second. This beyond caps out on that, so it really doesn't matter how large the bursts are so long as they are at least 240 to give everything a chance to form up.


This just gives an easy way to syncs up both single integers and streams of integers. It also happens to do it at the wc3 cap speed, so the syncs are as fast as they can be, which isn't all that fast, lolz.
 
Level 5
Joined
Jun 16, 2004
Messages
108
I am not sure you understand, and you apparently do not know how sync packets are formed either. The sync packets contain the game cache name, label, category, probably a CRC32 since it is an action packet, packet size, packet type, value, etc. Well, the important thing to note there is that it contains three strings whose lengths you control.

Not sure if you bothered to test it either, it is a one line change to try. My own sync system which I wrote 2-3 years ago and modified slightly synced 2500 integers in about 17-18 seconds with a cache name of "sc.cc" and dropped to 9-10 seconds with a cache name of "a".

Your File.w3x map which you deleted improved from 18-19 seconds to around 12 seconds. Perhaps you should try it?
 
Level 31
Joined
Jul 10, 2007
Messages
6,306
Thank you Halo. I followed your suggestions.

New speeds:

2500: 9 seconds from 16 seconds
10000: 1 minute 22 seconds from 2 minutes and 1 second

edit
The speed sharply drops as you add in more characters. 100,000 syncs took 16 minutes and 48 seconds whereas 2500 again took 9 seconds. By all accounts, the 100k sync if done correctly should only take 6 minutes. If I do it using only single chars, it'd be even faster.

What I am pondering doing now is using many gamecaches in order to minimize the sizes. I'm using base 90 atm to minimize label sizes. If I used 90 gamecaches, I could go up to 8010 while still using 1 char for each label ;o. This should drop the 10000 sync from 1 minute and 22 seconds to more like 15 seconds =O. Of course, I am weary of using 90 gamecaches because of the wc3 cache limit, but I will probably go for it.
 
Last edited:
Level 17
Joined
Apr 27, 2008
Messages
2,455
So gamecache speed is a limiter here? I thought we were bottlenecked by the Internet speed not the hard drive speed?

I suppose it's hardware dependant, for example you would get a really different result with a ssd.

EDIT : Or not when i read again, nothing has been said like that, the sync is slower with longer string labels names, because of the packet size.
But it's still an interesting test.
 
Level 17
Joined
Apr 27, 2008
Messages
2,455
Well, remember that actually the jass was clearly not done for all this silly stuff.
We are abusing it in every ways.
Plus it makes sense that the datas which actually really need to be synced have the priority, i mean the ones handled by the wc3 engine.
 
Level 31
Joined
Jul 10, 2007
Messages
6,306
Planned updates

AES Cipher = Cipher("password") //done via MD5
MD5 checksum = checksum(data in base 256)

module with methods like
broadcast size
data size (all data, for progress bar)

essentially dumping that nasty loop into a module


data = all in base 256 for easy AES. Can't be stored in base 256, has to be stored in base 64. Base conversion between 64 and 256 is extremely fast.


For syncing, up to 8100 or so values can be sync'd at a time. Can do creation/destruction, but this would add a lot of ops to each broadcast.

Can do a queue of syncs to be done. Data registers to the queue and the queue is gone over 1 at a time, this way the entire buffer can be used by 1 set of data, but this can cause issues with small pieces of data. Need to split the buffer up.

One possibility is to give each player in the game their own data sync buffer, but this can be problematic as connection speeds differ.

Furthermore, the buffer indexes are only known locally (for the player broadcasting the data), so dynamic segmentation isn't really possible.

Final Idea: split buffer into chunks of 675. Players can take over as many chunks as they need. As progress is known, the chunks can be given up when the data size grows smaller, thus transferring the chunk to the next player.

edit
new finding on data sync.. max sync rate is 90^3, not 90^2. Data can ofc be broadcasted from multiple players at once. The buffer can be split up among the players (90/n*90^2, or in the case of 12, 56700 values per buffer). The buffer can then be reused at the end of each burst. A much better use would actually be making players instantiate buffers, where a buffer can do 8100 values. The buffers can be linked together via keys. The buffers instantiated are based off of how much data the player is going to broadcast. Essentially, in a given broadcast loop, all players will be broadcasting with their own localized buffers. At the end, all of the buffers will be gone over so that all of the data for all players is updated.

Another way is to just sync and process one player at a time, this way the data doesn't have to be stored for all players, but rather for 1 at a time. However, players have both upload/download, so in order to sync as quickly as possible, all 12 player uploads should be utilized at the same time.

edit
A look at the new upcoming features

DataStream
The stream of values (stores the actual values for syncing etc). These streams can be expanded and contracted dynamically in intervals of 8100!

single value syncing and multi value syncing have been rewritten to be able to sync values from multiple players rather than values from 1 player with the use of data streams

The annoying burst etc loop has been thrown into a module. If the entire buffer is gone through in 1 set of bursts, attempt to expand the buffer, otherwise read the values out of the buffer and reuse it.
 
Last edited:
Level 31
Joined
Jul 10, 2007
Messages
6,306

Unnamed tab 1


Here you go everyone, the new DataSync 2.0.0.0, completely untested. This thing, once it is proven to be working completely, will be a beast.

Essentially, it can sync up to 90^3 integers per instant. The previous could sync infinity, but the 90^3 syncs in this one are high speed. By high speed, I mean that they are at least 2x-3x+ faster than the ones in the older version.

The streams dynamically size themselves as well!

Check out how easy it is to use compared to the old version =O.

Code

Demo


JASS:
library IntegerSync /* v2.0.0.0
*************************************************************************************
*
*   Used to sync integers between players.
*
*           Special Thanks to Halo7568
*
*************************************************************************************
*
*   */uses/*
*
*       */ Thread       /*  hiveworkshop.com/forums/submissions-414/snippet-thread-218269/
*
************************************************************************************
*
*************************************************************************************/
    private keyword DataInit
    private struct Data extends array
        private static constant string DATA_KEY = "!#$%&()*+'-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{}~"
        static constant integer KEY_LENGTH = StringLength(DATA_KEY)
        static constant integer KEY_LENGTH_2D = KEY_LENGTH*KEY_LENGTH
        
        private static string array keyStr
        private static gamecache array data
        
        static method write takes integer k1, integer k2, integer k3, integer v returns nothing
            call StoreInteger(data[k1], keyStr[k2], keyStr[k3], v)
            call SyncStoredInteger(data[k1], keyStr[k2], keyStr[k3])
        endmethod
        static method read takes integer k1, integer k2, integer k3 returns integer
            return GetStoredInteger(data[k1], keyStr[k2], keyStr[k3])
        endmethod
        static method flush takes integer k1, integer k2, integer k3 returns nothing
            call FlushStoredInteger(data[k1], keyStr[k2], keyStr[k3])
            call SyncStoredInteger(data[k1], keyStr[k2], keyStr[k3])
        endmethod
        static method has takes integer k1, integer k2, integer k3 returns boolean
            return HaveStoredInteger(data[k1], keyStr[k2], keyStr[k3])
        endmethod
        static method flushP takes integer k1 returns nothing
            call FlushGameCache(data[k1])
            set data[k1] = InitGameCache(keyStr[k1])
        endmethod
        
        implement DataInit
    endstruct
    private module DataInit
        private static method initKeys takes nothing returns nothing
            local integer k = KEY_LENGTH
            loop
                set k = k - 1
                set keyStr[k] = SubString(DATA_KEY, k, k + 1)
                exitwhen 0 == k
            endloop
        endmethod
        private static method initData takes nothing returns nothing
            local integer k = KEY_LENGTH
            loop
                set k = k - 1
                set data[k] = InitGameCache(keyStr[k])
                exitwhen 0 == k
            endloop
        endmethod
    
        private static method onInit takes nothing returns nothing
            call initKeys()
            call initData()
        endmethod
    endmodule
    
    private keyword DataTableInit
    private struct DataTable
        private static hashtable array table
        private static integer instanceCount = 0
        private static integer array recycler
        
        method read takes integer pid, integer k returns integer
            return LoadInteger(table[pid], this, k)
        endmethod
        method write takes integer pid, integer k, integer v returns nothing
            call SaveInteger(table[pid], this, k, v)
        endmethod
        
        static method create takes nothing returns thistype
            local thistype this = recycler[0]
            if (0 == this) then
                set this = instanceCount + 1
                set instanceCount = this
            else
                set recycler[0] = recycler[this]
            endif
            
            return this
        endmethod
        
        method destroy takes nothing returns nothing
            local integer i = 11
            loop
                call FlushChildHashtable(table[i], this)
            
                exitwhen 0 == i
                set i = i - 1
            endloop
        endmethod
        
        implement DataTableInit
    endstruct
    
    private struct DataStream extends array
        private static constant integer LIMIT = Data.KEY_LENGTH
    
        private static integer instanceCount = 0
        private static integer array recycler
        
        private thistype wk1
        private integer wk2
        private integer wk3
        
        private thistype next
        private thistype prev
        
        private integer count
        readonly integer size
        
        private thistype pk1
        private integer pk2
        private integer pk3
        
        private method contract takes nothing returns nothing
            local thistype des = prev
            
            set prev = des.prev
            set prev.next = this
            
            set recycler[des] = recycler[0]
            set recycler[0] = des
            
            set count = count - 1
            set size = size - Data.KEY_LENGTH_2D
        endmethod
        
        method getProgress takes nothing returns integer
            local integer progress = 0
            
            if (pk1 == -1) then
                return 0
            endif
            
            loop
                exitwhen not Data.has(pk1, pk2, pk3)
                
                set progress = progress + 1
                
                set pk3 = pk3 + 1
                if (pk3 == Data.KEY_LENGTH) then
                    set pk3 = 0
                    set pk2 = pk2 + 1
                    if (pk2 == Data.KEY_LENGTH) then
                        set pk2 = 0
                        set pk1 = pk1.next
                        
                        if (pk1 == this) then
                            set pk1 = -1
                            return progress
                        endif
                    endif
                endif
            endloop
            
            return progress
        endmethod
        
        method dump takes DataTable table, integer pid, integer index, integer remainingData returns integer
            local thistype k1 = this
            local integer k2 = 0
            local integer k3 = 0
        
            loop
                call table.write(pid, index, Data.read(k1, k2, k3))
                set index = index + 1
            
                set k3 = k3 + 1
                if (k3 == Data.KEY_LENGTH) then
                    set k3 = 0
                    set k2 = k2 + 1
                    if (k2 == Data.KEY_LENGTH) then
                        set k2 = 0
                        set k1 = k1.next
                    endif
                endif
                
                exitwhen k1 == wk1 and k2 == wk2 and k3 == wk3
            endloop
            
            set k1 = this
            loop
                set k1 = k1.next
                call Data.flushP(k1)
                exitwhen k1 == this
            endloop
            
            set wk1 = this
            set wk2 = 0
            set wk3 = 0
            
            set pk1 = this
            set pk2 = 0
            set pk3 = 0
            
            loop
                exitwhen size - Data.KEY_LENGTH_2D < remainingData
                call contract()
            endloop
            
            return index
        endmethod
        
        method expand takes nothing returns boolean
            local thistype new = recycler[0]
            
            if (0 == new) then
                set new = instanceCount + 1
                if (new == LIMIT) then
                    return false
                endif
                set instanceCount = new
            else
                set recycler[0] = recycler[new]
            endif
            
            set new.prev = prev
            set new.next = this
            set prev.next = new
            set prev = new
            
            set new.wk2 = 0
            set new.wk3 = 0
            
            set count = count + 1
            set size = size + Data.KEY_LENGTH_2D
            
            return true
        endmethod
        
        method write takes integer v returns boolean
            call Data.write(wk1, wk2, wk3, v)
            
            set wk3 = wk3 + 1
            if (wk3 == Data.KEY_LENGTH) then
                set wk3 = 0
                set wk2 = wk2 + 1
                if (wk2 == Data.KEY_LENGTH) then
                    set wk2 = 0
                    set wk1 = wk1.next
                    
                    if (wk1 == this) then
                        return false
                    endif
                endif
            endif
            
            return true
        endmethod

        static method create takes nothing returns thistype
            local thistype this = recycler[0]
            local thistype new
            local thistype pnew
            
            if (0 == this) then
                set this = instanceCount + 1
                if (this == LIMIT) then
                    return 0
                endif
                set instanceCount = this
            else
                set recycler[0] = recycler[this]
            endif
            
            set wk2 = 0
            set wk3 = 0
            
            set pk1 = this
            set pk2 = 0
            set pk3 = 0
            
            set next = this
            set prev = this
            set wk1 = this
            
            set count = 1
            set size = Data.KEY_LENGTH_2D
            
            return this
        endmethod
        
        method destroy takes nothing returns nothing
            set prev.next = 0
            loop
                exitwhen 0 == this
                
                set recycler[this] = recycler[0]
                set recycler[0] = this
                
                call Data.flushP(this)
                
                set this = next
            endloop
        endmethod
    endstruct
    
    private module DataTableInit
        private static method onInit takes nothing returns nothing
            local integer i = 11
            loop
                if (GetPlayerSlotState(Player(i)) == PLAYER_SLOT_STATE_PLAYING and GetPlayerController(Player(i)) == MAP_CONTROL_USER) then
                    set table[i] = InitHashtable()
                endif
                
                exitwhen 0 == i
                set i = i - 1
            endloop
        endmethod
    endmodule
    
    struct Sync extends array
        private static integer instanceCount = 0
        private static integer array recycler
        
        static method create takes nothing returns thistype
            local thistype this = recycler[0]
            
            if (0 == this) then
                set this = instanceCount + 1
                if (this == Data.KEY_LENGTH) then
                    return 0
                endif
                set instanceCount = this
            else
                set recycler[0] = recycler[this]
            endif
            
            return this
        endmethod
        
        method operator [] takes integer pid returns integer
            return Data.read(0, pid, this)
        endmethod
        method has takes integer pid returns boolean
            return Data.has(0, pid, this)
        endmethod
    
        method sync takes integer data, boolean source returns nothing
            local Thread thread = Thread.create()

            call TriggerSyncStart()
            if (source) then
                call Data.write(0, GetPlayerId(GetLocalPlayer()), this, data)
            else
                call Data.flush(0, GetPlayerId(GetLocalPlayer()), this)
            endif
            call TriggerSyncReady()
            
            call thread.sync()
            
            loop
                call TriggerSyncStart()
                exitwhen thread.synced
                call TriggerSyncReady()
            endloop
            
            call thread.destroy()
        endmethod
        
        method destroy takes nothing returns nothing
            set recycler[this] = recycler[0]
            set recycler[0] = this
        endmethod
    endstruct
    
    module SyncStream
        method operator Data takes nothing returns DataTable
            return this
        endmethod
    
        //all arrays accessed via arr[playerId]
    
        /*
        *   Amount of data to be sync'd
        */
        static integer array dataSizes
        
        /*
        *   The streams
        */
        private static DataStream array streams
        
        /*
        *   Amount of data left to be sync'd
        */
        private static integer array dataRemaining
        
        /*
        *   How much data is current going through the stream. When the stream
        *   is full, attempt to expand the stream. If expansion failed, dump the stream.
        *   Dumping requires a full synchronization. As all streams will be sychronized, dump
        *   all of the other streams as well.
        */
        private static integer array streamWritten
        
        /*
        *   Data stored in data table (this). This isn't very real-time.
        */
        private static integer array dataSynced
        
        private static integer array progress
        
        /*
        *   Creates streams, threads, and syncs data sizes
        */
        private method syncDataSizes takes integer dataSize, boolean source returns integer     //returns amount of syncs to do
            local Sync syncer = Sync.create()
            local integer i = 11
            local integer syncs = 0
            
            call syncer.sync(dataSize, source)
            
            loop
                if (syncer.has(i)) then
                    set dataSizes[i] = syncer[i]
                    set streams[i] = DataStream.create()
                    set dataRemaining[i] = dataSizes[i]
                    set syncs = syncs + 1
                endif
            
                exitwhen 0 == i
                set i = i - 1
            endloop
            
            call syncer.destroy()
            
            return syncs
        endmethod
        
        static method create takes nothing returns thistype
            /*
            *   This is a data table
            *   A data table can store values for 12 players, so all data can be stored into this
            */
            local thistype this = DataTable.create()
            
            /*
            *   A call to startSync both initialize the stream (user code) and returns
            *   the amount of data to be sync'd (local value)
            */
            local integer dataSize = startSync()
            
            /*
            *   if the amount of data to be sync'd isn't 0, then the local player is a broadcasting player
            *   that is, they have data to broadcast
            */
            local boolean source = dataSize != 0
            
            /*
            *   This is the local data stream. There is also an array of these for each player.
            */
            local DataStream stream
            
            /*
            *   This refers to the number of remaining players to be sync'd. A player is synchronized
            *   when their synchronization thread is true.
            */
            local integer syncs
            
            /*
            *   A burst is how much data to send in an instant
            *   Too high of a burst will cause op limit or desync
            */
            local integer burst
            
            /*
            *   Player iterators
            */
            local integer i
            local integer k
            
            /*
            *   The data to be broadcasted. This needs to be stored in a variable
            *   to prevent data from being skipped over.
            */
            local integer dat
            
            /*
            *   Used with dumping
            */
            local boolean didSync
            local Thread syncThread
            
            set syncs = syncDataSizes(dataSize, source)
            
            set stream = streams[GetPlayerId(GetLocalPlayer())]
            
            /*
            *   Get the first data to be broadcasted
            */
            if (source) then
                set dat = getNextBroadcast()
            endif
            
            /*
            *   The sync loop
            */
            loop
                /*
                *   Broadcast section, surrounded by trigger syncs to prevent desyncs from local code
                */
                call TriggerSyncStart()
                if (source) then
                    set burst = Data.KEY_LENGTH/10
                    loop
                        /*
                        *   Loop until either the stream is full, the burst is finished, or the data has
                        *   all been broadcasted
                        *
                        *   With the default settings, a burst will never be stopped due to a full stream
                        *   The reason for this is because the burst size divides evenly into the stream unit
                        *   size. However, if the stream unit size is changed, the burst may not divide evenly into it.
                        */
                        exitwhen stream.write(dat) or 0 == burst or 0 == dataSize
                        set dat = getNextBroadcast()
                        set dataSize = dataSize - 1
                        set burst = burst - 1
                    endloop
                    
                    /*
                    *   If the data has all been broadcasted, unmark as broadcasting player
                    *   and send a thread sync
                    */
                    if (0 == dataSize) then
                        set source = false
                    endif
                endif
                call TriggerSyncReady()
                
                /*
                *   Data stream dumping and expansion section
                */
                set i = 11              //player iterator
                set didSync = false     //this refers to whether something was dumped or not
                loop
                    /*
                    *   If the player has data to broadcast
                    */
                    if (streams[i] != 0) then
                        /*
                        *   Make sure that the player is still here
                        */
                        if (GetPlayerSlotState(Player(i)) == PLAYER_SLOT_STATE_PLAYING and GetPlayerController(Player(i)) == MAP_CONTROL_USER) then
                            /*
                            *   Increase stream written by burst size
                            */
                            set streamWritten[i] = streamWritten[i] + Data.KEY_LENGTH/10
                            
                            /*
                            *   If stream written is bigger than the stream size, then have to either
                            *   expand or dumb the stream as the stream will be full
                            */
                            if (streamWritten[i] >= streams[i].size) then
                                /*
                                *   Add values not written to data remaining. All values will be subtracted later (no special case).
                                *   This ensures that the values not broadcasted yet aren't subtracted.
                                */
                                set dataRemaining[i] = dataRemaining[i] + streamWritten[i] - stream.size
                                if (not streams[i].expand()) then
                                    /*
                                    *   The stream attempted to expand to avoid a dump. Dumps are rather costly.
                                    *   The expansion failed, so the stream is full, meaning that a dump is required.
                                    *   To do a dump, the entire thread must be sync'd, meaning that all of the data
                                    *   that has been broadcasted from all streams will be sync'd.
                                    *
                                    *   Only perform the costly thread sync if it has not already been done on this iteration.
                                    */
                                    if (not didSync) then
                                        /*
                                        *   These 3 lines = the thread sync
                                        */
                                        set syncThread = Thread.create()
                                        call syncThread.sync()
										loop
											call TriggerSyncStart()
											exitwhen thread.synced
											call TriggerSyncReady()
										endloop
                                        call syncThread.destroy()
                                        
                                        set didSync = true
                                        
                                        /*
                                        *   Dump all previously hit streams on this iteration (might as well)
                                        *   This is to minimize thread syncs
                                        */
                                        set k = i
                                        loop
                                            exitwhen 0 == k
                                            set k = k - 1
                                            
                                            if (streams[k] != 0) then
                                                set dataSynced[k] = streams[k].dump(this, k, dataSynced[k], dataRemaining[k])
                                                set streamWritten[k] = 0
                                            endif
                                        endloop
                                    endif
                                    
                                    /*
                                    *   Dump the stream
                                    */
                                    set dataSynced[i] = streams[i].dump(this, i, dataSynced[i], dataRemaining[i])
                                    set streamWritten[i] = 0
                                endif
                            elseif (didSync) then
                                /*
                                *   If the stream isn't full but a sync occurred, dump the stream
                                */
                                set dataSynced[i] = streams[i].dump(this, i, dataSynced[i], dataRemaining[i])
                                set streamWritten[i] = 0
                            endif
                            
                            /*
                            *   Now reduce data remaining by the burst size
                            */
                            set dataRemaining[i] = dataRemaining[i] - Data.KEY_LENGTH/10
                            if (dataRemaining[i] == 0) then
                                /*
                                *   If there is no data remaining, dump and then destroy the stream
                                */
                                if (not didSync) then                   //dump
                                    set syncThread = Thread.create()
                                    call syncThread.sync()
									loop
										call TriggerSyncStart()
										exitwhen thread.synced
										call TriggerSyncReady()
									endloop
                                    call syncThread.destroy()
                                    
                                    set didSync = true
                                    
                                    set k = i
                                    loop
                                        exitwhen 0 == k
                                        set k = k - 1
                                        
                                        if (streams[k] != 0) then
                                            set dataSynced[k] = streams[k].dump(this, k, dataSynced[k], dataRemaining[k])
                                            set streamWritten[k] = 0
                                        endif
                                    endloop
                                endif
                                
                                /*
                                *   Destroy the stream
                                */
                                call streams[i].destroy()
                                set streams[i] = 0
                                
                                /*
                                *   Amount of syncs left
                                */
                                set syncs = syncs - 1
                            endif
                        else
                            /*
                            *   If the player left, destroy their stream and reduce # of syncs left
                            */
                            call streams[i].destroy()
                            set streams[i] = 0
                            
                            set syncs = syncs - 1
                            
                            set dataSizes[i] = 0
                        endif
                    endif
                    exitwhen 0 == i
                    set i = i - 1
                endloop
                
                /*
                *   % Progress Tracking
                */
                static if thistype.broadcastPercentComplete.exists then
                    call TriggerSyncStart()
                    set i = 11
                    loop
                        if (dataSizes[i] != 0 and i != GetPlayerId(GetLocalPlayer())) then
                            if (didSync) then
                                set progress[i] = dataSizes[i] - dataRemaining[i]
                            else
                                set progress[i] = progress[i] + streams[i].getProgress()
                            endif
                        else
                            set progress[i] = -1
                        endif
                        
                        call broadcastPercentComplete(i, (progress[i] + 0.)/dataSizes[i]*100)
                    endloop
                    call TriggerSyncReady()
                endif
                
                exitwhen 0 == syncs
            endloop
            
            return this
        endmethod
        method destroy takes nothing returns nothing
            local integer i = 11
            loop
                if (dataSizes[i] != 0) then
                    set dataSizes[i] = 0
                    set dataRemaining[i] = 0
                    set streamWritten[i] = 0
                    set dataSynced[i] = 0
                    set progress[i] = 0
                endif
                
                exitwhen 0 == i
                set i = i - 1
            endloop
        
            call DataTable(this).destroy()
        endmethod
    endmodule
endlibrary

JASS:
struct SyncStreamTest extends array
        private method startSync takes nothing returns integer
            return 0        //returns amount of data to be sync'd
                            //0 means that the player will be waiting for other players to sync
        endmethod
        private method getNextBroadcast takes nothing returns integer
            return 0        //the next piece of data to be sync'd
        endmethod

        //optional
        private method broadcastPercentComplete takes integer sourcePlayerId, real percentComplete returns nothing
        endmethod
    
        implement SyncStream
    endstruct

 
Top