1. Head to the 33rd Modeling Contest Poll and drink to your heart's desire.
    Dismiss Notice
  2. Choose your means of doom in the 17th Mini Mapping Contest Poll.
    Dismiss Notice
  3. A slave to two rhythms, the 22nd Terraining Contest is here.
    Dismiss Notice
  4. The heavens smile on the old faithful. The 16th Techtree Contest has begun.
    Dismiss Notice
  5. The die is cast - the 6th Melee Mapping Contest results have been announced. Onward to the Hive Cup!
    Dismiss Notice
  6. The glory of the 20th Icon Contest is yours for the taking!
    Dismiss Notice
  7. Check out the Staff job openings thread.
    Dismiss Notice
Dismiss Notice
60,000 passwords have been reset on July 8, 2019. If you cannot login, read this.

[vJASS] Sync (Game Cache)

Discussion in 'JASS Resources' started by TriggerHappy, May 14, 2016.

  1. Frotty

    Frotty

    Wurst Reviewer

    Joined:
    Jan 1, 2009
    Messages:
    1,479
    Resources:
    11
    Models:
    3
    Tools:
    1
    Maps:
    5
    Tutorials:
    1
    Wurst:
    1
    Resources:
    11
    Hi, my major problem now is that
    Code (Text):
    .addString()
    hits the oplimit at a length of about 170 characters, because Char2I is too slow.
    It would be nice if it could add at least 209 characters as this is the limit of a FileIO line.
    I was able to improve the performance by replacing Char2I with a faster alternative using this [vJASS] - StringHashEx guaranteed no-collision string hash:
    Code (vJASS):

        public function Char2IFast takes string c returns integer
            return LoadInteger(ht, 0, StringHashEx(c))
        endfunction
     

    Where ht contains the precomputed positions for all chars of the ALPHABET.
    It is enough to increase it above 200 chars, but still not a significant improvement. Additionally a FileIO file might contain up to 15 of these lines.

    I think the most convenient way to fix this would be to cache the strings added via .addString and convert them to integers later when .start is called. (Via timers to avoid hitting the oplimit)

    What do you say? Will you address this or do I need to keep my hacks?
     
  2. TriggerHappy

    TriggerHappy

    Code Moderator

    Joined:
    Jun 23, 2007
    Messages:
    3,789
    Resources:
    22
    Spells:
    11
    Tutorials:
    2
    JASS:
    9
    Resources:
    22
    @Frotty Why can't you just do something like:

    Code (vJASS):
    call .addString.execute(SubString(mystring, 0, 149))
    call .addString.execute(SubString(mystring, 150, 300))

    But yea I'll look into converting them after .start is called.
     
  3. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    478
    Resources:
    0
    Resources:
    0
    If you go for optimizing
    Char2I
    here are my 2c.

    You could use Ascii and then something like
    Char2Ascii(c)-' '
    .

    Or use the same technique but optimized for only the printable characters which could save both space and time. Or only use the optimized method when
    LIBRARY_Ascii
    wasnt found.

    Code (vJASS):


    library FastPrintableIndex initializer init

        globals
            private integer array idx
            private string array chr
        endglobals

        private function hash takes string s returns integer
            return StringHash(s) / 3183177 + 641
        endfunction

        function char2index takes string s returns integer
            local integer h = hash(s)
            if chr[h] != s then
                return idx[h+100]
            else
                return idx[h] -1
            endif
        endfunction

        private function init takes nothing returns nothing
            local string charmap = " !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~"
            local integer i = 0
            local integer len = StringLength(charmap)
            local integer h
            local string c

            loop
            exitwhen i == len
                set c = SubString(charmap, i, i+1)
                set h = hash(c)
               
                if idx[h] == 0 then
                    set chr[h] = c
                    set idx[h] = i+1
                else
                    set idx[h+100] = i
                endif

                set i = i +1
            endloop
        endfunction

    endlibrary
     
  4. Aniki

    Aniki

    Joined:
    Nov 7, 2014
    Messages:
    560
    Resources:
    6
    Tools:
    1
    Maps:
    1
    Spells:
    1
    JASS:
    3
    Resources:
    6
    There's another intresting way of implementing "Char2I"/ord.

    Because all strings in Jass are stored in a table (strings are really integers/offsets in that table) one can implicitly initialize it so that the 48th entry is "0", 65th entry is "A", 97th entry is "a", etc. and use type casting to get the integer from the string.
    You can see it being done here (search for chrord2). If the game is saved then loaded the string table needs to be reinitialized using the EVENT_GAME_LOADED for example.
     
  5. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    478
    Resources:
    0
    Resources:
    0
    Yes both ways are possible too and especially the
    GetLocalizedHotkey
    method is cute. But using Ascii is probably the least complicated.
     
  6. Waffle

    Waffle

    Joined:
    Jul 30, 2013
    Messages:
    280
    Resources:
    0
    Resources:
    0
    hmm w3 is utf8 tho.. at least in theory it should be possible to transmit the whole set of representable values.

    in practise perhaps it would not be useful.., or you could reduce utf-8 to an ascii representation before transmission.. but it is a tiny wart in many jass2 libraries that they can only handle a miniscule fragment of all the text usable by war3.

    perhaps there is room for a separate library that can escape utf8 text for transmission in pure ascii and then decode it later?
     
  7. Frotty

    Frotty

    Wurst Reviewer

    Joined:
    Jan 1, 2009
    Messages:
    1,479
    Resources:
    11
    Models:
    3
    Tools:
    1
    Maps:
    5
    Tutorials:
    1
    Wurst:
    1
    Resources:
    11
    Any news?
    Btw I get this message when using this lib and hosting with ENT bots:
    [​IMG]
     
  8. Waffle

    Waffle

    Joined:
    Jul 30, 2013
    Messages:
    280
    Resources:
    0
    Resources:
    0
    i dont think that actually has anything to do with this lib tho..

    i think thats some ghost++ warning.
     
  9. Frotty

    Frotty

    Wurst Reviewer

    Joined:
    Jan 1, 2009
    Messages:
    1,479
    Resources:
    11
    Models:
    3
    Tools:
    1
    Maps:
    5
    Tutorials:
    1
    Wurst:
    1
    Resources:
    11
    If I import this lib into my map and use it, the message shows.
    If I don't use it, it doesn't show.

    So what would be your definition of "having to do with this lib" exactly?
    [​IMG]
     
  10. TriggerHappy

    TriggerHappy

    Code Moderator

    Joined:
    Jun 23, 2007
    Messages:
    3,789
    Resources:
    22
    Spells:
    11
    Tutorials:
    2
    JASS:
    9
    Resources:
    22
    I'll update soon.

    Anyway I use this library on makemehost and nothing like that shows. I'm assuming ENT has some API which uses game cache, and this interferes. Not really my fault but if you have a solution then sure.

    EDIT: I also use it on my own personal ghost++ bot.
     
    Last edited: Feb 25, 2017
  11. Frotty

    Frotty

    Wurst Reviewer

    Joined:
    Jan 1, 2009
    Messages:
    1,479
    Resources:
    11
    Models:
    3
    Tools:
    1
    Maps:
    5
    Tutorials:
    1
    Wurst:
    1
    Resources:
    11
    Yes, it seems to only appear on ENT bots (maybe you can confirm this?). It doesn't seem to cause any issues, just wanted to let you know.
    If you so desire I could ask the admins on ENT-forum what it's about.
     
  12. TriggerHappy

    TriggerHappy

    Code Moderator

    Joined:
    Jun 23, 2007
    Messages:
    3,789
    Resources:
    22
    Spells:
    11
    Tutorials:
    2
    JASS:
    9
    Resources:
    22
    Updated, v1.2.3.
    • optimized addString to allow much larger strings without hitting the OP limit.
    • isPlayerIdDone is now a wrapper for isPlayerDone.
    For anyone curious, the way I solved it was through preloading the values of Char2I in a gamecache. By using the character and alphabet as keys, I can lookup the value through an inline-able function. However since gamecache isn't case sensitive, I add a simple "0" or "1" in front of the key which determines the case.

    The longest string I tested was a length of 1000.

    Relevant Code

    Code (vJASS):
    static method bool2I takes boolean b returns integer
        if (b) then
            return 1
        else
            return 0
        endif
    endmethod

    static method char2I takes string alphabet, string c returns integer // requires preloading cache with data
        return GetStoredInteger(SyncData.Cache[2], alphabet, I2S(bool2I(StringCase(c, true) == c)) + c)
    endmethod

    private static method preloadChar2I takes nothing returns nothing
        local integer i = 0
        local string c

        loop
            exitwhen i >= ALPHABET_BASE

            set c = I2Char(ALPHABET, i)

            call StoreInteger(Cache[2], ALPHABET, I2S(bool2I(StringCase(c, true) == c)) + c, Char2I(ALPHABET, c))

            set i = i + 1
        endloop
    endmethod

    Example
    Code (vJASS):

    scope GameCacheSyncTest initializer Init

        private function SyncComplete takes nothing returns boolean
            local SyncData d = GetSyncedData()
            local string s = d.readString(0)
       
            call BJDebugMsg("Length = " + I2S(StringLength(s))) // Hello
            call BJDebugMsg(s)
            // clean up
            call d.destroy()

            return false
        endfunction

        private function StartTest takes nothing returns nothing
            local SyncData d = SyncData.create(Player(0))
            local string s = ""
            local string ten = "A12345678a"
            local integer len = 400
            local integer i = 0

            loop
                exitwhen i > len

                set s = s + ten

                set i = i + 10
            endloop

            call d.addString(s, len) // must define the (max) length of the string
            call d.addEventListener(Filter(function SyncComplete))

            call DisplayTimedTextToPlayer(User.Local, 0, 0, 15, "Syncing...")

            // begin syncing
            call d.start()
        endfunction

        private function Init takes nothing returns nothing
            call TimerStart(CreateTimer(), 0, false, function StartTest)
        endfunction

    endscope
     
    Last edited: Feb 28, 2017
  13. TriggerHappy

    TriggerHappy

    Code Moderator

    Joined:
    Jun 23, 2007
    Messages:
    3,789
    Resources:
    22
    Spells:
    11
    Tutorials:
    2
    JASS:
    9
    Resources:
    22
    Updated, v1.2.4.
    • Updated to support new version of SyncInteger.
    • Char2I is now preloaded through a hashtable rather than a gamecache.
    I updated demo map on the first post. It contains a little benchmark.
     
  14. IcemanBo

    IcemanBo

    Joined:
    Sep 6, 2013
    Messages:
    6,532
    Resources:
    23
    Maps:
    3
    Spells:
    11
    Template:
    1
    Tutorials:
    5
    JASS:
    3
    Resources:
    23
    Submission: Sync v1.2.4
    Date: 1 March 2017
    Status: Approved
    Note:

    The desync issue seems fixed and all works for me.
    Triggerhappy made improvements and added also more functionality, it is very good.
    No reason why it should not be used, approved.
     
  15. Kazeon

    Kazeon

    Joined:
    Oct 12, 2011
    Messages:
    3,296
    Resources:
    38
    Icons:
    2
    Tools:
    1
    Maps:
    7
    Spells:
    21
    Tutorials:
    3
    JASS:
    4
    Resources:
    38
    Looks like calling
    ReloadGameCachesFromDisk()
    will bug this system if called after the system's initialization: on sync callback readString will return empty string with length of specified max string length + 1. The solution is to use init module for calling that ReloadGameCachesFromDisk function so it would run before the system's initialization. I think it worth noticing.
     
  16. Waffle

    Waffle

    Joined:
    Jul 30, 2013
    Messages:
    280
    Resources:
    0
    Resources:
    0
    hmm.. fails to compile with pjass error :( :

    "Cannot convert player to index" in SyncData.readBuffer

    Code (vJASS):

    ...
    // if everything has been synced
        if ( b ) then
            if ( not s__SyncData_localFinished[data] ) then // async
                set s__SyncData_localFinished[data]=true
                // notify everyone that the local player has recieved all of the data
                call SyncInteger(s__SyncData_LocalPlayer , data) //THIS LINE
            endif
       
        endif

    ...
     



    "Cannot convert filterfunc to code" in SyncData.onInit
    Code (vJASS):

        call OnSyncInteger(Filter(function s__SyncData_updateStatus)) // THIS LINE
               
     
     
  17. TriggerHappy

    TriggerHappy

    Code Moderator

    Joined:
    Jun 23, 2007
    Messages:
    3,789
    Resources:
    22
    Spells:
    11
    Tutorials:
    2
    JASS:
    9
    Resources:
    22
  18. Waffle

    Waffle

    Joined:
    Jul 30, 2013
    Messages:
    280
    Resources:
    0
    Resources:
    0
    thx, will try in a moment.
     
  19. TriggerHappy

    TriggerHappy

    Code Moderator

    Joined:
    Jun 23, 2007
    Messages:
    3,789
    Resources:
    22
    Spells:
    11
    Tutorials:
    2
    JASS:
    9
    Resources:
    22
    Updated, v1.2.5.
    • Updated to support SyncInteger's SelectionSync struct.
    • Fixed some errors in the documentation.
    This system seems to work much faster on the newest patch. Here are some benchmarks I took using this system:

    [1.28.5]

    1MB (250,000 ints) : 1044.739 seconds (957 bytes/s)
    10KB (2500 ints) : 0.590 seconds (4,327 bytes/s)
    40KB (10000 ints) : 26.882 seconds (1,487 bytes/s)

    [1.26.0]

    1MB (250,000 ints) : ? seconds (? bytes/s)
    10KB (2500 ints) : 19.887 seconds (502 bytes/s)
    40KB (10000 ints) : 124.035 seconds (322 bytes/s)
     
  20. Kazeon

    Kazeon

    Joined:
    Oct 12, 2011
    Messages:
    3,296
    Resources:
    38
    Icons:
    2
    Tools:
    1
    Maps:
    7
    Spells:
    21
    Tutorials:
    3
    JASS:
    4
    Resources:
    38
    I wonder is the synchronization fully reliable? I just recognized that sometimes it's failed to sync things yet it still fires the "on sync" event which obviously leads to sudden desync.

    I'm using it to synchronize player's mouse coordinate. What I can notice when the desync happen is that all sent data became zero. I will try to break it down:

    1. This is how I sync things:
    Code (vJASS):

    set data = SyncData.create(p)
    call data.addReal(GetMouseX())
    call data.addReal(GetMouseY())
    call data.addInt(playerId)
    call data.addEventListener(Filter(function OnSync))
    call data.start()
     


    2. This is how OnSync function looks like:
    Code (vJASS):

        private function OnSync takes nothing returns boolean
     
            local SyncData d = GetSyncedData()
            local real x = d.readReal(0)
            local real y = d.readReal(1)
            local integer id = d.readInt(0)
            local player p = d.from
            local string s
     
            if Locale == p then
                set s = "war3mapImported\\point_target.mdx"
            else
                set s = ""
            endif
            call DestroyEffectTimed(AddSpecialEffect(s, x, y), 5.) // Use TimedHandle from Jass section
            call Fighter[id].move(x, y) // << Desync here. Basically it orders main unit to move to (x, y)
     
            return false
        endfunction
     


    3. I used multiplayer emulator to test my map. I noticed the syncing failure by controlling client-2's character (
    Fighter[1]
    ) by moving it around. What I noticed when it's desync'ed is that it's not client-2's character who's receiving the order, instead it's client-1's character (
    Fighter[0]
    ), and it's moving toward the center of the map. It means that
    Code (vJASS):
            local real x = d.readReal(0)
            local real y = d.readReal(1)
            local integer id = d.readInt(0)

    all returned zero. It's like what you are gonna see in this video:


    If you watch carefully, you may notice that when the desync happened, the archer (in client-2's side) is heading toward the correct target point. Meaning that the sent data were correct. While in client-1's side all sent data became zero, that's why in client-1's side Fighter[0] was the one who's receiving the order (sent id became 0) and it's moving toward the center of the map (x & y became 0).

    I'm using latest version of this system and patch version 1.27b. If I'm doing wrong in my code please let me know.
    I hope you can solve this issue. :) Thanks in advance!


    EDIT:
    Damn, I'm sorry before. I've found out that it's all because of a mistake on my end. I forgot to destroy the instance when it's finished syncing. So that hasInt, etc will return true immediately when it uses the same mission key. That's why the sync event got fired before things are actually synced. Sorry again. : (
     
    Last edited: Aug 31, 2017