1. Updated Resource Submission Rules: All model & skin resource submissions must now include an in-game screenshot. This is to help speed up the moderation process and to show how the model and/or texture looks like from the in-game camera.
    Dismiss Notice
  2. DID YOU KNOW - That you can unlock new rank icons by posting on the forums or winning contests? Click here to customize your rank or read our User Rank Policy to see a list of ranks that you can unlock. Have you won a contest and still havn't received your rank award? Then please contact the administration.
    Dismiss Notice
  3. The Lich King demands your service! We've reached the 19th edition of the Icon Contest. Come along and make some chilling servants for the one true king.
    Dismiss Notice
  4. The 4th SFX Contest has started. Be sure to participate and have a fun factor in it.
    Dismiss Notice
  5. The poll for the 21st Terraining Contest is LIVE. Be sure to check out the entries and vote for one.
    Dismiss Notice
  6. The results are out! Check them out.
    Dismiss Notice
  7. Don’t forget to sign up for the Hive Cup. There’s a 555 EUR prize pool. Sign up now!
    Dismiss Notice
  8. The Hive Workshop Cup contest results have been announced! See the maps that'll be featured in the Hive Workshop Cup tournament!
    Dismiss Notice
  9. Check out the Staff job openings thread.
    Dismiss Notice
Dismiss Notice
60,000 passwords have been reset on July 8, 2019. If you cannot login, read this.

[vJASS] Sync (Game Cache)

Discussion in 'JASS Resources' started by TriggerHappy, May 14, 2016.

  1. Frotty

    Frotty

    Wurst Reviewer

    Joined:
    Jan 1, 2009
    Messages:
    1,437
    Resources:
    11
    Models:
    3
    Tools:
    1
    Maps:
    5
    Tutorials:
    1
    Wurst:
    1
    Resources:
    11
    Hi, my major problem now is that
    Code (Text):
    .addString()
    hits the oplimit at a length of about 170 characters, because Char2I is too slow.
    It would be nice if it could add at least 209 characters as this is the limit of a FileIO line.
    I was able to improve the performance by replacing Char2I with a faster alternative using this [vJASS] - StringHashEx guaranteed no-collision string hash:
    Code (vJASS):

        public function Char2IFast takes string c returns integer
            return LoadInteger(ht, 0, StringHashEx(c))
        endfunction
     

    Where ht contains the precomputed positions for all chars of the ALPHABET.
    It is enough to increase it above 200 chars, but still not a significant improvement. Additionally a FileIO file might contain up to 15 of these lines.

    I think the most convenient way to fix this would be to cache the strings added via .addString and convert them to integers later when .start is called. (Via timers to avoid hitting the oplimit)

    What do you say? Will you address this or do I need to keep my hacks?
     
  2. TriggerHappy

    TriggerHappy

    Code Moderator

    Joined:
    Jun 23, 2007
    Messages:
    3,672
    Resources:
    22
    Spells:
    11
    Tutorials:
    2
    JASS:
    9
    Resources:
    22
    @Frotty Why can't you just do something like:

    Code (vJASS):
    call .addString.execute(SubString(mystring, 0, 149))
    call .addString.execute(SubString(mystring, 150, 300))

    But yea I'll look into converting them after .start is called.
     
  3. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    451
    Resources:
    0
    Resources:
    0
    If you go for optimizing
    Char2I
    here are my 2c.

    You could use Ascii and then something like
    Char2Ascii(c)-' '
    .

    Or use the same technique but optimized for only the printable characters which could save both space and time. Or only use the optimized method when
    LIBRARY_Ascii
    wasnt found.

    Code (vJASS):


    library FastPrintableIndex initializer init

        globals
            private integer array idx
            private string array chr
        endglobals

        private function hash takes string s returns integer
            return StringHash(s) / 3183177 + 641
        endfunction

        function char2index takes string s returns integer
            local integer h = hash(s)
            if chr[h] != s then
                return idx[h+100]
            else
                return idx[h] -1
            endif
        endfunction

        private function init takes nothing returns nothing
            local string charmap = " !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~"
            local integer i = 0
            local integer len = StringLength(charmap)
            local integer h
            local string c

            loop
            exitwhen i == len
                set c = SubString(charmap, i, i+1)
                set h = hash(c)
               
                if idx[h] == 0 then
                    set chr[h] = c
                    set idx[h] = i+1
                else
                    set idx[h+100] = i
                endif

                set i = i +1
            endloop
        endfunction

    endlibrary
     
  4. Aniki

    Aniki

    Joined:
    Nov 7, 2014
    Messages:
    524
    Resources:
    4
    Spells:
    1
    JASS:
    3
    Resources:
    4
    There's another intresting way of implementing "Char2I"/ord.

    Because all strings in Jass are stored in a table (strings are really integers/offsets in that table) one can implicitly initialize it so that the 48th entry is "0", 65th entry is "A", 97th entry is "a", etc. and use type casting to get the integer from the string.
    You can see it being done here (search for chrord2). If the game is saved then loaded the string table needs to be reinitialized using the EVENT_GAME_LOADED for example.
     
  5. LeP

    LeP

    Joined:
    Feb 13, 2008
    Messages:
    451
    Resources:
    0
    Resources:
    0
    Yes both ways are possible too and especially the
    GetLocalizedHotkey
    method is cute. But using Ascii is probably the least complicated.
     
  6. Waffle

    Waffle

    Joined:
    Jul 30, 2013
    Messages:
    271
    Resources:
    0
    Resources:
    0
    hmm w3 is utf8 tho.. at least in theory it should be possible to transmit the whole set of representable values.

    in practise perhaps it would not be useful.., or you could reduce utf-8 to an ascii representation before transmission.. but it is a tiny wart in many jass2 libraries that they can only handle a miniscule fragment of all the text usable by war3.

    perhaps there is room for a separate library that can escape utf8 text for transmission in pure ascii and then decode it later?
     
  7. Frotty

    Frotty

    Wurst Reviewer

    Joined:
    Jan 1, 2009
    Messages:
    1,437
    Resources:
    11
    Models:
    3
    Tools:
    1
    Maps:
    5
    Tutorials:
    1
    Wurst:
    1
    Resources:
    11
    Any news?
    Btw I get this message when using this lib and hosting with ENT bots:
    [​IMG]
     
  8. Waffle

    Waffle

    Joined:
    Jul 30, 2013
    Messages:
    271
    Resources:
    0
    Resources:
    0
    i dont think that actually has anything to do with this lib tho..

    i think thats some ghost++ warning.
     
  9. Frotty

    Frotty

    Wurst Reviewer

    Joined:
    Jan 1, 2009
    Messages:
    1,437
    Resources:
    11
    Models:
    3
    Tools:
    1
    Maps:
    5
    Tutorials:
    1
    Wurst:
    1
    Resources:
    11
    If I import this lib into my map and use it, the message shows.
    If I don't use it, it doesn't show.

    So what would be your definition of "having to do with this lib" exactly?
    [​IMG]
     
  10. TriggerHappy

    TriggerHappy

    Code Moderator

    Joined:
    Jun 23, 2007
    Messages:
    3,672
    Resources:
    22
    Spells:
    11
    Tutorials:
    2
    JASS:
    9
    Resources:
    22
    I'll update soon.

    Anyway I use this library on makemehost and nothing like that shows. I'm assuming ENT has some API which uses game cache, and this interferes. Not really my fault but if you have a solution then sure.

    EDIT: I also use it on my own personal ghost++ bot.
     
    Last edited: Feb 25, 2017
  11. Frotty

    Frotty

    Wurst Reviewer

    Joined:
    Jan 1, 2009
    Messages:
    1,437
    Resources:
    11
    Models:
    3
    Tools:
    1
    Maps:
    5
    Tutorials:
    1
    Wurst:
    1
    Resources:
    11
    Yes, it seems to only appear on ENT bots (maybe you can confirm this?). It doesn't seem to cause any issues, just wanted to let you know.
    If you so desire I could ask the admins on ENT-forum what it's about.
     
  12. TriggerHappy

    TriggerHappy

    Code Moderator

    Joined:
    Jun 23, 2007
    Messages:
    3,672
    Resources:
    22
    Spells:
    11
    Tutorials:
    2
    JASS:
    9
    Resources:
    22
    Updated, v1.2.3.
    • optimized addString to allow much larger strings without hitting the OP limit.
    • isPlayerIdDone is now a wrapper for isPlayerDone.
    For anyone curious, the way I solved it was through preloading the values of Char2I in a gamecache. By using the character and alphabet as keys, I can lookup the value through an inline-able function. However since gamecache isn't case sensitive, I add a simple "0" or "1" in front of the key which determines the case.

    The longest string I tested was a length of 1000.

    Relevant Code

    Code (vJASS):
    static method bool2I takes boolean b returns integer
        if (b) then
            return 1
        else
            return 0
        endif
    endmethod

    static method char2I takes string alphabet, string c returns integer // requires preloading cache with data
        return GetStoredInteger(SyncData.Cache[2], alphabet, I2S(bool2I(StringCase(c, true) == c)) + c)
    endmethod

    private static method preloadChar2I takes nothing returns nothing
        local integer i = 0
        local string c

        loop
            exitwhen i >= ALPHABET_BASE

            set c = I2Char(ALPHABET, i)

            call StoreInteger(Cache[2], ALPHABET, I2S(bool2I(StringCase(c, true) == c)) + c, Char2I(ALPHABET, c))

            set i = i + 1
        endloop
    endmethod

    Example
    Code (vJASS):

    scope GameCacheSyncTest initializer Init

        private function SyncComplete takes nothing returns boolean
            local SyncData d = GetSyncedData()
            local string s = d.readString(0)
       
            call BJDebugMsg("Length = " + I2S(StringLength(s))) // Hello
            call BJDebugMsg(s)
            // clean up
            call d.destroy()

            return false
        endfunction

        private function StartTest takes nothing returns nothing
            local SyncData d = SyncData.create(Player(0))
            local string s = ""
            local string ten = "A12345678a"
            local integer len = 400
            local integer i = 0

            loop
                exitwhen i > len

                set s = s + ten

                set i = i + 10
            endloop

            call d.addString(s, len) // must define the (max) length of the string
            call d.addEventListener(Filter(function SyncComplete))

            call DisplayTimedTextToPlayer(User.Local, 0, 0, 15, "Syncing...")

            // begin syncing
            call d.start()
        endfunction

        private function Init takes nothing returns nothing
            call TimerStart(CreateTimer(), 0, false, function StartTest)
        endfunction

    endscope
     
    Last edited: Feb 28, 2017
  13. TriggerHappy

    TriggerHappy

    Code Moderator

    Joined:
    Jun 23, 2007
    Messages:
    3,672
    Resources:
    22
    Spells:
    11
    Tutorials:
    2
    JASS:
    9
    Resources:
    22
    Updated, v1.2.4.
    • Updated to support new version of SyncInteger.
    • Char2I is now preloaded through a hashtable rather than a gamecache.
    I updated demo map on the first post. It contains a little benchmark.
     
  14. IcemanBo

    IcemanBo

    Joined:
    Sep 6, 2013
    Messages:
    6,190
    Resources:
    22
    Maps:
    3
    Spells:
    11
    Template:
    1
    Tutorials:
    4
    JASS:
    3
    Resources:
    22
    Submission: Sync v1.2.4
    Date: 1 March 2017
    Status: Approved
    Note:

    The desync issue seems fixed and all works for me.
    Triggerhappy made improvements and added also more functionality, it is very good.
    No reason why it should not be used, approved.
     
  15. Quilnez

    Quilnez

    Joined:
    Oct 12, 2011
    Messages:
    3,268
    Resources:
    37
    Icons:
    2
    Tools:
    1
    Maps:
    7
    Spells:
    21
    Tutorials:
    2
    JASS:
    4
    Resources:
    37
    Looks like calling
    ReloadGameCachesFromDisk()
    will bug this system if called after the system's initialization: on sync callback readString will return empty string with length of specified max string length + 1. The solution is to use init module for calling that ReloadGameCachesFromDisk function so it would run before the system's initialization. I think it worth noticing.
     
  16. Waffle

    Waffle

    Joined:
    Jul 30, 2013
    Messages:
    271
    Resources:
    0
    Resources:
    0
    hmm.. fails to compile with pjass error :( :

    "Cannot convert player to index" in SyncData.readBuffer

    Code (vJASS):

    ...
    // if everything has been synced
        if ( b ) then
            if ( not s__SyncData_localFinished[data] ) then // async
                set s__SyncData_localFinished[data]=true
                // notify everyone that the local player has recieved all of the data
                call SyncInteger(s__SyncData_LocalPlayer , data) //THIS LINE
            endif
       
        endif

    ...
     



    "Cannot convert filterfunc to code" in SyncData.onInit
    Code (vJASS):

        call OnSyncInteger(Filter(function s__SyncData_updateStatus)) // THIS LINE
               
     
     
  17. TriggerHappy

    TriggerHappy

    Code Moderator

    Joined:
    Jun 23, 2007
    Messages:
    3,672
    Resources:
    22
    Spells:
    11
    Tutorials:
    2
    JASS:
    9
    Resources:
    22
  18. Waffle

    Waffle

    Joined:
    Jul 30, 2013
    Messages:
    271
    Resources:
    0
    Resources:
    0
    thx, will try in a moment.
     
  19. TriggerHappy

    TriggerHappy

    Code Moderator

    Joined:
    Jun 23, 2007
    Messages:
    3,672
    Resources:
    22
    Spells:
    11
    Tutorials:
    2
    JASS:
    9
    Resources:
    22
    Updated, v1.2.5.
    • Updated to support SyncInteger's SelectionSync struct.
    • Fixed some errors in the documentation.
    This system seems to work much faster on the newest patch. Here are some benchmarks I took using this system:

    [1.28.5]

    1MB (250,000 ints) : 1044.739 seconds (957 bytes/s)
    10KB (2500 ints) : 0.590 seconds (4,327 bytes/s)
    40KB (10000 ints) : 26.882 seconds (1,487 bytes/s)

    [1.26.0]

    1MB (250,000 ints) : ? seconds (? bytes/s)
    10KB (2500 ints) : 19.887 seconds (502 bytes/s)
    40KB (10000 ints) : 124.035 seconds (322 bytes/s)
     
  20. Quilnez

    Quilnez

    Joined:
    Oct 12, 2011
    Messages:
    3,268
    Resources:
    37
    Icons:
    2
    Tools:
    1
    Maps:
    7
    Spells:
    21
    Tutorials:
    2
    JASS:
    4
    Resources:
    37
    I wonder is the synchronization fully reliable? I just recognized that sometimes it's failed to sync things yet it still fires the "on sync" event which obviously leads to sudden desync.

    I'm using it to synchronize player's mouse coordinate. What I can notice when the desync happen is that all sent data became zero. I will try to break it down:

    1. This is how I sync things:
    Code (vJASS):

    set data = SyncData.create(p)
    call data.addReal(GetMouseX())
    call data.addReal(GetMouseY())
    call data.addInt(playerId)
    call data.addEventListener(Filter(function OnSync))
    call data.start()
     


    2. This is how OnSync function looks like:
    Code (vJASS):

        private function OnSync takes nothing returns boolean
     
            local SyncData d = GetSyncedData()
            local real x = d.readReal(0)
            local real y = d.readReal(1)
            local integer id = d.readInt(0)
            local player p = d.from
            local string s
     
            if Locale == p then
                set s = "war3mapImported\\point_target.mdx"
            else
                set s = ""
            endif
            call DestroyEffectTimed(AddSpecialEffect(s, x, y), 5.) // Use TimedHandle from Jass section
            call Fighter[id].move(x, y) // << Desync here. Basically it orders main unit to move to (x, y)
     
            return false
        endfunction
     


    3. I used multiplayer emulator to test my map. I noticed the syncing failure by controlling client-2's character (
    Fighter[1]
    ) by moving it around. What I noticed when it's desync'ed is that it's not client-2's character who's receiving the order, instead it's client-1's character (
    Fighter[0]
    ), and it's moving toward the center of the map. It means that
    Code (vJASS):
            local real x = d.readReal(0)
            local real y = d.readReal(1)
            local integer id = d.readInt(0)

    all returned zero. It's like what you are gonna see in this video:


    If you watch carefully, you may notice that when the desync happened, the archer (in client-2's side) is heading toward the correct target point. Meaning that the sent data were correct. While in client-1's side all sent data became zero, that's why in client-1's side Fighter[0] was the one who's receiving the order (sent id became 0) and it's moving toward the center of the map (x & y became 0).

    I'm using latest version of this system and patch version 1.27b. If I'm doing wrong in my code please let me know.
    I hope you can solve this issue. :) Thanks in advance!


    EDIT:
    Damn, I'm sorry before. I've found out that it's all because of a mistake on my end. I forgot to destroy the instance when it's finished syncing. So that hasInt, etc will return true immediately when it uses the same mission key. That's why the sync event got fired before things are actually synced. Sorry again. : (
     
    Last edited: Aug 31, 2017