[Crash] Desync/Crash on map init, but just sometimes?

[Eikonium]

Hi guys,

I'm currently troubleshooting a desync/crash occuring with my current project and need a bit of help on brainstorming its possible cause.

The issue is:
Map sometimes crashes in online multiplayer (privately hosted) for some of the players in the moment the map goes from loading screen to ingame state. The unaffected players will still be able to play the game, seeing that the crashing players have left the game.

I know how to debug fatal errors, but I can't really grasp this one, because:

• The game will just sometimes crash (between 10% and 75% of hosting attempts, different every day).
• When the game crashes, it only does for some of the players. The others can proceed just as normal.
• Crashes only happen in multiplayer, never in singleplayer (i.e. never with the world editor testmap functionality).
• Also happens, when all players are playing in classic mode (no Reforged graphics).

Such a crash can't be properly reproduced (thus I struggle investigating), but at least crashing players do receive a proper crash report. I've attached one of them for you (in case interesting).

So the fact that only some players crash does hint for a causative desync, right? So a desync that only happens sometimes and causes one group of players to crash in the second of entering the game.

If anybody has an idea about where this might come from, I'd be very happy

Thanks in advance for your help and best regards
Eikonium

 

[IcemanBo]

^Address where is crashes.

The above highlighted line will run, which is a killcode which will always crash. The 1 line above "rax" calls a function, that would (probably) prevent from ever returning, so then the killcode would never run.

Code design looks like anti cheat protection. Maybe it's some sort of timing problem, inside the "call rex" that will lead to errors, so the function will properly return and killcode will run.

From logs there were hooks by other programs which don't belong to the game. Example

• Sophos (anti virus program)
• Display Fusion (probably software you use for gaming)

do maybe all players you play with use Dusplay Fusion? Could you try to disable both listed programs? Maybe they affect wc3 and give some bad timing problems. If you have, you can upload more "crash.txt" files to compare.

 

[Dr Super Good]

Reforged graphics? If so is your GTX 960 the 2 or 4 GB model? The 2 GB model could easily run into OoM problems when running Reforged at high textures.

 

[Eikonium]

Code design looks like anti cheat protection. Maybe it's some sort of timing problem, inside the "call rex" that will lead to errors, so the function will properly return and killcode will run.

Are you referring to warcraft internal code or to the JASS script I've written into the map? I'm honestly not too familiar with the details you elaborated on

do maybe all players you play with use Dusplay Fusion? Could you try to disable both listed programs? Maybe they affect wc3 and give some bad timing problems. If you have, you can upload more "crash.txt" files to compare.

Only I am using Display Fusion, while Sophos might actually be installed for my friends as well. The crashs however are not restricted to a specific person. Sometimes it's me, sometimes others. I'll test with deactivated programs on next occasion, but would believe that these programs are not the cause, because the crashes are not bound to my machine. Thanks for the suggestion!
I've attached a second crash.txt coming from a different machine. Let me know, if you need more. I can definitely ask friends to send their log files across. Thanks a lot!!

Reforged graphics? If so is your GTX 960 the 2 or 4 GB model? The 2 GB model could easily run into OoM problems when running Reforged at high textures.

The 4GB model and we've all been playing on SD graphics (furthermore it's sometimes just the others crashing). Thanks for your suggestion!

 

[IcemanBo]

Are you referring to warcraft internal code or to the JASS script I've written into the map?

It's not your map code, but warcraft code.

At second crash, it crashed at exact same position.
Sophos and Display Fusion, and also no other hooks... nothing there. It's a other map that crashes here, but this crash at least was probably not by external programs.

Graphic card should not be problem either.
1st graphic card is a dedicated Nvidia, crash sometimes
2nd graphic card is on board Intel, crash sometimes, too

Is it only in your project? I ask because

1st crash.txt => TowerRush.w3x
2nd crash.txt => Privat/Brawl 1.3.w3x

maybe crashes are random for other maps, too, not related to your project? This might be important to know.

===

Crash is definitly expected at this very position, as said it's some killcode that will always lead to crash. But seems like Warcraft 3 bug that it even jumps there, if you don't try to cheat or so.

Maybe it's testable if there is specific code in the map, that leads to the crash. Maybe some hacky stuff happens "onInit" triggers, or so? If you're very unsure you could try to switch from init to "Game Time 0 Seconds elapses" or so... but not sure it does help.

rash.txt coming from a different machine. Let me know, if you need more.

I think not required atm.

 

[Eikonium]

The two maps are actually the same project, just different versions (yes, both are my map).
Init triggers just setup data structure for the most part, but I will play around with them and see, if I can achieve results with replacing "Map Init" with "Game Time Elapsed 0".
Will be hard to see, though, because the crashs come so rarely on some days that I can never be sure that I fixed it even after 20 successful hosting attempts.

I'm still wondering, if the desynchronous crashing behaviour is telling us something... Are there any known reasons for desynced crashs during loading screen?

Just to check, might code like this lead to issues?

function Trig_A_Player_Leaves_Game_Actions takes nothing returns nothing
    //actions
endfunction

//===========================================================================
function InitTrig_A_Player_Leaves_Game takes nothing returns nothing
    local integer i = 1
    set gg_trg_A_Player_Leaves_Game = CreateTrigger(  )
    loop
        exitwhen i > udg_NUM_MAX_PLAYERS //referring to GUI variable here, which is initialized with a value in the variable editor. Can this be desynchronous at times? (Might GUI var initializing come after this init function for some players?)
        call TriggerRegisterPlayerEventLeave( gg_trg_A_Player_Leaves_Game, ConvertedPlayer(i) )
        set i = i+1
    endloop
    call TriggerAddAction( gg_trg_A_Player_Leaves_Game, function Trig_A_Player_Leaves_Game_Actions )
endfunction

 

[IcemanBo]

Is there a message that they desynced, maybe it just crashes?

The code is fine. And GUI vars are always initialisized first, before triggers run. As in JASS there's no multiple threads or so, nothing runs parallel, it also shouldn't be a problem.

Normal JASS should never crash wc3, as it's just being interpreted, and no native code. It's always a bug when it does.
There are some code that wiill just always lead to crash, like creating multiboard on init, but the problem is that it happens here only sometimes makes it weird / makes believe it's an timing problem for warcraft.exe.

Still would be interesting, if crahes occur for random maps, or specifically at the project

 

[Eikonium]

I have guessed this to be a desync, because the crash often just occurs for some players. Remaining players will receive an ingame message "Player X has left the game" right when the game starts. Is there any method of truely knowing that the players desynced before crashing?
I am used to the fact that wrong JASS code (like referring to Player(-1)) would either crash all players or none.

It only crashes for my project (as far as I'm concerned).

Sounds like a timing problem indeed!

 

[IcemanBo]

Sadly idk what exact game messages at desync will show, or if by default any message would show at all, without custom triggers. Playing too rarely atm. :/
Since it crashes only for your project, would you mind to share the map (pm/discord would work)?

 

[Eikonium]

Sure thing, I've sent it via Discord! (Thanks a lot!)

 

[Teo_omant]

Hi guys,

I'm currently troubleshooting a desync/crash occuring with my current project and need a bit of help on brainstorming its possible cause.

The issue is:
Map sometimes crashes in online multiplayer (privately hosted) for some of the players in the moment the map goes from loading screen to ingame state. The unaffected players will still be able to play the game, seeing that the crashing players have left the game.

I know how to debug fatal errors, but I can't really grasp this one, because:

• The game will just sometimes crash (between 10% and 75% of hosting attempts, different every day).
• When the game crashes, it only does for some of the players. The others can proceed just as normal.
• Crashes only happen in multiplayer, never in singleplayer (i.e. never with the world editor testmap functionality).
• Also happens, when all players are playing in classic mode (no Reforged graphics).

Such a crash can't be properly reproduced (thus I struggle investigating), but at least crashing players do receive a proper crash report. I've attached one of them for you (in case interesting).

So the fact that only some players crash does hint for a causative desync, right? So a desync that only happens sometimes and causes one group of players to crash in the second of entering the game.

If anybody has an idea about where this might come from, I'd be very happy

Thanks in advance for your help and best regards
Eikonium

Hello. I have exactly the same problem. Did you manage to fix it?

 

[Eikonium]

It just stopped from happening at some point, don't know why, maybe due to some patch or after I converted it to lua.
Also I haven't tested my project in multiplayer for a while, so I'm not even sure, if the bug is really fixed.
Are you using the latest patch?
Besides, if you are using Lua, the pairs()-function apparently sometimes iterates in different order for different players, as @Forsakn pointed out here, possibly leading to desyncs. So try to avoid that iteration method, when possible.

If you happen to find a solution, please let me know!

 

[Forsakn]

It just stopped from happening at some point, don't know why, maybe due to some patch or after I converted it to lua.
Also I haven't tested my project in multiplayer for a while, so I'm not even sure, if the bug is really fixed.
Are you using the latest patch?
Besides, if you are using Lua, the pairs()-function apparently sometimes iterates in different order for different players, as @Forsakn pointed out here, possibly leading to desyncs. So try to avoid that iteration method, when possible.

If you happen to find a solution, please let me know!

Yup, using pairs() in multiplayer maps is a no-no, the bigger the table the more likely you are to get a desync. If you use pairs() you can add the snippet down below (it's from the Lua wiki) to your map and replace your pairs() calls with orderedPairs(). It should be "plug-and-play", then you won't have to do any big rewriting to find out if pairs() was the cause of your desyncs, then you might want to use ipairs or regular for-loops if performance is a concern (I doubt it will ever matter, I just tend to do this).

Spoiler: OrderedPairs.lua

--[[
Ordered table iterator, allow to iterate on the natural order of the keys of a
table.

Example:
]]

function __genOrderedIndex( t )
    local orderedIndex = {}
    for key in pairs(t) do
        table.insert( orderedIndex, key )
    end
    table.sort( orderedIndex )
    return orderedIndex
end

function orderedNext(t, state)
    -- Equivalent of the next function, but returns the keys in the alphabetic
    -- order. We use a temporary ordered key table that is stored in the
    -- table being iterated.

    local key = nil
    --print("orderedNext: state = "..tostring(state) )
    if state == nil then
        -- the first time, generate the index
        t.__orderedIndex = __genOrderedIndex( t )
        key = t.__orderedIndex[1]
    else
        -- fetch the next value
        for i = 1, #t.__orderedIndex do
            if t.__orderedIndex[i] == state then
                key = t.__orderedIndex[i+1]
            end
        end
    end

    if key then
        return key, t[key]
    end

    -- no more value to return, cleanup
    t.__orderedIndex = nil
    return
end

function orderedPairs(t)
    -- Equivalent of the pairs() function on tables. Allows to iterate
    -- in order
    return orderedNext, t, nil
end

 

[Teo_omant]

I code in jass. I haven't found the reason or the resolution yet.

 

[Eikonium]

@Forsakn I have recently tried to verify the asynchronous behaviour of pairs() online with a friend by printing through a few non-array tables, but we couldn't manage to print any different order on our screens. Do you maybe have some example code that could help us with the verification?

 

[Dr Super Good]

@Forsakn I have recently tried to verify the asynchronous behaviour of pairs() online with a friend by printing through a few non-array tables, but we couldn't manage to print any different order on our screens. Do you maybe have some example code that could help us with the verification?

It may be non trivial to verify it. At least you want to repeatedly add and remove mappings from the table and allow a few garbage collection cycles to pass. Garbage collection cycles can be detected by creating and throwing away a large, finite number of tables and waiting until Warcraft III memory usage drops down.

In base Lua there is no guarantee as to the order the table is iterated. However it is possible that with one of the patches they added a well defined order.

 

[Eikonium]

I wasn't aware of that garbage collection can affect the behaviour of pairs(). Isn't it deactivated in the current patch anyway?

Yeah, base Lua pairs doesn't have a guaranteed iteration order. Which is, why we thought Blizzard might have overwritten pairs to make it synchronous, until @Forsakn reported desyncs with the Global Initialization library, when it was still using pairs.

 

[Teo_omant]

Have you used Wc3 map optimizer in your map?

By the way, do you still have the jass version of your map that crashing? It would help me to find at least some pattern.

 

[Eikonium]

Sure. Attached is a very early JASS state of development and I remember it did desync/crash on this version on nearly every map start in multiplayer.

I later tried to find the issue with a few friends by starting it over and over in multiplayer, deactivating more and more triggers, until the desyncs stop happening. But at the time we tried, the issue was already gone. At least we couldn't reproduce it anymore.

If it still crashes for you, that's maybe something you can try?

 

[Teo_omant]

THANKS!!! I found the problem! The TriggerRegisterPlayerEvent() and with bj function called when creating a trigger at the start of the game results in desync. I assigned mouse and keyboard events after 0.00 - crashes stopped.

function InitTrig_PlayerMousePosition takes nothing returns nothing
local integer i= 1
set gg_trg_PlayerMousePosition=CreateTrigger()
loop
exitwhen i > udg_NUM_MAX_PLAYERS
call TriggerRegisterPlayerMouseEventBJ(gg_trg_PlayerMousePosition, ConvertedPlayer(i), bj_MOUSEEVENTTYPE_MOVE)
set i=i + 1
endloop
call TriggerAddAction(gg_trg_PlayerMousePosition, function Trig_PlayerMousePosition_Actions)
endfunction

You had a crash because of this piece of code

 

[Eikonium]

Wonderful, how did you spot the issue so quickly?
Did your own map have the same event, so your issue is also fixed?

I wonder, why TriggerRegisterPlayerMouseEventBJ desyncs before map start. Maybe loading is not synchronized and the event can actually fire during loading screen?
I suppose, the same thing happens with the non-BJ version TriggerRegisterPlayerEvent?

Thanks a lot for finding the reason!

 

[Teo_omant]

I suspected this function, but did not pay attention, because its action was turned off for me. But it doesn't matter. Desync occurs from the event itself. The meaning is this: triggers are created during the loading screen, and events are created, so the mouse move event is already working. And all players can have different loading speeds. Because of this, desync.

I checked other similar functions, like BlzTriggerRegisterPlayerKeyEvent. If you register, for example, F1 when initializing the trigger and press F1 during loading - desync. But the chance is not 100%.

 

[Eikonium]

Really good find, thank you for checking!