[Requires investigation] The easiest method to synchronize Z values...

[Macadamia]

Hello dear community !

Reading about the issues caused by the asynchronous Z values returned by the different related native, I have found a workaround, with the feedback of @crunch.

The idea is really simple : the issue with the Z values is that the natives to get them can potentially return different values, depending on your operating system and your graphic settings, and other things like terrain deformations not affecting the height equally across clients.

Some clever people are even currently working on ways to synchronize the entire map's Z values, or even create a Height Map at compile time.
The Height Map would probably work perfectly, but is not that easy to implement for the normal map maker.
As for synchronizing the entire z values with SendSyncData native, in spite of the amazing work done by @ScrewTheTrees, it currently still takes too much time.

Trying to synchronize Z values on the fly is unpractical, because of the delay introduced, the Z values cannot be used immediately.


My method doesn't need anything complex or heavy at all.

It is based on the idea of Reasonable Precision Loss Compromise.

The principle is really basic : there is a difference between Z values returned by the natives.
Let's call this difference ZDelta.

If you reduce the Z values precision by ZDelta, you will remove all differences between Z values returned by the natives.

To make sure this was possible, I first made a map with messy and uneven terrain, with walkable doodads, various heights, various cliff heights, different water depths, etc...
Then I placed units around at map init.
I added a function triggered by a key event that displays all the units Z values given by BlzGetUnitZ.

For my test, I ran 2 instances of Reforged in LAN :
- one instance with the lowest graphic settings on SD
- one instance with the highest graphic settings on HD

Then I tested the map and compare the results for both clients.
My results showed that the highest ZDelta was just over 16.0.
Some might argue that it is a bit high, but if you synchronize Z values, you'll have the same problem.

So If we consider this value with a small extra margin, we end up with a maximum ZDelta of 20.0

I believe a precision of 20.0 on the Z axis is not too bad considering height values can go from 0.0 to 8,000.0. You could also try slightly smaller or bigger ZDelta to favor Precision or Safety.

Now let's use the simple maths :

All we need is reduce the Z natives precision to never be lower than 20.0.
The easiest way to do that is simply by dividing the Z value by 20.0, Round the result to the nearest integer, and multiply that integer by 20.0.

Here is an example function made in Lua :

function Round(r)
    if (r<0) then return math.floor(r-0.5) else return math.floor(r+0.5) end
end

function GetApproximateZ( myUnit )
    return (20.0 * ( Round( BlzGetUnitZ( myUnit ) / 20.0 ) ) )
end

You can apply this simple trick to any complex system using Z values, and you will have real Z values, with the only issue being a loss of precision.

You could also hook all Z related natives and keep your code identical in Jass, Lua and GUI or other scripting languages.

NOTE THAT THE LOSS OF PRECISION CANNOT HAVE AN IMPACT WORSE THAN SYNCHRONIZING THE Z VALUE OF ONE OF THE CLIENTS.

I hope this will help people with desync issues related to the Z natives...

EDIT :
IT SEEMS THIS WAS A FALSE HOPE :
The issue is that, because the Z async values are continuous, there is no way to predict what bound of the precision interval the approximated value will end up.
I am currently trying to identify a pattern that would allow to differentiate between the precision interval bounds.

 

[IcemanBo]

It sums up some thing I wasn't aware of, thank you!

How is ensured Z values are sync?

• ClientA UnitZ=90 -> will approximate to 100
• ClientA UnitZ=85 -> will approximate to 80

am I missing something?

 

[Macadamia]

Yes, because we have to chose a ZDelta.

The higher the ZDelta, the more we lose precision to gain safety, and vice-versa.

I chose 20.0 as it was a safe value for the maximum differences I experienced between low SD and high HD settings.

Note most values are usually not async. On cliffs, they seem identical.
Ramps, bridges and other walkables seem to be where the async values start to show.

Also note that a value of 20.0 for Z is not too noticeable.

EDIT :

Sorry I misunderstood your point.
You are right, rounding the values is not the proper method, we probably need to calculate an average rather than a simple rounded value.
Will update the first post.

 

[IcemanBo]

Sorry, I still don't understand. If there's a delta in result, it's not sync? From my example, clients will have delta of 20, while in real they would have delta of 5?

 

[Macadamia]

Actually maybe I was too optimistic there.
My idea was that reducing precision would make ZDelta become neglictible and thus would not produce different Z values.

The problem is that we don't know at what "distance" the Z returned by the native will be from the others.
I will look into this, but it might have just been false hope, in what case I will modify the first post accordingly.

 

[IcemanBo]

Have you maybe recognized a pattern of different Z values? Like maybe one could calibrate/sync some test values at map start, to find out clien'ts graphic conditions, and to know if to go to upper bound/lower bound in later game.
Most likely there's no pattern, but just an idea.

 

[Macadamia]

Yes I have.

The lower values were all on SD with low graphic settings.

I am not 100% sure but it might be possible to use this to chose the bottom final value.

EDIT :
Actually there is an exception that behaves in an opposite manner.
I suppose I should multiply the different situations and try to find a reliable pattern.

UPDATE :

I will simply post here what I posted on Discord to expose the latest findings with the help of @ScrewTheTrees :


Is anyone running on a Mac or an unconventional PC ?
First tests seem to imply that terrain made in the editor(saved in w3e) is actually synced even on different hardware*
But we've only compared on a i7-6700HQ (mobile proc) and a i9-9900K (so desktop) machines so far
Graphic settings (especially SD vs HD) seem to be only impacting Z where there is a walkable/flyable widget not on original terrain(edited)
If this is true it would be very easy to sync the limited amount of impacted tiles (except on big maps with tons of walkable widgets
In other words the terrain itself would already be sync, and preventing LocationZ and UnitZ desyncs would be really easy
On maps creating these widgets in the map script rather than in the WE, it would be even easier as one could save the terrain height for the tiles occupied by the widget before this one is placed.
Then GetLocationZ or UnitZ would just need to return the usual value, except on the tiles occupied by widgets that would then be saved.

The fact is, if this was confirmed with a few more tests, that would completely make sense as the terrain data is stored in the w3e map file and read by the game engine, so there is no reason why the terrain itself should be async, even on different hardware (we could confirm this with results from more different hardware.

I am attaching the quick map I made that we used to compare our results, and the results were identical on Reforged with highest HD settings.
- We have not tested with SD as I already noticed that SD returns different Z values ONLY where there are widgets placed on the terrain.
- All you need to do is run the map on Reforged with max HD settings and once ingame press the "Z" key to display the Z values, a,d then take a screenshot..

Here are the identical results we both obtained :

(Note that I have not tested this with trees and flying units above)


Thanks to anyone who would also run the test on their machine to confirm the hardware does not impact terrain Z in normal conditions.

 

[IcemanBo]

Yes it sounds plausible that it's a model HD/SD problem that results in different height.
Something like terrain size or cliff height are defined by blizzard constants, and should be always the same for all graphics.

Edit:
Here's a thread about detecting HD/SD. [vJASS] - Detect Reforged

Edit2:
@Blarto has also a thread about GetZ. Reforged - [DESYNC] - 2 Possible causes found
He also explains, aginst the opinion I had, that terrain without models can cause desyncs, if it's steep cliffs.
( steep cliffs is if you highten your terrain about max using modified MiscData.txt you locate under "Warcraft III\war3.mpq\UI\MiscData.txt" and have a modified/higher "MaxSlope" value, above "86" to have no height limit) Maybe you haven't considered steep cliffs in your test, and it would bringt bad results.

After readig all this, I get more opinion that it's probably better not to rely on using synced Z values.

 

[Macadamia]

Yes I see you point.

I indeed didn't use steep cliffs. But I suppose people ready to alter the game limits have to expect collateral issues.

Fact is, I strongly believe that when keeping in the bounds the game was set up for would be safe, except of course when models are not rendered identically.
In such circumstances, there is no harm in trying to find workarounds, like the community has spent so much timer trying to do.

Fact is, a fix that would just take models into account would never suffer from dysfunction if Blizzard fixes this bug.

As the steep cliffs, maybe the engine simply has its own set limits dating from old times. If that is the case Blizzard could easily fix this. But will they ?

 

[IcemanBo]

Steep cliffs height causing desync is imo a bit weird. But more, I believe syncing height values, for gameplay, others than pure visuals, is probably more relying on game-altering than using steep cliffs.
Especially because of blizzard does not want it synced by nature: common.j -- patch 1.31.1 | HIVE

// This function is asynchronous. The values it returns are not guaranteed synchronous between each player.
//  If you attempt to use it in a synchronous manner, it may cause a desync.
native GetLocationZ            takes location whichLocation returns real

 

[Macadamia]

I see your point.

The problem relies on system that need to use Z like projectiles and such.
Mind you, thinking about it, I don't really know what prevents these systems to use flying height instead unless they'd need negative values.

IIRC the current flying height of a unit is a sync property.

 

[IcemanBo]

FlyHeight alone can be async, as in @Blarto 's thread from above, it's widely discussed. But also there in thread, it's mentioned that it becomes a problem in combination with steep cliffs.
Modern missile systems should rely on efffects, which are just pure visuals and don't affect gameplay, so can be async for sure.

 

[Macadamia]

Yes thats probably the best way to avoid issues I admit

 

[Blarto]

Hi, I got mentioned and I have a workable solution (already time tested in live games). However I didn't finish developing it for general use, so I'll just describe what I did.

Firstly I'd like to say Blizzard fixed several desync issues as pertaining to Bridges and Walkables. They might have also resolved the steep cliff issue but not sure, I stopped caring about it as I've fixed it in a different way.

Here is what is Known and ASSUMED
- (FACT) Syncing 256x256 values is not practical, let alone 512x512
- (FACT) The [vJASS] - Missile uses xefx, thus FlyHeight, thus potential desyncs
- (ASSUMPTION) It is too expensive for you to write&test a new Missile/projectile library
- (ASSUMPTION) You have access to 1.32 and 1.31 map making tools (1.31 doesnt require bnet launcher, has reforged jass features, thus superior)

How to solve
Easiest solution is to assume 100 Z height and be done with it. it will screw your missiles but map wont desync anymore.

Proper solution would be storing Z values for lookup, but it needs synchronisation, or does it? Instead of syncing 256x256 values, then using those values to calculate 256x256 continuous planes which you can then use to extrapolate accurate Z values, you "sync" at compile time, thus you are single player, for 256x256 map, calculate 512x512 or 1024x1024 grid of Z values, then encode all calculated values into base 64 string. Then you cram 100,000+ values into 128 Hashtables for fast lookup. This compile process takes several minutes as jass/lua + writing to file isnt exactly fast. In addition, you need an external tool to parse the resulting files back into usable scripts.

At actual play time, all the Z data is already there in form of jass/lua script, and all you need is to decode it. Here is an example of how that might look like.

This approach has several disadvantages, namely the Z's aren't 100% accurate, and every time you change the terrain you have to run a lengthy script. I stopped developing this tool since I'm pretty sure most people wont care, since this only affects the high-end maps that are actually multiplayer and played, which you can count with 2-5 hands.

    function GetZ takes real x, real y returns real
        return LoadReal(stripes[4000 + R2I(x/number_hastables)], R2I(x/32), R2I(y/32))
    endfunction

 

[Blarto]

Forgive the double post but this is pertaining to working the approach described above into something universal.

The best way in my opinion is to decode Z values directly from the map file (raw) directly, and then calculating the base64 strings externally.

This is possible but I didn't do it since jass already has natives to access (GetLocationZ+FileIO).

So my current toolchain is
Jass -> run map "in terrain mode" -> files -> java+parse files back to jass code -> back to jass -> (ready for general play) play map

A proper toolchain would be
java (or python or whatever) parse terrain files into Z values, calculate planes and save Z values into jass code -> (ready for general play) play map

 

[IcemanBo]

Firstly I'd like to say Blizzard fixed several desync issues as pertaining to Bridges and Walkables. They might have also resolved the steep cliff issue but not sure

Maybe it really was a bug, as terrain altering itself should probably really not have different gameplay values. In comparison, The thing with model height is of course something that may always occur with different models in HD/SD.

@AGD has worked on a new Missile library with using effects, which goes on in the end of your linked thread. [vJASS] - Missile

 

[KW~]

Yes I have.

The lower values were all on SD with low graphic settings.

I am not 100% sure but it might be possible to use this to chose the bottom final value.

EDIT :
Actually there is an exception that behaves in an opposite manner.
I suppose I should multiply the different situations and try to find a reliable pattern.

UPDATE :

I will simply post here what I posted on Discord to expose the latest findings with the help of @ScrewTheTrees :


Is anyone running on a Mac or an unconventional PC ?
First tests seem to imply that terrain made in the editor(saved in w3e) is actually synced even on different hardware*
But we've only compared on a i7-6700HQ (mobile proc) and a i9-9900K (so desktop) machines so far
Graphic settings (especially SD vs HD) seem to be only impacting Z where there is a walkable/flyable widget not on original terrain(edited)
If this is true it would be very easy to sync the limited amount of impacted tiles (except on big maps with tons of walkable widgets
In other words the terrain itself would already be sync, and preventing LocationZ and UnitZ desyncs would be really easy
On maps creating these widgets in the map script rather than in the WE, it would be even easier as one could save the terrain height for the tiles occupied by the widget before this one is placed.
Then GetLocationZ or UnitZ would just need to return the usual value, except on the tiles occupied by widgets that would then be saved.

The fact is, if this was confirmed with a few more tests, that would completely make sense as the terrain data is stored in the w3e map file and read by the game engine, so there is no reason why the terrain itself should be async, even on different hardware (we could confirm this with results from more different hardware.

I am attaching the quick map I made that we used to compare our results, and the results were identical on Reforged with highest HD settings.
- We have not tested with SD as I already noticed that SD returns different Z values ONLY where there are widgets placed on the terrain.
- All you need to do is run the map on Reforged with max HD settings and once ingame press the "Z" key to display the Z values, a,d then take a screenshot..

Here are the identical results we both obtained :

View attachment 358468

(Note that I have not tested this with trees and flying units above)


Thanks to anyone who would also run the test on their machine to confirm the hardware does not impact terrain Z in normal conditions.

cpu:1950x

 

[Macadamia]

cpu:1950x

Thanks, it again seems hardware has no impact on height getters.
In my own map, I have used an alternate route for dealing with the different results these gatters can give between SD and HD mode : I just don't use direct Z getters at all, and just use fixed Z values depending on the cliff level, and between ground and flying units that need a special effect / custom projectile.

This way there is no possible async Z values. But I also understand that this simplification is not at all appropriate in many circumstances...

Seeing that only SD / HD seem to impact Z, many desync issues could be fixed if Blizzard would normalize all model heights.

 

[Prometheus3375]

Ryzen 5 3550H, GeForce GTX 1650

 

[Macadamia]

Thanks, this is another confirmation.

So basically the terrain Z returns identical values.
The issue definitely lies within destructables / doodads or any kind of walkable as models don't have the same height in SD and HD.

An easy workaround would be to just save the cells that have such objects. Then the custom GetLocationZ could check if the location is on one of these cells, and if that is the case, return a value increased with a generic height, and if it is not on one of these cells, return the original value of the native.

Such a custom function would return the same Z value on every client, and thus no need for a sync...