Synchronized Heightmap of GetLocationZ().

[ScrewTheTrees]

Introduction

This is not a finished solution that is ready for normal usage, i don't recommend using my example project in a serious project since its very rough and unpolished Proof of concept.
Its somewhat of a deep dive into my findings when experimenting with with my theory of a "SafeLocationZ()" native of a synced HeightMap.

I had two general ideas on how to easily achieve this, Compile time and OnMapStart. After careful consideration i made the decision that OnMapStart would be easier for a quick proof of concept as CompileTime would require me to more properly read/write model data to generate the destructables heights and all that.

While i already have data containers for reading map files and model files, combining these and generating said HeightMap would be incredibly hard to do true to form, i have however considered trying something like that for HiveWE in the future.

I also uncovered some MISC data about BlzSendSyncData()

Resources
ScrewTheTrees/HeightMapSyncSafe - My test project.


The Data

First of all, i work mainly with typescript as a personal preference, this is also very doable in LUA, but for JASS i do not recommend this, hence my warning at the start, its also very rough and not something i would distribute as a library.

Let's get to the meat of things.

Warcraft 3 has for long had a deep resentment for the Z coordinate, it has always been unsafe to use and lately it has been even unsafer even with all players being on SD, my idea was that instead of fetching that, one would pre-generate a map that is synchronous for all users and fetch coordinates from that.

So i settled on simple standards: 480x480 map, with a tile resolution of 128x128, this amount to 230,400‬ nodes that need to be sent and made synced. Luckily the height is not supposed to be above/below 8000 which helps out later.
Naturally that would be quite a high step between nodes so i used a Gradient interpolation to smooth out the results.

Now to the sending of the data, i opted for the BlzSendSyncData native and events as they seem the most practical for my uses.
So the first test was simple, Iterate all nodes and send them as CSV. ( X;Y;Z )
While in single player this was instant, my performance tests were done over LAN and should be somewhat similar over the internet due to the low bit usage, also due to another reason brought up later.

So to start:

Build 7:
CSV, one per sync:
1213 seconds.
5 bytes - 15 bytes

Clearly this is far above any usability, people do not want to wait 20 minutes to start playing the game.
At this stage i was obviously worried this was not gonna get anywhere but my next experiment, occurring after quite a bit of refactoring was more promising.

Build 94:
16 bits to 2 char:
904 seconds
Always 6 bytes;

3/4th of the time of the previous attempt! that is quite an improvement by chopping down the average number of bytes by approx half (edge cases and all that).
Now it just writes X,Y,Z as 2 bytes each.

However at the time i didn't know but, this test run is not entirely truthful, the encoding i wrote was quite primitive, squashing the number into 2 chars :

Spoiler: Int16ToChars

Int16ToChars(num: number): string {
        let num1 = (num + this.int16Offset) % 0x100;
        let num2 = ((num + this.int16Offset >>> 8)) % 0x100;

        return string.char(num1)
            + string.char(num2);
    }

StringToInt16(str: string): number {
        let char1 = str.charAt(0);
        let char2 = str.charAt(1);

        let x = (string.byte(char1));
        x += (string.byte(char2) << 8);
        x -= this.int16Offset;

        return x;
    }

(>>> is due to typescript to lua not liking >>)

This had a huge flaw, 0, you see, it seems like strings passed to BlzSendSyncData are null terminated, which means that string.char(0) would cause the string to be far shorter and exclude data.

Build 157:
14 bits to 2 char, assemble into byte chunks of x,y,z:
345 seconds
240 bytes;

The huge difference from the previous attempt was simply that instead of just sending 6 bytes of data per sync, i instead filled up XYZ up to 240 bytes per sync string.
This seems to have removed a lot of overhead that happens in the background, allowing for synchronization for a lot more data.

As you can see i reduced 16 bits to 14 bits, this was the easiest way to dodge the issue of null terminated strings yet still large enough for my purposes.


Now we are reaching some type of victory lap, that is when my good friend @MindWorX suggested me to do a proper chunk system to cut down the amount of data further.

With a proper chunk system we could exclude the majority of XY boilerplate and effectively reduce the packet size with slightly less than 66%, for those who want to know, one simply writes x,y for starting position, and optionally width/height for more dynamic sizes once at the start of the packet, then you just loop through and write the raw Z data in programmatic order.

Build 200:
14 bits to 2 char, assemble into 8x8 chunks with x/y/width/height once in header and the rest raw Z data:
120 seconds.
128+8 bytes

Neat, 120 seconds, that is almost acceptable!
Now there is some technical details about how i deal with this, i have a lua coroutine, writing one of these packets every 0.01 seconds, at the start i sent it all instantly but it had its issues so i had to do it that way.
You see there is this issue where BlzSendSyncData() prevents you from ordering units.
However this test still allowed for the ordering of units, leading to me grasping that this native steals the normal bandwidth of giving units orders, something that also clued me into it is that it seems synced to warcraft 3's turn timer (only occuring every 0.1 seconds/100ms in batches).

This became more clear with this:

Build 202:
14 bits to 2 char, assemble into 11x11 chunks with x/y/width/height once in header and the rest raw Z data:
108 seconds.
242+8

Unlike build 200, 202 would cause the syncing player to not be able to give orders, essentially overflowing some buffer with limitations, gotta follow those Dial up standards.

In the end i opted for 8x8 chunks as my final approach to this problem.

Conclusion

Synchronization of the entire map Z seems to some degree possible, there are however a lot of challenges related to this.

• My testing case used a quite wide resolution of 128x128, this means that cliffs and destructibles act horrendously with the interpolation, one would need a lower resolution to achieve that with good results.
• There is a waiting time until the map is 100% filled in, this can however be mitigated by synchronizing the chunks according to some priority queue that uses the positions of projectiles/units as priority spots.
• It requires LUA to be done effectively, JASS would be some kind of headache, but you are welcome to try.
• If you want to update the map after deformations one would have to implement updating of chunks.

For now i will put this on the shelf, i might revisit the concept later when i have a more proper idea for improving it, i do have some ideas but nothing set in stone yet.



Thank for for reading this needlessly long and perhaps useless article, however for persevering through it do have some possibly useful information:

Spoiler: BlzSendSyncData Information

BlzSendSyncData:
* Max length: 255
* Is null terminated, so dont use string.char(0)
* There is some max quota when sending data, but sending large strings is far more efficient than many strings, so probably some overhead.
* Running too much data through this at once is gonna stop all player interaction. (Like giving unit orders)
* Only stops the player interactions for the players who is sending the data.
* Seems synchronized to the turn timer of the game (100ms / 10fps).

 

[Zwiebelchen]

Its a shame you didn't really get this to a satisfying conclusion with a working system. But nice finds about BlzSendSyncData either way. Might be useful for those going nuts with codeless save/load systems.

 

[ScrewTheTrees]

Its a shame you didn't really get this to a satisfying conclusion with a working system. But nice finds about BlzSendSyncData either way. Might be useful for those going nuts with codeless save/load systems.

Yeah, i have a few theories on how to possible get it working for general use cases, but i expect it to take quite a bit of time and research along with a few headaches. Due to data limitations when sending data through wc3 there will definitly be compromises in such as system such as initial imprecision or a period before use.

Another idea is a priority queue that tries to load in chunks closer to specific points (like heroes) first and allows gameplay while doing its bushiness in the background.

Thanks for reading

 

[Cokemonkey11]

Instead of synchronising game state, I can recommend using virtual heightmap. This even works with terrain deformations.

I can extract this to a wurst library if it would be useful to others:

Cokemonkey11/BastardsAndConquerors

 

[ScrewTheTrees]

Instead of synchronising game state, I can recommend using virtual heightmap. This even works with terrain deformations.

I can extract this to a wurst library if it would be useful to others:

Cokemonkey11/BastardsAndConquerors

Yes it was my other idea for this, instead i opted to this since i wanted to adhere to walkable destructables and have if follow the same logic with water.
Your approach is currently far more usable as my approach needs more R&D to even work in a map at all