• 🏆 Texturing Contest #33 is OPEN! Contestants must re-texture a SD unit model found in-game (Warcraft 3 Classic), recreating the unit into a peaceful NPC version. 🔗Click here to enter!
  • It's time for the first HD Modeling Contest of 2024. Join the theme discussion for Hive's HD Modeling Contest #6! Click here to post your idea!

Precomputed, synchronized height map

Antares

Spell Reviewer
Level 21
Joined
Dec 13, 2009
Messages
512
While doing 3D collisions, someone brought up to me the problem with terrain height not being synced between players, a problem I had completely forgotten about. I searched here on the forums if this is a problem that has been solved yet, but I found only posts around 2020 discussing the issue, but not providing any concrete solutions:

Synchronized Heightmap of GetLocationZ
[DESYNC] - 2 Possible causes found

Since there wasn't anything newer that I could find, I assume no one really tried to tackle that problem since then, at least not here on hive?

So, because syncing that much data in real-time doesn't work, I thought the next-best solution would be to simply precompute the height map.

You create the height map with a spacing of 128 on map initialization:

Lua:
    local function CreateHeightMap()
        local xMin = (worldMinX // 128)*128
        local yMin = (worldMinY // 128)*128
        local xMax = (worldMaxX // 128)*128 + 1
        local yMax = (worldMaxY // 128)*128 + 1
        local x = xMin
        local y
        local i = 1
        local j
        while x <= xMax do
            heightMap[i] = {}
            y = yMin
            j = 1
            while y <= yMax do
                heightMap[i][j] = GetLocZ(x,y)
                j = j + 1
                y = y + 128
            end
            i = i + 1
            x = x + 128
        end
        iMax = i - 2
        jMax = j - 2
    end

To get the terrain height at any location, you use this function:

Lua:
        ---@param x number
        ---@param y number
        ---@return number
        GetTerrainZ = function(x, y)
            --Find the bottom-left corner of the tile that the coordinates are in.
            local rx = (x - worldMinX)/128 + 1
            local ry = (y - worldMinY)/128 + 1
            local i = rx // 1
            local j = ry // 1
            rx = rx - i
            ry = ry - j
            --Safety check to prevent nil values if quering a location outside the world bounds.
            if i < 1 then
                i = 1
                rx = 0
            elseif i > iMax then
                i = iMax
                rx = 1
            end
            if j < 1 then
                j = 1
                ry = 0
            elseif j > jMax then
                j = jMax
                ry = 1
            end
            --Find out in which triangle the coordinates are in, then interpolate based on three grid points.
            if rx + ry > 1 then --In top-right triangle
                return (rx + ry - 1)*heightMap[i+1][j+1] + ((1 - rx)*heightMap[i][j+1] + (1 - ry)*heightMap[i+1][j])
            else
                return (1 - rx - ry)*heightMap[i][j] + (rx*heightMap[i+1][j] + ry*heightMap[i][j+1])
            end
        end

This function is 100% accurate and is even twice as fast as the traditional method:

Lua:
    local function GetLocZ(x, y)
        MoveLocation(moveableLoc, x, y)
        return GetLocationZ(moveableLoc)
    end
So, even if you're not worrying about desyncs, this is still nice to have.

This should take care at least of desyncs caused by terrain deformations.

But according to the one thread I linked, there could still be desyncs even with a height map calculated at map initialization because of Reforged. Therefore, I looked at how feasible it is to calculate the height map in single-player and write it to a file, then import that data into the map and use it to generate it for every subsequent map launch. I came up with this:

Lua:
    local chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
    local MINIMUM_Z = -256

    local function ReadHeightMap()
        local charPos = 0
        local numRepetitions = 0
        local charValues = {}
        for i = 1, string.len(chars) do
            charValues[string.sub(chars, i, i)] = i - 1
        end
        local firstChar = nil
        for i = 1, #heightMap do
            for j = 1, #heightMap[i] do
                if numRepetitions > 0 then
                    heightMap[i][j] = heightMap[i][j-1]
                    numRepetitions = numRepetitions - 1
                else
                    local valueDetermined = false
                    while not valueDetermined do
                        charPos = charPos + 1
                        local char = string.sub(HeightMapCode, charPos, charPos)
                        if char == "-" then
                            charPos = charPos + 1
                            char = string.sub(HeightMapCode, charPos, charPos)
                        end
                        if tonumber(char) then
                            local k = 0
                            while tonumber(string.sub(HeightMapCode, charPos + k + 1, charPos + k + 1)) do
                                k = k + 1
                            end
                            numRepetitions = tonumber(string.sub(HeightMapCode, charPos, charPos + k)) - 1
                            charPos = charPos + k
                            valueDetermined = true
                            heightMap[i][j] = heightMap[i][j-1]
                        else
                            if char == nil or char == "" then
                                heightMap[i][j] = 0
                            end
                            if firstChar then
                                if charValues[firstChar] and charValues[char] then
                                    heightMap[i][j] = charValues[firstChar]*52 + charValues[char] + MINIMUM_Z
                                else
                                    heightMap[i][j] = 0
                                end
                                firstChar = nil
                                valueDetermined = true
                            else
                                firstChar = char
                            end
                        end
                    end
                end
            end
        end
        HeightMapCode = nil
    end

    local function WriteHeightMap(subfolder)
        local preloadString = 'HeightMapCode = "'
        PreloadGenClear()
        PreloadGenStart()
        local numRepetitions = 0
        local firstChar
        local secondChar
        local stringLength = 0
        for i = 1, #heightMap do
            for j = 1, #heightMap[i] do
                if j > 1 then
                    if math.abs(heightMap[i][j-1] - heightMap[i][j]) <= 0.5 then
                        numRepetitions = numRepetitions + 1
                    else
                        if numRepetitions > 0 then
                            preloadString = preloadString .. numRepetitions
                        end
                        numRepetitions = 0
                        firstChar = (heightMap[i][j] - MINIMUM_Z) // 52 + 1
                        secondChar = heightMap[i][j]//1 - MINIMUM_Z - (heightMap[i][j]//1 - MINIMUM_Z)//52*52 + 1
                        preloadString = preloadString .. string.sub(chars, firstChar, firstChar) .. string.sub(chars, secondChar, secondChar)
                    end
                else
                    if numRepetitions > 0 then
                        preloadString = preloadString .. numRepetitions
                    end
                    numRepetitions = 0
                    preloadString = preloadString .. "-"
                    firstChar = (heightMap[i][j] - MINIMUM_Z) // 52 + 1
                    secondChar = heightMap[i][j]//1 - MINIMUM_Z - (heightMap[i][j]//1 - MINIMUM_Z)//52*52 + 1
                    preloadString = preloadString .. string.sub(chars, firstChar, firstChar) .. string.sub(chars, secondChar, secondChar)
                end
                stringLength = stringLength + 1
                if stringLength == 100 then
                    Preload(preloadString)
                    stringLength = 0
                    preloadString = ""
                end
            end
        end
        if numRepetitions > 0 then
            preloadString = preloadString .. numRepetitions
        end
        preloadString = preloadString .. '"'
        Preload(preloadString)
        PreloadGenEnd(subfolder .. "\\heightMap.txt")
        print("Written Height Map to CustomMapData\\" .. subfolder .. "\\heightMap.txt")
    end

Basically, you start the map, and it encodes the height map into a string that is saved to a file with the Preload native. One then has to remove all the extra characters from it and simply copy/paste it back into the map, and the next time the map loads, it will decode the height map from the string. A mapper can simply generate the height map normally during development and then precompute it once for the release version.

I tested it for various maps and for a giant map like Divide and Conquer, the string size is ~60kb, so it's no problem at all. With some more optimization, I got it down to 44kb.

Tell me what you think! What's the current state of GetLocationZ desyncs?
 

~El

Level 17
Joined
Jun 13, 2016
Messages
558
Nice.

I'm sure you can get the embedded string size to be even smaller if you use base64 (for which optimized lua implementations are easily available). Besides, after the MPQ compression is applied to the script file it's probably even smaller than that.

Although to be fair at this scale the size might as well not matter.

One thing I would suggest is to add a sanity check on heightmap load that ensures that it is not out of date, at least for some debug mode. I.e. read the actual heightmap from the game and compare it to the one stored in the script and compare the values with an epsilon value of like 0.001 to make sure its close enough.

An alternate way of synchronizing the heightmap is by using sync natives and just streaming it from a single player for some "source of truth".
 

Antares

Spell Reviewer
Level 21
Joined
Dec 13, 2009
Messages
512
Nice.

I'm sure you can get the embedded string size to be even smaller if you use base64 (for which optimized lua implementations are easily available). Besides, after the MPQ compression is applied to the script file it's probably even smaller than that.

Although to be fair at this scale the size might as well not matter.
Hm, I'm not quite sure if that would help. As far as I understand it from my quick search, base64 encodes 6 bits of data into each ASCII character, which is 8 bits. Right now, I'm rounding height values to integers and the minimum height is -256 and the maximum height is 1536. So, I need at least 11 bits of data to store the information for one grid point. Two ASCII characters use 16, so there's not a lot of wasted space to begin with.

I can tell you my compression algorithms and you can tell me if you can find a way to improve it:
  • Scan the terrain in columns starting at the bottom-left.
  • Round height values to the nearest integer.
  • 52 alphabet characters for values. Two characters encode one value.
  • If the terrain is flat, write a number that represents the number of repetitions. So "ef19" means, use value "ef" for this and the following 19 vertices.
  • If the terrain is flat enough, the terrain height can be represented as the difference to the previous value with only one character. "|" starts a block where two characters encode the absolute value, "+" starts a block where a single characters denotes the height increase to the previous vertex, and "-" a block where it denotes a decrease.
Example:
Lua:
|ua+MEpc21n-n49|qL1vH2qQvR+hhgd-bipj3

One thing I would suggest is to add a sanity check on heightmap load that ensures that it is not out of date, at least for some debug mode. I.e. read the actual heightmap from the game and compare it to the one stored in the script and compare the values with an epsilon value of like 0.001 to make sure its close enough.
That's a good idea. You can just check the loaded heightmap against a newly generated one and print a warning that it's out of date (or invalid because you messed up the string).

An alternate way of synchronizing the heightmap is by using sync natives and just streaming it from a single player for some "source of truth".
Well, according to @ScrewTheTrees in the first thread I linked, it's too much data for this to be viable.
 

~El

Level 17
Joined
Jun 13, 2016
Messages
558
Hm, I'm not quite sure if that would help. As far as I understand it from my quick search, base64 encodes 6 bits of data into each ASCII character, which is 8 bits. Right now, I'm rounding height values to integers and the minimum height is -256 and the maximum height is 1536. So, I need at least 11 bits of data to store the information for one grid point. Two ASCII characters use 16, so there's not a lot of wasted space to begin with.
Allow me to get very pedantic and technical here :) Please forgive me my autism.

Technically since you use 2 chars per height point, and you only use 11 bits for the height, you have a data density of 68.75%, as opposed to b64 which has a 75% data density. A whole 6.25% difference!

There is another point to be made here in that since you use text to represent your numbers for your RLE compression, you're losing data density there too (using 1-3 characters to represent a typically 1-8 bit value). Same with the other metadata markers. Now, the nice thing about b64 is that it can encode any arbitrary byte buffer. This means, that you can have a bitstream interface to write/read your data directly as bits, so you can use e.g. only 4 bits for RLE (or some varint implementation to cut down on very small numbers!!), exactly 11 bits for the height values, and so on. Then, you can also apply bog-standard DEFLATE or bzip2 over the byte buffer that you have to do some extra lossless compression. In fact, almost every generic compression algorithm out there will have RLE anyway, so it'll pick up on patterns like that in your data (though it only usually works on byte boundaries).

None of this matters in my opinion though and your implementation is perfectly fine as-is for the use-case that is presented here.

Well, according to @ScrewTheTrees in the first thread I linked, it's too much data for this to be viable.

I disagree with STT there as it is totally viable and there are maps that routinely synchronize much larger payloads. It's just going to take a while as the transfer speeds are really slow, and the scaffolding required to do the synchronization safely is a bit unwieldy - but its doable. IIRC retail has a sync speed of around 1-4kb/s, so it'd take about ~10-20 seconds to sync a 40kb heightmap. Which, honestly, you could probably get down even lower if you went for absolute data density. Slow? Sure. Doable? Absolutely.

Since we're on the topic anyway, there's 2 things I'd like to mention:

Lua:
if math.abs(heightMap[i][j-1] - heightMap[i][j]) <= 0.5 then
  numRepetitions = numRepetitions + 1
else
This might yield unexpected results on very gradual slopes, e.g. if the terrain rises at a rate of .4 units per tile. Likely? Nay. But, correctness!!!

Lua:
preloadString = preloadString .. "-"
When you're building strings in Lua, it's much more efficient to put the individual string parts in a table with table.insert, and then use table.concat to join them together all at once. What this does is it avoids extra allocations from intermediate string results and also lessens the stress on the GC engine. This can actually yield noticeable performance increases in code that builds large strings, like yours, especially as the string grows to a point where it is already several kilobytes in size, and each new appended character will effectively re-allocate the whole thing (sans some possible optimizations in Lua that I'm unaware of, but it'd still have to happen occasionally)
 

Antares

Spell Reviewer
Level 21
Joined
Dec 13, 2009
Messages
512
Allow me to get very pedantic and technical here :) Please forgive me my autism.

Technically since you use 2 chars per height point, and you only use 11 bits for the height, you have a data density of 68.75%, as opposed to b64 which has a 75% data density. A whole 6.25% difference!
Alright, you got me :).

There is another point to be made here in that since you use text to represent your numbers for your RLE compression, you're losing data density there too (using 1-3 characters to represent a typically 1-8 bit value). Same with the other metadata markers. Now, the nice thing about b64 is that it can encode any arbitrary byte buffer. This means, that you can have a bitstream interface to write/read your data directly as bits, so you can use e.g. only 4 bits for RLE (or some varint implementation to cut down on very small numbers!!), exactly 11 bits for the height values, and so on. Then, you can also apply bog-standard DEFLATE or bzip2 over the byte buffer that you have to do some extra lossless compression. In fact, almost every generic compression algorithm out there will have RLE anyway, so it'll pick up on patterns like that in your data (though it only usually works on byte boundaries).
Ok, won't all that code you need for this demand more space than the space you're actually saving in the string? So, it's 8.5% less with b64, then I count 10k other characters for RLE and +/-/| markers in the Divide and Conquer height map, so that's 10kb. Let's say you can compress that by 50%, so you'd get from 44 to 35 kb approximately. So, you'd have to write the functions that do all this stuff in 9k characters or less (unless we're talking about even bigger maps).

However, having a well-optimized system like this, in general, could be quite nice if you want to precompute more stuff, not only terrain height. I was also thinking about the possibility to create in-game replays: Let's say you make an RTS. Whenever you play against a buddy, you record all the player inputs, and if there's a particularly great game, you load that data into the map, and add it to an in-game library that other players can watch.

None of this matters in my opinion though and your implementation is perfectly fine as-is for the use-case that is presented here.
I do like getting obsessed over optimizations that don't matter, so this is right up my alley :plol:

Since we're on the topic anyway, there's 2 things I'd like to mention:

Lua:
if math.abs(heightMap[i][j-1] - heightMap[i][j]) <= 0.5 then
  numRepetitions = numRepetitions + 1
else
This might yield unexpected results on very gradual slopes, e.g. if the terrain rises at a rate of .4 units per tile. Likely? Nay. But, correctness!!!
You are correct. It should compare with the previous rounded value instead of the previous raw value. Easy fix. Good find!

When you're building strings in Lua, it's much more efficient to put the individual string parts in a table with table.insert, and then use table.concat to join them together all at once. What this does is it avoids extra allocations from intermediate string results and also lessens the stress on the GC engine. This can actually yield noticeable performance increases in code that builds large strings, like yours, especially as the string grows to a point where it is already several kilobytes in size, and each new appended character will effectively re-allocate the whole thing (sans some possible optimizations in Lua that I'm unaware of, but it'd still have to happen occasionally)
Good to know, thanks! That should also be easy to fix.

I will try to make this into a fully working resource. If you like to have a go at a better compression algorithm, I could add that later and credit you, but while I understand what you're saying, it's too high level for me to implement it myself :plol:.
 

~El

Level 17
Joined
Jun 13, 2016
Messages
558
Ok, won't all that code you need for this demand more space than the space you're actually saving in the string? So, it's 8.5% less with b64, then I count 10k other characters for RLE and +/-/| markers in the Divide and Conquer height map, so that's 10kb. Let's say you can compress that by 50%, so you'd get from 44 to 35 kb approximately. So, you'd have to write the functions that do all this stuff in 9k characters or less (unless we're talking about even bigger maps).
Shhhhhhh, we don't talk about that.

To be fair the size is a bit irrelevant anyway since the max map size on retail is like, 256 MiB? Or at least 128 MiB. There's really not much point to compressing it further unless you want to send it over the network in-game.
If you like to have a go at a better compression algorithm, I could add that later and credit you,
My warcraft days are over for the most part. It's all a curious exercise but that's about it for me.
 
Top