Generate random terrain

[sethmachine]

Hi,

Are there any tools that can create new Warcraft 3 map files with random but realistic terrain? Or does anyone have experience editing the Warcraft 3 map files directly to change terrain? Note I am not talking about terrain generation in game, but before when creating a Warcraft 3 map.

Ideally I'd want a tool that I could feed in examples of terrain I like (either Warcraft 3 maps or even photos of real landscapes) and the tool could generate the terrain appropriately from some random sampling/stochastic process.

 

[GhostWolf]

My code can easily apply an height map given an image.
If you mean biomes with different ground textures, fitting doodads, rivers, mountains, cliffs, etc., then I don't know of any tool.
That being said, there is no difference between implementing that for WC3 maps than any other kind of game, or indeed even from implementing it in Jass, except you get additional freedom.
Except for cliffs. Good luck on figuring the logic behind their creation (maybe in the future).

 

[Dr Super Good]

I do recall seeing one long ago. It could generate some kind of random melee map to play. That said good luck finding it, it probably never was that good and probably will not run anymore.

Making random terrain generators is a lot harder than one might think. If you want some idea where to start look at 0 A.D. as they have random terrain generators for a variety of different kinds of RTS map.

 

[sethmachine]

To be very clear, I am talking about creating terrain in the editor, not during gameplay.

So is there an API I can use to create terrain in a Warcraft 3 map file? For example, I might give it the coordinates/dimensions of a rectangle and then choose the terrain type, after which I would see it in the Warcraft 3 map file, when opened in the editor or played in game.

My code can easily apply an height map given an image.
If you mean biomes with different ground textures, fitting doodads, rivers, mountains, cliffs, etc., then I don't know of any tool.
That being said, there is no difference between implementing that for WC3 maps than any other kind of game, or indeed even from implementing it in Jass, except you get additional freedom.
Except for cliffs. Good luck on figuring the logic behind their creation (maybe in the future).

Which code? Can you point me to this? I agree that's its just a general problem of creating some kind of 2D image that fits some stylistics /standards (i.e. it's not random noise).

I do recall seeing one long ago. It could generate some kind of random melee map to play. That said good luck finding it, it probably never was that good and probably will not run anymore.

Making random terrain generators is a lot harder than one might think. If you want some idea where to start look at 0 A.D. as they have random terrain generators for a variety of different kinds of RTS map.

I'd be willing to go as far as using something like a GAN--General Adversarial Neural network, which I would train to learn to generate terrains similar to the ones I trained a different (neural network) model on. I'd take samples from maps on the hive whose terrain/styles I liked. GANs have been used to generate images and other complex information that look very real but don't actually exist or have been seen before.

I just need a way to generate a map terrain programmatically and also a way to turn existing terrain data into usable features for such an approach.

 

[Dr Super Good]

Many such libraries exist already. I have written at least 3 different w3e (terrain mesh file) interfaces for Java.

 

[sethmachine]

Many such libraries exist already. I have written at least 3 different w3e (terrain mesh file) interfaces for Java.

Ah excellent. Does it also allow placement of doodads?

Where can I download the Java code that allows this interface and any documentation?

I did not see anything here: https://www.hiveworkshop.com/members/dr-super-good.116773/

 

[Dr Super Good]

Does it also allow placement of doodads?

Doodads are a separate file (.doo), but other people wrote interfaces for those as well.

Where can I download the Java code that allows this interface and any documentation?

Here was my last attempt for the .w3e file. In retrospect I am not too happy with the IO mechanics so in future I might change them. A TerrainMesh object can represent and manipulate a Warcraft III .w3e file. I used it to create various procedural terrains such as waves.

Get a w3e file, place it in the provided object input stream type. Get the TerrainMesh object from it. Manipulate it. Use object output stream type provided to write TerrainMesh to the .w3e file. One can also make a TerrainMesh object from scratch.

 

[GhostWolf]

It is indeed not particularly hard to change terrain heights, ground textures, place doodads, and other such manipulations.
There is however no easy to use API for any of this at this point. At least not in my codebase.

Currently the most challenging parts are probably cliffs and ramps. We know how to read them, but nothing into the logic of creating them.
That being said, this is mostly due to no one yet using them in any way other than reading for rendering.
Making cliffs normally should probably not prove too difficult, but the rules for how to manipulate the whole terrain when you raise cliffs by more than 2, for example, need to be checked closely (there are many weird and buggy behaviors that WE exhibits when editing terrain that need to be looked at, but to be honest, this might not really be relevant to generated terrain).

As far as pathing goes, @eejin has been looking into it, and can edit the pathing on a more fine-grained level than is achievable in WE in the first place, so you're good on that front (well, at some point, when there exists one codebase that can do everything, or at the very least executables that can be chained).

 

[sethmachine]

Here was my last attempt for the .w3e file. In retrospect I am not too happy with the IO mechanics so in future I might change them. A TerrainMesh object can represent and manipulate a Warcraft III .w3e file. I used it to create various procedural terrains such as waves.

Get a w3e file, place it in the provided object input stream type. Get the TerrainMesh object from it. Manipulate it. Use object output stream type provided to write TerrainMesh to the .w3e file. One can also make a TerrainMesh object from scratch.

Thanks for the Java code. I downloaded it and started looking at it in Netbeans. Could you provide any real examples where you use your code to create terrain in a map? Even a simple "Hello world" example would be very appreciative.

Doodads are a separate file (.doo), but other people wrote interfaces for those as well.

What tools/code exists for creating and reading from the doodad layer? So if I have an existing map, how can I read the positions and IDs of each doodad placed? And then is there an API for editing the .doo file?

If I created a terrain mesh file and a .doo file, how would I go about combining these into a usable warcraft 3 map file?

I did read through the wikis @eejin kindly put together:

war3map.w3e Terrain · stijnherfst/HiveWE Wiki · GitHub

war3map.doo Doodads · stijnherfst/HiveWE Wiki · GitHub

What would be great would a simple "Hello World" example that writes terrain and doodad data in Java or C++ code and creates a usable Warcraft 3 map file.

 

[GhostWolf]

It might not be so straightforward, because that involves creating new things.
Much like cliffs, no one really looked into for instance creating a new terrain (as far as I know).
For example, the size of the map is stored in multiple built-in files, and I for one didn't look into what you need to do to update all of them correctly.
So far I only edited maps made in WE, so it just never mattered.
But again, that being said, it might be that all of these things are very easy to figure out, I just never tried.

 

[sethmachine]

It's also fine if the hello world examples take a blank map as a "start" and then writes terrain/doodads to the map. I may end up creating this hello world example myself if nobody can come forward with their own example.

 

[sethmachine]

As an update: I ported @eejin 's Terrain w3e code (HiveWE/Terrain.cpp at master · stijnherfst/HiveWE · GitHub) into Python, and I was able to read a w3e file. I also could write the data back to a new w3e file, which had no diffs from the original (good sanity check). I've attached a JSON representation of the w3e data (for any data less than 1 byte, I did not yet turn it into human readable values, e.g. water_and_edge for a corner). Apparently can't upload .json files, so I changed the extension to .txt (the first field is my metadata, so ignore it).

If I understand correctly, I can now manipulate the tilepoint/corner texture values, height, and variation to programmatically produce terrain, which I'll then import into the .w3x MPQ archive.

I'm also bringing in some abandoned Python project that put a wrapper around the StormLib C++ API (GitHub - mikeboers/PyStorm: Python wrapper around the StormLib library for manipulating MPQ archives (for Blizzard games).), so I can actually chain the terrain manipulation with publishing a map.

 

[GhostWolf]

You don't need to ask - try and see

Do note that the specs are still not quite correct, there are many wrong assumptions made by people many years ago, which we are now over time proving false.
Some of them involve tile points. Cliffs are pretty weird too.
Problem is, the actual truth is also not quite known yet.

 

[sethmachine]

You don't need to ask - try and see

Do note that the specs are still not quite correct, there are many wrong assumptions made by people many years ago, which we are now over time proving false.
Some of them involve tile points. Cliffs are pretty weird too.
Problem is, the actual truth is also not quite known yet.

Do you mean that the Terrain parser (for w3e files) of @eejin is incorrect? That is what I based my Python off of. A good thing that I probably won't care very much about actual cliffs--what I've seen and done is simply using elevation combined with unwalkable tiles to create "cliffs."

C++ code: HiveWE/Terrain.cpp at master · stijnherfst/HiveWE · GitHub

I guess I haven't actually tried to write terrain yet using this parser. I'll report on that once I do so.

Is the doodad specification also incomplete/incorrect? war3map.doo Doodads · stijnherfst/HiveWE Wiki · GitHub

Also, when I update the w3e/doo file, don't I need to update the pathing map? Or is there a way to have Warcraft rebuild the pathing map on its own once the terrain/doodad file is generated? Obviously without opening the map in World Editor GUI and then saving it. @eejin perhaps you can answer this?

 

[GhostWolf]

Those specs are the most current information that we know, however it is clear there are still problems with them. You can use them, don't worry.

I believe eejin said that WE indeed bakes the doodad pathing information, which means you must handle that too.
eejin's code should be able to do what you want at some point probably not far in the future, and already can some of it, you can ask him nicely to add some form of CLI to it

 

[Dr Super Good]

Also, when I update the w3e/doo file, don't I need to update the pathing map?

Yes you need to change the pathing map if you want it to reflect the terrain and doodads on it. Further more the resulting pathing map is the pathing map one accesses via triggers, since the natives do not factor in destructables and units (buildings).

 

[eejin]

Yeah, destructibles aren't baked into the pathing map, but other doodads are. Some tiles (like rocks) are unbuildable which is also encoded in the pathing map.

The doodad specification seems to be largely "correct" in that most doodads load just fine and we haven't noticed any missing behaviour. Do keep in mind that there are fields which are unknown entirely or just unclear what their exact behaviour is. Using it should be fine.

 

[sethmachine]

@eejin thanks for chiming in.

So if I want to correctly update a map's terrain/doodads programmatically, I need to do the following steps?

1. Update terrain (warmap3.w3e)
2. Update doodads (warmap3.doo) (does this also include trees and destructibles?)
3. Go through each tile in the war3map.wpm and somehow incorporate the information I changed in the terrain/doodads?

What happens if I don't update the pathing map? Will the units AI be bugged?

How do I map each "corner" in the terrain to a "tile" in the pathing map?

Or do you have a script that can build the pathing map from reading the terrain/doodads file?

 

[Dr Super Good]

What happens if I don't update the pathing map? Will the units AI be bugged?

Nothing to do with AI. It will mean units will walk in a way which is not consistent with how the terrain looks. For example ground units might happily traverse up/down cliffs and through deep water or suddenly run into an obstacle on flat land.

How do I map each "corner" in the terrain to a "tile" in the pathing map?

Technically each tile is made up of 4 tile points. However I recall each tile having major influence from only 1 of the corner points, and that is used to decide the pathing for the area.

 

[eejin]

The pathing map divides each tile into 4x4 blocks (though the editor usually fills this in as 2x2 blocks). If you place an unbuildable tile at a corner then it will have an effect on the 4 tiles surrounding it.

 

[sethmachine]

I see, so the doodads/terrain/destructible don't have any pathing information encoded in them. The pathing map encodes this information. But why do I have to define the pathing for each tile/position on the map? Shouldn't I just be able to define the pathability of each texture/doodad, e.g. the ashenvale rock texture is unwalkable globally? Similary for trees, doodad rocks, etc.

In any case what I'm looking:

Automatically generate the correct pathing map given updates to the war3map.w3e and war3map.doo files

 

[GhostWolf]

Destructibles have their own pathing texture attached, since they are dynamic (can be moved, destroyed, etc.)
Doodads have the same, but they get baked, since they will never change.

You do define the pathability of the texture/doodad, ashenvale rock texture is unwalkable globally, but that's just a boolean defined in the SLK, you need actual data for the real time pathing.

Once eejin finishes his demo pathing painter, I think this will be more obvious, and also a very useful tool indeed. And of course, that has everything to do also with doodads, and other things, which is why I said you might have a tool doing what you want in the not too far future

 

[sethmachine]

Update: I combined StormLib API with @eejin 's Terrain parser and I was able to randomize the textures+variation of all the tiles on a stock W3X file, (2)GlacialThaw.w3x and then write back the new war3map.w3e.

I was able to start a 2 player melee game with no crashes or other serious issues (I didn't check pathing).

However, while visually the terrain/textures were different from the original map, the minimap did not reflect the new terrain--it showed the old terrain (mostly snow)

How do I update the minimap to reflect the updated war3map.w3e file?

I've attached the modified GlaicalThaw.w3x file for your inspection.

I also played around with cliff texture, cliff variation, and layer height. I randomized all these, however while @eejin says layer height is 4 bits (0 - 15 possible values), my map would crash upon loading if I used certain layer heights within that range. Setting all tiles to layer height = 5 would put most of the map underwater. Apparently some layer heights are illegal or can cause a crash?

I also randomized ground height between 0 and 8192 * 2 (the min/max apparently, ground zero is 8192). It's a very humorous result with peaks and gorges (uploaded too).

 

[WaterKnight]

The minimap information is stored in .mmp, which the WE creates from the terrain/placed objects on save, as it calculates the .wpm (pathing). Not sure where the colors come from, of course should be from TerrainArt.slk but do not remember such attribute or maybe in one of the editor UI files. Layer 15 is indeed illegal. The water height/flag is also in each node definition of the .w3e file.

 

[GhostWolf]

There are 14 layers, each one is 128 units high.
I am not sure if the number 14 is actually hardcoded, or if the game dynamically calculates it based on the values from one of the ini files that defines the min/max height of cliffs (don't remember which, check eejin's wiki page).

In addition, each tile has its own ground height, which is in the range [-2048, 2047.8], however as you can clearly tell, those are NOT the numbers you supply.
As far as I can tell with experimenting, there are two possibilities - the ground height is a 14bit number, or the logic handling it acts like a 14bit number.
With that in mind, the range is [0, 0x4000], and 0x2000 is indeed "ground zero".
To map that to the real range, you'd need a division by 4, and subtracting 2048.

I don't know what's the deal with the maximum being 2047.8, probably that 0.2 gets lost in some division + conversion to floats?

In addition, for some reason WE adds -256 to all heights, probably because of said min/max height cliff numbers, and indeed uses it also for doodad/unit placements, so don't get confused by that.
For example, make a new map at layer 0, and check the Z coordinate anywhere with your mouse.
I don't know what the game itself reports, e.g. via Jass, because no one would make a test map for me

The final height of a tile point follows something along the lines of

height = (groundHeight14Bits / 4) - 2048 - 256 + layer * 128
// groundHeight14Bits is in the range [0, 0x4000]
// groundHeight14Bits / 4 gives us the range [0, 0x1000]
// -2048 to get [-2048, 2048]
// -256 for the base cliff level or whatever it is
// layer * 128 is self explanatory

So how to use this? let's say you want some height N.
Take 0x2000, and add N*4 to it.
For example, height 25 (same as raising the ground 4 times in WE) is 0x2000+100.

Note that none of this is in eejin's specs yet.

All of this follows suit with the fact that the water level is also not a 16bit integer - we know there is a boundary flag at the 15th bit.
It might very well be that there are 3 more flags we don't know about. Or not. Who knows.

As a side note, a lot of this is untestable directly in vanila WE, as it has all sorts of constraints.
For example, it seems to enforce a maximum relative height of 150 between neighboring tiles, it seems to enforce a maximum final height based on only the cliff max height (14*128), and all sorts of other such things.
It will still gladly load a map that doesn't use these constraints, but don't think about editing the terrain in vanila WE, because it will ruin your terrain by moving it in funny ways to match these constraints

 

[sethmachine]

The minimap information is stored in .mmp, which the WE creates from the terrain/placed objects on save, as it calculates the .wpm (pathing). Not sure where the colors come from, of course should be from TerrainArt.slk but do not remember such attribute or maybe in one of the editor UI files. Layer 15 is indeed illegal. The water height/flag is also in each node definition of the .w3e file.

How do I create a new .mmp using information from the .w3e file? I'm surprised the minimap is a static asset--I would figure it should be drawn dynamically, but I guess that's why Blizzard doesn't offer a lot of ways to change terrain in game. I can't find .mmp documentation anywhere in @eejin 's project or these others (how do they relate to @eejin 's work?):

Home · stijnherfst/HiveWE Wiki · GitHub
Home · ChiefOfGxBxL/WC3MapTranslator Wiki · GitHub
GitHub - ChiefOfGxBxL/Ice-Sickle: Next-generation WC3 world editor built on Electron

Is there a way I can "trick" the WE to generate the .mmp/.wpm on the command line, e.g. programmatically tell the WE to open the map and save it? Or other tools that could accomplish this, without opening the WE GUI?

There are 14 layers, each one is 128 units high.
I am not sure if the number 14 is actually hardcoded, or if the game dynamically calculates it based on the values from one of the ini files that defines the min/max height of cliffs (don't remember which, check eejin's wiki page).

In addition, each tile has its own ground height, which is in the range [-2048, 2047.8], however as you can clearly tell, those are NOT the numbers you supply.
As far as I can tell with experimenting, there are two possibilities - the ground height is a 14bit number, or the logic handling it acts like a 14bit number.
With that in mind, the range is [0, 0x4000], and 0x2000 is indeed "ground zero".
To map that to the real range, you'd need a division by 4, and subtracting 2048.

As you said, in the .w3e file, ground height is a 14 bit number with 8192 representing ground level zero. Since my problem doesn't have to render terrain and the relation between the ground heights bits and actual height is linear, for my purposes 8192 is ground level zero. I only need to write back valid data to a new .w3e file. I'm planning on using these fields as input to a machine learning algorithm and I can always normalize later or afterwards.

All of this follows suit with the fact that the water level is also not a 16bit integer - we know there is a boundary flag at the 15th bit.
It might very well be that there are 3 more flags we don't know about. Or not. Who knows.

This is something I don't understand either--below is the code @eejin uses for getting water height and the boundary flag:

            uint16_t water_and_edge = reader.read<uint16_t>();
            corner.water_height = ((water_and_edge & 0x3FFF) - 8192.f)/ 512.f;
            corner.map_edge = water_and_edge & 0xC000;

The mask used to get water height is 0011 1111 1111 1111, which is indeed 14 bits. But the mask to get the map edge flag gets both the 15th and 14th bit (0 indexing I'm assuming):

0xC000 is 1100 0000 0000 0000

If boundary is just the last bit (15th), then what role does the 14th (second significant bit) play. Are there 4 flags instead of 1, e.g. 11, 10, 00, 01 ? In my terrain parser I just extract these two bits as the boundary flag. If the boundary is only the 15th bit, then I just flip the 15th bit if I want to change boundary (I haven't tested yet).

 

[WaterKnight]

How do I create a new .mmp using information from the .w3e file?

Convert each terrain node into a pixel? respectively scale it. I have no access to the exact algorithm. The WE also offers some options iirc when exporting it >File >Export Minimap but that is as TGA or something like that.

W3M and W3X Files Format
GitHub - inwc3/wc3libs: general java wc3libs

I'm surprised the minimap is a static asset--I would figure it should be drawn dynamically, but I guess that's why Blizzard doesn't offer a lot of ways to change terrain in game.

Wc3 does draw some dynamic stuff on top like units, heroes, blight but the background is firm.

Is there a way I can "trick" the WE to generate the .mmp/.wpm on the command line, e.g. programmatically tell the WE to open the map and save it? Or other tools that could accomplish this, without opening the WE GUI?

None that I know of. Of course the algorithm should not be a super complex one.

If boundary is just the last bit (15th), then what role does the 14th (second significant bit) play. Are there 4 flags instead of 1, e.g. 11, 10, 00, 01 ?

Probably just a mistake/relaxation as the other bit is mostly zero, in which case it does not matter.