Data transfer between maps

[AMarkov]

Disclaimer: This is not even a question, but rather concerns sharing existing experience on a broader scale.

It concerns Reforged and its advantage of working with maps as directories. The problem and its solution are more global and automated. Surely, there are already some ideas and methods for implementing this in other projects.

By default, we import/export all data from Map A (custom Triggers, Units, Decorations, Models, Icons, Textures), and then import it with full replacement into Map B.
But what if the project includes multiple maps? How can we do this quickly while avoiding critical errors?
And what if each map needs to contain its own unique set of triggers, in addition to a common set for all maps? Can we separate .j or .lua files and then compile the projects as a whole? Are there any similar compilers available at the moment?

I am a web developer, so in theory, I could write a PHP script for my own needs and tasks.

To specify the tasks: my project is WarCraft Legend, and I want to manage the project not just for a single map but for multiple maps, while ensuring that the entire codebase applies to all maps. Each map also carries its own unique logic—custom victory and defeat conditions, and perhaps unique decorations and units specific to that map. The problem is that I want to integrate pre-existing maps into the project with minimal time investment, such as a campaign map, which already has its own set of triggers.

From my own experience, I can share the following: working with a map and linking it to a GitHub repository helps avoid creating local copies of the map, as I can track only the changes and maintain version history in a remote repository. Using my IDE, I also manage models and icons efficiently. Once a model is integrated, I save the project in the editor to apply the changes.

 

[AMarkov]

So, I managed to perform the data transfer, but I haven’t retrieved all the lines yet. This was done hastily through a custom parser written in PHP (Laravel framework).

There are missing lines (Varian - King of Stormwind). I’d prefer not to overwrite the map description and name in the .j file. It’s likely that the parser will need to add string replacements for different projects.

The object data, rawcodes, should be universal. So, if I want to introduce new objects in the donor map, I need to create them beforehand.

As for the JASS code, at the moment it’s written in a single file, so it needs to be written in such a way that it doesn’t directly reference objects on the map, allowing for a quick resave.

Sharing experiences is still relevant. Has anyone had a similar experience? What pitfalls did you encounter, and how did you solve various issues?

 

[AMarkov]

File	Description	Share between map	Comment
war3map.j	code	true
war3map.w3e	terrain texturing	false
war3map.shd	shadow map	false
war3map.wpm	passability map	false
war3map.doo	info about trees	false
war3mapUnits.doo	information about all objects placed on map	false
war3map.w3i	Various information about the map, which is set in the editor in the scenario section	false
war3map.wts	String values	true
war3mapMap.blp	Minimap	false
war3map.mmp	Minimap icons during initialization	false
war3map.w3u	Units - Objects	true
war3map.wtg	Trigger and variable names	true
war3map.w3r	Info by regions	false
war3map.w3s	Sounds are set	false
war3map.wct	map script + any text script	true
war3map.imp	contains info about imported files	true
war3mapMisc.txt	constants	true
war3mapExtra.txt	editor settings	true
conversation.json	are needed to match portrait talking animations to their respective sound file lines	false
war3map.w3a	Abilities - Objects	true
war3map.w3b	Destructables - Objects	true
war3map.w3c	Camera parameters	false
war3map.w3d	Doodads - Objects	true
war3map.w3h	Buffs - Objects	true
war3map.w3q	Upgrades - Objects	true
war3map.w3t	Items - Objects	true
war3mapSkin.w3a	Skin data	true
war3mapSkin.w3b	Skin data	true
war3mapSkin.w3d	Skin data	true
war3mapSkin.w3h	Skin data	true
war3mapSkin.w3q	Skin data	true
war3mapSkin.w3t	Skin data	true
war3mapSkin.w3u	Skin data	true

Can someone clarify some unclear points?

 

[Nichilus]

I don't understand what you are trying to achieve, to be honest. I get it that you want to import data from one map to another, but I fail to understand what you mean by 'But what if the project includes multiple maps? How can we do this quickly while avoiding critical errors?'
Like, what should be the output of this entire thing? A single map that contains data from multiple other maps? Like A -> C, B -> C?
Or do you mean just copying data from one map to another, like A -> A2, B -> B2
If it is the latter, why not just make a copy of the whole map?
Otherwise I don't understand how you have evaluated that it is safe to just copy & paste entire war3map.w3* files into one single output file.

As for the table, from what I know:

• war3mapSkin.w3* files contain object editor data that are bound to change based on localization or other reasons (i.e. teen filter, chinese version, etc.). It contains stuff like model path, unit names, unit sounds, etc.
• conversation files are needed to match portrait talking animations to their respective sound file lines.
• war3map.wct contains map script + any text script (be it a gui-converted trigger, or lua/jass script file)
• war3map.imp contains info about imported files. Considering how buggy the import manager is, I am not surprised if it gets constantly updated even if you don't make any change yourself.
• war3mapUnits.doo contains information about all objects placed on map

 

[AMarkov]

I don't understand what you are trying to achieve, to be honest. I get it that you want to import data from one map to another, but I fail to understand what you mean by 'But what if the project includes multiple maps? How can we do this quickly while avoiding critical errors?'
Like, what should be the output of this entire thing? A single map that contains data from multiple other maps? Like A -> C, B -> C?
Or do you mean just copying data from one map to another, like A -> A2, B -> B2
If it is the latter, why not just make a copy of the whole map?
Otherwise I don't understand how you have evaluated that it is safe to just copy & paste entire war3map.w3* files into one single output file.

As for the table, from what I know:

• war3mapSkin.w3* files contain object editor data that are bound to change based on localization or other reasons (i.e. teen filter, chinese version, etc.). It contains stuff like model path, unit names, unit sounds, etc.
• conversation files are needed to match portrait talking animations to their respective sound file lines.
• war3map.wct contains map script + any text script (be it a gui-converted trigger, or lua/jass script file)
• war3map.imp contains info about imported files. Considering how buggy the import manager is, I am not surprised if it gets constantly updated even if you don't make any change yourself.
• war3mapUnits.doo contains information about all objects placed on map

I sincerely apologize if my initial request was not clear enough.

Allow me to explain. I believe I am working on quite a unique challenge.

I am developing a co-op mode similar to StarCraft 2's for WarCraft 3. The key feature of this mode isn't just the unique commanders with their own races (which is essentially an Altered Melee), but also a map with unconventional victory/defeat conditions.

This leads to a particular issue. I need not only to handle data transfer but also to ensure simultaneous support for maps with different sets of "personalized" triggers. Alternatively, I may need to create a "Unified Trigger Pool" system that activates the appropriate triggers for each map. At the same time, there needs to be some level of automation to keep all maps relevant for the future, especially with the addition of new maps and commanders.

I've already developed a prototype, though it is still far from perfect. I was hoping to hear insights from fellow mapmakers who have experience working in this direction.

Thank you so much for clarifying some of the points that I had not fully understood.

 

[AMarkov]

war3mapSkin.w3* files contain object editor data that are bound to change based on localization or other reasons (i.e. teen filter, chinese version, etc.). It contains stuff like model path, unit names, unit sounds, etc.

Are these files mandatory for import between maps? Will the Chinese version of the map break if the file isn't transferred?

 

[Nichilus]

I sincerely apologize if my initial request was not clear enough...

I see, so basically, you have a master map that contains the shared logic (i.e. commanders and their abilities, etc.) and then you have scenario maps that you are actually playing and that contain their own unique object editor data and triggers (i.e. in SCII's terms, one map is about defeating wild zergs, while another is about defending from Tal'darim protoss).
And the aim is to import master map's data into scenario maps, right?
If so, then that is the basically 'a single map that contains data from multiple maps' approach that I wrote about.
In that case you are in for a world of hurt
I am not sure how much you've read through any documentation/tutorial about map file's definitions, but most files, like war3map.w3* files use table structure. At the beginning of the file there is a UINT representing the size of the table. So if you, for example, just append unit data from master map into scenario map, but do not update the table's length appropriately, you will most likely encounter bugs.
The worst file is the .wtg file that represents GUI triggers. Its structure is complicated, but also very unintuitive (my guess is that Blizz intended some changes to Trigger Editor, but never finished it, so the .wtg file now has horrible, buggy format, basically a work-in-progress that never got cleaned up).

Are these files mandatory for import between maps? Will the Chinese version of the map break if the file isn't transferred?

Yes, they are mandatory, but not because it would break Chinese/teen version. That's part of what Blizz uses those files for, I believe. For example you cannot have a zombie showing bones in China, so most of the undead units look differently there than in the rest of the world. I believe the teen version is less gory (or was supposed to be, I've never played it like that ). One of the data that war3mapskin file contains is unit's model path. So during map load I believe Blizz updates the data from those war3mapskin files to load different assets.
But the main reason are the audio-visual aspects of your units.

To give an example, both war3map.w3u and war3mapskin.w3u files contains changes to units in object editor (basically each and every magenta-colored text). From all those changed fields the war3map.w3u file contains only static ones related to gameplay, i.e. health, movement type, attack, damage, tech, etc.
On the other hand, the war3mapskin.w3u contains data like unit's soundset, unit's model, icons, unit name - all of those are subject to change based on localization.
So as you can see, you cannot really omit the war3mapskin, because you would be missing part of object editor data that make up your units.

 

[Elprede]

Yes, they are mandatory, but not because it would break Chinese/teen version. That's part of what Blizz uses those files for, I believe. For example you cannot have a zombie showing bones in China, so most of the undead units look differently there than in the rest of the world. I believe the teen version is less gory (or was supposed to be, I've never played it like that ). One of the data that war3mapskin file contains is unit's model path. So during map load I believe Blizz updates the data from those war3mapskin files to load different assets.
But the main reason are the audio-visual aspects of your units.

To give an example, both war3map.w3u and war3mapskin.w3u files contains changes to units in object editor (basically each and every magenta-colored text). From all those changed fields the war3map.w3u file contains only static ones related to gameplay, i.e. health, movement type, attack, damage, tech, etc.
On the other hand, the war3mapskin.w3u contains data like unit's soundset, unit's model, icons, unit name - all of those are subject to change based on localization.
So as you can see, you cannot really omit the war3mapskin, because you would be missing part of object editor data that make up your units.

If you were to merge the data from war3mapSkin file with the data from its corresponding object data file, then there would be no need to import these war3mapSkin files. Map Adapter has this feature for map conversion.

 

[Luashine]

You can hack-and-patch the .wtg format because it doesn't have any lengths encoded. For example, you can find a trigger's name and in case of a long custom code just replace the text until the NULL.

I wonder if the w3o format is read when it's inside the MPQ next to other files. If yes and if it is read before the .w3u, w3b etc. only then it'd be possible to use .w3o as a shared file (editable master and shared to downstream maps).

The -skin files are not required. The localization is done by overriding the paths in the MPQ-way. Same for teen and HD and "game data set" (balance files between v1.0, custom, current). If your explanation is accurate, I'd love to see it in action.

The map name is taken from .w3i, currently the functions in the script are only called in map selection and lobby views.

The common ground before was to use textual object definitions (JassHelper's //! macros) that were injected into the object files at compile time. It's a common theme among developers, isn't it, to make the resulting artifact reproducible, for that it's made scriptable and so ends up in a text form. I think you'll all agree editing and creating objects in text will be pure pain. So your build system will need to have at least two functions:

1. Open editor with the blueprint map (upstream map / base map / other synonym names)
2. Share the changes into downstream maps

PS: Thumbs up from me even though it's extremely niche. I don't know how campaigners do it. But I concluded that a "custom race" project must have exactly what you describe to easily port the changes between melee maps.

 

[Nichilus]

If you were to merge the data from war3mapSkin file with the data from its corresponding object data file, then there would be no need to import these war3mapSkin files. Map Adapter has this feature for map conversion.

What happens if you merge data from war3mapSkin into war3map file, open the map, make some object editor changes and save it? Will a war3mapSkin be created again?

 

[AMarkov]

First of all, thank you to everyone who responded! It means a lot to me.

In that case you are in for a world of hurt
I am not sure how much you've read through any documentation/tutorial about map file's definitions, but most files, like war3map.w3* files use table structure. At the beginning of the file there is a UINT representing the size of the table. So if you, for example, just append unit data from master map into scenario map, but do not update the table's length appropriately, you will most likely encounter bugs.
The worst file is the .wtg file that represents GUI triggers. Its structure is complicated, but also very unintuitive (my guess is that Blizz intended some changes to Trigger Editor, but never finished it, so the .wtg file now has horrible, buggy format, basically a work-in-progress that never got cleaned up).

GitHub - ChiefOfGxBxL/WC3MapTranslator: Translate war3map ⇄ json formats for WarCraft III .w3x maps

Translate war3map ⇄ json formats for WarCraft III .w3x maps - GitHub - ChiefOfGxBxL/WC3MapTranslator: Translate war3map ⇄ json formats for WarCraft III .w3x maps

github.com

Home

Translate war3map ⇄ json formats for WarCraft III .w3x maps - ChiefOfGxBxL/WC3MapTranslator

github.com

GitHub - ChiefOfGxBxL/WC3MapSpecification: The specification of WC3 .w3x files as a living document

The specification of WC3 .w3x files as a living document - ChiefOfGxBxL/WC3MapSpecification

github.com

I see your point, but it still doesn't seem like a dead end. While exploring GitHub, I came across a system for parsing w3* files into JSON in both directions. Could this be the holy grail for solving the problem of a unified repository containing all the data? The unified repository would remain, but individual data for different maps could be selectively integrated.

Working with TS/JS and Node JS server is also not a problem for me.

 

[AMarkov]

The worst file is the .wtg file that represents GUI triggers. Its structure is complicated, but also very unintuitive (my guess is that Blizz intended some changes to Trigger Editor, but never finished it, so the .wtg file now has horrible, buggy format, basically a work-in-progress that never got cleaned up).

I looked at my generated .wtg file, it looks like it contains all the comments I left for the GUI triggers (in the window above the GUI trigger area), as well as the names of the trigger "files" inside the editor.

 

[AMarkov]

I’ve also outlined a system for cleaning up the .wts file from developer-specific data in the naming of units, abilities, etc. For easier development, instead of using a suffix, I use a prefix before the name for easier navigation. Personally, I’ve already accepted that many units in the game have a prefix, but it seems that I’ll be able to spare regular players from this 'developer clutter.'

Example of my data:
STRING 1204
// Abilities: A020 ([Arthas] Harvest a soul), Name (Name)
{
[Arthas] Harvest a soul
}

 

[Nichilus]

yes, it seems the tool would do a good job, however it does not support parsing of wtg files.
The wtg files, specifically gui trigger definitions, contain multiple strange constructs - whether those are present depends on which action/condition you chose.
I've mapped the odd behavior when I was working on my own tool, so I may as well just post info about it into tutorial section, since on current Reforged there are some constructs that I have not seen mentioned anywhere else.

 

[AMarkov]

yes, it seems the tool would do a good job, however it does not support parsing of wtg files.
The wtg files, specifically gui trigger definitions, contain multiple strange constructs - whether those are present depends on which action/condition you chose.
I've mapped the odd behavior when I was working on my own tool, so I may as well just post info about it into tutorial section, since on current Reforged there are some constructs that I have not seen mentioned anywhere else.

Thanks a lot! Let me know when you update the tutorial, it's very interesting.

 

[AMarkov]

yes, it seems the tool would do a good job, however it does not support parsing of wtg files.
The wtg files, specifically gui trigger definitions, contain multiple strange constructs - whether those are present depends on which action/condition you chose.
I've mapped the odd behavior when I was working on my own tool, so I may as well just post info about it into tutorial section, since on current Reforged there are some constructs that I have not seen mentioned anywhere else.

And if all the code is translated into Jass, will the .wtg file stop being generated?

 

[Nichilus]

No, it will still be generated. The wtg file contains the definition of the trigger editor's tree view (the left side panel in trigger editor that starts with map name as root node).

 

[AMarkov]

I didn’t think about it, but our favorite site has already integrated a parser. And it handles all types of triggers quite successfully (GUI, Jass, Lua)

Warcraft 3 Trigger Format Specification (WTG!)

Warcraft 3 Trigger Format Specification (WTG!) The past few weeks I have been working on a tool to display the triggers of any map on the site. As some of you know, we had a similar system on the old site. The most difficult task in the creation of this tool is the parsing of the file...

www.hiveworkshop.com

 

[Nichilus]

As I wrote - the Reforged version contains additional constructs that I have not seen covered anywhere. The tutorial you posted covers pre-Reforged version. (although it is a good source of information, many structures remained same/similar in Reforged).

 

[AMarkov]

I'm considering creating a project specification where, for each map, I list the triggers that need to be activated. During the parsing of projects across directories, I would search for matches and use code to replace the deactivation string with the activation of the trigger.

Do you think this approach could work?

*I'm still working on the project architecture in light of new insights.

 

[AMarkov]

I'm modifying my system for hero trigger management. Let me explain: all hero triggers are initially disabled when the map is initialized. When I select a specific hero, I activate one trigger, which then initializes the entire set of triggers associated with that hero.

What I want to implement is a general trigger that will start the whole set of triggers. The activation condition will be a global variable, where I'll store the map's name. In the parser, I'll be replacing this global variable dynamically.

This way, there will be a unified repository, from which all child maps will be populated.

Trying to mess around with converting .w3* files to JSON and back could end up being a huge waste of time. I’m not a system architect. Similarly, attempting to clean up the .j file doesn't seem worthwhile at this stage.

I might revisit this concept later, once I’ve gathered more data. Maybe cleaning up the file will make sense if the map load times turn out to be excessively long. How much impact do triggers have that are created but not active in the map code?

 

[AMarkov]

yes, it seems the tool would do a good job, however it does not support parsing of wtg files.
The wtg files, specifically gui trigger definitions, contain multiple strange constructs - whether those are present depends on which action/condition you chose.
I've mapped the odd behavior when I was working on my own tool, so I may as well just post info about it into tutorial section, since on current Reforged there are some constructs that I have not seen mentioned anywhere else.

You might find this interesting. I'm working on a project using Git for version control, along with a remote GitHub repository. For some reasons, I had to roll back to a previous version, and it seems the wtg file got partially corrupted, leading to errors in the generation of the final .j file. When I tried to save the project, the number of triggers became -12,890,539 (a negative value). Despite this, I didn’t encounter any crashes in the Warcraft editor.

Reverting to the current version restored the map's functionality. Between version changes, I made sure to close and reopen the editor.
The GUI display of triggers is normal. However, there are significant errors in the .j file. Some // before comments have been deleted.

Additionally, when I try to open the file in a text editor, more than 50% of the content doesn't display. Is that normal?

 

[Luashine]

Ralle's wtg: GitHub - HiveWorkshop/Formats: This project aims to document and create Kaitai grammars for the various binary formats in Warcraft III.

@WaterKnight I think improved upon that and standardized to his approach: GitHub - WaterKnight/Warcraft3-Formats-KaitaiStruct: KaitaiStruct schemas for various formats of Warcraft III

me, documentation-only so far and I excluded wtg (or not?), WIP: GitHub - Luashine/wc3-file-formats: Project to document the proprietary file formats used by Warcraft 3

 

[Nichilus]

Sorry, I've been away from home due to work, so I had no time to write the wtg doc yet.

You might find this interesting. I'm working on a project using Git for version control, along with a remote GitHub repository. For some reasons, I had to roll back to a previous ...

One can only thank Blizzard for this experience I once had hope that they would fix all the broken stuff over time, but Reforged launched ~4.5 years ago and still no fixes in sight :/

Ralle's wtg: GitHub - HiveWorkshop/Formats: This project aims to document and create Kaitai grammars for the various binary formats in Warcraft III.
...

Honestly, to this day I had no idea that something like Katai Struct existed in the first place. I had to struggle a bit to understand the logic there, but I think I get the gist of it. Anyway, I've checked WaterKnight's Warcraft3-Formats-KaitaiStruct/w3-wtg-131.ksy at main · WaterKnight/Warcraft3-Formats-KaitaiStruct but I still think it would incorrectly evaluate a few specific, albeit very simple, triggers - that is, unless he does some cleanup via some other files.
I'll write a doc in similar format to Ralle's Warcraft 3 Trigger Format Specification (WTG!) - it will be interesting to compare notes with rest of you guys

 

[Nichilus]

Update: I've just posted the specification. It took me a while to write and format the examples.
See here: Warcraft 3 Trigger Format Specification (WTG!) - Reforged 1.36