Share scripts among custom campaign maps

[hatepoem]

# Given game version 1.27.1, how to effectively inject a common Jass script snippet into every map of a custom campaign?

## Problem

I intend to make a custom campaign for a legacy version of the game (1.27.1). The most critical problem at the moment for me is to make a chunk of my Jass script available for every map in my campaign. I could not come up with a satisfactory solution myself, so I ask here how to do it.

## Motivation

Most importantly, I want the scripts for my custom spells to be easily packaged with the campaign. Less important, but still desireable, to share helper functions and such, and simultaneously keep those separated from map-specific scripts.

I used to manually open every map and insert the code into the "custom scripts" section of the trigger editor. This approach is extremely error prone and time consuming. It won't do if I am ever making another campaign.

## What was tried so far

Here is how I approached the problem before asking here.

Create a skeleton campaign file (w3n) with native World Editor.

Use an MPQ editor. I tried the following implementations (backend):
- WinMPQ (SFmpq.dll);
- smpq (libstorm);
- Ladik's MPQ Editor (libstorm);
- jmpq3 (jmpq3)

Extract all map files. Patch all the map files by replacing "war3map.j" script in each extracted file map with purposefully modified ones (that much is trivial). Finally, patch the campaign file by replacing every original map file that is contained in the campaign archive with the patched map files. Fail.

Patching map files works as intended, even if it's much clunkier than I hoped. Individual maps, patched in this manner, given they are executed without the campaign archive, are parsed by the game as expected. The problem is that the patched campaign file produces fatal errors at runtime and is not recognized by the game.
In other words, I can patch scripts in maps, but I cannot patch map files in campaign.

Each MPQ editor tool I tried fails slightly differently. But the gist of it, no matter what options I choose, it never produces the desired result. It all ends at a fatal runtime error of some form or another. libstorm backend produces error message when the map is selected in campaign screen reading "corrupted file". SFmpq backend hangs to a black screen without any message before the campaign screen even shows. With certain configurations MPQ editors refuse to modify the campaign archive to begin with. smpq "single-unit" option does nothing. WinMPQ "embedded mpq" seems like it's intended for something else, I couldn't even figure out how that feature works.

The other approach I though about is patching the game MPQ archive by replacing their Blizzard.j file. I would rather avoid this. It seems less maintainable and more difficult to install for end users.
The last approach I thought of is patching map files without extracting them. For that I will probably need to write and compile my own program relying on the libraries. But it seems to me that it's the problem of the libraries themselves and not their front-ends. I might be missing some obvious configuration option, but I read up pretty much everything I could find on the topic and still can't produce a workable solution.

So, how do I inject my Jass script snippets into all my custom campaign maps automatically?

 

[hatepoem]

This thread suggest something that I suspected that I need to edit some other metadata in the campaign for it to be able to locate patched map files. I cannot edit campaign maps.

But first I am not sure if it's my issue. And second I couldn't figure out how to edit campaign metadata. Sure, I could read them, but any attempts at modifying them ended up in corrupted archives.

What I am trying to do should be possible, considering that this tool claims that it uses SFMpq (WinMPQ backend) to modify maps and patch campaign file accordingly. Campaign Processing Helper v1.1

 

[hatepoem]

I did an experiment. I extracted all Campaigns/DemoCampaign.w3n files. Then I re-created the archive using smpq and MPQEditor.exe. Both with libstorm9 back-end.

#!/bin/sh
APP="wine64 ${HOME}/local/share/mpqeditor/x64/MPQEditor.exe"
${APP} new demo-ladik.w3n
${APP} add demo-ladik.w3n Demo03.w3m
${APP} add demo-ladik.w3n Demo04.w3m
${APP} add demo-ladik.w3n Demo05.w3m
${APP} add demo-ladik.w3n Demo-Loading-BotLeft.tga
${APP} add demo-ladik.w3n Demo-Loading-BotRight.tga
${APP} add demo-ladik.w3n Demo-Loading-TopLeft.tga
${APP} add demo-ladik.w3n Demo-Loading-TopRight.tga
${APP} add demo-ladik.w3n DemoLocationIcons.tga
${APP} add demo-ladik.w3n OrcSymbol.tga
${APP} add demo-ladik.w3n star1.tga
${APP} add demo-ladik.w3n war3campaign.imp
${APP} add demo-ladik.w3n war3campaign.w3f
${APP} add demo-ladik.w3n war3campaign.w3t
${APP} add demo-ladik.w3n war3campaign.w3u
${APP} add demo-ladik.w3n war3campaign.wts
${APP} add demo-ladik.w3n war3campImported/d03_blue.ai
${APP} add demo-ladik.w3n war3campImported/D05_brown.ai
${APP} add demo-ladik.w3n war3campImported/D05_green.ai
${APP} add demo-ladik.w3n war3campImported/D05_lightblue.ai
${APP} add demo-ladik.w3n war3campImported/DemoLoad03.mdx
${APP} add demo-ladik.w3n war3campImported/DemoLoad04.mdx
${APP} add demo-ladik.w3n war3campImported/DemoLoad05.mdx
${APP} add demo-ladik.w3n war3campImported/scorescreen-hero-seawitch.tga

smpq is the same except it's native to Linux. +++ Also, smpq aims at supporting more modern versions of MPQ archives, like World of Warcraft ones. The MPQ version 0, which all Warcraft 3 archives are supposed to be, isn't even listed in the manual. Ladik's was build originally to support Warcraft 3 archives specifically, therefore I expect it to work better. But they both produce malformed archives this way.

The result campaign archive cannot be read by the game. It just doesn't appear in the file browser. I did no modifications to the component game files themselves.

I tried using WinMPQ for the same purpose but it gave me some weird warnings and did nothing. I suspect it wouldn't have worked either way.

The only file I suspect is distinct from the original campaign archive to this re-constructed one is the attributes meta file, I am guessing. Going to check that one in detail.

 

[hatepoem]

It doesn't seem like attributes file has anything to do with custom campaigns.

 

[hatepoem]

MPQ archives may be parts of other files. In that case the library somehow appends metadata to indicate to the game where the MPQ archive starts in a file. libstorm docs mentioned something like this.

Looking at the DemoCampaign.w3n in a hexeditor, sure enough it appears that every map is a separate MPQ archive with a distinct header clearly visible. I think it's the header described in seciton 2) of this article. W3M and W3X Files Format W3M and W3X Files Format

Sure enough, it's this map header that gets lost when smpq or any other editor try to append or overwrite a map in an existing archive.

I am not sure how to best recreate these headers or how to prepend them, but I think I am on the right track.

 

[hatepoem]

I managed to make an MPQ archive appear in the file browser inside the game. Indeed, the WarCraft 3 map header was all that was required.

I still can't make any campaign map file to load. From what I understood from staring at the hexeditor output, campaign maps are appended to a campaign file and not included in an MPQ archive. So that's what I tried to mimick. Create header, append baseline campaign MPQ to it, then append individual map files. This much works for making it show in the file browser, but not enough to make it load. This doesn't look right in the hexeditor. Maybe I use wrong tool to append the map MPQs. Stuck again.

+++ Something like creating header, appending baseline campaign MPQ archive, then using smpq to include map files doesn't work either. The original DemoCampaign.w3n has every map file clearly visible by it's header in the hexeditor. Using any MPQ editor strips those. So valid custom campaign file should have those map headers present as well. But just appending those with cat does nothing. So the map files are indeed included in the baseline MPQ somehow, and not just separate MPQs shipped in one file. I just don't understand the format enough.

+++ I give up for now. I already easily create individual maps that successfully embed my custom scripts automatically. Since it's the most repeated job, it is the one that requires to be automated. Packaging a campaign archive, whatever it's exact format, is only done at release, and will be repeated at most a few times a month. I can bear with it and probably save myself more time. I can't figure this thing out.

 

[hatepoem]

Here is the program that I used to generate map file headers. It accepts a single argument from command line and makes it the map name. It writes the resulting header to standard output. This snippet isn't worth a Git repository, but I want to save it somewhere.

#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

const char HEADER_MAGIC[4] = "HM3W";
const char FOOTER_MAGIC[4] = "NGIS";
const size_t HEADER_SIZE = 0x200;

enum map_flag_t {
    MINIMAP_HIDE = 0x0001,
};

struct map_header_t {
    char magic[4];
    int32_t unknown;
    char map_title[36];
    int32_t map_flag_mask;
    int32_t map_player_quantity_max;
};

struct map_footer_t {
    char magic[4];
    char data[256];
};

int main(int argc, char **argv)
{
    FILE *file_p = stdout;
    if (NULL == file_p) {
        perror("could not open file for writing");
        return EXIT_FAILURE;
    }

    char *buffer;
    buffer = calloc(HEADER_SIZE, sizeof(char));
    fwrite(buffer, HEADER_SIZE, 1, file_p);
    free(buffer);

    rewind(file_p);

    struct map_header_t header_p = {
        .magic = "HM3W",
        .unknown = 0,
        .map_flag_mask = MINIMAP_HIDE,
        .map_player_quantity_max = 1,
    };
    strcpy(header_p.map_title, argv[1]);

    fwrite(&header_p, sizeof(header_p), 1, file_p);

    fflush(file_p);
    fclose(file_p);

    return EXIT_SUCCESS;
}

 

[hatepoem]

I figured it out.

In order to successfully overwrite a map inside a custom campaign, the map file must be valid (with valid map header and MPQ header). Then, the campaign archive must contain metadata about campaign buttons that are rendered at runtime in game.

Finally, the issue that I was experiencing, and what I couldn't understand, is as follows. Additional arguments must be passed to the MPQ tool to successfully overwrite a map file. The map file must be added as a single unit (not divided into sectors like most other files in the MPQ archive for easy streaming). It also must be added without compression (this is how I lost map header repeatedly).

I use smpq that's based on libstorm same as Ladik's. For example.

smpq --add --overwrite --single-unit --compression none mycampaign.w3n map1.w3x

Note, I didn't test caching and hero loading yet. But I can patch scripts and other metadata just fine with this, so for now I am happy.

All of my musings in this and related threads up until this point are irrelevant. What actually needs to be done I just described.