Opening w3x/w3m files

[Chaosy]

So I was considering some options regarding working on a map without having wc3 installed.

I set up a quick web server to run locally but quickly realized I found no library to use to open and extract the .j file

Could anyone give a hint on how to do it?

GitHub - ladislav-zezula/StormLib: Official GitHub repository of the StormLib library created by Ladislav Zezula (author) seems like a solid option but there are 0 instructions on how to use it, and it is written in C, I got no clue how to transfer that to javascript.

 

[GhostWolf]

If it's only extracting, my code supports inflate compression, which accounts for most of the MPQ files:
mdx-m3-viewer/parser.js at master · flowtsohg/mdx-m3-viewer · GitHub

Although I suspect what you really want is only line 25 which creates the MPQ:
mdx-m3-viewer/src/handlers/mpq/parser at master · flowtsohg/mdx-m3-viewer · GitHub

Nothing there supports writing though.

 

[Chaosy]

Well that is a pity since I would need to insert the file again.

It was only an idea on my part, so I am not too sad if that is not doable.

 

[Dr Super Good]

Chances are it is doable but would need a lot of development work for a pure javascript solution.

A hybrid solution might be more viable. For example one could use javascript to send commands to a server which has a Java or C++ process running that executes the commands to manipulate the map archive files.

If people worked more on Java libraries, one could use those to write a third party editor to work on maps in a cross platform way. However chances are one will still need some install of WC3, even if it is just for the main data MPQ archive files. In theory such an editor could even be ported to run on Android devices.

 

[GhostWolf]

@Chaosy getting there.

fetch('War3_low.mpq')
    .then((request) => request.arrayBuffer())
    .then((buffer) => {
        let archive = new MpqArchive();
     
        archive.load(buffer);
      
        let listfile = archive.get('(listfile)'); // => MpqFile
        console.log(listfile.text()); // also has .arrayBuffer()
      
        archive.set('test.txt', stringToBuffer('this is a test hello')); // => true
      
        archive.has('test.txt'); // => true

        archive.delete('test.txt'); // => true
      
        saveAs(new Blob([archive.save()], { type: 'application/octet-stream'}), 'test.mpq');
    });

 

[Chaosy]

@GhostWolf

Oh, neat.

I was toying around with alternatives. I thought that if my editor could just generate a .j file, and then use the import vjass functionality to load it.
I have not added this yet.

Currently I load a .j file from a map with only GUI triggers (since I think that is way easier than reverse compiling vjass)

Hidden

I successfully detect all the triggers and their associated code, kind of.
Right now I just check for the code inside the initTrig function, however some GUI actions create extra functions. ForGroup and if-blocks being the ones shown in my demo triggers.
So that is not working yet, but it is on my todo list.
Right now it gets detected as a custom script which is not ideal, but it works.

 

[GhostWolf]

@Chaosy

It's at the point where the API above works, but doesn't handle some edge cases, doesn't yet compress added files, etc.
I want to give you a copy so you can test it.
What kind of environment do you work in? (browser, node, do you have webpack, ...)

Note that it still doesn't support maps, but once MPQ works, it can easily be extended to maps.
That is, you can load a map as an MPQ, but you'll only get the MPQ part without the header.

/Edit
It now compresses added files with zlib where it can.
Is there any point in allowing to encrypt files?

 

[Chaosy]

@GhostWolf
Node.js


No, I do not think encrypting is needed.

 

[GhostWolf]

@Chaosy
My own internal code now uses the new archive, and so far it works well.
I have committed all of my changes, technically you can use it now, but you will have to setup webpack and run it, due to V8 not supporting ES6 imports/exports.
I am not quite sure if there's a way around this, I don't have too much experience with Node.
I would like to be able to compile files together into one require()able file.

In case you don't need instructions, just npm install mdx-m3-viewer.
I attached an example webpack config that should work for compiling stuff in the viewer.
One of the things the library exports is MpqArchive.
For a smaller import size, since you probably don't care about the rest of the viewer, you can import the archive directly, it depends on almost nothing from the rest of the viewer.

The source is here mdx-m3-viewer/src/handlers/mpq/parser at master · flowtsohg/mdx-m3-viewer · GitHub


/Edit

Added map support.
It is exported as W3xMap, and I exported also the stringToBuffer utility function if you need it.
I don't have time to do tests now, but just from opening 3 Blizzard maps and saving them, it seems that (attributes) is consistently encrypted with a key that depends on its position in the archive.
I don't yet support this when saving (you can read it), so if you encounter this (watch the console), you can delete the file if you want.
In addition, for a copy of (4)LostTemple.w3x at the very least, MPQEditor couldn't load the (listfile), however the map worked fine in WE, so I need to check that.

 

[Chaosy]

According to: ECMAScript Modules | Node.js v8.6.0 Documentation
I can actually enable import.

Though I have not used imports before since they are not in the standard yet, so I do not know how to use it regardless.

edit: I managed to run the webpack and get the viewer.min.js file
I linked it into my HTML file, now I just need to write the client side script.

HTML:
<script src="/viewer.min.js"></script> //your module
<script src="/test.js"></script> //the file I write the test code in

function readFile() {
    return new Promise(function(res, rej) {
        var fileReader  = new FileReader();
        fileReader.onload = function(){
            res(this.result);
        }
        fileReader.readAsArrayBuffer("/test.w3x");
    });
}


let archive = new MpqArchive();


readFile().then(function(buffer) {
    archive.load(buffer);

    let listfile = archive.get('(listfile)'); // => MpqFile
    console.log(listfile.text()); // also has .arrayBuffer()

    archive.set('test.txt', stringToBuffer('this is a test hello')); // => true

    archive.has('test.txt'); // => true

    archive.delete('test.txt'); // => true

    saveAs(new Blob([archive.save()], { type: 'application/octet-stream'}), 'test.mpq');
    console.log("oh shiet waddup");
});

The idea here is to load a local w3x file and then insert the test.txt into it.
However I get
"test.js:12 Uncaught ReferenceError: MpqArchive is not defined"

 

[GhostWolf]

You said you work with Node
The attached webpack config will result in the file having the global variable ModelViewer, which contains MpqArchive/W3xMap etc.

I did see that my listfile problem is actually that all compressed files are now bugged for some reason, even though they worked fine before. Need to fix that.

 

[Chaosy]

Yeah I do. Does not really make a difference client side though, does it?

The reason I did it client side is because your example did that.

 

[GhostWolf]

Just a different way of importing the code. I don't really know how to import directly with Node because of the ES6 imports.
Everyone's waiting for V8 to support them.

Right now compression fails when saving maps, but works when saving archives. Not sure what's going there - all of the data loaded is correct, but some flags are gone.

 

[Chaosy]

Welp. I compiled the most recently provided webpack.

console.log(ModelViewer);

Does not seem to contain any variables.

 

[GhostWolf]

This is easier for now, need to figure how to properly share code.

For some reason when I save maps, all of the data is correct, but MPQEditor and WE seem to ignore a flag I use and not even show it, which stops the maps from being valid.
The flags are used correctly when opening a MPQ.
I cannot understand what's going on, maybe MPQs in maps don't support it.
For now everything is saved uncompressed, so at least it will work and you'll be able to test it.

If you want to load a MPQ/map rather than using an empty one, either pass your buffer to the constructor, or to the load() method.

 

[Chaosy]

Seems to work fine.

Quick guide on how to load a w3x file now that I know how to save it?

edit: what I mean is, in order to use the .get I need to select an existing file.

edit: Nevermind, got it to work.

 

[GhostWolf]

Do you mean how to know what files the MPQ has?
You can either get the (listfile), or look at archive.files
But we already know what files a map should/could have:

Files

(listfile)
(signature)
(attributes)
war3map.w3e
war3map.w3i
war3map.wtg
war3map.wct
war3map.wts
war3map.j
war3map.shd
war3mapMap.blp
war3mapMap.b00
war3mapMap.tga
war3mapPreview.tga
war3map.mmp
war3mapPath.tga
war3map.wpm
war3map.doo
war3mapUnits.doo
war3map.w3r
war3map.w3c
war3map.w3s
war3map.w3u
war3map.w3t
war3map.w3a
war3map.w3b
war3map.w3d
war3map.w3q
war3mapMisc.txt
war3mapSkin.txt
war3mapExtra.txt
war3map.imp
war3mapImported\*.*

war3map.j can also be in scripts/war3map.j, used by protectors. I didn't check yet if that bothers WE or not.

You might notice that the map object already parses some of the internal files.
I use these for the viewer.
If you want files that are not parsed to be parsed, tell me and I'll implement them.
I'll probably move the parsing to another function that needs to be called explicitly, in case it's not needed.

 

[Chaosy]

I got it to work.

It is a bit annoying that I have to choose a local file each time while debugging, but they disabled doing something like fileReader.readAsArray("./myFile.w3x"); because of security reasons, websites could otherwise scan your PC.
So not I have a <input> tag for the user to choose a file.

With that out of the way, actual development on the GUI viewer is back online ;P
Or more likely, code restructure to prepare for looping through all arguments.

 

[GhostWolf]

Changed the encoding to use sectors - compression should be better, and also actually work for both MPQs and maps. I still have no idea why the other way doesn't work in maps.

If what you're after is parsing Jass (I have no clue what you're doing), I have a parser stolen from Ralle.
I used it to run Jass code in the viewer, which is pretty amusing.
I might actually do some stuff with it relating to the new writeable map.

 

[GhostWolf]

Added support for saving files with an encryption key based on their offset, when their offset changed.
This should remove the annoying errors with (attributes), and in some cases other files.

Also added MpqArchive.prototype.getFileNames() just for convenience.

If everything will continue working fine, the only thing left to add is support for more compression types, especially explode() which seems to be used by all sorts of older (listfile)s.
Unless there's something missing that I am not thinking about.

I also optimized some parts and removed memory allocations.


/Edit

Turns out there was still a lot to optimize, very easy to see when loading a big MPQ like war3x.
I optimized decoding files greatly, but there are still optimizations left.
Should be a lot faster and cheaper on memory in the next patch.

 

[Chaosy]

@GhostWolf

Basically an online GUI trigger editor.

And since I need to make the viewer interactive at some point I cannot use Ralle's.
Part of the fun is coming up with your own solution ^^

Since I am nowhere close to being able to save a file just yet I will not update the file, I will do that when the times comes ^^
Thanks a lot for your hard work though

 

[GhostWolf]

Optimized the code, it should be a lot faster for all operations, and take far less memory.

New files for which the compressed data would be bigger than decompressed data are stored decompressed properly again.
Doing otherwise results in their data being garbage once read by WE or any other loader.

The map file loaders (terrain, units, ...) will no longer get called when loading a map, but rather must be called explicitly.

As to what you're doing - you need to parse the WTG file, see Ralle's specs in the stickies.
I might implement it at some point.

 

[Chaosy]

Until I know how to read the wtg file I will just save in a txt file and insert it.
Along with the .j file, this would make it work ingame, but not in the normal editor.

Which is obviously not ideal, but it is ok for the time being. For me it is only a proof of concept anyway

 

[GhostWolf]

Improved some stuff, and changed the API.

Rather than allowing set(name, buffer, true) to override files, you must now explicitly call edit(name, buffer). I am still unsure about this, since it by design makes you get the file twice, via has() and edit().
In addition there's now rename(name, newName).
And an extra utility function bufferToString(buffer) which is internally used by MpqFile.text().
I also exposed BinaryReader and BinaryWriter in case you want to experiment with reading/writing the WTG, they make it easier.

As far as I can tell, everything is working, and there isn't much that could be optimized further.
Maybe I'll add in the future more compression types and more map internal file parsers.

/Edit
Added MpqFile.rename(newName), MpqFile.edit(buffer), and MpqFile.delete().
This will allow to skip get()s if you are going to reuse the file anyway.

 

[GhostWolf]

Whoops, turns out I was corrupting any encrypted file that was decoded. Fixed that.
MpqArchive.edit() -> MpqArchive.set(), I changed it back, but you don't need any additional boolean to set an existing file. Just like the standard Map.
MpqFile.edit() -> MpqFile.set().
Optimized file decoding for non-encrypted files.

 

[Chaosy]

Dope. My progress does not seem to keep up with yours though xD

 

[GhostWolf]

Added support for imports in maps. When adding non-internal files to the MPQ without them being in the import list, they will be deleted by WE.

W3xMap.getFileNames() - shortcut
W3xMap.getImportNames()
W3xMap.import(name, buffer) - MpqArchive.set(), but adds the entry to the imports too.
W3xMap.set(name, buffer) - shortcut
W3xMap.get(name) - shortcut
W3xMap.has(name) - shortcut
W3xMap.delete(name) - MpqArchive.delete(), but removes the entry from the imports too.
W3xMap.rename(name, newName) - shortcut

I don't know what happens if there's a name in the imports list but the file itself was deleted, so prefer to use W3xMap.delete() when using maps.

Try to parse the WTG

 

[Chaosy]

@GhostWolf

this is my client side code for the .j file

function readFile(file) {
    return new Promise(function(res, rej) {
        var fileReader  = new FileReader();
        fileReader.onload = function(){
            res(this.result);
        }
        fileReader.readAsArrayBuffer(file);
    });
}

let MpqArchive = ModelViewer.MpqArchive,
    W3xMap = ModelViewer.W3xMap,
    stringToBuffer = ModelViewer.stringToBuffer;

var fileInput = $('#fileInput');
var socket = io();

fileInput.change(function() {
    readFile(this.files[0]).then(function(file) {
        console.log(1);
        let map = new W3xMap();
        map.load(file);
        let listfile = map.archive.get('war3map.j');
        console.log(listfile.text());
        /*$("#map").submit(function() {
            socket.emit("j", listfile.text());
        });*/
    });
});

Gives "Uncaught (in promise) unknown compression method"

 

[GhostWolf]

Ah, interesting. I think the zlib library failed to decompress the data.
Can you give me the map?

 

[Chaosy]

@GhostWolf
asd | HIVE

 

[GhostWolf]

My bad.

 

[Chaosy]

I get a bunch of information, but I am uncertain if there is an error in the parser or that it is somehow intended to look like this:

Which is basically what it looks like if I open it with notepad.
The file is war3map.wtg

 

[GhostWolf]

Yes, it's binary data. You need to follow the specifications Ralle wrote (stickied in this forum), and parse the data.
That's why I exposed BinaryReader/Writer.
They allow to read/write typed binary data with a pretty simple API.

 

[GhostWolf]

Started working on WTG parsing anyway, since it's needed for other things too.
You'll probably be able to use it later for both reading and adding new GUI.
All sorts of map editing features might be exposed.
We'll see.

 

[Kakerate]

All sorts of map editing features might be exposed.

ya just be sure not to expose anything too useful or it will be patched out