[Reforged] Keeping map/campaign imports in local/external files to reduce save time?

[Macielos]

I have a large campaign file (820MB and growing fast) and I was wondering if I could speed up saving time (which now takes ~40 seconds or so) by keeping imports outside the campaign file.

I tried local files - I enabled them, extracted my imports into retail folder, loaded a "thin" campaign file in editor and in game - works perfectly, the campaign weighs 13MB now, I'll pack it back before the release. The problem is, when I save a map or a campaign, all imports are saved back to the campaign file. Can I somehow prevent that?

I know I could put custom stuff in War3Mod.mpq pre-1.31, but I think this option is gone with migration to CASC. I suppose there's no way to use custom MPQs now?

Or any other tricks you guys might have to have your campaigns and maps save faster? (nope, I can't throw stuff away, in fact it's gonna put even more)

 

[PurgeandFire]

Cool idea! First, how did you "thin" your campaign file to begin with? I'm guessing you used a CASC editor and extracted all the imports to your local files directory and deleted them from the campaign? Did you delete the imported files via the CASC editor, or did you use the world editor and delete the individual imports?

And are you deleting the assets from the campaign file or from the individual maps? (or both?) Are most of your imports imported at the campaign level, or are they imported into each map individually?

My thinking is that if you deleted the assets from the campaign file but left the import list as-is. The import list is stored as a war3map.imp within the map/campaign itself and is a simple list of assets imported into the map. My best guess is that when the editor is building the archive (e.g. the w3n or the w3x), it will look at the import list and then it'll "search" for each file (which I'm guessing will look first within the map archive itself and then look within the local files directory) before packing it all up into the final archive. But I haven't tested it.

If my theory is right, then to get your workflow working you'd probably need to remove the imports from the actual import list too. But it probably would kinda suck to have to manually re-import everything down the road, trying to get all the paths right. But some simple scripts could maybe smooth out the process there. Regardless, I'd recommend making a back-up of your campaigns/maps as-is first--and once I know your process maybe I can make some more specific recommendations!

 

[Macielos]

Cool idea! First, how did you "thin" your campaign file to begin with? I'm guessing you used a CASC editor and extracted all the imports to your local files directory and deleted them from the campaign? Did you delete the imported files via the CASC editor, or did you use the world editor and delete the individual imports?

Ladik's MPQ Editor to delete folders from MPQ campaign file, and occasionally mpqmaster because the former sometimes messes up the campaign file (making dummy change with mpqmaster fixes it) while the latter can't delete folders. Yeah, I know...

And are you deleting the assets from the campaign file or from the individual maps? (or both?) Are most of your imports imported at the campaign level, or are they imported into each map individually?

From the campaign file. I have very few imports in individual maps, mostly AIs and single texture, tile etc. overrides, so they don't bother me.

My thinking is that if you deleted the assets from the campaign file but left the import list as-is. The import list is stored as a war3map.imp within the map/campaign itself and is a simple list of assets imported into the map. My best guess is that when the editor is building the archive (e.g. the w3n or the w3x), it will look at the import list and then it'll "search" for each file (which I'm guessing will look first within the map archive itself and then look within the local files directory) before packing it all up into the final archive. But I haven't tested it.

If my theory is right, then to get your workflow working you'd probably need to remove the imports from the actual import list too. But it probably would kinda suck to have to manually re-import everything down the road, trying to get all the paths right. But some simple scripts could maybe smooth out the process there. Regardless, I'd recommend making a back-up of your campaigns/maps as-is first--and once I know your process maybe I can make some more specific recommendations!

Well, someone on Discord adviced me to remove .imp file. For some reason removing it didn't work on my main campaign yet (somehow import list gets recreated), but it did on a clean one, with a single map and a single local file import. I imported it to the campaign, created a copy under Warcraft III\retail\war3campImported\, cleared .imp file, relaunched the campaign - the import's still working, it is not saved into campaign file, and import tab is empty. So far, so good.

EDIT:
Okay, it did work with main campaign, by mistake I extracted .imp backup into retail folder .

EDIT2:
Okay, so far everything works smoothly from local files and campaign/maps save time went down drastically. So it's great for work on missions. Now I need to work out a flow of making changes that involve the imports. I believe I'd have to restore .imp file, add/modify imports, then export them to local files (or just make the same changes) - but I need a modified .imp file, so I need to save the campaign - which would produce "fat" file again... Ehhh... Damn, I'll just write a script to scan local files and generate the .imp file for me.

I'd also need my imports back whenever I want to use a sound because imported sounds are fetched from imports and they're gone without the .imp file.

 

[PurgeandFire]

Okay, so far everything works smoothly from local files and campaign/maps save time went down drastically. So it's great for work on missions. Now I need to work out a flow of making changes that involve the imports. I believe I'd have to restore .imp file, add/modify imports, then export them to local files (or just make the same changes) - but I need a modified .imp file, so I need to save the campaign - which would produce "fat" file again... Ehhh... Damn, I'll just write a script to scan local files and generate the .imp file for me.

I'd also need my imports back whenever I want to use a sound because imported sounds are fetched from imports and they're gone without the .imp file.

nice! Awesome that it is working. I def agree on having a script to generate the .imp file for you. If you need any specs on the file format, you can find it here:

• WC3MapSpecification/Imports/1.md at master · ChiefOfGxBxL/WC3MapSpecification
• W3X_Files_Format

Honestly you can probably just give ChatGPT some context for the file format and then it can help write a script to generate that for you based off a folder. That way all the changes you want to make (e.g. renames, moving things around, etc.) can just be done in the file system and then you don't really need to touch anything in the editor until you want to pack it all together.

And then to support the sound use case, perhaps add an option to generate the war3map.imp only for the sound files or for a specific allow-list. That way you can test out your sounds and stuff without having to pack absolutely everything into the campaign.

Let me know if you need any help getting the scripts sorted out!

 

[Macielos]

Okay, I did it and I have no bloody idea why I waited so long to give this idea a try. So here's my flow:

1. Enabled local files
2. Created an SVN repo (yeah, I know, SVN is old, but artistic part of my team thinks git is Matrix and that scares them), commited all the imports from my campaign there
3. Cloned SVN repo to retail folder, standard blizz files are kept in .svnignore
4. Removed all imports from the campaign, as well as .imp file

5. From that moment I work on local files, saving the campaign lasts mere seconds instead of a minute, all changes in imports are commited to SVN and then updated by the co-workers. Issues I noticed because of no import list:
a) I can't use a new sound as music in sound editor - in order to do so, the easiest way it to re-import it normally, use it and remove later
b) When I pick a model for custom unit/ability/etc., a model overview in the left bar is not working - not a problem really

6. Once I want to share a "fat" campaign file with someone, I run this script to generate a new import list:

GitHub - Macielos/WarcraftImportListGenerator

Contribute to Macielos/WarcraftImportListGenerator development by creating an account on GitHub.

github.com

I put generated file into the campaign via MPQ editor, open the campaign, click save - all the imports are loaded back into the campaign.

Voila.

 

[Macielos]

Just for the record - i expanded the above tool to be able to add content of a directory to an existing .imp file under proper paths (e.g. lots of icons under ReplaceableTextures\...\BTNblabla.blp). I needed it because - surprisingly - there was no simple way to update the index after putting lots of new files to MPQ archive using e.g. Ladik's MPQ Editor.