MiniWc3

[Retera]

A friend asked me to compress the file size of the game of Warcraft 3 to as small as possible, a goal that I could respect to reduce the time it takes to copy my legally purchased Warcraft 3 game to a couple of lab computers in college that wipe their harddrives on reboot in order to play the game with some friends on a given evening.

So, I set about it, and got to a barebones version of the game that is 100 MB and can still play in custom games and melee games on Battle.net without disconnecting because all of the unit data is consistent with what it needs to be.

Does anybody else have any use for the results of that compression project, and does anybody know if it would be legal to share a version where you have to install or build it out of your existing WC3 install so I'm not giving people the game who don't own it already?

 

[Dr Super Good]

So, I set about it, and got to a barebones version of the game that is 100 MB and can still play in custom games and melee games on Battle.net without disconnecting because all of the unit data is consistent with what it needs to be.

You might still be banned on BattleNet from melee as it may detect the installation has been modified which violates the terms of use. Generally I would say custom games on BattleNet are safe, especially since they are mostly run by third party robots.

Does anybody else have any use for the results of that compression project, and does anybody know if it would be legal to share a version where you have to install or build it out of your existing WC3 install so I'm not giving people the game who don't own it already?

As far as I am aware you can download Warcraft III freely. Technically you do need a licence (CD keys) to install and play it through. One can get most of the Warcraft III assets by downloading StarCraft II which can be done without even a Blizzard account from the offline mode of the Blizzard app.

I see you have heavily upped the JPEG content BLP compression. I would hardly call it "like real game" as it is highly noticeable, especially on the terrain.

If one plays at the highest visual settings one can remove mipmaps from certain textures. Specifically terrain and any UI element like command card buttons and unit icons do not need mipmaps when played on maximum visual settings. The result is a saving of ~1/4 of the texture file size.

 

[DracoL1ch]

wc3 have NO checks for integrity, you may modify w/e you want except for a few files.
personally I cant see the purpose. fucking 8gb flashdrives being sold in russia for $4, in other countries prolly even cheapers. and you can't fit all game resources, meaning some maps will miss models

 

[Dr Super Good]

wc3 have NO checks for integrity, you may modify w/e you want except for a few files.

Are you certain? I would imagine Warden and such would monitor integrity like it monitors background apps and other outside influences.

personally I cant see the purpose. fucking 8gb flashdrives being sold in russia for $4, in other countries prolly even cheapers.

Downloading from a flash drive is not the fastest. It might take minutes setup per computer to copy the 1.1 GB base install of Warcraft III.

Of course one could buy everyone a <$4 8 GB USB stick and run the full install of Warcraft III off the USB stick directly. If the computer systems have 2+ GB of memory, the I/O bottleneck of the USB stick could probably be masked by forcing all the Warcraft III MPQ files into memory in the background as a sequential read operation (fastest speed).

and you can't fit all game resources, meaning some maps will miss models

He has fitted in all game assets so custom maps will still work. However the assets are heavily compressed with lossy compression, as his demonstration image shows on the terrain where JPEG compression artefacts are clearly visible.

 

[DracoL1ch]

well I saw opt'd models before - polygons trying to escape it, making model awfuly. if there's no such kind of bugs - ok.
idk about warden, but the thing that it ignored by ANY maphack means warden is useless and definitely not about checks. about anything else - google for dota promode. I have insanely altered wc3, for instance.

 

[Kyrbi0]

I see you have heavily upped the JPEG content BLP compression. I would hardly call it "like real game" as it is highly noticeable, especially on the terrain.

Perhaps to you, maven of technology and overall wizard. : )

 

[Dr Super Good]

idk about warden, but the thing that it ignored by ANY maphack means warden is useless and definitely not about checks. about anything else - google for dota promode. I have insanely altered wc3, for instance.

Warden only ran on custom games for about a year around the time of the hashtable patch. As I said chances are you will not be banned from custom games today as all those robots are not being banned showing either no warden or a lack of maintenance to use warden to ban anyone from custom games.

Warden should still be running for melee games, especially automated tournament matches. It might detect the tampered with install of Warcraft III and flag you as a possible cheater. If you get banned for this is another question as ultimately Blizzard might ignore it as you are not using it to cheat or they might not care about cheats at all anymore (busy migrating to BattleNet 2.0 which does perform automated integrity checks).

Perhaps to you, maven of technology and overall wizard. : )

You can see the 8*8 JPEG compression block artefacts on the terrain textures... This is not due to the screenshot being uploaded as it is a png and the artefacts are bigger than 8*8 pixels and projected on a surface. Standard Warcraft III terrain on high does not look like that, as this experimental screenshot shows (ignore left as that was a seamless texture test so not standard).

big image

 

[Kyrbi0]

You can see the 8*8 JPEG compression block artefacts on the terrain textures... This is not due to the screenshot being uploaded as it is a png and the artefacts are bigger than 8*8 pixels and projected on a surface. Standard Warcraft III terrain on high does not look like that, as this experimental screenshot shows (ignore left as that was a seamless texture test so not standard).

big image

I hear what you're saying, but... I can't. That's my point. You can, 'one could', even, but I cannot.

The picture is difficult to navigate in-browser, so I'm not quite seeing what the comparison is. But I appreciate it nonetheless.

 

[WhiteFang]

You can see the 8*8 JPEG compression block artefacts on the terrain textures... This is not due to the screenshot being uploaded as it is a png and the artefacts are bigger than 8*8 pixels and projected on a surface. Standard Warcraft III terrain on high does not look like that, as this experimental screenshot shows (ignore left as that was a seamless texture test so not standard).

big image

I see it

 

[Dr Super Good]

I hear what you're saying, but... I can't. That's my point. You can, 'one could', even, but I cannot.

How visible they are varies depending on display quality, dpi and how good ones eye sight is.

Low quality or poorly calibrated displays might have such a low dynamic range the colour difference is not very noticeable and hence the artefacts are less noticable. High density displays, or displays viewed from far away, might display the image so perceptually small that the artefacts are not perceivable. If one suffers from an underlying eye condition such as colour blindness, short/long sightedness, retinal damage, fracture of brain into two halves etc then one might not be able to perceive the artefacts due to a reduction in what one can perceive.

 

[Kyrbi0]

How visible they are varies depending on display quality, dpi and how good ones eye sight is.

Low quality or poorly calibrated displays might have such a low dynamic range the colour difference is not very noticeable and hence the artefacts are less noticable. High density displays, or displays viewed from far away, might display the image so perceptually small that the artefacts are not perceivable. If one suffers from an underlying eye condition such as colour blindness, short/long sightedness, retinal damage, fracture of brain into two halves etc then one might not be able to perceive the artefacts due to a reduction in what one can perceive.

Sure, yep.

 

[pyf]

How visible they are varies depending on display quality, dpi and how good ones eye sight is.

Low quality or poorly calibrated displays might have such a low dynamic range the colour difference is not very noticeable and hence the artefacts are less noticable. High density displays, or displays viewed from far away, might display the image so perceptually small that the artefacts are not perceivable. If one suffers from an underlying eye condition such as colour blindness, short/long sightedness, retinal damage, fracture of brain into two halves etc then one might not be able to perceive the artefacts due to a reduction in what one can perceive.

Maybe I have all of the above


On a more serious note,
@Retera : I am interested in a tutorial on how you did it.

I myself usually reduce the HDD footprint of my already installed video games. As an example, the 24 bpp textures from the QRP Map textures for Quake i use, can safely be reduced to 8bpp without a significant loss in visual quality ingame imho (tip: do not use Floyd-Steinberg dithering).

I have trimmed my already-installed StarCraft, Diablo II and Warcraft 3, by removing nearly all of the solo campaign stuff / useless music / unused characters in solo play. I never play online, btw.

I am pointing out, by trimming/removing MPQ contents, one may afterwards be unable to update the game to the next version by using the official Patches. It happened to me when I wanted to update from a trimmed WC3 1.26a to version 1.27a. The reason was I had removed all the maps from the root of the MPQs. Since I always keep an unmodified backup on an external HDD. I simply restored the two unmodified MPQs and I was then able to update to 1.27a.

 

[Dr Super Good]

myself usually reduce the HDD footprint of my already installed video games.

Why are people bothering with this when a 1 TB HDD costs less than 40 GBP...

It will not speed up load time on a modern computer. It will not improve performance on a modern computer.

 

[pyf]

My 2005 laptop uses a 55 GB HDD @5400 RPM. I use an external 500 GB one for my backups.

I personally see no point in having any kind of data use more HDD space than necessary. Unless it significantly improves loading times, that is.

Among other things, converting all the textures of a game to 256 colors may allow for a 50% loading speed boost. It also saves memory.

 

[Deleted member 219079]

laptop

Install Linux Mint on it, 55GB will be plenty! WC3 works decently with Wine.

 

[pyf]

Thanks for the suggestion.

55GB already is much with XP installed. I already removed some of its useless data.

 

[Dr Super Good]

Among other things, converting all the textures of a game to 256 colors may allow for a 50% loading speed boost. It also saves memory.

Decoding every mipmap level of every texture (JPEG and BLP) in the game with additional checks using JAVA takes less than 30 seconds. Most loading time will always be taken up by object data and other game state initialization related tasks.

Decoding JPEG textures is not as slow as you might think. the DFTs used are highly optimized, the block size regular and both JPEG and direct (indexed) colour BLPs have some kind of compression which needs to be undone be it as part of the file (JPEG) or the MPQ (direct colour). I would recon changing all textures to direct (indexed) sample model might speed up load times in the average long loading map by about <5%.

The problem with direct (indexed) colour model is the mipmaps which are forced to share the same palate. They are colour wise not accurate and techniques like dither do not work at all (they only work without any scaling). This is why most games today use real time optimized fixed ratio compression with optional lossy secondary compression, minimal visual impact with great saving of GPU memory and file size.

It also saves memory.

Does it? I have a feeling the game internally converts the images to RGBA8 anyway as the indexed colour model used by WC3 and WoW is very unusual, having a separate 0, 1, 4 or 8 bit alpha component. I am not sure the legacy fixed pipeline of GPUs support this as although they do support mappings such as used by indexed colour, they generally do not like having to mix component sources for texel data.

That said, it only affects GPU memory, as that is where the textures have to reside. Even the cheapest, 30 GBP odd, graphic card has far more memory than WC3 can ever use. Hence why Blizzard removed the graphic memory system requirements from Warcraft III's official listing as it is impossible to buy new consumer level hardware that has less than enough memory to easily load every texture in the game one could ever possibly use.

For people interested, every BLP file inside WC3 stored losslessly in png files with only basic lossless compression takes up 400 MB. This includes every mipmap level in a total of 42,198 files. For comparison the GeForce GT 710, the lowest priced budget NVidia GPU one can get at the moment costing as little as $35 USD, has at least 1 GB of memory and can max WC3 out with AA as it is more powerful than an 8800GT.

As for memory... I just load the entire 1 GB of WC3 into memory when I want to play it so I have ~0 I/O loading times. This takes <10 seconds thanks to sequential I/O on mechanical drives (~100 MB/sec) and gives me random access I/O rate faster than any SSD (SSDs still slower than memory). It also uses up 1/18 GB of my memory, which WC3 itself can only use up at most 2/18 GB before it crashes due to being 32 bit and likely not built to use extended 32 bit address range for a full 4 GB of virtual memory space.

with XP installed

How can anyone use that in 2017... Well outside of playing very old games which only work on XP by running it in a virtual machine.

 

[pyf]

The 50% loading speed boost may be achieved on Quake or Doom source ports, by using high-res 24/32 bit textures converted to 8 bpp. I have done these tests myself many years ago. Please note, the average is more in the 35-40% range for such games.

Source ports support many image file formats such as PCX, TGA, PNG, JPG... Which is great for experimenting. Caveat: source ports are no created equal. Therefore, variations in loading times exist for the same set of optimized textures.

Maximum PNG compression may severely hamper loading times; in this specific case, less compression is a desirable thing, even though it means more hdd space used. For JPEG files, it is said libjpeg-turbo can improve decoding times, but idk by what margin.


One may use IrfanView to toy with images. The image / Information menu allows to know how much memory a picture uses, as well as gives a rough estimate of its loading time (in milliseconds) among other things.

Since AGP, textures may use some of the computer's memory:
Accelerated Graphics Port - Wikipedia
Not sure this is still being used nowadays...

How can anyone use that in 2017... Well outside of playing very old games which only work on XP by running it in a virtual machine.

Warcraft 3 *is* a very old game. How can anyone use that in 2017...
This game still works best on XP atm afaik.

 

[Dr Super Good]

The 50% loading speed boost may be achieved on Quake or Doom source ports, by using high-res 24/32 bit textures converted to 8 bpp. I have done these tests myself many years ago. Please note, the average is more in the 35-40% range for such games.

Warcraft III is neither Quake or Doom engine based though...

Maximum PNG compression may severely hamper loading times; in this specific case, less compression is a desirable thing, even though it means more hdd space used. For JPEG files, it is said libjpeg-turbo can improve decoding times, but idk by what margin.

WC3 does not support PNG. In fact it has to use BLP for anything that needs mipmaps and internally BLP either uses an indexed colour model or is 4 channel JPEG. In theory the indexed colour model textures will load faster but they have a lot of limitations.

The fastest textures to load are real time compressed textures, for example DXT/S3TC, because they can be loaded into the GPU compressed so have lowest bandwidth overhead. Depending on I/O characteristics it is either fastest to load them from an uncompressed source (file in file cache, limited by memory and PCIe bandwidth) or with some sort of simple compression (not in file cache and reading from mechanical drive, limited by read rate of mechanical drive).

One may use IrfanView to toy with images. The image / Information menu allows to know how much memory a picture uses, as well as gives a rough estimate of its loading time (in milliseconds) among other things.

Which is still trivial compared with loading the state in WC3. If one was to bench mark loading times one might find texture loading is at most 5-10% of the time.

Not sure this is still being used nowadays...

Was obsoleted a long time ago by PCIe. In theory PCIe also allows the GPU to use system memory except unlike AGP it does not have strict controller based limits so any memory limit is entirely up to the driver. In reality it is impractical for discrete graphic cards to use system memory due to buss latency (data locality) so this functionality is only used for specialist copy to/from RAM buffers, buffering data/command transfers and if the GPU runs out of texture memory to prevent OOM crashes. To put it in perspective, a modern GPU using PCIe 3.0 can in theory achieve transfer rates of 15.7 GB/sec compared with the 2.1 GB/sec of AGP. Using system memory to extend discrete GPU memory is no longer considered much anymore due to the smallest desktop GPUs having at least 1 GB of memory. On the other end of the spectrum you have AMDs answer for game consoles (XBO and PS4) which is to use a common memory pool for both CPU and GPU which both are on the same physical chip.

This game still works best on XP atm afaik.

It works the same on more modern OS... Probably even better than XP due to performance improvements with the scheduler, memory management and concurrency of the operating system. Assuming multi core CPU of course (might work slightly worse with single core due to other overheads).

 

[pyf]

I am explaining why a lower HDD footprint is desirable for *anything* as a general rule, as long as it does not affect performance negatively in a significant manner. I am providing ways to experiment with it for anyone with his / her own hardware.

I am pointing out, I do not have only Warcraft 3 / Blizzard games installed.

For Blizzard games, I only deleted files so far (some solo play data / useless music / useless FMVs / useless characters for solo play). Rough HDD space gain estimates for the French version of Warcraft 3 are:

Spoiler: for war3.mpq

- unmodified
426 MB

- no campaign dialogue (/sound/dialogue/*campaign)
357 MB

- no maps (*.w3m) - can not patch the game anymore afterwards with the official patches
349 MB

- no support files (*.html, *.jpg, *.css, *.js)
348 MB

- no files in the root *except* (attributes), (listfile), config.txt, *.mpq
341 MB

- muzak (/sound/music/mp3music/HOUE1-2-3)
depends - keep them if you like mods / cinematics

Spoiler: for war3x.mpq

- unmodified
384 MB

- no campaign dialogue (/sound/dialogue/*expcamp)
231 MB

- no maps nor campaign (*.w3m, *.w3n, *.w3x) - can not patch the game anymore afterwards with the official patches
206 MB

- no support files (*.html, *.jpg, *.css, *.js)
203 MB

- no files in the root *except* (attributes), (listfile), war3x.txt, *.mpq
195 MB

- muzak (/sound/music/mp3music/HOUN X1)
depends - keep them if you like mods / cinematics

Spoiler: Summary for RoC + TFT

810 MB
588 MB (nodialogue for campaigns)
555 MB (nodialogue nomaps)
551 MB (nodialogue nosupport)
536 MB (nodialogue nosupport noracine)
--- MB (nodialogue nosupport noracine nomuzak)

(taken from my raw notes)


The loading time of textures depends from game to game, and from game engine to game engine.
Suffice to say, milliseconds add to milliseconds.

There are no Warcraft 3 source ports available (for obvious reasons); therefore the possibilities to experiment are limited. Plus, Blizzard games/engines are not designed to be very versatile / modder friendly afaik.


My integrated Intel i915GM PCIe 1.x chipset allows for an AGP aperture size a shared VGA memory size of:
- 64 MB
- 128 MB
- variable size (thanks to the DVMT 3.0 feature)
I personally use the 64 MB fixed value (configurable from the BIOS settings)


If one has more than sufficient RAM, then one may wish to load the entire game from a RAMDisk (requires additional registry modifications; not tested). The smallest the HDD footprint, the more desirable this can be, since the data has to be copied once onto the RAMDisk.


Warcraft 3 does *not* work the same way on more modern OS. I personally do not experience:
- unexplained slowdows since the W10 Anniversary Update
- black / washed out / not working FMVs
- buildings sometimes disappearing (seems to be a video card driver issue?)
- WC3 suddenly asking to insert the game's original CD-ROM
...nor whatever other issue, on XP SP3


About the subject of trimming and CD-ROM games, this reminds me:

One may create an ISO of a CD-ROM, install the game from this ISO by doing a minimum install if applicable, and then trim the ISO of any useless stuff (installation files, third party stuff, audio CD tracks eventually...). When this is done, one may convert the trimmed ISO (Mode 1) to the ISZ file format with minimum compression. This may reduce HDD footprint for pre-2000 CD-ROM games. I personally call these 'PlayCDs', because they do not allow to install the game anymore, yet allow to play it without a hassle (also, no need to hunt for any no-cd etc...).

 

[Dr Super Good]

- no campaign dialogue (/sound/dialogue/*campaign)

Many custom maps use campaign dialog...

Suffice to say, milliseconds add to milliseconds.

Which in the case of Warcraft III ends up probably a lot less than the random access IO latency for reading on mechanical drives if you do not file cache the MPQ archives. All WC3 textures are at most 512*512 which is trivial in 2017.

There are no Warcraft 3 source ports available (for obvious reasons); therefore the possibilities to experiment are limited. Plus, Blizzard games/engines are not designed to be very versatile / modder friendly afaik.

It is designed to be quite optimized. That is part of the reason it runs on practically everything.

My integrated Intel i915GM PCIe 1.x chipset allows for an AGP aperture size a shared VGA memory size of:
- 64 MB
- 128 MB
- variable size (thanks to the DVMT 3.0 feature)
I personally use the 64 MB fixed value (configurable from the BIOS settings)

I do not know of any such processor currently in production. Even the worst modern intel integrated GPU has access to a driver determined graphic memory size similar to how XBO and PS4 work as they are both combination CPU/GPU on a single chip.

If one has more than sufficient RAM, then one may wish to load the entire game from a RAMDisk (requires additional registry modifications; not tested). The smallest the HDD footprint, the more desirable this can be, since the data has to be copied once onto the RAMDisk.

No, one can just abuse the system file cache. This is even faster than a RAM disk as basically no I/O needs to be done at all where as with a RAMDisk it still has to perform I/O with the drivers. It will remain in the system file cache as long as one uses the data and does not put a large data volume of other files into the system file cache.

The system file cache is a side effect of how memory mapping and I/O works. Since page files are supported to back virtual memory pages when the system is low of free memory, the same system can be used to support I/O of any file. Instead of deleting the read in pages once the I/O operating completes, they can take advantage of free memory by being retained as pages in memory with a high clear priority, known as the file cache. If I/O operations request a page and the page is inside the file cache then it will have access to the page after a very fast remapping operation without having to perform any disc I/O, skipping the I/O driver and completing the read/write at RAM speed (with same system call overhead). This is why a lot of memory is so important for modern computer systems, my 18 GB of memory means that I have both the game fully in memory and any files it touches so that subsequent reads from game data files happen at RAM speeds, giving high levels of performance for a 2010 mechanical drive backed system.

- unexplained slowdows since the W10 Anniversary Update

Turns out the slowdown was due to me messing with the option registry entries for the game, probably from back when it did not have widescreen support. Since I deleted all registry entries for Warcraft III to restore it to default the game runs at silky smooth 60 FPS. Specifically I think I tampered with the frame rate setting to try and optimize GPU usage however in the case of WC3 it cause the frame rate to be unstable in a bad way.

- black / washed out / not working FMVs

That was caused by the last patch not properly fixing the problem. They are working on fixing it but until then one has to wait. Not like one cannot simply go to YouTube and view them until then and that one actually views them that often (I last used mine over 10 years ago).

- buildings sometimes disappearing (seems to be a video card driver issue?)

Never had this issue on Windows Vista, 7 or 10 (assuming fresh start of WC3, and not having played a widgitized map before). From all the people reporting it, it is a driver bug with Intel GPUs on laptops that have had a long up time. Intel always has driver bugs so this is no surprise.

- WC3 suddenly asking to insert the game's original CD-ROM

This is a bug with a failure in the bootstrap loader. If any sort of error occurs it prints that message because it was intended for the failure of the copy protection on the disc. However the message is retained but the copy protection has been removed so it will still produce that message if another error occurs. I think this is mostly permission related so making the Warcraft III install folder user accessible fixes it.

...nor whatever other issue, on XP SP3

Like? I have been using non-XP for over 7 years, during which I did play quite a bit of WC3. Practically no issues showed up until Windows 10 and all those are solved.

One may create an ISO of a CD-ROM, install the game from this ISO by doing a minimum install if applicable, and then trim the ISO of any useless stuff (installation files, third party stuff, audio CD tracks eventually...). When this is done, one may convert the trimmed ISO (Mode 1) to the ISZ file format with minimum compression. This may reduce HDD footprint for pre-2000 CD-ROM games. I personally call these 'PlayCDs', because they do not allow to install the game anymore, yet allow to play it without a hassle (also, no need to hunt for any no-cd etc...).

Only works with some of the most basic copy protection forms. Others will seemly randomly sample parts of the CD data and so fail if this is done. Not like it matters seeing how on a £40 HDD you can fit ~2,000 such ISO files and USB 2.0 speeds are more than enough to play such games from an external HDD.

 

[Kyrbi0]

As for memory... I just load the entire 1 GB of WC3 into memory when I want to play it so I have ~0 I/O loading times. This takes <10 seconds thanks to sequential I/O on mechanical drives (~100 MB/sec) and gives me random access I/O rate faster than any SSD (SSDs still slower than memory). It also uses up 1/18 GB of my memory, which WC3 itself can only use up at most 2/18 GB before it crashes due to being 32 bit and likely not built to use extended 32 bit address range for a full 4 GB of virtual memory space.

Wait, how is that done? You basically load the entire game directly into RAM?

 

[Dr Super Good]

Wait, how is that done? You basically load the entire game directly into RAM?

I wrote a simple java program that touched every byte of all the MPQ files (most of WC3 data). It added every byte of the main MPQs into a sort of checksum to prove that it had iterated every byte, usually taking ~5-10 seconds to complete. The File Cache takes care of the rest. Use RAMMap to prove file residency in file cache (warning RAMMap itself uses a lot of memory so can eject pages from file cache). It is not perfect as not all files are pushed into RAM, however it pushes most of the data WC3 maps load (primary MPQs) which saves most of the time.

One can also prove the difference it makes as RAMMap allows one to purge the file cache. This will make WC3 take around as long to load as initially at system start (excluding DLLs). The difference is non-trivial for startup load. It also gets the data into filecache a lot faster than WC3 can as it pushes a sequential read rather than random access to individual files inside the MPQ. I recommend at least 2 GB of free memory for this as otherwise the file cache may be purged by other files and applications.

 

[pyf]

Wait, how is that done? You basically load the entire game directly into RAM?

@Kyrbi0 : with this imho:
http://www.hiveworkshop.com/posts/2137546/

 

[Dr Super Good]

@Kyrbi0 : with this imho:

That particular implementation worked by memory mapping the main MPQ files, touching every byte to load the pages into memory and calculate a checksum which is displayed (to prevent optimizers from potentially not touching every byte, could also be done with a volatile as optimizers cannot touch those but I wanted something I could confirm every byte was being touched), and then keeping the mapping open to raise retention priority (part of an active file mapping so hopefully less likely to be discarded than just being in the file cache).

In theory one could have made a directed implementation that only loaded the pages of the MPQs that WC3 will use, lowering the required file cache footprint. However chances are this will be slower as it is no longer a big file sequential read but a lot of small reads so might be subject to random access latency.

This is also entirely pointless with a SSD. Not only does the file cache behave differently with SSDs (less willing to write out changes, more willing to forget previously read pages), but with the average consumer grade SSD the difference between file cache memory speed and SSD random I/O speed is only 1-2 orders of magnitude, which is fast enough to be considered trivial compared with processing the content loaded. The only benefit of caching the MPQ archives in the file cache is to eliminate the bad random access I/O performance of mechanical drives reading various files inside the MPQ archives which can add 10+ seconds to loading a complex map. SSDs do not suffer from this as they are capable of random access I/O speeds in the order of 100 MB/sec or more with similar latency to sequential reads (could process all pages of all WC3 MPQ files in a completely random order in well under 10 seconds, similar time to sequentially reading them on a mechanical drive).

 

[Retera]

I worked for several days to get it as small as I did, I don't remember everything, but here was the general gist:

Spoiler: Retera's Process

1. Take contents of war3.mpq, war3x.mpq, war3xlocal.mpq, and war3patch.mpq and put them into 4 separate folders (make sure to have full listfile)

2. Copy these contents all into the same folder, so that when we are done we will just have 1 "game.mpq" which does not have repeat files
(Still not using MPQs right now)

3. Delete all those support files and root files just like pyf did (*.html, *.jpg, *.css, *.js)

4. Extract the contents of the Tileset-MPQs (A.mpq, B.mpq, etc.) also into a separate folder. Run all squisher scripts on these folders (see below)

5. Extract the contents of War3_low.mpq and War3x_low.mpq into and on top of the files in the main folder, in that order, to help models compress to slightly smaller size

6. Delete all TGA files that are not inside of PathTextures\ as other TGAs are literally wastes of space created by 3ds max export during Blizzard's work flow that they forgot to remove

7. Write a model squisher script akin to MdxSquisher that:

1. goes to every 32 bit floating point value in every MDX file in the game and bit masks it with 0xFFFF0000 so that the low-order bits of the mantissa are 0's, since nobody cares about those anyway
2. replaces all animation keyframes with Linear type frames so that they do not have InTans and OutTans and thus take 1/3 as much space
3. remove any keyframe that comes between two other, similar keyframes to further simplify animations

8. Run "blplabcl.exe", command line BLP Lab that Shadow Daemon gave me to build the matrix eater a while back on every texture in the game

• Tell it to compress to 50% or 25% quality.
• Create mipmaps for anything in "units", "abilities", "buildings", "textures", "environment", "doodads", "sharedmodels", "objects\inventoryitems", or "replaceabletextures\splats"
• Do not create mipmaps for other textures, to save space

9. Vaporize any file with ".mp3" or ".wav" extension because the goal was the smallest working copy of the game possible, and sound is not a requirement it's an extra feature

10. All those automated compressor scripts sometimes crashed, and logged which files they crashed on. Made another script to replace any file in the archive with the original source material from normal Warcraft on any case where a script crashed for a given file

11. Delete Campaign Screen and Menu Screen model files, and replace all blps in their subdirectories with 1x1 alpha black file, so that they take no space, but can still be used by models in custom maps (sometimes fan model authors use campaign tex to save space, you need to have the model load in custom maps or else units disappear)

12. Delete anything else I just felt like deleting that I probably wasn't going to use ("Melee_V0" folder deleted, "Maps" folder deleted, "Sound" folder deleted, "UI\SoundInfo" deleted, "UI\Captions" deleted, all "UI\Mac**" deleted because I'm on Windows)

13. Replace most of the contents of "ReplaceableTextures\Cliff" and "ReplaceableTextures\Water" with the 1x1 alpha black file, since any file in these folders with a prefix like "G_" in the name is not used by the game, and only by fanmade models.

14. Replace most of "ReplaceableTextures\Shadows" with the 1x1 alpha black file because who needs shadows, you just have to have "Shadow.blp" and "ShadowFlyer.blp" use the output of the compressor scripts and be valid files, the rest are buildings and trees and whatever [You can use Nirvana hacks to have real-time shadows anyway, don't need these probably, was my thinking]

15. Delete all the "arthasandillidanfight" stuff, but leave the 1x1 alpha black textures for fan models to still load

16. Delete everything in "Scripts\" except for the Magic 7 ("Blizzard.j", "common.j", "common.ai", "elf.ai", "orc.ai", "human.ai", "undead.ai")

• Preloader whatever is nice but the game runs without it, my target goal was minimum file size

17. Delete everything in "Fonts\" except for "FRIZQT_.TTF" because that's the only one you probably actually need, it's the English one and official language of THW is English I read somewhere I think

And...

Now you have a 92 MB mpq you can use, you can replace the others with table size 4 dummy empty files, and so you have < 100 MB for the main game files, then you just need maps and other stuff (Don't need war3patch.mpq because you can't patch this monster and the game runs without it)

See attachment. It's a proof of concept with a bunch of models and unit data, but you need to provide your own CD key to use it and stuff (obviously can't post a runnable one, pretty sure that's not legal or something)

 

[Chaosy]

Interesting.

Not useful for me personally but I can imagine it being decent for some.

Still, why not copy wc3 to an USB?

 

[Retera]

I had it on USBs for years, but they always seemed to break or get lost.
But, with that said, the potential to do so still exists, and in that case usually 1 GB is no problem.

 

[Kyrbi0]

Once I realized that it could be run from a flash drive (took an embarrassingly long time to figure out), I copy pasted the files over & never looked back.
Nowadays, with the issues my computer has with running Warcraft (permissions and all that), I pretty much work entirely off of that flash drive.

 

[pyf]

I worked for several days to get it as small as I did, I don't remember everything, but here was the general gist:
[...]
See attachment. It's a proof of concept with a bunch of models and unit data, but you need to provide your own CD key to use it and stuff (obviously can't post a runnable one, pretty sure that's not legal or something)

Thanks a lot @Retera.

Fyi, @LeP just updated his MPQ Compressor tool.
Very early testing on one stock Blizzard map, shows a very impressive (and unexpected tbh) compression ratio.

What do you think?