An error in the program. It must get itself into an infinite write loop.
I am unsure, but I think it is caused by the MPQ libraries used, and not the optimizer program itself. Similar bugs exist in many of the old MPQ editors which likely all used the same old base library.
Unlike the tools, the library itself has and still is being upgraded with such problems fixed. Unfortunately I do not think one can upgrade the library used by the tool without having the tool source code and re-building.
I recommend messing around with the various options available. Especially ones archive related and possible JASS related as those are the areas which the tool does the most work. I remember one of them helping stop the problem but forget which as it has been several years.
Do be aware that Vexorian Optimizer only supports "unprotected" maps. Using it on a protected map will very likely cause the behaviour described. Most map protectors damage the MPQ archive in some way which breaks old MPQ libraries badly.