I used to do all my early models in vertex modify too, and i remember that the overflow issue usually was connected to something special you did, like extruding a surface that lacked a side (i don't remember what it was, really).
Another problem i know occur oftenly is that the extrude tool sometimes misses a polygon, creating an ugly hole in the mesh; as DonDustin said, the tool is very, very bugged.
@Atmosferic: changing language settings or decimal symbol is only if the program won't start at all; since the program won't start with the wrong decimal symbol, it can't be related to this error.
Vertex modify is a very primitive and buggy tool, but it can actually be a very useful free alternative for making simple, static models. What you have to remember though is to save the model often, and also oftenly take a look at it in magos every now and then to make sure the mesh is not messed up. You can also make models in several steps, and connect them as geosets, and finally merge them together. This can help in allowing you to UV objects from several angles. Just remember to merge the geosets, since too many of them can cause graphical bugs in warcraft.
I've done so many models in that program, and after some frustration, they usually turn out good. Most models i did for my halo map, like the wall doodads and the antenna thing i had in the screenshot for my flood model, not to mention the needler and the spiker, are made in VM. I've never tried MDLVis though, perhaps it's better.
