Widget memory footprint

HappyTauren · Apr 1, 2016

Okay, so we all know that units, destructables and items leave some memory footprint even after we've destroyed them, nulled all pointers to them, et cetera, because that's just the way blizzard did it. So, how much memory is it, you ask?

Items: 0.2KB of memory per instance
Ordinary units: 0.2KB of memory per instance
Hero units: 0.5KB of memory per instance
Destuctables: 2.0KB of memory per instance

Wait. Did I just say that destructables are about as taxing to create/destroy and leave in memory, as 4 freaking heroes?

Yep, you heard it right. Destructables take 4x as much memory as heroes once destroyed.

So, does anybody have any idea why this would be so? I had a system that used a ton of dynamic destructables but I now want to find a different solution because this is just ridiculous.

Marine · Apr 1, 2016

Though I can't say exactly why destructibles are so leaky, I can say definitively that they are very poorly optimized compared to other entities in wc3. For example, they're one of the few things (besides units) that are actually rendered even when they're not viewed. Considering there is typically several hundred of them on the map at the same time it puts a lot of strain on the system. Add to that the Walkability and pathing of destructibles and it is a mess. So I'd hazard a guess and say that Blizzard skimped on optimizing destructibles altogether, including post-removal memory.

HappyTauren · Apr 1, 2016

Units and destructables are not rendered if their center (and a bit around that) is not viewed, I don't know what you're talking about.

BPower · Apr 1, 2016

Yes units are not rendered if not beeing visible on the screen.
Doodads are not rendered if their center, not sure if it's always the center, of the model isn't on the screen.

Destructables however are always rendered.
That's why Zwiebelchen made DestructableHider.
Pretty cool system if you use a lot of destructables in your map.

IcemanBo · Apr 1, 2016

I was pretty sure former tests had shown a unit perma-leak in scale of 0.04 kb.
Does someone maybe know more about it's execution or even it's original thread? We could compare.

HappyTauren · Apr 1, 2016

BPower said:
Yes units are not rendered if not beeing visible on the screen.
Doodads are not rendered if their center, not sure if it's always the center, of the model isn't on the screen.

Destructables however are always rendered.
That's why Zwiebelchen made DestructableHider.
Pretty cool system if you use a lot of destructables in your map.

My experience suggests following things:
1) Units are not displayed if some range around their center is not displayed
2) The same goes for destructables, but like units have overhead from pathfinding and so on, destructables don't actaully show, instead they ARE ALWAYS CHECKING WHETHER THEY SHOULD BE SHOWN, so this causes overhead instead of them actually rendering
3) Doodads actually use model extents, so if model extents are within range, it's displayed

I have never witnessed this "destructables are always rendered" thing, in fact, I've changed tall destructables for doodads because they DIDN'T render when I needed them to, but it's obvious they cause some other kind of overhead, as Zwiebelchen has documented.

^IcemanBo: This is a pretty rough estimate, the unit I tested was footman, it may be that the more data you add to the unit the more memory it takes, that's why a hero unit takes so much more memory.

Zwiebelchen · Apr 2, 2016

Have you monitored this with a script that slowly generates more widgets and then cleans them up again?

Because I noticed memory consumption in WC3 to be somewhat difficult to track. WC3 has the tendency to spiral upwards in memory until hitting a certain point, then suddenly falls back to a lower level again. It seems that there is some sort of garbage collection going on behind the scenes that cleans up these memory footprints, but it is only fired after a certain memory treshold is reached.

But yes, destructables are taxing as fuck, be it in pure ingame performance or in memory consumption. I think this is because of the possibility of being walkable.

HappyTauren · Apr 3, 2016

Oh, right! If destructables always include a pathing matrix of some sort, no wonder they take up so much memory!

Haven't thought of that, thanks!
BTW, I have a question, why do people still use Pathing Blockers as destructables, when they're obviously better as doodads unless you need them dynamic?

PurgeandFire · Apr 3, 2016

We usually just use whatever Blizz gives us. I don't think too many people associate destructables with being problematic. But making it into doodads is a pretty neat idea.

I had an idea for a tool a while back to convert all pathing blockers (or just the ones the user picks) to static pathing (each map has a pathmap, wpm, that defines the pathing of the entire map). I wonder if it would help performance at all. I never had time to check. It would probably shed a few bytes off your map to boot.

HappyTauren · Apr 3, 2016

We usually just use whatever Blizz gives us. I don't think too many people associate destructables with being problematic.

I am pretty sure quite a number of maps did use doodad pathing blockers back in the day when destructible limits were hard to break.

I wonder if it would help performance at all.

Doodads' pathing is already compiled into the pathing file, but it'd be a good idea to remove unneeded doodads anyway, though it'd still have a minimal effect on performance as most CPU/GPU use from Doodads comes from those that have complex geometry and huge bounds.

Basically, Doodads really are just geometry performance-wise, the pathing that's calculated of them is done at map compilation, not runtime, so if they have no geometry, then they're almost irrelevant.

Zwiebelchen · Apr 3, 2016

PurgeandFire said:
We usually just use whatever Blizz gives us. I don't think too many people associate destructables with being problematic. But making it into doodads is a pretty neat idea.

That's the main problem there. People just don't think about it.

I made that choice on purpose, for the simple reason that I wrote a script that removes all those pathing blockers placed at map init and replaces them with a SetTerrainPathable call. That way I don't have to increase my doodad count but also get the correct pathmap and very little extra overhead.

But looks like I was wrong with that, if the memory footprint is that high.

The question is: what consumes more resources, a removed destructable or a "living" doodad?

...
And let's face it: even with 2000 pathing blockers as destructable, after removing all of them, the total footprint would be a mere 4 Megabyte. Hardly something that will make a huge difference in the game.

Anyway, removing destructable pathing blockers and replacing them with SetTerrainPathable is a must have if you use pathing blockers based on destructables. I can probably share the script I used on that if people are interested. The existing one here on hive has a bug as the creator assumed that the smallest possible cell in WC3 is 64x64, when it's actually 32x32.

I had an idea for a tool a while back to convert all pathing blockers (or just the ones the user picks) to static pathing (each map has a pathmap, wpm, that defines the pathing of the entire map). I wonder if it would help performance at all. I never had time to check. It would probably shed a few bytes off your map to boot.

If you want to make such a tool (possibly also with a user-defined registry), that could probably help a lot of people (including myself), as I noticed that destructable pathing blockers have some nasty side effects on the pathing algorithm aswell (for example, SetItemX() and Y will place the item on destructable blocked terrain, but not on doodad blocked terrain.

Dr Super Good · Apr 4, 2016

For example, they're one of the few things (besides units) that are actually rendered even when they're not viewed.

I am pretty sure they are not rendered when out of view otherwise even melee maps would not be playable at the time WC3 was released.

Anyway, removing destructable pathing blockers and replacing them with SetTerrainPathable is a must have if you use pathing blockers based on destructables. I can probably share the script I used on that if people are interested. The existing one here on hive has a bug as the creator assumed that the smallest possible cell in WC3 is 64x64, when it's actually 32x32.

If WE was not so terrible you could paint your pathing directly into the pathing map file. No destructibles and no JASS needed.

People need to keep in mind that the smallest unit of memory allocation is a page and it is often done many pages at a time to reduce OS calls. These pages make up the allocated virtual memory address space of an application. Single pages, or the entire unit which was allocated, cannot be freed if any part still contains useful or referenced data. Hence it is very easy for allocated application virtual address space to expand but much more difficult for it to shrink. Eventually it should reach an equilibrium where by the number of pages allocated has sufficient free space in them that no more pages need to be allocated, and hence memory usage stops growing. Another approach is the garbage collection approach where in the presence of a handle reference scheme you can compact data hence allowing pages to be freed.

WaterKnight · Apr 5, 2016

If you want to paint pathing in WE (terrain pathing), use doodads or GUI rects + one of my postproc tools converting them.

@memory: Sounds freaking expensive, destructables are used for elevated effects, too.

Wareditor · Apr 5, 2016

Zwiebelchen said:
Anyway, removing destructable pathing blockers and replacing them with SetTerrainPathable is a must have if you use pathing blockers based on destructables. I can probably share the script I used on that if people are interested. The existing one here on hive has a bug as the creator assumed that the smallest possible cell in WC3 is 64x64, when it's actually 32x32.

It would be nice if you do share it

Dr Super Good · Apr 5, 2016

This is the sort of thing that people at Blizzard need to look into with patches, and disclose the findings. There is too much speculation and ambiguity for us to say for sure.

Just because virtual address space is increasing does not mean there are any leaks. A leak only occurs if the address space is never recycled or freed after it is doing being used.

Zwiebelchen · Apr 6, 2016

Wareditor said:
It would be nice if you do share it

Sure, here you go:

JASS:

library APB initializer init

globals
    private constant integer UNWALKABLE = 'YTpb'
    private constant integer UNWALKABLE_LARGE = 'YTpc'
    private constant integer UNWALKABLE_HORIZONTAL = 'B007'
    private constant integer UNWALKABLE_VERTICAL = 'B008'
    private constant integer UNWALKABLE_DIAGONAL1 = 'B006'
    private constant integer UNWALKABLE_DIAGONAL2 = 'B009'
endglobals

private function Replace takes nothing returns boolean
    local destructable d = GetFilterDestructable()
    local integer i = GetDestructableTypeId(d)
    local real x
    local real y
    if i == UNWALKABLE then
        set x = GetWidgetX(d)
        set y = GetWidgetY(d)
        call SetTerrainPathable(x-16, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x-16, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x+16, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x+16, y-16, PATHING_TYPE_WALKABILITY, false)
        call RemoveDestructable(d)
    elseif i == UNWALKABLE_LARGE then
        set x = GetWidgetX(d)
        set y = GetWidgetY(d)
        call SetTerrainPathable(x -16, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -48, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -48, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +48, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +48, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y-48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y+48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y-48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y+48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -48, y-48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -48, y+48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +48, y-48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +48, y+48, PATHING_TYPE_WALKABILITY, false)
        call RemoveDestructable(d)
    elseif i == UNWALKABLE_HORIZONTAL then
        set x = GetWidgetX(d)
        set y = GetWidgetY(d)
        call SetTerrainPathable(x -16, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -48, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -48, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +48, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +48, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -80, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -80, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +80, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +80, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -112, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -112, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +112, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +112, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -144, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -144, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +144, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +144, y+16, PATHING_TYPE_WALKABILITY, false)
        call RemoveDestructable(d)
    elseif i == UNWALKABLE_VERTICAL then
        set x = GetWidgetX(d)
        set y = GetWidgetY(d)
        call SetTerrainPathable(x -16, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y-48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y+48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y-48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y+48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y-80, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y+80, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y-80, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y+80, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y-112, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y+112, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y-112, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y+112, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y-144, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y+144, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y-144, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y+144, PATHING_TYPE_WALKABILITY, false)
        call RemoveDestructable(d)
    elseif i == UNWALKABLE_DIAGONAL1 then
        set x = GetWidgetX(d)
        set y = GetWidgetY(d)
        call SetTerrainPathable(x -16, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -48, y-48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -48, y-80, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -80, y-48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -80, y-80, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -112, y-112, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -112, y-144, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -144, y-112, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -144, y-144, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +48, y+48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +48, y+80, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +80, y+48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +80, y+80, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +112, y+112, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +112, y+144, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +144, y+112, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +144, y+144, PATHING_TYPE_WALKABILITY, false)
        call RemoveDestructable(d)
    elseif i == UNWALKABLE_DIAGONAL2 then
        set x = GetWidgetX(d)
        set y = GetWidgetY(d)
        call SetTerrainPathable(x -16, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -16, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y-16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +16, y+16, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -48, y+48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -48, y+80, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -80, y+48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -80, y+80, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -112, y+112, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -112, y+144, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -144, y+112, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x -144, y+144, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +48, y-48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +48, y-80, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +80, y-48, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +80, y-80, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +112, y-112, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +112, y-144, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +144, y-112, PATHING_TYPE_WALKABILITY, false)
        call SetTerrainPathable(x +144, y-144, PATHING_TYPE_WALKABILITY, false)
        call RemoveDestructable(d)
    endif
    set d = null
    return false
endfunction


private function init takes nothing returns nothing
    local rect r = GetWorldBounds()
    local integer i = 0
    call EnumDestructablesInRect(r, Filter(function Replace), null)
    call DestroyBoolExpr(Filter(function Replace))
    call RemoveRect(r)
    set r = null
    set initcheckerbool2 = true
endfunction

endlibrary

As you can see; I added four custom pathing blockers that aren't a default in WC3. Those use the pathing textures of the fence doodads (vertical, horizontal and two diagonal versions) to save the terrainer some time. The pathing textures for these are:
PathTextures\StoneWall1Path.tga
PathTextures\StoneWall2Path.tga
PathTextures\StoneWall3Path.tga
PathTextures\StoneWall4Path.tga

Widget memory footprint

Similar threads