Data Space for map in Game

[Dark-Zalor]

Hello Everyone.

I'm working on my map Pokemon World.

I added 550 Pokemons, 18 types, 450 attacks.

All this is stored with thousands of structures and links between them by reference. All this data is useful for a lot of actions in the map. So it is stored in the ram.
And the game still load all that.

1/ My question is what space does it takes as a max before the game can't store more data?

I read somewhere that the number of hashtable is limited.
But is there a limitation for variables. Or number s of structs. (Due to the fact the game is managing an array of variables behind)
Or it depends only on the computer that runs the map.

2/ what are the good practice about strings for example? Especially icon paths. Because they are pretty long.
I created a lot of variable with the complete path of icons.
Is it better to store the complete name? Or store the suffix "BTN_MyIcon.blp" for example and use a function that concatenante the prefix? Is the concatenation less difficult?

What are the good practice for that?
It's just a thread to discuss about that. Share practices or feelings about it.
My computer run it but it should be annoying if other players have an experience ruined because their computer is less powerful.

 

[PurgeandFire]

For #1, I mentioned a few details for structs specifically in another thread: Nbr Instance Structs Allocated MAX

The limit for the number of instances of a single struct is 8190, which gets reduced if you use array variables in the struct. But there really isn't a definitive limit on the number of different structs you can have. You should be fine in your case with thousands or even tens of thousands of instances of data. As far as I know, there really isn't a limit until your map starts taking up too much RAM (which for modern wc3, depends on your computer).

Keep in mind that the variables you store (which are simple pointers) typically have a much smaller memory footprint compared to actual game objects. That's why for memory usage, we typically focus our attention towards memory leaks where game objects are not properly destroyed when they are no longer needed. That will give you the most bang-for-your-buck when optimizing your map.

Memory Leaks

C0Memory leaks For a quick lookup on most important actions, read Things That Leak. Introduction Object Leaks Reference Leaks Miscellaneous Conclusion C1Introduction If your computer's memory keeps occupied with stuff you already lost access to, it's called memory leak. It slows down the...

www.hiveworkshop.com

Pay special attention to any of your periodic triggers, as those are the ones that are usually the biggest culprit. If you have a leak in a periodic trigger that runs every 0.03 seconds, that can eat up a lot of memory very quickly. The one-off leaks that happen for a trigger that fires once usually won't make a huge difference in the end (but it is still a good practice to clean them up wherever you can!).

For #2, it depends on what you're trying to optimize--but since this topic is about memory, I would recommend just using the complete paths. Each time you use a unique string during a map's session (in-game), it'll create an entry into an in-memory string table for caching purposes. So if you use full paths, you'll get entries like this:

"ReplaceableTextures\\CommandButtons\\BTN_FirstIcon.blp"
"ReplaceableTextures\\CommandButtons\\BTN_SecondIcon.blp"
"ReplaceableTextures\\CommandButtons\\BTN_ThirdIcon.blp"
...

If you use concatenation, e.g. "ReplaceableTextures\\CommandButtons" + myIconName, it'll end up generating entries for each fragment like this:

"ReplaceableTextures\\CommandButtons"
"BTN_FirstIcon.blp"
"ReplaceableTextures\\CommandButtons\\BTN_FirstIcon.blp"
"ReplaceableTextures\\CommandButtons\\BTN_SecondIcon.blp"
"BTN_SecondIcon.blp"
"ReplaceableTextures\\CommandButtons\\BTN_ThirdIcon.blp"
"BTN_ThirdIcon.blp"
...

The full paths would technically be more memory efficient since they'd create fewer entries into the string table. But it honestly won't make a big difference since these sort of things are static.

I'd put most of your attention towards the dynamic things in your game, like any potential memory leaks. You can use Windows task manager to look at how much memory Warcraft 3 is taking up when running your map. If you notice it continually spiking up significantly after playing it for a long time, chances are you'll want to take a look at your code and look for any leaks!

 

[Nichilus]

The limit for the number of instances of a single struct is 8190

I don't know how it is for vjass structs, but standard jass arrays had the limit increased to ~32.760 instances since at least v1.31.

 

[PurgeandFire]

I don't know how it is for vjass structs, but standard jass arrays had the limit increased to ~32.760 instances since at least v1.31.

Yeah sadly vJASS structs have still kept the same instance limit (8190) in reforged since it was hardcoded into the jasshelper compiler.

 

[Dark-Zalor]

Thanks for your very complete answer @PurgeandFire

I had a strange cas with functions.
I had a trigger launched after a timout (0.02 in my case) that trigger inside that trigger it calls thousand, and thousand of functions. Not in the same one.
And the game stop the thread.
No errors just it stopped....

I put multiple call of function on a timer to call them over the time and it finally worked. I didn't know too many calls of function would stop a thread.

3/ question about triggers and events.
Is there a limit of triggers withe the same event that can exist simultaneously?
And of course launched when the event occurs?

Thanks for your complete answers guys

 

[Dark-Zalor]

Hi,
I had other questions about my data structure used like a DataBase

struct Poke
  static thistype array P
  static integer PT = 0

  string name
  integer id
  PokeType pType    // other struct with type for example (water, fire) that has same structure

  private method destroy

  static method getById takes integer id returns thistype

  static method create takes string name, integer id returns thistype
endstruct

Here are the schema of my data structs. Static method to get them.
And they are referenced together (between resources)

The differents structures are initiated with the code below.

function initPokes takes nothing returns nothing
  call Poke.create("Pikachu", 'h001') // This call duplicated 550 times for each Pokémon with given data
  ...
endfunction

Theses function are called at the beginig of the map. In the trigger that starts after 0.02 seconds of game time.

The question is pretty vague.
1/ Is it a good way to create a Data Base like? with array of structs like above?
2/ the function before, is it the most efficient way to fill it? (huge amount of function calls). The management of data in each call is not a problem.

All my map uses that kind of database, it's pretty fast (instant in game) but if you have suggestions to create a system less heavy, faster, or whatever...
I'm interesting in your advises or suggestions.

 

[PurgeandFire]

Thanks for your very complete answer @PurgeandFire

I had a strange cas with functions.
I had a trigger launched after a timout (0.02 in my case) that trigger inside that trigger it calls thousand, and thousand of functions. Not in the same one.
And the game stop the thread.
No errors just it stopped....

I put multiple call of function on a timer to call them over the time and it finally worked. I didn't know too many calls of function would stop a thread.

Ah yeah, you likely ran into the operation limit (or "op limit"). Wc3 puts an operation limit in place (roughly 300000 "operations"), likely to prevent any faulty code from looping indefinitely. But sometimes that limit can cause issues for map devs if you run a lot of operations on a single "thread".

There isn't really a way to "check" how many operations you've ran so far, but generally most people will do what you did--if you notice you're running a LOT of code for something, it may be worth splitting the work across multiple threads.

The reason the timer worked for you is because when you do TimerStart, the callback will be executed on a fresh "thread" (so it'll start at 0 operations, and you can do up to 300000 new operations).

But it is also possible to run a function on a new thread without a timer (in case you want it to all happen instantly), for example:

TriggerExecute

TriggerEvaluate

vJASS Execute

ExecuteFunc

globals
    trigger TriggerB = CreateTrigger()
endglobals

function A takes nothing returns nothing
    call TriggerExecute(TriggerB) // this will run function B on a new thread
endfunction

function B takes nothing returns nothing
    // do heavy work
endfunction

function Init takes nothing returns nothing
    call TriggerAddAction(TriggerB, function B)   
endfunction

globals
    trigger TriggerB = CreateTrigger()
endglobals

function A takes nothing returns nothing
    call TriggerEvaluate(TriggerB) // this will run function B on a new thread
endfunction

function B takes nothing returns boolean // must return boolean for use as a `Condition`
    // do heavy work
    return false
endfunction

function Init takes nothing returns nothing
    call TriggerAddCondition(TriggerB, Condition(function B))   
endfunction

function A takes nothing returns nothing
    call B.execute() // this will run B on a new thread via a trigger internally
endfunction

function B takes nothing returns nothing
    // do heavy work   
endfunction

function A takes nothing returns nothing
    call ExecuteFunc("B") // this will run B on a new thread
endfunction

function B takes nothing returns nothing
    // do heavy work   
endfunction

The TriggerEvaluate method is technically the fastest, but it requires you to format your function so that it returns a boolean. I personally recommend using the vJASS built-in .execute(), because that generates a trigger under-the-hood and makes your code a bit easier to maintain compared to the other solutions. It also supports struct methods easily. I don't personally recommend ExecuteFunc, as that is the slowest (it has to search for the function by name) and you need to use special prefixes to be able to use it with private/public vJASS methods.

3/ question about triggers and events.
Is there a limit of triggers withe the same event that can exist simultaneously?
And of course launched when the event occurs?

Thanks for your complete answers guys

No, there is no practical limit to the number of triggers with the same event. And there isn't a practical limit to how many times the trigger can run either--I remember running a test map that fired an event like 5000 times a second and it was still running just fine even after being called over a million times. Each trigger execution gets its own fresh thread, so it firing multiple times doesn't really matter when it comes to the op-limit.

The main thing to just consider is that all your scope/InitTrig initializers will be run on the same thread, so you generally don't want to do a ton of work in your initializers. If you do plan on doing a bunch of work, I recommend using the techniques mentioned above to do the heavy work on a new thread (like with .execute()) or using library or struct initializers--since those each get run on a fresh thread.

Hi,
I had other questions about my data structure used like a DataBase
...
Here are the schema of my data structs. Static method to get them.
And they are referenced together (between resources)

The differents structures are initiated with the code below.

function initPokes takes nothing returns nothing
  call Poke.create("Pikachu", 'h001') // This call duplicated 550 times for each Pokémon with given data
  ...
endfunction

Theses function are called at the beginig of the map. In the trigger that starts after 0.02 seconds of game time.

The question is pretty vague.
1/ Is it a good way to create a Data Base like? with array of structs like above?
2/ the function before, is it the most efficient way to fill it? (huge amount of function calls). The management of data in each call is not a problem.

All my map uses that kind of database, it's pretty fast (instant in game) but if you have suggestions to create a system less heavy, faster, or whatever...
I'm interesting in your advises or suggestions.

For #1, yes, I think that is a great way to make a database for your game. You may want to split up the work across the threads (as mentioned above) if you do a ton of stuff, but otherwise that is very efficient.

If you don't plan on ever destroying the Poke instances, then you can technically make a custom initializer that just increments a counter for use as the instance ID:

struct Poke extends array // "extends array" will remove the default allocator/deallocator
    static integer instanceCount = 0

    static method allocate takes nothing returns thistype
        set thistype.instanceCount = thistype.instanceCount + 1
        return thistype.instanceCount
    endmethod

    static method create takes string name, integer id returns thistype
        local thistype this = thistype.allocate()
        // ...
        return this
    endmethod
endstruct

It doesn't make a huge difference though performance-wise, so it isn't super necessary. Just wanted to point that out since you mentioned you were running into op-limits.

The other main thing I'd recommend is just making sure the way you access it is a fast, constant look-up. For example, if you wanted to find the Poke data for "Pikachu", it would be pretty inefficient to loop through up to 550 pokemon to find him. Instead, I recommend using a hashtable so you can quickly look-up the pokemon by name or unit-id:

struct Poke
    private static hashtable ht = InitHashtable()

    string name
    integer id

    static method getByName takes string name returns thistype
        return LoadInteger(ht, 0, StringHash(name))
    endmethod

    static method getById takes integer id returns thistype
        return LoadInteger(ht, 1, id)
    endmethod

    static method create takes string name, integer id returns thistype
        local thistype this = thistype.allocate()
        set this.name = name
        set this.id = id
        call SaveInteger(ht, 0, StringHash(name), this) // save "this" instance under (0, name)
        call SaveInteger(ht, 1, id, this) // save "this" instance under (1, id)
        return this
    endmethod
endstruct

For #2, I think the way you're filling it is just fine. It is tedious, but almost all in-game databases are about doing a lot of tedious stuff up-front in one place so you can access them quickly later. So I just recommend splitting the work across multiple threads and you'll be good to go!

 

[Dark-Zalor]

Wow thanks a lot for your very complete answer.
It's really interesting

I have an other question quite noobish

is it better to use during boolean comparison

not(myCondition())

myCondition() == false

I think the second one is more optimized, because they both check the value, but the first one call the 'not' function in addition

or we don't care it's only a syntax

 

[PurgeandFire]

I have an other question quite noobish

is it better to use during boolean comparison

not(myCondition())

myCondition() == false

I think the second one is more optimized, because they both check the value, but the first one call the 'not' function in addition

or we don't care it's only a syntax

It is just a syntax difference. not isn't actually a function--it is just an operator (similar to ! in other programming languages). So you won't really notice a speed difference. I don't think anyone has benchmarked it but usually those sorts of optimizations aren't going to really make a difference beyond a few nanoseconds.

So in your case, I recommend using whichever one feels more readable to you!

 

[Dark-Zalor]

That's really great to talk about code with someone that knows it, like you

My 2 last questions I promise

1/ I have a question with threads due to the facts that some of my structs are referencing each other
these ones should be in the same thread right?

2/ How many threads can be created and running at the same time?
Is there a limit? Each trigger that runs get a new thread? (By trigger right? not by time it has been triggered)

If I have:
Trigger A
Trigger B

if trigger A is triggered 5 times it is managed by 2 threads, not by 5 + 1 => 6 threads
am I right?

 

[PurgeandFire]

That's really great to talk about code with someone that knows it, like you

My 2 last questions I promise

1/ I have a question with threads due to the facts that some of my structs are referencing each other
these ones should be in the same thread right?

2/ How many threads can be created and running at the same time?
Is there a limit? Each trigger that runs get a new thread? (By trigger right? not by time it has been triggered)

If I have:
Trigger A
Trigger B

if trigger A is triggered 5 times it is managed by 2 threads, not by 5 + 1 => 6 threads
am I right?

Np, feel free to ask as many as you'd like.

For #1, it depends. With Warcraft 3, we just have virtual threads (very different from operating system threads), so structs and data don't really "belong" to a single thread. So it really just depends on when and where you're accessing the data. For example, if I'm accessing those structs in one trigger--that might use thread A. If I access it in another trigger, that might use a different thread B.

For #2, I don't think any map makers ran into a limit with the number of threads. Just note that Warcraft 3's virtual threads aren't truly "running" at the same time (scripts in Warcraft 3 run on a single-thread in reality)--instead you can think of each thread as its own "task" of work to complete with its own op-limit. Once it finishes running all its work or reaches a suspension point (like TriggerSleepAction), it'll switch to any other threads that need to run and process those.

So in your example, if Trigger A is triggered 5 times--that means that the trigger's action function would get run with 5 threads (a new one each time). We don't really know if wc3 recycles threads internally or not, but it shouldn't really matter for our purposes conceptually.

 

[Dark-Zalor]

Hi,
Ok It's clear now

I have a few structs that does not extends array.
Does it needs to be changed to improve the performance of the map?
Maybe because theses strucs are lighter....

 

[PurgeandFire]

Hi,
Ok It's clear now

I have a few structs that does not extends array.
Does it needs to be changed to improve the performance of the map?
Maybe because theses strucs are lighter....

Honestly it probably won't make much of a difference converting the structs to extend array. That removes the allocation/deallocation functionality (and then you can put in your own)--but I wouldn't expect that to make any noticeable difference in your map. I mostly recommended that for the case where you're running into the op-limit--but that is separate from "performance".

I assume with performance you mostly mean your map's performance--like FPS, stutters, etc. For those, graphics are usually the heaviest (especially particle effects). For scripts, you'll usually want to focus on the things that run periodically first (e.g. every 0.03 seconds)--but only if you notice that they are actually causing a problem.

Premature optimization is often the devil of programming--a lot of programmers will look to their scripts and try to spend weeks to optimize them to save a frame or two, when in reality the thing eating up all the FPS is a super high-poly model with 4k textures, lol. That isn't to say that script optimization is bad, but you just want to be strategic about it--otherwise you'll waste a lot of time in the wrong areas.

 

[Dark-Zalor]

Ok thanks for the good advises.
I'm working on memory leaks.

Thanks a lot @PurgeandFire

 

[Dark-Zalor]

The other main thing I'd recommend is just making sure the way you access it is a fast, constant look-up. For example, if you wanted to find the Poke data for "Pikachu", it would be pretty inefficient to loop through up to 550 pokemon to find him. Instead, I recommend using a hashtable so you can quickly look-up the pokemon by name or unit-id:

I did the hashtable to index my structs

It's really fast and cool
Thanks a lot

 

[Dark-Zalor]

Hi,
I saw you wrote the SaveInteger function like this:

call SaveInteger(ht, 0, StringHash(name), this)

I fill my hashtable using that syntax

call SaveInteger(ht, StringHash(name), 0, this)

Is it the same impact for the game than what you wrote?
The hashtable init just 1 row in your case? is it right? And mine multiple rows with only 1 entry?

How is managed a hashtable? Is it an array of array?
Or table of map (key, value)?

If in a row (parent key) I save a value at the key 5 for example does it create an array of 6 elements, or just the key [5] of the map will be created?

Thanks for your knowledge share mate

 

[PurgeandFire]

How is managed a hashtable? Is it an array of array?
Or table of map (key, value)?

If in a row (parent key) I save a value at the key 5 for example does it create an array of 6 elements, or just the key [5] of the map will be created?

Thanks for your knowledge share mate

Your way of saving it is 100% fine. It shouldn't make a noticeable difference. I'm not sure about the exact implementation under-the-hood, but it is definitely some form of a key-value map (or possibly multiple maps). But it is definitely not an array, so you don't have to worry about which key you choose.

The main reason to use a parent key is if you intend to flush a "group" of values later on via FlushChildHashtable. (flushing just clears out the table) Otherwise, either one is fine depending on whatever organization system you prefer!