1. Updated Resource Submission Rules: All model & skin resource submissions must now include an in-game screenshot. This is to help speed up the moderation process and to show how the model and/or texture looks like from the in-game camera.
    Dismiss Notice
  2. The 15th Mini-Mapping Contest came to an end. The Secrets of Warcraft 3 are soon to be revealed! Come and vote in the public poll for your favorite maps.
    Dismiss Notice
  3. The 12th incarnation of the Music Contest is LIVE! The theme is Synthwave. Knight Rider needs a song to listen to on his journey. You should definitely have some fun with this theme!
    Dismiss Notice
  4. Join other hivers in a friendly concept-art contest. The contestants have to create a genie coming out of its container. We wish you the best of luck!
    Dismiss Notice
  5. Check out the Staff job openings thread.
    Dismiss Notice
Dismiss Notice
60,000 passwords have been reset on July 8, 2019. If you cannot login, read this.

How to reduce loading times of Object Editor data

Discussion in 'General Mapping Tutorials' started by Xonok, May 15, 2013.

  1. Xonok

    Xonok

    Joined:
    Mar 27, 2012
    Messages:
    3,042
    Resources:
    8
    Spells:
    3
    Tutorials:
    5
    Resources:
    8
    One of the main things that makes big maps lag is inefficient use of the object editor. This tutorial is meant to bring out some things that can be used to prevent that. This tutorial is not limited to object editor stuff, but it somewhat focuses on it.

    Purple Fields
    Theory
    Whenever you change something in the object editor it is written down. The base values are all unchangeable. What the editor really does is create a new unit and write down how it is different from the base unit.
    Purple color marks the fields that have been changed and thus, are written into the map file.

    "So what should I do?"
    Whatever you do, one thing is for certain. You can only avoid purple fields if you are making a melee map.
    What you can do, however, is use the purple fields wisely.

    Common mistake 1:
    Picture
    Too many levels.JPG

    Making an ability with 100 levels can be all fine and dandy, but lets imagine that each ability level takes 0.05 seconds to load. Quite little time, isn't it? But now suddenly your map loads 5 seconds longer from just 1 ability.
    This is not a real life example, but it can be used to show the difference.
    This is all because in any practical way if you make an ability with lots of levels, then it will have many purple fields.
    Note that number values are very cheap, as seen in Benchmarks.

    Now imagine what would happen if you made an ability with 10 000 levels? You can try it out, but be warned that it will lag the editor(and increase loading times).
    Ask yourself these questions
    Does each level really make a considerable difference ingame?
    What is the lowest amount of levels that this ability must have.

    I used to use 100-level abilities for making units gain combat experience. So that a unit with critical strike ability would have a greater chance to crit when it actually has been in a fight.
    The solution to this is coded criticals.
    Note that this is not an easy solution(aww, triggers...), but it saves huge amounts of power.
    What to do
    • Get a damage detection system
    • Get a unit indexer
    • Create a variable array for the critical chance of units(preferrably integer, because integers are less expensive than reals. Not that it matters that much. )
    • Every time a unit deals damage check if it is allowed to have critical strikes(if it has ability). If it is, then add some amount to critical hit chance

    This can be applied to any stat. I just used critical hits as an example. Due to numbers being cheap you do not need such system for criticals, you only need to avoid having lots of text fields(only create text for the first field).
    Changing stats such as health can be tricky, that's why you should combine the previous approach with a Bonusmod system.

    If you use a custom editor such as JNGPE, then you can even make hero units out of non-heroes and the other way around, without using purple fields. On highly customized heroes this might even be worth it if you know a non-hero unit similar to your hero idea.
    *When one of the text fields is empty, then the editor uses the text from the previous field, if the previous is also empty, then it goes one more step back, and so on.
    *Always keep brush list disabled. It hasn't got anything to do with loading times, but it makes mapping much faster.


    Widgetizing

    Widgetizing is not a mistake, but a way of optimizing the map. Essentially it makes the map use different base files(slk/txt), which simply load a lot faster due to being in a different format.
    One of the limitations of widgetizing is that abilities are limited to 3 levels(no room in the tables).
    In a sense the widgetizer fixes one of the mistakes that blizzard made.
    Widgetizer
    Widgetizing has been told to decrease loading times by up to 80%, but increases map size.
    However, note that widgetizing can cause crashes when you play a non-widgetized map right after playing a widgetized map.


    Initialization
    A lot of the loading time can be taken up by various things that happen at initialization. Some of those things can be done at other times though.
    How it's typically done:
    • Trigger
      • Events
        • Map initialization
      • Conditions
      • Actions
        • -------- Doing stuff here --------

    How it should be done:
    • Trigger
      • Events
        • Time - Elapsed game time is 0.01 seconds
      • Conditions
      • Actions
        • -------- Doing stuff here --------

    Doing so pushes the actions of that trigger to happen after the map has loaded, resulting in a lag spike right when players have finished loading, but reduces loading times. By my experience this lag spike feels shorter than the loading time that is avoided with this method, but that's mostly an illusion.

    Preloading

    Typically the game loads things only when it needs to. Things can be loaded before they are really needed by placing them on the map or though a special JASS function.
    This example preloads the wisp model:
    Code (vJASS):
    call Preload( "units\\nightelf\\Wisp\\Wisp.mdx" )

    The Preload function is typically used in melee games to prevent lag when a new unit type appears or some spell is cast for the first time.
    Anything that happens during map initialization, including creation of preplaced units, makes the loading time longer, so it's better to create things after the map has loaded.

    Models are loaded when they first appear.
    Abilities are loaded the first time they are given to a unit.

    This means that if you have an ability with 9999999999999999999999 levels and there's a preplaced unit that has it, it will make the loading time very long.
    However, if you create the unit just 0.01 seconds after loading, it instead creates a huge lag spike after the game starts.

    When you have a map with lots of abilities it's reasonable to preload everything over a long time so players won't notice any lag.
    Preloading is useful because it lets you prevent random lag spikes by making them happen at a specific time.

    Benchmarks

    Testmap with 1 custom critical strike(neutral)
    Test1: No purple fields - 4 seconds
    Test2: 101 purple fields - 4 seconds
    Test3: 1001 purple fields - 4 seconds
    Test4: 3001 purple fields - 4 seconds
    On each of those tests there was 1 integer field and the rest were reals. There might have been up to a 0.5 second loading time difference, but I consider it insignificant, considering the amount of fields.
    Test5: 6001 purple fields - 35 seconds.
    On this test I filled 3000 text fields. Elseway it's like test 4.
    Test6: 101 purple fields - 8 seconds.
    This time it was the levels field and 100 tooltip fields.
    Quite apparently strings take way more time to load than reals.
    Testmap with 1 custom Raise Dead
    Test1: 1001 purple fields - 8 seconds
    1000 of the fields are unittype fields.
    Apparently unittype fields are somewhat costly. It seems that a unittype field is a string with 4 letters.
    The reason to strings/unittypes being expensive seems to be that while every number variable is 32 bits a string is 32 for every letter.
    It also seems that it has something to do with strings beings processed in an inefficient manner.
     

    Attached Files:

    Last edited: Apr 28, 2016
  2. deathismyfriend

    deathismyfriend

    Joined:
    Oct 24, 2012
    Messages:
    6,532
    Resources:
    14
    Spells:
    12
    Tutorials:
    2
    Resources:
    14
    u forgot to meantion about unselecting brush list in the window tab.
     
  3. Xonok

    Xonok

    Joined:
    Mar 27, 2012
    Messages:
    3,042
    Resources:
    8
    Spells:
    3
    Tutorials:
    5
    Resources:
    8
    I thought about adding it, but it has nothing to do with reducing loading times. I guess I'll still add it.
     
  4. deathismyfriend

    deathismyfriend

    Joined:
    Oct 24, 2012
    Messages:
    6,532
    Resources:
    14
    Spells:
    12
    Tutorials:
    2
    Resources:
    14
    thats true but it helps a ton with editing time. it may not be a perfect fit with the tutorial but its still in the right area. IMO
     
  5. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,021
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    They are unchangeable within the editor. You can however overwrite the slk/txt files that harbor them by overshadowing their paths in the map. This format is loaded much, much faster and it is initialized altogether on map loading. There are two, maybe three cons. You cannot have data for levels/variations > 4, the fields for them do not exist. Those would again have to be realized via magenta modifications. Second, when replacing the txt files, those are normally language-dependent, so you lose the localization there. And lastly, in case you want to additionally use the standard data, you would have to reimport the whole files, which boosts map size. In reality, you should discard stuff you do not need.
     
  6. Xonok

    Xonok

    Joined:
    Mar 27, 2012
    Messages:
    3,042
    Resources:
    8
    Spells:
    3
    Tutorials:
    5
    Resources:
    8
    Does this have a tutorial somewhere? If yes, then I'd link it. If no, then I'd add it to this tutorial.
     
  7. PurgeandFire

    PurgeandFire

    Code Moderator

    Joined:
    Nov 11, 2006
    Messages:
    7,427
    Resources:
    18
    Icons:
    1
    Spells:
    4
    Tutorials:
    9
    JASS:
    4
    Resources:
    18
  8. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,021
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    You could use the hack of JNGP and add a tool that transforms your object editor data upon saving. The problem is that the editor cannot work with custom slks effectively. It does not load imported slks from the map, so when you reopen your map, you will not have access to your data. And if someone would succeed in modifying the editor this way, it would have to be 100% accurate in order not to kill your file or at least store a backup of the modification data.

    I am doing it the way that I separate editor map and playable file. When saving in editor, it copies the map to elsewhere and alters the duplicate (and also issues to playtest it, so I do not have to traverse the os everytime).
     
  9. Xonok

    Xonok

    Joined:
    Mar 27, 2012
    Messages:
    3,042
    Resources:
    8
    Spells:
    3
    Tutorials:
    5
    Resources:
    8
    Would be nice if someone made it. I don't know how though.
     
  10. WaterKnight

    WaterKnight

    Joined:
    Aug 18, 2009
    Messages:
    4,021
    Resources:
    5
    Maps:
    1
    Tutorials:
    4
    Resources:
    5
    It's not only about oe data. In general, maps can be optimized and there are approaches to go away from the standard editor. To join hands, the separation into development data and processed map would be recommendable.
     
  11. Xonok

    Xonok

    Joined:
    Mar 27, 2012
    Messages:
    3,042
    Resources:
    8
    Spells:
    3
    Tutorials:
    5
    Resources:
    8
    Wish someone actually made a specialized editor though, not just a hack.
     
  12. PurgeandFire

    PurgeandFire

    Code Moderator

    Joined:
    Nov 11, 2006
    Messages:
    7,427
    Resources:
    18
    Icons:
    1
    Spells:
    4
    Tutorials:
    9
    JASS:
    4
    Resources:
    18
    Sorry about not giving a review.

    I think it had the "This is under construction" text on it so I hadn't done a thorough review yet. I really like this info though. It is great to consider when making a map. I don't really like having the entire tutorial in spoilers (it is a bit hard on the eyes) but to each his own. :)

    Nice job. Approved and useful!
     
  13. Xonok

    Xonok

    Joined:
    Mar 27, 2012
    Messages:
    3,042
    Resources:
    8
    Spells:
    3
    Tutorials:
    5
    Resources:
    8
    Yes, I had a large text that said it's still in writing.
    After a while I realized I'm out of ideas, so I just decided it's good enough for now.

    EDIT: I believe the most important part of this are the benchmarks, as they even changed my original hypothesis.
    I now know that number values don't have a noticeable impact, while strings can kill your map.
    It makes sense though, since each number is 32 bits while each letter in a string is 32.
     
    Last edited: Jun 15, 2013
  14. Xonok

    Xonok

    Joined:
    Mar 27, 2012
    Messages:
    3,042
    Resources:
    8
    Spells:
    3
    Tutorials:
    5
    Resources:
    8
    Added a new section called "Initialization".
    I'm considering reworking this tutorial to be more thorough.
     
  15. Ceday

    Ceday

    Joined:
    Feb 22, 2010
    Messages:
    1,033
    Resources:
    0
    Resources:
    0
    I don't agree with initialization part.If you are going to do something that early which doesn't require game to start(multiboard, timer window etc) you should do all your stuff in initialization/loading.That is what loading means anyway.
     
  16. Xonok

    Xonok

    Joined:
    Mar 27, 2012
    Messages:
    3,042
    Resources:
    8
    Spells:
    3
    Tutorials:
    5
    Resources:
    8
    Well, the point is to reduce the time spent in the loading screen. As far as I've seen, having a lag spike at the start of the game isn't even noticeable even with lots of things loaded that way.
     
  17. laserdemon

    laserdemon

    Joined:
    Aug 4, 2012
    Messages:
    188
    Resources:
    0
    Resources:
    0
    Maybe you should discuss on how to reduce purple field, I think it's really hard to avoid changing them if you want to make your map different from Warcraft 3
     
  18. Xonok

    Xonok

    Joined:
    Mar 27, 2012
    Messages:
    3,042
    Resources:
    8
    Spells:
    3
    Tutorials:
    5
    Resources:
    8
    Usually it's nothing tragic when you have hundreds of custom abilities. This tutorial is just meant to give people some idea where exactly the long loading times might come from.
    Note that purple fields only matter on abilities that are actually on some unit. If you don't have any units with that ability, it doesn't get loaded.

    I think the Preloading part might even be more important than purple fields, not everything necessarily happens during the loading.
    If you have lots of things in the map you can significantly increase performance by just loading them when they're needed, instead of right away.
    In an ORPG this might mean organizing the map as areas where you only spawn creeps when the players first enter that area.
    I've seen many ORPG maps where the creeps are preplaced. In most cases that's just not necessary.
     
  19. laserdemon

    laserdemon

    Joined:
    Aug 4, 2012
    Messages:
    188
    Resources:
    0
    Resources:
    0
    I think this becomes a problem when you have heroes with hundreds of levels and custom abilities with 100 levels, the purple field for abilities data and tooltip make the abilities data exceed 2mb which causes the loading time increase dramatically. Preloading make the game lag and probably disconnect when player's computer is too old
     
  20. Xonok

    Xonok

    Joined:
    Mar 27, 2012
    Messages:
    3,042
    Resources:
    8
    Spells:
    3
    Tutorials:
    5
    Resources:
    8
    2MB of ability data is something I have never seen. An ability with 8000 levels of floats(the ability is spell shield) freezes the game for over 10 seconds. Yet, the map only takes 98 KB with that.
    If you have 2MB of ability data, you're probably doing things so wrong that it doesn't even matter if you preload things or not.
    In any case, you shouldn't preload lots of things at the same time. If you don't load many things at a time the lag won't be noticeable.