Exploring File I/O Tricks and Techniques

[Lizreu]

As reported by @TriggerHappy, the "Allow Local Files" requirement has been lifted from the and now anyone can (apparently) read files, with or without this setting.

This is a huge deal breaker, and as the maintainer of Wurst's (rather high-level) Persistence API, this has made me reconsider some of the choices I made in the library. In particular, I'd like to investigate, and propose, some alternative methods of reading data from Preload files.

Note: All of this is heavily inspired by @TriggerHappy's past work on FileIO, so huge credit goes to him for coming up with all of this.

Note2: If you didn't know, Wurst has a very high-level and relatively easy-to-use Persistence API which handles all the grunt work behind the scenes, starting from reading the data from disc, to synchronizing it between players, and deserializing it into a nice comfy class for you to use, while handling various edge and error cases. Oh, and did you also know that it can handle arbitrarily large files as well?

With that said, let me revise some of the methods that have already been discovered by others, and some of the more obscure ones I came up with:

1. SetPlayerName / GetPlayerName

The classic. The preloader file contains consecutive calls to `SetPlayerName(Player(x), "payload")`, which are then read by the map's script, and the names reset. This has a very severe limitation of only allowing MAX_PLAYERS * 1024 (2048 if you really wanna push it) bytes per file. While definitely more than most people would ever need, there are certain scenarios in maps where you might need to save more than that. This is the method that Wurst currently uses, and it splits the total payload into multiple files in the case that more than 1 is needed.

2. SetUnitName / GetUnitName

This is a very weird one I came up with not so long ago... The idea is that the preloader file contains consecutive calls to `SetUnitName(CreateUnit(...), "payload")`. The gotcha that prevents desyncs here is that all other players must create the required amount of units at the same time as the loader runs the file. I didn't go anywhere with this, but in my tests, it worked, without desyncing.

3. BlzSetAbilityTooltip

This is one pioneered by @TriggerHappy in his latest FileIO library, however he only uses 1 ability with 1 level, which limits the payload to around 2048 chars, since (AFAIK) that's the length limit of strings in WC3 before they go bad. 1-2KB is more than most people would ever need, but we are here to push limits! So...

4. BlzSetAbilityTooltip on multiple ability levels

This one requires a custom dummy ability with a certain number of levels. I came up with this shortly after looking at TriggerHappy's new lib, and has the nice benefit of being VERY expandable, having minimal side effects, and able to have files of up to 128MB in payload. The file consists of consecutive calls to `BlzSetAbilityTooltip(DUMMY_ID, "payload", i)`, and in my tests, one JASS thread can handle around 100-200 thousand of these calls, making this method arguably the best one, although I'm still yet to test it thoroughly.

And that's about it, as far as my own knowledge on this goes. This is, of course, entirely orthogonal to the problem of actually synchcronizing the data, which I covered in my tutorial here.

There's 3 reasons why I'm posting about this here:
1. To spread the knowledge about File IO in WC3.
2. To see if anyone else has dabbled in this and knows any other alternative methods that may be superior to the ones I outlined here.
3. To see what you guys think about it.

Thanks for reading!

 

[Daffa]

Thank the new natives I guess for giving more opportunities.

This interesting, though I haven't work on IO stuff myself. I'll just gonna keep track of this progress.

I do need to note the two last method are restricted to 1.29+, and with the stuff going on, it might be a risky road for back compat.

 

[Lizreu]

Thank the new natives I guess for giving more opportunities.

This interesting, though I haven't work on IO stuff myself. I'll just gonna keep track of this progress.

I do need to note the two last method are restricted to 1.29+, and with the stuff going on, it might be a risky road for back compat.

Oh yes, it's something I should've mentioned too, thanks for pointing it out.
In fact, the second method is also 1.29+ only, since it relies on BlzSetUnitName, which is a new native as well.

 

[Daffa]

Yeah, that too.

Anyway, I think the fourth method might be the solution to high amount of data. Though personally I doubt many map would need a lot of data to store to be honest.
This can be useful for 2P Campaign project, might want to give @Gismo359 a heads up.

 

[Lizreu]

Yeah, that too.

Anyway, I think the fourth method might be the solution to high amount of data. Though personally I doubt many map would need a lot of data to store to be honest.
This can be useful for 2P Campaign project, might want to give @Gismo359 a heads up.

Yeah, not many maps requires that much data, but I'm a bit of a special snowflake in this regard, since my map can easily require up to 20-60KB of storage per user. @Frotty's map also supports some kind of save/load thingy for his mazes, which also require more data than what most people need.

The biggest game-changer here is the fact that everyone will be able to use it now without having to enable local files explicitly.

 

[TriggerHappy]

The reason I didn't use ability levels was because I wanted the script to be easily copy and pastable without requiring any object data. If the native worked for levels that didn't exist in the ability then I could have implemented a line system, or simply extended the max length, but I want to keep the library lightweight. I quickly tested my FileIO with a string of length 4000 (concatenated strings can reach 4097) and it worked fine. I imagine most people wouldn't need more data than that in a file so I figured I wouldn't complicate the script any further.

There are tons of ways to pass data back and forth through preload files. You can store and retrieve raw integers with SetPlayerTechResearched and GetPlayerTechResearched (almost no limit of data amount). You can even pass data with gamecache, but the only problem is calling InitGameCache raises the handle count even if you flush the previous cache. I even managed to use hashtables through a I2Hashable function, but it required me creating a new hashtable each time the preload file ran. You pass the handle id of your main hashtable, and the dummy hashtable can cast to it and store data. Of course this is almost useless as you can only have 255 hashtables.

In the end I think my current implementation with the ability tooltips is the best solution.

EDIT: Well, instead of multiple levels you could have an array of abilities.

 

[Lizreu]

The reason I didn't use ability levels was because I wanted the script to be easily copy and pastable without requiring any object data. If the native worked for levels that didn't exist in the ability then I could have implemented a line system, or simply extended the max length, but I want to keep the library lightweight. I quickly tested my FileIO with a string of length 4000 (concatenated strings can reach 4097) and it worked fine. I imagine most people wouldn't need more data than that in a file so I figured I wouldn't complicate the script any further.

There are tons of ways to pass data back and forth through preload files. You can store and retrieve raw integers with SetPlayerTechResearched and GetPlayerTechResearched (almost no limit of data amount). You can even pass data with gamecache, but the only problem is calling InitGameCache raises the handle count even if you flush the previous cache. I even managed to use hashtables through a I2Hashable function, but it required me creating a new hashtable each time the preload file ran. You pass the handle id of your main hashtable, and the dummy hashtable can cast to it and store data. Of course this is almost useless as you can only have 255 hashtables.

In the end I think my current implementation with the ability tooltips is the best solution.

EDIT: Well, instead of multiple levels you could have an array of abilities.

Thanks for the info. I definitely agree that most people won't need more than 4 kb of data, and even that much is way overkill for the majority of use cases. Or, perhaps, nobody's used more data because traditionally data was saved using chat codes, which severely limits the whole concept. Maybe with proper FileIO available to everyone, people will be able to come up with more interesting ideas on what to do with it.

As I mentioned previously, I'm personally rather invested in this since the save files for my map routinely reach the range of 10-200kb of data, so I'm always looking for ways to push this forward.

Didn't know that strings could store up to 4kb of data, this is something new. I read somewhere long ago that strings maxed out at about 2047 characters. Do you know if this is specific to the new patch or has it always been this way?

I agree with you that the tooltip solution is the best, easy to use and has very little overhead. Going to be using it in the future in Wurst's standard lib.

 

[TriggerHappy]

Thanks for the info. I definitely agree that most people won't need more than 4 kb of data, and even that much is way overkill for the majority of use cases. Or, perhaps, nobody's used more data because traditionally data was saved using chat codes, which severely limits the whole concept. Maybe with proper FileIO available to everyone, people will be able to come up with more interesting ideas on what to do with it.

As I mentioned previously, I'm personally rather invested in this since the save files for my map routinely reach the range of 10-200kb of data, so I'm always looking for ways to push this forward.

Didn't know that strings could store up to 4kb of data, this is something new. I read somewhere long ago that strings maxed out at about 2047 characters. Do you know if this is specific to the new patch or has it always been this way?

I agree with you that the tooltip solution is the best, easy to use and has very little overhead. Going to be using it in the future in Wurst's standard lib.

The concatenated string limit has been 4097 for as long as I can remember ( or 4099? [Documentation] String Type ).

If you have to store that much data then maybe make an array of unused standard abilities or find some abilities id's in sequential order and use those instead of the level parameter. This way you can avoid creating object data which (for some reason) takes the game forever to load.

 

[Lizreu]

The concatenated string limit has been 4097 for as long as I can remember ( or 4099? [Documentation] String Type ).

If you have to store that much data then maybe make an array of unused standard abilities or find some abilities id's in sequential order and use those instead of the level parameter. This way you can avoid creating object data which (for some reason) takes the game forever to load.

As far as I can tell, loading time depends on how many custom fields you have in your map. If you create an ability with a few thousand of levels, but don't modify them, then that ability only has 1 custom field, which is the level field. In my tests, even 16000 levels didn't affect load times at all, which is why I proposed this idea.

Also, the thread you linked to mentions that the limit is 2048-2052 bytes, for some reason. Do you have another source?

EDIT: Oh, actually it says in one place that it's ~2000, and then down the road that it's ~4000. Weird.

 

[PurgeandFire]

EDIT: Oh, actually it says in one place that it's ~2000, and then down the road that it's ~4000. Weird.

sorry, lol. I updated it in one place and forgot to update it in the other. I've fixed that now.