- Joined
- Jan 13, 2017
- Messages
- 88
Introduction
[Edit] Just to be perfectly clear, previously local files could only be saved using a registry flag, this has been changed in previous patches, 1.30 or similar and one can always save files now.
Greetings fellow modders, Time for another write-up, and this time with actual results!
Some may recall my previous write-up on synchronization of the Z height map and its synchronization speed, that ended with promising but pretty useless results.
Today i bring something a bit more, useful. Something that is very well possible to use for maps in actuality. Its also relatively simple and might be portable to JASS (Although i do not recommend it).
Pogo jumping on my previous thread this is slightly different, being more of a technical explanation behind the development and core ideals of the code instead of a timeline of improvements / changes.
Let's get started.
Resources
ScrewTheTrees/SaveLoadBigData , The GIT repository that i used, its MIT licensed so feel free to use the code however you feel like.
The magical file where i wrote the code , This is the file in the repository where my primary code was written.
Basics of Saving/Loading using preloads
This is the basics, if you already know the basics of preloading you can just jump to my implementation further down.
Now, i did take inspiration from TriggerHappy's Codeless Save/Load to learn about preload natives and how they are used so huge props to him.
My Implementation
Now i assume you know the very basics so lets get into my slightly different implementation.
Originally my idea was simply to implement a simple save/load framework into my WC3-TreeLib typescript library so that i could easily save/load data.
Hence the code i use is typescript, however it should be easily transferable to raw lua if that is what you need, the basics are relatively simple.
Now TriggerHappy's design used the setting of ability tool-tips to share the data from the preload files, this is great way and is instant to do without a problem, My approach was a tiny bit more complex and not at all JASS friendly, instead if decided to experiment with my old friend "BlzSendSyncData()".
When doing my write-up on synchronized height map i got well acquainted to that native, and i simply wondered, could i use it to send data from a preload file to the game?
The first test was very simple but yielded the results i wanted, the preload string i wrote in, was synchronized to all players using the event directly from the file.
This was intriguing to me and i wondered, how much can it handle?
I started populating large amount of these but yet, it kept synchronizing, the data.
This is where my initial ideas popped and i got to work with a basic flow:
To Save:
* Saving
My implementation was relatively simple, you can find it inside the writeFile method on the sync class.
What i simply did was encode it to Base64, and feed it into the preload file.
Now its not necessary, but i did add a header to every string that is used to identify currentchunk/maxChunk size, This is not necessary as they always should arrive in order, but i generally like it so it got to stay. This was written as simple HEX values as the first 16 chars of the string.
Now, i removed the original S2I i had at the end as i realised i didnt need it, this happened as i was writing it and decided to implement said fix.
So, saving was easy and this is the general gist of it:
And done deal, this is the magical snippet that makes sure the preload files share their data.
It not very complex.
* Loading / Reading
Now this is pod racing...
The general difference of my implementation is, that the file contents are automatically synchronized to all players. So one does not have to do it manually.
As for the performance of this, its very acceptable and i will take it up in a bit.
My handling of the reading was simple, i would use a Promise, aka when all the data is assembled it would use a callback and set a flag on that class/table, people who has used javascript webdevs should know promises quite well, their headaches and usability.
Then i ran the preload, originally i would run a "is finished" method once the current chunk size hit the max chunk size, but this naturally caused an issue When no file was loaded. Since no sync would be sent, forever leaving the promise unfullfilled, so then i added another SyncString with another prefix that would be sent after the file was preloaded to signal that the file was ready, this made my chunk system a bit useless but i kept it in for simplicity and maybe future debugging.
Now, my system is limited to the reading of 1 file per player at once using the framework, it does however throw a warning if one tries to read more than one file for a player at once and deny the user to do so.
There is also a method for checking if a player has a promise already active.
Results
Now to the best part, results, and how i used them.
My first use case was simple, save and load the entire Bee Movie script and then read it for the players line by line.
This amounted to 55308 characters that needed to be saved, loaded & synchronized.
The results? well, in LAN it only took 3 seconds to send the entire script.
That is most likely far more than anyone would ever need to save to file in a normal map,
The code to make it happen was simple:
And to show the lines to the players:
And finally how it looks like ingame:
It did not desync the players, so a huge success i would say in that regard.
I did another experiment where added hotsaves for the locations of all a players own units when hitting a hotkey, and then creating units at said locations when hitting another hotkey, this was also safe, and almost instant with my limited amount of players, it was also part of the SyncSaveLoadTest.ts file where you can see how i accomplished it.
Final words
Unlike my previous writeup, this on can be considered useful by people who wants to save large amounts of data.
Personally i do like my approach to it, since data is Warcraft 3 needs to be synchronous anyway one might as well just take care of it ASAP to make sure the data is the same for everyone.
The Base64 encoding and all that also allows me to save any byte i want, so which means i could run it through ciphers or other encryptions, in one of my testcases i ran it through a simple Caesar Cipher with no problems.
I am quite interested in how i can apply this to maps, not only can i save pretty much any amount of data i want that Warcraft 3 can handle, and i can save any byte data i want.
This should mean that one could make like a "custom dungeon editor" and then share and play the maps with other players later, or saving of other data like global worldflags/states for which gates are open and what triggers a player has already touched. Or just the location of every single unit on the map if that is your cup of tea.
I make sure these type of projects use MIT as their license so people can rewrite/use/alter the code freely with minimal restrictions so feel free to experiment with your own solutions or maybe porting it to pure Lua and maybe even JASS!
If you got this far, thanks for reading my yet again overly long article.
I hope you have either learned something, or find any type of use for this for your own projects.
See you later, space cowboy!
[Edit] Just to be perfectly clear, previously local files could only be saved using a registry flag, this has been changed in previous patches, 1.30 or similar and one can always save files now.
Greetings fellow modders, Time for another write-up, and this time with actual results!
Some may recall my previous write-up on synchronization of the Z height map and its synchronization speed, that ended with promising but pretty useless results.
Today i bring something a bit more, useful. Something that is very well possible to use for maps in actuality. Its also relatively simple and might be portable to JASS (Although i do not recommend it).
Pogo jumping on my previous thread this is slightly different, being more of a technical explanation behind the development and core ideals of the code instead of a timeline of improvements / changes.
Let's get started.
Resources
ScrewTheTrees/SaveLoadBigData , The GIT repository that i used, its MIT licensed so feel free to use the code however you feel like.
The magical file where i wrote the code , This is the file in the repository where my primary code was written.
Basics of Saving/Loading using preloads
This is the basics, if you already know the basics of preloading you can just jump to my implementation further down.
Now, i did take inspiration from TriggerHappy's Codeless Save/Load to learn about preload natives and how they are used so huge props to him.
Now, preload files are interesting, they allow for execution of JASS code using files, and to write so called preload files, on its own this is quite interesting, the problem is that it seems to run in its own little world.
This makes the saving/loading of data a minor hassle, but due to some good natives like setting the extended tooltip of abilities (aka TH's approach), one can transfer the data from the Preload files to the game engine.
Now when it comes to writing preload files it was quite simple initially, the fun part comes when exploiting the generating of said preload files.
So by using:
We can write to files with a very specific file format:
Now this is where it becomes abuse able, by abusing the system we can write our own Code snippets inside jass, for example:
This would run the function DoNothing() when preloading this file, by instead using something like BlzSetAbilityExtendedTooltip() you can change information inside the players game.
There is one more step to this, it needs to be synchronized before being used, using a sync event or similar.
This way you can simply save/load strings of anything you really want to files and load them up and decipher them in game at a later date.
Now Preload("") only supports up to 255 characters and null terminates the string so generally you have to make it preload safe by removing scary letters like % and char(0).
Also some of the length is lost as you need to use ")\n to end the preload function early and then some code at the end to make sure the function ends with valid code, my choice is calling S2I as it was a short function name.
This makes the saving/loading of data a minor hassle, but due to some good natives like setting the extended tooltip of abilities (aka TH's approach), one can transfer the data from the Preload files to the game engine.
Now when it comes to writing preload files it was quite simple initially, the fun part comes when exploiting the generating of said preload files.
So by using:
JavaScript:
PreloadGenClear();
PreloadGenStart();
Preload("data up to 255 chars long and this native is repeatable");
PreloadGenEnd("myFile.txt");We can write to files with a very specific file format:
Code:
function PreloadFiles takes nothing returns nothing
call PreloadStart()
call Preload( "Data up to 255 chars long and this native is repeatable" )
call PreloadEnd( 0.1 )
endfunctionNow this is where it becomes abuse able, by abusing the system we can write our own Code snippets inside jass, for example:
JavaScript:
Preload("\")\ncall DoNothing()\ncall S2I(\"");This would run the function DoNothing() when preloading this file, by instead using something like BlzSetAbilityExtendedTooltip() you can change information inside the players game.
There is one more step to this, it needs to be synchronized before being used, using a sync event or similar.
This way you can simply save/load strings of anything you really want to files and load them up and decipher them in game at a later date.
Now Preload("") only supports up to 255 characters and null terminates the string so generally you have to make it preload safe by removing scary letters like % and char(0).
Also some of the length is lost as you need to use ")\n to end the preload function early and then some code at the end to make sure the function ends with valid code, my choice is calling S2I as it was a short function name.
My Implementation
Now i assume you know the very basics so lets get into my slightly different implementation.
Originally my idea was simply to implement a simple save/load framework into my WC3-TreeLib typescript library so that i could easily save/load data.
Hence the code i use is typescript, however it should be easily transferable to raw lua if that is what you need, the basics are relatively simple.
Now TriggerHappy's design used the setting of ability tool-tips to share the data from the preload files, this is great way and is instant to do without a problem, My approach was a tiny bit more complex and not at all JASS friendly, instead if decided to experiment with my old friend "BlzSendSyncData()".
When doing my write-up on synchronized height map i got well acquainted to that native, and i simply wondered, could i use it to send data from a preload file to the game?
The first test was very simple but yielded the results i wanted, the preload string i wrote in, was synchronized to all players using the event directly from the file.
This was intriguing to me and i wondered, how much can it handle?
I started populating large amount of these but yet, it kept synchronizing, the data.
This is where my initial ideas popped and i got to work with a basic flow:
To Save:
- Assemble raw string data to save.
- Encode said data to Base64 to make it JASS safe.
- Start preloading.
- Write orderly chunks of data inside BlzSendSyncData() (my selected size: 180 chars)
- End the preload.
- Call Preloader() on the file for a player.
- Use a SyncData event to catch these and assemble them back into a full string.
- Decode the Base64 to the original string.
- Call a callback with the assembled string.
* Saving
My implementation was relatively simple, you can find it inside the writeFile method on the sync class.
What i simply did was encode it to Base64, and feed it into the preload file.
Now its not necessary, but i did add a header to every string that is used to identify currentchunk/maxChunk size, This is not necessary as they always should arrive in order, but i generally like it so it got to stay. This was written as simple HEX values as the first 16 chars of the string.
Now, i removed the original S2I i had at the end as i realised i didnt need it, this happened as i was writing it and decided to implement said fix.
So, saving was easy and this is the general gist of it:
JavaScript:
for (let i = 0; i < toCompile.length; i++) {
assemble += toCompile.charAt(i);
if (assemble.length >= chunkSize) {
let header = EncodingHex.To32BitHexString(noOfChunks) + EncodingHex.To32BitHexString(math.ceil(i / chunkSize));
Preload(`")\ncall BlzSendSyncData("${this.syncPrefix}","${header + assemble}`);
assemble = "";
}
}And done deal, this is the magical snippet that makes sure the preload files share their data.
It not very complex.
* Loading / Reading
Now this is pod racing...
The general difference of my implementation is, that the file contents are automatically synchronized to all players. So one does not have to do it manually.
As for the performance of this, its very acceptable and i will take it up in a bit.
My handling of the reading was simple, i would use a Promise, aka when all the data is assembled it would use a callback and set a flag on that class/table, people who has used javascript webdevs should know promises quite well, their headaches and usability.
Then i ran the preload, originally i would run a "is finished" method once the current chunk size hit the max chunk size, but this naturally caused an issue When no file was loaded. Since no sync would be sent, forever leaving the promise unfullfilled, so then i added another SyncString with another prefix that would be sent after the file was preloaded to signal that the file was ready, this made my chunk system a bit useless but i kept it in for simplicity and maybe future debugging.
Now, my system is limited to the reading of 1 file per player at once using the framework, it does however throw a warning if one tries to read more than one file for a player at once and deny the user to do so.
There is also a method for checking if a player has a promise already active.
Results
Now to the best part, results, and how i used them.
My first use case was simple, save and load the entire Bee Movie script and then read it for the players line by line.
This amounted to 55308 characters that needed to be saved, loaded & synchronized.
The results? well, in LAN it only took 3 seconds to send the entire script.
That is most likely far more than anyone would ever need to save to file in a normal map,
The code to make it happen was simple:
JavaScript:
this.syncSaveLoad = SyncSaveLoad.getInstance(); //I know Singletons are an antipattern.
this.syncSaveLoad.writeFile("test.txt", EntireBeeMovieScript); //This is a huge constant string.
this.syncSaveLoad.read("test.txt", Player(0), (promise) => { //Player 1 Red is the only one reading the file
xpcall(() => {
this.parseLines = StringFuncs.UnpackStringNewlines(promise.finalString); //This support function removes useless whitespace to make sure each line has actual lines in it.
}, Logger.critical);
});And to show the lines to the players:
JavaScript:
step(): void { //Part of my entity system, executes once a second in this class.
if (this.currentLine < this.parseLines.length) {
print(this.currentLine, ": ", this.parseLines[this.currentLine]); //parseLines is empty unless the file read goes through.
this.currentLine += 1;
}
}And finally how it looks like ingame:
It did not desync the players, so a huge success i would say in that regard.
I did another experiment where added hotsaves for the locations of all a players own units when hitting a hotkey, and then creating units at said locations when hitting another hotkey, this was also safe, and almost instant with my limited amount of players, it was also part of the SyncSaveLoadTest.ts file where you can see how i accomplished it.
Final words
Unlike my previous writeup, this on can be considered useful by people who wants to save large amounts of data.
Personally i do like my approach to it, since data is Warcraft 3 needs to be synchronous anyway one might as well just take care of it ASAP to make sure the data is the same for everyone.
The Base64 encoding and all that also allows me to save any byte i want, so which means i could run it through ciphers or other encryptions, in one of my testcases i ran it through a simple Caesar Cipher with no problems.
I am quite interested in how i can apply this to maps, not only can i save pretty much any amount of data i want that Warcraft 3 can handle, and i can save any byte data i want.
This should mean that one could make like a "custom dungeon editor" and then share and play the maps with other players later, or saving of other data like global worldflags/states for which gates are open and what triggers a player has already touched. Or just the location of every single unit on the map if that is your cup of tea.
I make sure these type of projects use MIT as their license so people can rewrite/use/alter the code freely with minimal restrictions so feel free to experiment with your own solutions or maybe porting it to pure Lua and maybe even JASS!
If you got this far, thanks for reading my yet again overly long article.
I hope you have either learned something, or find any type of use for this for your own projects.
See you later, space cowboy!
Last edited:
it just takes some time to wrap ones head around how it all works, took me quite a bit of time for sure, along with quite a few crashes and accidentally corrupting the sent data.
