Hey everyone.
I would like to offer a few separate propositions, each of which would individually facilitate working with save/load systems in WC3 and make them easier to write and use.
Currently, it is already possible to do most of what I am about to offer - you can look in Wurst's StdLib2 "Network" and "MultifileIO" packages to have a taste of how it can be done, but there are some issues with what can be currently achieved:
1. There is no nice way of reading/saving files. You can exploit the Preload API, but it is far from being even remotely nice to use:
1.1 You have to use SetPlayerName to be able to read data from files, which limits you to 16 lines per file. This means that if you want to write more than that, you have to split your data into multiple files (You can see in Wurst's StdLib2 how I accomplish this, for instance),
1.2 There is no way to check the existence of a file or directory,
1.3 There is no way to write binary (or even text) data to a file without "dead weight" of the Preload API,
1.4 There is no way to read contents from a file other than using GetPlayerName or other stateful APIs,
1.5 You have to enable a specific registry value in order to be able to read files,
2. There is no nice way of sending data over the network. You can use GameCaches and the Sync natives, but:
2.1 The SyncStoredString native is broken, requiring you to serialize strings as int-streams,
2.2 The API is weird and unintuitive, keys are strings (would be more convenient to have them as ints for this use-case),
2.3 GameCaches are very slow compared to even HashTables, making syncing code rather resource-intensive, especially for large payloads,
2.4 The API is not synchronous and is very hard to use in JASS or even vJASS. You can write an asynchronous API using callbacks in Wurst, which is nicer, but it is still not very easy to use,
2.5 You are requried to constantly restart threads if you want to handle large payloads
3. There is no nice way of working with raw binary data. This issue is actually not specific to just save/load, but also other applications:
3.1 No bitwise operations. Bitwise ops have to be coded in using lookup tables or MemoryHack,
3.2 No char-int conversions. To get an int value of a char you have to use lookup tables, and vice-versa
Hence, I propose several additions, each of which addresses these issues. The simplest to implement here would be #3, so let's start with that:
JASS:
// Each function will be annotated with it's C++ equivalent
// ~a
native BitwiseNot takes integer a returns integer
// a & b
native BitwiseAnd takes integer a, integer b returns integer
// a | b
native BitwiseOr takes integer a, integer b returns integer
// a ^ b
native BitwiseXor takes integer a, integer b returns integer
// a << b
native BitwiseShl takes integer a, integer b returns integer
// a >> b
native BitwiseShr takes integer a, integer b returns integer
// Returns the byte value of the first character in the string, or -1 if string is empty
// Return value is guaranteed to be in the bound of [0, 256)
native Char2Byte takes string a returns integer
// Converts the byte into a single character corresponding to that character's value
// The argument should be in the bound of [0, 256), if it is not, the returned string should be null
native Byte2Char takes integer a returns string
Next is a proposal for a File IO API. This API should work out of the box, without requiring any tweaks to the registry:
JASS:
constant integer FILEMODE_READ = 1
constant integer FILEMODE_WRITE = 2
constant integer FILEMODE_READWRITE = 3
type file extends handle
type directory extends handle
native FileExists takes string path returns boolean
native DirectoryExists takes string path returns boolean
// Returns an iterator to a directory's contents
// Returns null if path doesn't exist or isn't a directory
native GetDirectory takes string path returns directory
// Callback-style iteration
native ForDirectory takes directory dir, code callback returns nothing
native GetEnumPath takes nothing returns string
// Pops a single path from this directory object, allowing us to iterate it using a while loop
native PopDirectoryPath takes directory dir returns string
// Opens a new file, creating it if necessary
// There should probably be a restriction on which extensions are allowed
// Good candidates are probably ".txt" and ".dat"
native OpenFile takes string path, integer mode returns file
// Opens a new file only for a certain player
// This is necessary to allow reading/writing locally, because creating a handle locally would desync, so
// we need to create it in shared code
// Reading/Writing for other players on this handle will have no effect, unless SyncFile has been called,
// then reading operations should become available
native OpenFile takes string path, integer mode, player owner returns file
// (I'm not sure if this should be added or not) Tries to send the contents of an entire file from one players to the rest
// Calling this in a local block will read the whole file and start sending it to other players, allowing us to read the file
// synchronously in shared code without issues
native SyncFile takes file f, player sender returns nothing
// If this can be implemented, it'd be really neat
// Pauses the current thread until the file has finished syncing
native SyncFileWait takes file f returns nothing
// Async events for syncing to signal when it's finished
native TriggerRegisterFileSync takes trigger t returns nothing
native GetSyncedFile takes nothing returns file
// Flushes the changes and closes the file
native CloseFile takes file f returns nothing
native GetFileSize takes file f returns integer
native SetFileSeekPos takes file f, integer pos returns nothing
native GetFileSeekPos takes file f returns integer
// Checks whether the file has hit EOF
native IsFileEOF takes file f returns boolean
// Reads the contents of the file into a string, up to specified amount of characters
native FileReadString takes file f, integer max returns string
// Writes the contents of the string into the file as-is, without null-terminators
native FileWriteString takes file f, string s returns nothing
// Natives for reading and writing bytes directly, rather than using a string
// Since WC3 runs only on little-endian platforms, these too should probably be little-endian
native FileWriteUInt8 takes file f, integer a returns nothing
native FileWriteUInt16 takes file f, integer a returns nothing
native FileWriteUInt32 takes file f, integer a returns nothing
native FileWriteInt8 takes file f, integer a returns nothing
native FileWriteInt16 takes file f, integer a returns nothing
native FileWriteInt32 takes file f, integer a returns nothing
// Natives for reading bytes directly
native FileReadUInt8 takes file f returns integer
native FileReadUInt16 takes file f returns integer
native FileReadUInt32 takes file f returns integer
native FileReadInt8 takes file f returns integer
native FileReadInt16 takes file f returns integer
native FileReadInt32 takes file f returns integer
// Writing/Reading reals
native FileWriteReal takes file f, real a returns nothing
native FileReadReal takes file f returns real
Last is the proposal to address the issue of networking data across players. There are two solutions I'd like to offer:
The first one is the most straightforward and requires minimal new natives
But for it to be worth it, the code should be optimized to make most use of the bandwidth available, and the fact that we are sending
data in a batch (instead of how SyncStored* GameCache natives seem to work, which are very slow, only 1-4 kbps max)
JASS:
// When called locally, will start syncing the contents of this hashtable to other players
native SyncHashtableParent takes hashtable ht returns nothing
// Same as SyncHashtable, but only for a child hashtable
native SyncHashtableChild takes hashtable ht, integer parentKey returns nothing
// Pauses the thread until the specified hashtable has finished syncing
native SyncHashtableWait takes hashtable ht returns nothing
// Events for hashtable syncs
native TriggerRegisterHashtableSync takes trigger t returns nothing
native GetSyncedHashtable takes nothing returns hashtable
Another way, which could turn out more efficient, would be to create a new type specifically for syncing
This type would be something like a byte-buffer with natives for reading, writing and syncing it's contents
It's internal C++ representation could be something as simple as std::vector<uint8_t>, allowing us to use it
as a dynamic array type, which has also been long since requested in the community
JASS:
type bytes extends handle
// self-explanatory
native CreateBytes takes nothing returns bytes
native DestroyBytes takes bytes b returns nothing
native BytesWriteUInt8 takes bytes b, integer pos, integer a returns nothing
native BytesWriteUInt16 takes bytes b, integer pos, integer a returns nothing
native BytesWriteUInt32 takes bytes b, integer pos, integer a returns nothing
native BytesWriteInt8 takes bytes b, integer pos, integer a returns nothing
native BytesWriteInt16 takes bytes b, integer pos, integer a returns nothing
native BytesWriteInt32 takes bytes b, integer pos, integer a returns nothing
native BytesReadUInt8 takes bytes b, integer pos returns integer
native BytesReadUInt16 takes bytes b, integer pos returns integer
native BytesReadUInt32 takes bytes b, integer pos returns integer
native BytesReadInt8 takes bytes b, integer pos returns integer
native BytesReadInt16 takes bytes b, integer pos returns integer
native BytesReadInt32 takes bytes b, integer pos returns integer
native BytesWriteReal takes bytes b, integer pos, real r returns nothing
native BytesReadReal takes bytes b, integer pos returns nothing
native BytesWriteString takes bytes b, integer pos, string s returns nothing
native BytesReadString takes bytes b, integer pos, integer amount returns string
// Get the amount of data in this buffer
native GetBytesSize takes bytes b returns integer
// Reserve capacity for this buffer
native ReserveBytesCapacity takes bytes b, integer capacity returns nothing
// Get capacity for this buffer
native GetBytesCapacity takes bytes b returns bytes
// Tries to sync the local contents of this bytes object to other players
native SyncBytes takes bytes b returns nothing
// Pauses the current thread until syncing has finished
native SyncBytesWait takes bytes b returns nothing
// Events
native TriggerRegisterBytesSync takes trigger t returns nothing
native GetSyncedBytes takes nothing returns bytes
