[Lua] FileIO (Lua-optimized)

[Trokkin]

So I made this snippet 3 years ago, and I can't remember where did I get the idea to use Lua inside preload files, and then that SetAbilityTooltip don't care for string length anymore... But anyways, here I optimized the well-established method to save and load local files. And as far as I've searched the Hive, there aren't any definitive or innovative solutions to the problem, so here you go. Of course, syncing the value in the multiplayer is still on user.

• The description is loaded with only one BlzSetAbilityTooltip invocation, so no file size limits
• Strings can contain both single and double quotes without protection, thanks to Lua multiline string syntax -- but not double square brackets
• The overhead is ~150 bytes bigger than in TH's FileIO preload files, but each chunk is bigger, which is better for huge strings
• Almost GUI friendly, meaning you only need to use one simple custom script line to load or to save.

Spoiler: Source code

if Debug then Debug.beginFile "FileIO" end
--[[
    FileIO v1a

    Provides functionality to read and write files, optimized with lua functionality in mind.

    API:

        FileIO.Save(filename, data)
            - Write string data to a file

        FileIO.Load(filename) -> string?
            - Read string data from a file. Returns nil if file doesn't exist.

        FileIO.SaveAsserted(filename, data, onFail?) -> bool
            - Saves the file and checks that it was saved successfully.
              If it fails, passes (filename, data, loadResult) to onFail.

        FileIO.enabled : bool
            - field that indicates that files can be accessed correctly.

    Optional requirements:
        DebugUtils by Eikonium                          @ https://www.hiveworkshop.com/threads/330758/
        Total Initialization by Bribe                   @ https://www.hiveworkshop.com/threads/317099/

    Inspired by:
        - TriggerHappy's Codeless Save and Load         @ https://www.hiveworkshop.com/threads/278664/
        - ScrewTheTrees's Codeless Save/Sync concept    @ https://www.hiveworkshop.com/threads/325749/
        - Luashine's LUA variant of TH's FileIO         @ https://www.hiveworkshop.com/threads/307568/post-3519040
        - HerlySQR's LUA variant of TH's Save/Load      @ https://www.hiveworkshop.com/threads/331536/post-3565884

    Updated: 8 Mar 2023
--]]
OnInit("FileIO", function()
    local RAW_PREFIX = ']]i([['
    local RAW_SUFFIX = ']])--[['
    local RAW_SIZE = 256 - #RAW_PREFIX - #RAW_SUFFIX
    local LOAD_ABILITY = FourCC('ANdc')
    local LOAD_EMPTY_KEY = '!@#$, empty data'

    local function open(filename)
        name = filename
        PreloadGenClear()
        Preload('")\nendfunction\n//!beginusercode\nlocal p={} local i=function(s) table.insert(p,s) end--[[')
    end

    local function write(s)
        for i = 1, #s, RAW_SIZE do
            Preload(RAW_PREFIX .. s:sub(i, i + RAW_SIZE - 1) .. RAW_SUFFIX)
        end
    end

    local function close()
        Preload(']]BlzSetAbilityTooltip(' .. LOAD_ABILITY .. ', table.concat(p), 0)\n//!endusercode\nfunction a takes nothing returns nothing\n//')
        PreloadGenEnd(name)
        name = nil
    end

    ---
    ---@param filename string
    ---@param data string
    local function savefile(filename, data)
        open(filename)
        write(data)
        close()
    end

    ---@param filename string
    ---@return string?
    local function loadfile(filename)
        local s = BlzGetAbilityTooltip(LOAD_ABILITY, 0)
        BlzSetAbilityTooltip(LOAD_ABILITY, LOAD_EMPTY_KEY, 0)
        Preloader(filename)
        local loaded = BlzGetAbilityTooltip(LOAD_ABILITY, 0)
        BlzSetAbilityTooltip(LOAD_ABILITY, s, 0)
        if loaded == LOAD_EMPTY_KEY then
            return nil
        end
        return loaded
    end

    ---@param filename string
    ---@param data string
    ---@param onFail function?
    ---@return boolean
    local function saveAsserted(filename, data, onFail)
        savefile(filename, data)
        local res = loadfile(filename)
        if res == data then
            return true
        end
        if onFail then
            onFail(filename, data, res)
        end
        return false
    end

    local fileIO_enabled = saveAsserted('TestFileIO.pld', 'FileIO is Enabled')

    FileIO = {
        Save = savefile,
        Load = loadfile,
        SaveAsserted = saveAsserted,
        enabled = fileIO_enabled,
    }
end)
if Debug then Debug.endFile() end

Spoiler: Example usage

FileIO.Save('test.pld', 'Hello ' .. 'World!')
local res = FileIO.Load('test.pld')
print(res)

function FileIO_StressTest(len)
    local s = {}
    for i=1, len do
        s[i] = ('%d'):format(GetRandomInt(0,9))
    end
 
    FileIO.SaveAsserted(
        ('teststring %d.pld'):format(len),
        table.concat(s),
        function(file, expected, actual)
            Debug.log(file, '<', expected)
            Debug.log(file, '>', actual)
            assert(expected == actual, "FileIO stresstest failed!")
        end
    )
end
FileIO_StressTest(10)
FileIO_StressTest(1024)
FileIO_StressTest(999999)

Spoiler: Example resulting preload file

function PreloadFiles takes nothing returns nothing

    call PreloadStart()
    call Preload( "")
endfunction
//!beginusercode
local p={}local i=function(s)table.insert(p,s)end--[[" )
    call Preload( "]]i[[FileIO is Enabled]]--[[" )
    call Preload( "]]BlzSetAbilityTooltip(1095656547,table.concat(p),0)
//!endusercode
function a takes nothing returns nothing
//" )
    call PreloadEnd( 3233451.8 )

endfunction

Optional requirements:

• [Lua] - Debug Utils (Ingame Console etc.) by @Eikonium
• [Lua] - Total Initialization by @Bribe (remove/replace OnInit if you don't use it)

Inspired by:

• @ScrewTheTrees's [Lua/Typescript] "Codeless" Saving/Synchronized loading of huge amounts of data.
• @TriggerHappy's [vJASS] - FileIO & Codeless Save and Load (Multiplayer) - v3.0.1
• @Luashine's LUA variant of TH's FileIO
• @HerlySQR's LUA variant of TH's Codeless Save and Load

 

[Eikonium]

Good stuff!
Just to mention, @Luashine has published a Lua-ported version of TriggerHappy's FileIO library here.
It's hard to find though, because she didn't create a dedicated thread for it. Might be worth to discuss pro's and con's.

 

[Trokkin]

Aye, and there is also one by @HerlySQR. I'll add them to inspirations just to mention all the work that has been done on the topic. Too bad I didn't post my stuff here back in the days
I'll do a benchmark between mine and these two some time later, but now I'm busy with the rest of save/load process.

 

[HerlySQR]

Question, I see that you are using 1 ability instead of various, so there is no string length limit or what?

 

[Trokkin]

Wrote that in the description, yeah. Turns out it is so, at least with Lua. I've done stresstesting and it is able to load hundred kilobytes of a string in one go -- well, as far as I trust my test-writing skills. See Example Usage script, try running it yourself

 

[Wrda]

Feels like a "fileExists" function would make sense for me to exist, so that we can load it later or try another file.
Pun intended by the way.

 

[Trokkin]

Hm, I should add a nil return in case Load doesn't find a file, since now it returns ANdc tooltip -- updated that.

Then a FileExists function would be as easy as checking if the Load returned something that isn't nil. Though I don't thing such a function is a good idea since that discards data, and can lead to structures like if(FileIO.FileExists(file)) then DoSomething(FileIO.Load(file)) end. which is not efficient at all.

 

[Arvin.Ma]

I need help !

FileIO.Save('test.pld', 'Hello ' .. 'World!')
local res = FileIO.Load('test.pld')
print(res)
FileIO.Save('test.pld', 'Hello ' .. 'World123!')
res = FileIO.Load('test.pld')
print(res)

Why output 2 times "Hello World!" ???
i think the seq is

First: output => "Hello World!"
Second: output => "Hello World123!"

 

[Arvin.Ma]

@Trokkin

 

[Trokkin]

Why output 2 times "Hello World!" ???
i think the seq is

First: output => "Hello World!"
Second: output => "Hello World123!"

It seems that during one game session the aggressive caching still applies. It shouldn't be a problem though; you only need to load any file exactly once per game, then you just manage the data from the code.

If you really want to override a file then read it again during the same game, you can implement a cache for saved file content, and check that cache first when loading. I won't do it though, it really is an unnecessary complication. Something like function save(name, content) cache[name] = content FileIO.Save(name, content) end and function load(name) if cache[name] == nil then cache[name] = FileIO.Load(name) end return cache[name] end would meet your expectations.