- Joined
- Jan 3, 2022
- Messages
- 364
I'm primarily concerned about Lua. I want more input or suggestions, because Preload is too slow for the amount of logs I wanted to generate.
Why did I overwrite the file with each new line? I think when the game desyncs, it stops the script execution completely so I can't have a line buffer with delayed flushing (i.e. keep lines in memory and only write the Preload files every 10 lines or so). I suppose it could be optimized with various tricks, but ultimately due to this limitation preload files will remain at O(n^2).
Hopefully anyone else has better ideas. Though I think the best there's are: optimized preload / memory dump extraction.
- DisplayTextToPlayer: there're only 16 lines on screen and it is shared with map messages. In singleplayer the lines are saved to journal (F12) but not in multiplayer - I'm debugging a desync issue. If they're saved to journal/printed at a high rate, they consume a lot of memory.
- BlzDisplayChatMessage: ~25 lines on screen, but still not enough (maybe more lines can be made visible if you move the UI frame). Not saved anywhere at all, even in multiplayer
- Preload files: the go-to method but pretty slow. Saving 1.5MB freezes the game for 1-3 seconds. Lines are limited in length. Post-processing needed.
- Game save files: iirc multiplayer saves were disabled for Lua maps. Potentially high memory overhead
- Somehow saving data to replay files. Replay files should only contain player input, so idk if it's possible. The file format must be parsed and is a separate nuissance.
- Process memory dumps: currently they're the only way to extract uncaught Lua error messages (especially when the map fails to load in lobby and script execution seems to stop). Something like storing a huge string in memory / lines in a table. Downsides: the simple approach with
cat -v | grep
orstrings | grep
is very slow, it takes minutes to finish a 1GB memory dump. Potential upsides: automatable - Hooking into process memory. What for exactly? Downside: hard, anti-cheat?
- Modifying game to fully enable Lua os and io libraries. Downside: hard, anti-cheat?
- clown method: paint colored squares on the screen, recognize the color and turn it into binary output data. You'd need to do screen recording and image recognition. Upsides: a streamable format that doesn't fill the game's memory. Downsides: clown
- Capturing network traffic that we cause through sync. iirc network traffic is now encrypted, impossible.
Why did I overwrite the file with each new line? I think when the game desyncs, it stops the script execution completely so I can't have a line buffer with delayed flushing (i.e. keep lines in memory and only write the Preload files every 10 lines or so). I suppose it could be optimized with various tricks, but ultimately due to this limitation preload files will remain at O(n^2).
Hopefully anyone else has better ideas. Though I think the best there's are: optimized preload / memory dump extraction.
Code:
Invalid UNC:
s=os.clock(); for i = 1, 10000 do Preload([[\\.\Line\]].. i ..[[.txt]]) end; e=os.clock()
-> 50ms
Invalid relative path:
s=os.clock(); for i = 1, 10000 do Preload([[somfilee-]].. i ..[[.txt]]) end; e=os.clock()
-> 140ms
--------
PreloadGenEnd:
-> 4.84s for 4.7MiB
So preload file saving has ~1 MiB/s speed
--------
1.2GB file (not cached): 50s - time strings -5 Warcraft\ III.dmp | grep war3map.lua
-> 24 MB/s
Last edited: