[General] Data Storage in MPQ

[Serenity09]

I'm trying to define a table that represents localization/translation of text content from my game across multiple languages. My current idea is that I can quickly get basic translations off of Google's translate service and then build up a table from that. The table will have the following columns:

ID,en,af,...,zu

And each row will represent a single piece of text, translated across the above languages, identified within my map by the ID column

I can translate the table into a representation that is compatible with a wc3 string, but then maximum length for wc strings will make this approach much harder than im looking for

So my next thought is placing a new file (ideally just a CSV representing the table) directly in the map MPQ and then somehow interacting with that via JASS. Are there any options along this path? Are there any other paths that might work better?

I'm really trying to avoid maintaining the localized content defined as a JASS representation (ie JASS hashtable) - it will be much easier to maintain a table of localized content externally and parse that into a JASS Table on map load

 

[Drake53]

maximum length for wc strings

This is a JASS limitation, not a warcraft III one. You can circumvent the issue by using lua. If lua is not an option for you, you could try working with war3map.wts as well.

 

[Serenity09]

This is a JASS limitation, not a warcraft III one. You can circumvent the issue by using lua. If lua is not an option for you, you could try working with war3map.wts as well.

That's interesting I didn't realize that, but yeah, far too late for lua for this map

I've realized that part of my goal is self defeating

it will be much easier to maintain a table of localized content externally and parse that into a JASS Table on map load

If i need to use JASS to parse the string, then i'll need to deal with the string being too large to represent in JASS eventually, which i already said i don't want to do

I'd probably rather write something external to preparse the table CSV into JASS Table syntax... but maybe another suggestion will come along...

 

[LeP]

You could try using GetLocalizedString. I think it reads from war3map.wts which you can either distribute different per map or maybe insert it into mpq under different locales. Don't know if it will work though. Also i dont know of any guide for this

 

[Serenity09]

You could try using GetLocalizedString. I think it reads from war3map.wts which you can either distribute different per map or maybe insert it into mpq under different locales. Don't know if it will work though. Also i dont know of any guide for this

That's a good suggestion, thanks. I'm having the hardest time finding anything on GetLocalizedString, or making it work for even a native entry, much less a non-native entry in the .wts file. Can anyone advise on the proper usage?

I've got two machines, one with an English install, one with French. The goal is to enable access to all the languages wc natively supports, but i like how French sounds, so that's where im starting. I get the same result for GetLocalizedString all the ways I've thought to call it

ideally i want to access a predefined entry via jass from the .wts file:

STRING 187
// Doodads: D00M (en), Name (Name)
{
en
}

then id add localizations for that string for all the languages wc3 supports

I've tried calling this string via the following, with no localization occurring for en -> fr

call DisplayTextToPlayer(GetLocalPlayer(), 0, 0, GetLocalizedString("STRING 187"))
call DisplayTextToPlayer(GetLocalPlayer(), 0, 0, GetLocalizedString("STRING_187"))
call DisplayTextToPlayer(GetLocalPlayer(), 0, 0, GetLocalizedString("en")) //just defaults to "en" due to implementation of GetLocalizedString
call DisplayTextToPlayer(GetLocalPlayer(), 0, 0, GetLocalizedString("D00M"))

This way i can push the localization codes into the wts file and not need to depend on Blizzard's specific translations of existing vanilla content. That said, I got frustrated with that approach, so I'm trying to use GetLocalizedString on a native string, like "Peon", and then comparing the result to determine language code, but that's not going well either

call DisplayTextToPlayer(GetLocalPlayer(), 0, 0, GetLocalizedString("Peon"))
call DisplayTextToPlayer(GetLocalPlayer(), 0, 0, GetLocalizedString("Knight"))
call DisplayTextToPlayer(GetLocalPlayer(), 0, 0, GetLocalizedString("Priest"))
call DisplayTextToPlayer(GetLocalPlayer(), 0, 0, GetLocalizedString("Archer"))

All calls return their English value when accessed on the French client. The French client otherwise shows French text for all vanilla WC text (such as default interface, and vanilla units like the Peon). I've tried running the above calls with both clients in the same game, and with just the French client in the game, and i still always get English values. I've also tried the above in a new map (created on the 1.31 editor) with the same result.

Any thoughts what I'm doing wrong?

 

[Dr Super Good]

One used to be able to use MPQ localization feature to do this. But apparently that no longer works.

 

[Serenity09]

One used to be able to use MPQ localization feature to do this. But apparently that no longer works.

Thanks for posting that, i was starting to get a bit crazy eyed

Okay so taking a step back, my original question was about storing the localization content - there are two options for representing localized data in wc's engine: Either in the .wts file, which hypothetically can be interacted with via GetLocalizedString, or embedded into script files
Neither option is great for the source of localization data, which will almost certainly be something simple and tabular, like a CSV. Trying to merge external content with the .wts file is a bum deal, so that leaves script files. This has been fine so far (for a 500kb localization table), but a fair bit heavier than .wts approach ought to be. In return, the JASS approach allows dynamic localization after game load, which is maybe kinda nice

I'm thinking that i'll use a HashTable to represent the data in JASS HashTable[ContentID].string[LanguageID] Will the memory used by these strings (the 500kb, not so much the Table overhead) be effected by flushing them when there's no chance of accessing them again? I'm wondering how that will effect the string cache and memory usage



The only thing i'm completely lacking at this point is access to a player's WC installation language via JASS. I don't need the .wts file to manage localized content at this point, I don't necessarily even need any of the built in localization functions, i just need a ID for each player's install language (ideally the conventional two letter code)

• I could almost get this using an approach which locally starts a localized ai file, then that file pushes the relevant language code into the local player's name, shown here -- except that this approach desyncs 5 minutes later for some reason
• GetLocalizedString is maybe the best option so far, except i've yet to see it work
• GetUnitName probably depends on GetLocalizedString, either way though, its currently just as big a mystery to me. Calling it on the French client returns an empty string for the Knight and Ghoul units, while it returns the English names of the English client

Any suggestions for this aspect?

 

[Pyrogasm]

Seems it would be simplest just to pop up a dialog when the map loads and ask the player which language they want to read.

 

[Serenity09]

Seems it would be simplest just to pop up a dialog when the map loads and ask the player which language they want to read.

That's a good idea, and I think that's probably the direction i should head until i get something programmatic that can reliably auto-detect install language. I think this is better than trying to hack something as much as i was for the ai option

 

[Dr Super Good]

I'm thinking that i'll use a HashTable to represent the data in JASS
HashTable[ContentID].string[LanguageID]
Will the memory used by these strings (the 500kb, not so much the Table overhead) be effected by flushing them when there's no chance of accessing them again? I'm wondering how that will effect the string cache and memory usage

Assuming one uses Lua tables the overhead would be trivial. A few MB of string constants is nothing and table access is near O(1) anyway.

 

[Serenity09]

I dont know how ive been doing so much on this and read the 500"kb" every time as "mb" up till right now, reading your post. I even wrote it as kb in my post first try, and i see its 300-500kb every single time i import it back to the map. Programming with a bad cold is wild man. Okay i feel a lot better about that scaling / staying in memory long-term. That's cool then, we're gonna have a table with a whole lot of shitty auto translations

I think i've got a direction for now. Thank you Drake, LeP, DrSuperGood and Pyrogasm for reading and helping. If anyone can correct the usage of GetLocalizedString(), please do, but otherwise I'll post back here if anything changes in 1.32. i think GetLocalizedString() is actually a good fit for auto detection, but who knows, maybe they'll add GetLanguageCode()

 

[Pyrogasm]

These threads might contain some nuggets of information you can build on:

Define more GetLocalizedHotkey strings?
What's the best way to provide a map with multiple languages?
GetWTSString

 

[Drake53]

For auto-detecting the locale, you can use a new native: BlzGetLocale. It returns "enUS" for me.

 

[Tasyen]

Since Warcraft 3 V1.30 the localization system of warcraft 3 was changed / improved.

With 1.31 one can create custom GetLocalizedStrings using fdf. UI: Reading a FDF.
Overwritting - Tips

 

[Serenity09]

These threads might contain some nuggets of information you can build on:

Define more GetLocalizedHotkey strings?
What's the best way to provide a map with multiple languages?
GetWTSString

Sweet, thank you for finding these! I didn't think to search for hotkeys or WTS strings. The GetLocalizedHotkey thread starts to mention the shift to fdf files to store localized strings, so it seems like this has been becoming a thing since 2016 somehow

For auto-detecting the locale, you can use a new native: BlzGetLocale. It returns "enUS" for me.

Hooooly shit, my man!! This is perfect! Any idea if blizzard has ever suggested that they'll support additional variants (en-US, en-BR, en-...) or just their chosen variant (en-US)? I'm approaching this assuming they'll only ever do the main variant

Since Warcraft 3 V1.30 the localization system of warcraft 3 was changed / improved.

With 1.31 one can create custom GetLocalizedStrings using fdf. UI: Reading a FDF.
Overwritting - Tips

Okay, cool, this is much more promising than the single .wts file was for storage, mainly because i could maintain my localization fdf files so the world editor never needs to edit them. Will you let me know how this alternative to the JASS HashTable sounds?
1. External tool gets localization table as CSV/json
2. External tool parses CSV/json into memory and iterates all included languages
3. A new file is created per language
4. Add the localization to the filename like how wc expects ({filename}#{localization}.fdf)
5. Iterate all content included in the iterating language
6. Use the key as the content ID and the value as the localized string

StringList {
    ContentID1 "Localization1"
    ContentID2 "Localization2"
    ...
}

7. Drag n drop all .fdf's into the mpq to include/update all localized content

To me, that sounds a bit nicer to maintain, and it might even perform better if it only loads the relevant localizations into memory based on the players on game load. That said, i don't actually really benefit or need either of those things at the moment, and the JASS HashTable is comparable in terms of enabling this feature's functionality. I think this is probably a good improvement to get to post release, if maintenance on localizations is becoming problematic, or if other people are interested in reusing this approach to localize wc content