wc3libs (Java)

[WaterKnight]

GitHub - inwc3/wc3libs: general java wc3libs

That one has been set up quite some time ago, no idea why I had not not made a hive thread yet. It is noticeable that there are a couple of libraries (also Java ones) for wc3 now, so probably should name it more uniquely at some point.

Anyway, the one I have initiated (and which is part of the inwarcraft collective) tries to be general purpose, preferably covering all of wc3's format and whatever may be useful in wc3 mapmaking and modding, which is of course a far but covetable goal, therefore sucks everything in and tries to layer-interface the applications as I lined out here a bit.

Nonetheless, there is already a range of features:

object mod files: w3a, w3u, ...
most slk files, a couple of txt files, profile, wts
mpq interface
terrain: w3e, wpm, shd, mmp, ...
placed objects: doo, w3c, w3r, ...
2d art: blp, tga, ...
3d art: mdx
audio: there seems like scarcely support for audio but maybe eventually

Heed of a warning: there are still loads of bugs/rudimentary things and few tests. This cannot compete with the subtle idosyncracies that were figured out over time like with BLP. These libraries mostly grew by application: implementation by demand. So still a lot of improvement potential. Complaints are welcome.

 

[Deleted member 219079]

What specs did you use? Any C/C++ equivalents you know of?

 

[HappyTauren]

Complaints are welcome.

Your face is ugly, I hope you die.

I have yet to explore this, I knew it existed for a while but I never got around to working with it. I am mostly interested in the w3e manipulation for now, since JNGP is broken for the newest version of wc3.

 

[WaterKnight]

What specs did you use? Any C/C++ equivalents you know of?

Mostly W3M and W3X Files Format

TGA: TGA format specification
BLP: Magos
MDX: Magos/GhostWolf

and of course from what I could figure out myself while writing the code and what Wc3 exposes. Was planning on adding the specifications to the repository at some point.

It currently uses JMPQ3 as mpq editor but once also used Ladik's, therefore wrote some interface.

Barade has written C++ stuff.

 

[Frotty]

You could mention that this is used in wurst and easily usable as dependency in any manager thanks to gradle-fication by yours truly .
Adding some up to date specs to a git repo sounds good, maybe outside of the lib though? There are too many different versions flying around tbh.

In any case, great stuff - looking forward to future wurst integrations (vscode asset previewer?)

Also, latest commit doesn't succeed, please fix

 

[GhostWolf]

There are too many versions flying around, so make another one.

Sarcasm aside, consider collaborating with eejin. Home · stijnherfst/HiveWE Wiki · GitHub

Still need to update some fixes for w3e, like removing those silly 0x2000, 0x200, and constant water levels.

 

[Frotty]

Most if not all other jvm compatible libs are incomplete, not maintained or inside another project (therefore not easily usable as dependency) and usually untested.
Just like jmpq3, the goal of this lib is to be an easy, continuously integrated and unit tested, plug and play solution for any jvm project that wishes to interface with wc3 data formats.
Barade's and eejin's stuff is very nice, but not in jvm language. I think the most useful collaboration would be a shared documentation repo like jassdoc with decisive specs about the wc3 data formats.

 

[GhostWolf]

I am talking about specs, which have nothing to do with Java. We are currently looking into terrain rendering, and have figured newer information. If you want to write your own, and stay out of the loop (e.g. by avoiding the Hive discord), that's up to you.
Honestly though, Frotty, I am not directing this at you, because so far you have shown zero will to actually even try and collaborate on anything, rather than just saying everything is bad because it doesn't fit your language of choice (which again, isn't even relevant at all to data/logic specs).

 

[WaterKnight]

Nah, I would write it up anew with some chic uniformity according to current implementation since I know there are flaws here and there but you can still insert the links to other specs.

I like to learn new stuff, so hit me with insights.

@Frotty: need wc3 installation on travis for integration tests

 

[Frotty]

@WaterKnight I don't think we can upload all wc3 binary files for a staging environment though :/ Maybe some mock interface?

 

[GhostWolf]

I get it now. You are a troll. Consistent messages that keep describing all of the things you do, like opening up your own private chat servers or "curated forums", wanting people to exchange information in private, and sharing exactly 0 information when asked in the past (you did give me some sliver of hope with the map header), all the while calling people names as if that gives you some moral high ground, and yet you talk about unification. I really cannot see any other reasoning beside a troll.

Regardless, choose whichever path you want, you don't have to collaborate with anyone outside of your small circle, It's just so absurd that you then open a thread like the "unification" one.

Have a good one.

 

[HappyTauren]

why the fuck are you two even fighting

 

[Barade]

Very useful. I would have hoped for help for my wc3lib project but I understand that not everyone wants to use C++.

 

[WaterKnight]

Your site https://wc3lib.org/ seems to be down, by the way.

 

[Barade]

I am not using the domain/server anymore. It's all on GitHub only now. I will add a README.md file soon the repository.

 

[WaterKnight]

Currently thinking about converting everything to kotlin. Any objections? The architecture will receive some modifications. For example, I started to write dedicated reader/writer classes in order to boost IO flexibility. We thought of supporting alternative custom formats like having binary files in plain JSON for easier usage and storing meta information.

 

[Dr Super Good]

BLP reader is wrong. Why do people keep trying to reinvent the wheel when I already have a good Java BLP0/1 API. Mine even uses the standard IIO API...

BLP2 supports BLPs larger than 512 in any dimension as it is used by WoW. WC3 supports BLP files larger than 512 in any dimension since the 2016 patches, but from such files it will only use mipmap levels less than 512 in any dimension. Older versions of WC3 (do not care) and the WC3 beta only supported BLP files of resolutions below 512 in any dimension.

Alpha support is also wrong. BLP Indexed colour content supports 0, 1, 4 and 8 bit alpha. Yes the field is alpha bits and not a boolean.

 

[Frotty]

BLP reader is wrong. Why do people keep trying to reinvent the wheel when I already have a good Java BLP0/1 API. Mine even uses the standard IIO API...

BLP2 supports BLPs larger than 512 in any dimension as it is used by WoW. WC3 supports BLP files larger than 512 in any dimension since the 2016 patches, but from such files it will only use mipmap levels less than 512 in any dimension. Older versions of WC3 (do not care) and the WC3 beta only supported BLP files of resolutions below 512 in any dimension.

Alpha support is also wrong. BLP Indexed colour content supports 0, 1, 4 and 8 bit alpha. Yes the field is alpha bits and not a boolean.

Mainly because it's not usable as maven/gradle dependency, at least for me. Also iirc for a logn time you didnt have/update this repo.
But yea, otherwise looks cool. Can you make it a gradle project and remove eclipse specific files?

 

[Dr Super Good]

Also iirc for a logn time you didnt have/update this repo.

I updated it a week ago. How regularly do you expect it to be updated? Does it need to be updated that regularly? What is there even for me to update, I can only fix what people report...

Can you make it a gradle project and remove eclipse specific files?

Those are so the project can be mounted in Eclipse. I would imagine them being ignored if one uses another IDE.

I will look to moving it to gradle. Currently one can use it as a simple JAR dependency thanks to the SPI interfaces of Java and get most functionality. I will also look to add the Java 9 module file to it.

At the moment I am having a lot of problem with Eclipse's Java 9 support. The suggestion and syntax checker keep crashing lol.

 

[Frotty]

No, I just recall you saying that you had changes made that weren't published to the repo yet at some point. I saw that it got recently updated.

And no, importing a static jar defies the concept of using a Dependency manager in the first place. I would instead copy your code into wc3libs to use it. But having yours as a dependency available would of course be Ideal. If you want i can even PR a gradlefication of the repo

I also would suggest not making your project Java 9 dependant just yet, because it is hardly adopted so far and support is still lacking.

 

[WaterKnight]

BLP reader is wrong. Why do people keep trying to reinvent the wheel when I already have a good Java BLP0/1 API. Mine even uses the standard IIO API...

Well, guess that's an integration question mostly. To use code of others of course you first have to understand it and trust that it works as needed. The blacker the box, the more opaque the behavior, the more insecure it is and if it should be required to make adjustments, that does not really work on a resource that should be independent unless it has well defined and the exact extension points as required. At least I would like to have uniformity in this monolithic library and going to another language would probably mean a different implementation, anyway. Or you build a sophisticated binding/adapter. But more important is that there is a reference implementation/knowledge as you have accrued, then it's also easier to fan out correct variants that fit specific demands.

The library as a whole should have a policy/architecture/infrastructure for simple development and usage but as mentioned, next to default processes, it appears wise to increase flexibility to allow the replacement of modules and the user to conjoin alien code as necessary.

BLP2 supports BLPs larger than 512 in any dimension as it is used by WoW. WC3 supports BLP files larger than 512 in any dimension since the 2016 patches, but from such files it will only use mipmap levels less than 512 in any dimension. Older versions of WC3 (do not care) and the WC3 beta only supported BLP files of resolutions below 512 in any dimension.

Game versions are somewhat problematic to cover, yes, since the logic could actually be totally different. I think the library and tools should focus on the most recent version of the game because it would be even more cumbersome to test, account and fork the knowledge bases for multiple versions while the game is expected to be patched by everyone either way. But otherwise it makes sense to solve this via state machine, like setting some global library variable.

Alpha support is also wrong. BLP Indexed colour content supports 0, 1, 4 and 8 bit alpha. Yes the field is alpha bits and not a boolean.

The breakdown is already in. I do not think it's used correctly at the moment, however.

edit: Programming Wisdom on Twitter

 

[Dr Super Good]

No, I just recall you saying that you had changes made that weren't published to the repo yet at some point. I saw that it got recently updated.

Sorry there was some confusion. That was going to be for the Java MPQ library but I discarded them mostly in an attempt to rewrite the entire thing to use Asynchronous IO. That attempt has currently stalled for a variety of reasons.

I also did not make my BLP editor GUI open source because frankly the code is a huge joke. The thing even slightly broke with Java 9.

There are not and have never been outstanding changes for the BLP IIO library. I push them out pretty much the instant I have made them.

And no, importing a static jar defies the concept of using a Dependency manager in the first place. I would instead copy your code into wc3libs to use it. But having yours as a dependency available would of course be Ideal. If you want i can even PR a gradlefication of the repo

IIO uses the service provider interface. To read BLP images one does not even need to import a single class from the implementation. One could even place it in the class path and existing applications will suddenly support reading BLP images.

In my opinion it warrants being a separate library simply because it is a stand alone IIO plugin. Its use cases might not overlap with WC3 map components or models, so coupling it to them potentially results in dependency bloat.

I do want it to be a maven/gradle package just currently I am unsure how one goes about publishing them to be automatically linkable (so the build system fetches them automatically, similar to LWJGL). Eclipse can easily turn the project into Maven or Gradle and supports both via mature plugins, however turning that into a public resource is not too clear.

I also would suggest not making your project Java 9 dependant just yet, because it is hardly adopted so far and support is still lacking.

So when do you expect people to move to it? Java 10 is already scheduled for release in less than 2 months and Java 11 will be out in under 8 months. Oracle/Sun have moved Java to a bi-yearly major release schedule.

Many libraries such as LWJGL are already Java 9 compatible. I am not even sure if Java 9 compatibility breaks backwards compatibility unless explicit Java 9 features are used.

t least I would like to have uniformity in this monolithic library and going to another language would probably mean a different implementation, anyway.

But your library is Java and the IIO plugin is Java... What different language?

Game versions are somewhat problematic to cover, yes, since the logic could actually be totally different. I think the library and tools should focus on the most recent version of the game because it would be even more cumbersome to test, account and fork the knowledge bases for multiple versions while the game is expected to be patched by everyone either way. But otherwise it makes sense to solve this via state machine, like setting some global library variable.

Technically the BLP1 spec has not changed, just Blizzard changed their implementation of it in the 2016 patches so it is more robust. This includes the ability for it to load images larger than 512*512 pixels, which acted as an error condition before then. As far as I can see the current library treats BLP1 files larger than 512*512 as an error condition even though WC3 can still load them. You do not have to care about BLP0 as that is only used by the WC3 RoC beta which outside of researchers no one really cares about.

The resulting decoded image behaviour should be as close to the game as possible. Only differences should be where the game is clearly doing something stupid such as buffer overrun errors or choosing not to use mipmap levels with a dimension larger than 512. If the latest version of WC3 can load an image, the API should also be able to load it and the image should appear as close as possible the same.

The breakdown is already in. I do not think it's used correctly at the moment, however.

The code I saw decoded the alpha bits field as an "has alpha" boolean. I did glance over it so maybe I misread.

 

[WaterKnight]

But your library is Java and the IIO plugin is Java... What different language?

Nah, was just referring to "reinventing the wheel". Switching to another language like if someone wanted it in C would make for an obvious reason to reimplement. The other reasons I mentioned like uniformity, integration, use case are somewhat more subtle. And the question still stands with kotlin, which ofc offers direct interop...

The resulting decoded image behaviour should be as close to the game as possible. Only differences should be where the game is clearly doing something stupid such as buffer overrun errors or choosing not to use mipmap levels with a dimension larger than 512. If the latest version of WC3 can load an image, the API should also be able to load it and the image should appear as close as possible the same.

Agreed. Wc3 libs should foremost be tended to wc3. Though the language wc3 speaks is not necessarily the best one for development of wc3 contents, hence why artificial stuff like vJass or wurst came to life and why I addressed intermediate formats and emphasized infrastructure above.

The code I saw decoded the alpha bits field as an "has alpha" boolean. I did glance over it so maybe I misread.

You did not. I just flattened it to boolean:

hasAlpha = ((alphaBits & 0x8) > 0);

Magos specs were used, iirc. Can switch to yours later... Those 2d art classes also use JavaFX, not sure if we want to require it. And about concurrency, of course that will be the common case in GUI applications at least and probably all of that IO stuff should be parallelizable. But should it be inbuilt with callbacks/consumers or are general methods sufficient? Ideally, the user would have full control and semantics while still simple.

 

[WaterKnight]

Please post more requests and shortcomings in Issues · inwc3/wc3libs · GitHub. I will consider the BLP stuff after 1.29 is out.

 

[Frotty]

inwc3/wc3libs has progressed quite a bit, @WaterKnight and me have been steadily adding features and fixing bugs. Most notably the object data transformation to and from SLK/Txt is now somewhat stable and we have been playtesting maps successfully. The Jass parser can now collect id references and the game's patch version can be determined. We also started to unit test most features and make the repository very easy to build.

inwc3/JMPQ3 also received some updates to improve handling of mpq anomalies, increasing the range of supported maps for modification.

As usual we welcome any contributions from the community - maybe @Retera or @Dr Super Good have some cool ideas?

 

[Dr Super Good]

The Jass parser can now collect id references and the game's patch version can be determined.

I am not sure what you mean by this feature? Does it generate some JASS code that uses handle ID numbers to determine the game patch version at map run time? Or is it something to do with World Edit interoperation for wurst detecting the version of World Edit?

maybe @Retera or @Dr Super Good have some cool ideas?

With 1.30 coming a CASC library will need to be created in order to access the main game data files. Maps will remain MPQ so JMPQ3 will still work for them unchanged by the patch.

I was going to look into the CASC library but currently my coding time is being spent elsewhere.

From preliminary research it might be enough to simply extract data from Blizzard app built local archives. This would give read only without list file support for the game data files which should be enough to extract assets such as textures and models. A more complete CASC library would be able to access the game data files directly through the internet with potentially its own local caching archive, however this would be a lot more complex and so is not really a priority.

 

[Frotty]

I am not sure what you mean by this feature? Does it generate some JASS code that uses handle ID numbers to determine the game patch version at map run time? Or is it something to do with World Edit interoperation for wurst detecting the version of World Edit?

No, just collecting integers from the script that represent ids to calculate used objects.

From preliminary research it might be enough to simply extract data from Blizzard app built local archives. This would give read only without list file support for the game data files which should be enough to extract assets such as textures and models. A more complete CASC library would be able to access the game data files directly through the internet with potentially its own local caching archive, however this would be a lot more complex and so is not really a priority.

Yeah, a small reader seems most reasonably for now.

 

[WaterKnight]

The Jass parser can now collect id references and the game's patch version can be determined. We also started to unit test most features and make the repository very easy to build.

I am not sure what you mean by this feature? Does it generate some JASS code that uses handle ID numbers to determine the game patch version at map run time? Or is it something to do with World Edit interoperation for wurst detecting the version of World Edit?

Those two things are totally unrelated.

You can use the jass parser/lexer to find all integer literals and those can be converted back to 4 letter ids to make an estimation what objects are used by the map script. We use that to whitelist required ids that the object merger must not delete, currently. I should also add a string variant, then.

We don't know the game version at runtime. For now, I would not know what information you could use to safely determine the game version ingame. I have just integrated a portable executable parser (Windows) to identify your game version from the game executables at build-time. So you basically know the target you build for automatically. No big feat.

But we are open to all kinds of API ideas in relation to Warcraft III. MPQ/CASC is another level, of course, with many rules that I am not aware of. Which is why I have been using Ladik's and am using JMPQ for now. With a complete specification, however, I wouldn't mind approaching that.

edit:

The big picture is to duplicate all of the game logic in conjunction with maps. I have been steadily mapping all those special files inside and outside of the MPQs. First, it's low-level access/mutators for all file types, then we have treatment for those hardcoded paths, common usages like merging pathing maps, relationships between different files like combining .w3e with .shd and higher structures. I have also tried to uniform and establish an infrastructure to make things easier to grasp, reduce own mistakes and provide better interoperability.

One thing I am currently looking into on a global level is to generate reader/writers for the individual file types from grammars because, frankly, it bloats and will multiply once I add other format variants like JSON or XML, more so error-/non-uniform-prone. I am not a fan of redundancy.