Java MDX lib

[oger-lord]

Java MDX lib + BLP

For my doodad merger I only had java classes for the basic mdx functions but now I parsed Magos complete Format definition http://www.wc3c.net/tools/specs/MagosMdxFormat.txt to java.
The result are 53 java classes for all the different data models can contain.

It can still contain mistakes but I tested it with some Heroes and Buildings and the saved output was identical with the input model.

Load a model

BlizzardDataInputStream in = new BlizzardDataInputStream(new FileInputStream(inputFile));
MdxModel model = StreamUtils.loadModel(in);

Save Model

BlizzardDataOutputStream out = new BlizzardDataOutputStream(outputFile);
model.save(out);
out.close();

Update
I moved the code the github and added features to write and load blp/tga files!
https://github.com/OgerLord/WcDataLibrary/

Download
Download Source (from github)

Credits go to magos for this format desciription and to gexxo for the Blizzard streams.

 

[GhostWolf]

There were some issues with Magos' specs that I don't quite remember now, check this for updated specs (you can ignore the sound part, since no model in existence uses it).

 

[Dr Super Good]

People never know how to write proper java I/O classes... Sigh.

1. Read to buffer, as much data as you are certain you will need.
2. Read from buffer in correct endian to native structures.

Reading as much as possible in single units avoids OS system calls while the buffer allocation overhead (if recycling not possible) is classed as usual JAVA overhead.
Using buffer operations for endian compiles to native code very efficiently and avoids messy ugly classes which re-invent the wheel.

 

[PurgeandFire]

Neat. But perhaps you should host this on some code repository? e.g. Github, code.google, or sourceforge, etc. So if you make updates, it is easily reflected (and you won't have to keep uploading a new rar each time you make a change).

 

[oger-lord]

People never know how to write proper java I/O classes... Sigh.

1. Read to buffer, as much data as you are certain you will need.
2. Read from buffer in correct endian to native structures.

Reading as much as possible in single units avoids OS system calls while the buffer allocation overhead (if recycling not possible) is classed as usual JAVA overhead.
Using buffer operations for endian compiles to native code very efficiently and avoids messy ugly classes which re-invent the wheel.

I am not quite sure what your problem is. The Blizzard streams use BufferedStreams and add the correct endian methods.
Does java already offer methods for all different endian types and stuff?

Neat. But perhaps you should host this on some code repository? e.g. Github, code.google, or sourceforge, etc. So if you make updates, it is easily reflected (and you won't have to keep uploading a new rar each time you make a change).

Maybe, but at the moment it is the easiest to upload the rar.

Made some changes:
-Variables are now written java style small
-Replaced unhandy char arrays with Strings
-Variables (in particular arrays) are now initialized

 

[Dr Super Good]

Does java already offer methods for all different endian types and stuff?

Yes. It is part of their nio package that was added a long time ago. You use the nio channels to read into buffers (ByteBuffer) and then use that to convert the data into native format. You can even use file mapping to do I/O using it but that is less useful (it performs well but not very convenient to use and has issues with freeing mapped files).

You can set the endian used for a ByteBuffer using "order(ByteOrder bo)" method. This is done at the lowest level allowed and most JVMs (especially Oracle's standard implementation) will optimize them very efficiently now.

ByteBuffers convert very efficiently into low level assembly when JIT compiled so perform the endian conversions very efficiently. Additionally you can read entire structures in a single call which saves on call overhead. Buffered streams might make some difference if you are reading a lot of irregularly shaped structures (tons of I/O of small sections of a file) however if you are reading an array of objects its better to self buffer and avoid unnecessary copy operations.

One possible implementation I used was to have two classes. One for native representation of the data and another as a buffer wrapper for I/O. Both could extend a standard interface of accessor and mutator methods so work interchangeably. Alternatively you could extend a base native implementation to add I/O to it. Generally it is best to avoid having I/O at the same time as the native implementation as that couples object creation to certain ways (maybe the objects could come from some other source in future?).

 

[BlinkBoy]

Update your library for GhostWolf's Spec. It's more complete:

MDX Specification

Also use 1 interfaces or abstract classes for Key Tracks that helps abstract their common behavior.

 

[Marchioly]

I had started on something like this like 2 days ago. I took an approach that was more like

public abstract class Chunk {

public abstract String identifier();

public int chunkSize;

public void decode(ByteBuffer buffer) {
chunkSize = buffer.getInt();
}

}

And then an example of the implementation would be like

public class VersionChunk extends Chunk {

public int version;

@Override
public String identifier() {
return "VERS";
}

@Override
public void decode(ByteBuffer buffer) {
super.decode(buffer);
this.version = buffer.getInt();
}

}

i havn't added functionality for re-saving to my system, because I am planning to use my code to load the models into a 3d java engine. (currently using jpct, I had planned on using a3d, but noticed that it was sorta dying).

P.S. yes, I know it has been ~ 2months since the last reply on this topic. However, this section isn't really that active

 

[oger-lord]

Because Crigges told me he had some difficulties to use this lib, here is an example how to create a Box:

Besides I added a Utils.java to this example that contains some useful methods, for example generating extents/normals of a geoset.

 

[oger-lord]

I added some java code to save/load blp (jpg & paletted) and tga files and moved all the stuff to github:
https://github.com/OgerLord/WcDataLibrary/

 

[oger-lord]

Today I found and fixed two bugs:
-Some third party tools generate custom chunks for models, for example Magos MdxLib create an INFO chunk. Now the library is able to ignore them.
-Special Doodads (for example the cliff cave) have a second list of texture coordinates for geosets.

 

[Ezekiel12]

Woh woh, could it be we are experiencing the birth of an successor to Magos' Editor here?

And it's Open Source .

 

[GhostWolf]

Today I found and fixed two bugs:
-Some third party tools generate custom chunks for models, for example Magos MdxLib create an INFO chunk. Now the library is able to ignore them.
-Special Doodads (for example the cliff cave) have a second list of texture coordinates for geosets.

Instead of ignoring unknown chunks, you might want to just read them as buffers, and write them when saving, just so you won't delete program-specific data (although I have no idea if anyone sticks anything worthwhile there).

Every geoset has N texture coordinate sets, where N is the uint32 after the UVAS identifier. They are indexed in the material layer's coordId member, if you follow Magos' naming (granted, 99.99% of the time there's one set and all of the indices are 0).

 

[oger-lord]

Woh woh, could it be we are experiencing the birth of an successor to Magos' Editor here?

At the moment its only a library for people who want to work with mdx models.
For example I am working on a shadow calculator and that means I need the model informations of doodads. (See attachment)

Instead of ignoring unknown chunks, you might want to just read them as buffers, and write them when saving, just so you won't delete program-specific data (although I have no idea if anyone sticks anything worthwhile there).

I can do that, but this feature does not have a high priority.

Every geoset has N texture coordinate sets, where N is the uint32 after the UVAS identifier. They are indexed in the material layer's coordId member, if you follow Magos' naming (granted, 99.99% of the time there's one set and all of the indices are 0).

That makes sense, thanks for the information.

 

[Kiwyh]

Hi !

I'm really interested by your project. What new features do you have in mind ?
Also, could you talk a bit about Blizzard I/O streams classes ? What is their purpose ?

I have in mind to create an M2 "extension" of your lib, and add some converting methods on the class to, for example, output an mdx model using your functions. Ripping WoW models would become really fast and easy.
I'm more used to C but I would like to expand my Java knowledge and I think this would be a good exercise. I like the way parts write themselves in OO, and under-the-hood optimizations. Anyway I think it may be easier to maintain than a C program.

When I'll have time, I think I may start this as a fork of your library.

 

[Dr Super Good]

He uses BlizzardDataInputStream as a basic data stream which is also buffered. Which is not very Java compliant.

The buffering should be provided by the source stream and not his deserialization stream (this avoids potential cases of unintended double buffering if the source stream is already buffered, eg reading from a MPQ file). Additionally it should automatically detect the MDX file object rather than relying on explicit creation. After all what is the point of having magic numbers to identify file type if you never use them?

I show off such a stream functionality with my W3E (Terrain Mesh) interface. There you can deserialize a TerrainMesh object much like you do with using Java's standard Serialization system. The stream could be extended to support many different object types.

 

[Kiwyh]

I/O will be the first thing I'll check I think, because lower layers should be reliable before playing with higher structures. For the Separation of Concerns I would say this should be a separate lib. Everyone should not be developing its own I/O classes to stream Blizzard files (provided they are as special and similar between them as I read).

I checked your source code (the files in /io at least) and I like your manners. It's rare to find properly documented code out there. I don't fully understand everything yet (some trouble with understanding the necessity of marshaling hints) but I will.

 

[Dr Super Good]

(some trouble with understanding the necessity of marshaling hints)

They are intended for hidden "features" of a file format. For example to undo possible "map protection" hacks or potentially insert hidden data (eg a signature) which is not part of the object format.

The need was considered after my experience with the MPQ format. MPQ archives have a version field which has been incremented over the years. D2 and WC3 use version 0, World of Warcraft used to use version 1 (changed to CASC) and StarCraft 2 uses and Diablo III used to use (changed to CASC) version 3. Some annoying pricks decided it would be "cool" to "protect" WC3 MPQ files by giving them a version number other than 0 (usually 1 since it started around the time of WoW which was the first non-0 version MPQ) resulting in a technically invalid MPQ archive (corrupted). WC3 treats all MPQs as version 0 since that is the only implementation it knows. However a proper MPQ library implementation should interpret the MPQ archive based on version as there are major feature and mechanical differences between the versions and the library should support all versions of MPQ. Opening one of these "protected" WC3 archives would obviously break the library as it would try and interpret it as not version 0 so load garbage. Hence the need for "hints" since in this case a hint could be implemented "MPQVer" or "LegacyMPQ" which overwrites the version field and would allow such "protected" archives to be loaded.

The use of hints is object dependent. Generally an object should not use any hints. They are only there for objects which need non-standard I/O behaviour at times (such as the MPQ example).

 

[Koward]

They are intended for hidden "features" of a file format. For example to undo possible "map protection" hacks or potentially insert hidden data (eg a signature) which is not part of the object format.

The need was considered after my experience with the MPQ format. MPQ archives have a version field which has been incremented over the years. D2 and WC3 use version 0, World of Warcraft used to use version 1 (changed to CASC) and StarCraft 2 uses and Diablo III used to use (changed to CASC) version 3. Some annoying pricks decided it would be "cool" to "protect" WC3 MPQ files by giving them a version number other than 0 (usually 1 since it started around the time of WoW which was the first non-0 version MPQ) resulting in a technically invalid MPQ archive (corrupted). WC3 treats all MPQs as version 0 since that is the only implementation it knows. However a proper MPQ library implementation should interpret the MPQ archive based on version as there are major feature and mechanical differences between the versions and the library should support all versions of MPQ. Opening one of these "protected" WC3 archives would obviously break the library as it would try and interpret it as not version 0 so load garbage. Hence the need for "hints" since in this case a hint could be implemented "MPQVer" or "LegacyMPQ" which overwrites the version field and would allow such "protected" archives to be loaded.
The use of hints is object dependent. Generally an object should not use any hints. They are only there for objects which need non-standard I/O behaviour at times (such as the MPQ example).

Okay, thanks (it's me I changed my account to uniformize my indentity with various other forums).
If I haven't missed anything, we could use your implementation by modifying only WC3Files.java and adding Magic Numbers and paths to classes.
Hmm, considering I have WoW models in mind I should make a WoWFiles.java also extending ObjectTypeProvider and there should be no problem as it's just for file recognition right ?

EDIT : I was expecting a WC3Files instance in TerrainMesh and there is none, so I guess in the end you did not used the automatic detection by magic number ?

EDIT2 : Is there a place where I can point a link to credit you for the code ? Like a Github or something ?

 

[Dr Super Good]

If I haven't missed anything, we could use your implementation by modifying only WC3Files.java and adding Magic Numbers and paths to classes.

That is intended for that package. Since it uses a service provider you could create another implementation (eg SC2Files, WoWFiles etc) in another package and it would automatically incorporate both. The idea was to make it fully extendable without having to change existing packages (reduced coupling).

Hmm, considering I have WoW models in mind I should make a WoWFiles.java also extending ObjectTypeProvider and there should be no problem as it's just for file recognition right ?

Yes, as long as its implementation is similar and meets the requirements it will be loaded thanks to the service provider API.

EDIT : I was expecting a WC3Files instance in TerrainMesh and there is none, so I guess in the end you did not used the automatic detection by magic number ?

One cannot link that way since the class is only loaded when needed and so if you try to demarshal an object the class will not have been loaded (no static initialization) so it will not have fed the service provider with its information and so the input stream will fail to resolve the object class.

In any case one cannot link the classes directly because that forces them to be loaded. If you have 100 odd object type classes that is considerable overhead when you may only want to load 1 object of a very specific type (eg TerrainMesh). To prevent this from happening the provider class and implementation classes cannot directly couple to each other as any form of direct coupling will cause the classes to be loaded. To work around this a service provider for Blizzard files has all the appropriate object classes hard-coded in with classes being referenced indirectly by fully qualified class name string rather than an actual class link. This class name string then gets resolved into an actual direct class reference as the magic number is processed.

End result is the object input stream will only load the object classes it needs, instead of all object classes it can provide. Same applies to the object output stream except that does not load any classes (instead it just resolves the appropriate magic number).

EDIT2 : Is there a place where I can point a link to credit you for the code ? Like a Github or something ?

I plan on eventually uploading it to GitHub, just been busy with Windows 10 migration and other programming recently.

Currently I am not too happy with how I have done area searches in the TerrainMesh object since looking back the area specification interface might not be that easily usable.

 

[Koward]

I would like to implement multiple WoW formats.
These have very different structures. The version number (int located right after the Magic "MD20") can be used to know how to handle the file.
If I'm right, the current WC3Utilities load a class based on the Magic string. The version should be handled during the unmarshaling. Considering the multiplicity of formats (1 per base game, so 6 so far), I would have naturally made subclasses but I can only map a magic number to a class.
I wonder what would be the most Java compliant solution to this problem. Do you have an idea ?

The only solution I could think of : the class WoWModel has an attribute "WoWModel son" and multiples classes WM1, WM2, ... , WM6 extends WoWModel. The unmarshal() method resolve the type and assign the object to the attribute. Each method() in WoWModel is then son.method().
But I think it's an aberration for the OO paradigm.

 

[Dr Super Good]

The idea with versions generally is you implement 1 class for the latest version and then use conditional tests during loading to only load the parts which existed in older versions (or in the order they existed). Having legacy members is allowed (members not used in newer versions of the format).

 

[Koward]

Thanks, I'll go with that.
I have now an interesting problem that is tied to the reason I'm joining the object oriented programming (aside the fact my teachers will not let me pass if I don't).
WoW model files are not chunked. There is a fixed size header then the typical way to read other data is revealed via a number and an offset. In C I handle that with dozen structures as there is no generic type. I hope Java can bring me an elegant solution.
Here is how far I am : http://pastebin.com/FyLNtq0W
Two things :
- The seek() method I'm using doesn't exist. I can't see any equivalent to move to a given offset in the file.
- The ??? are what I think is the hardest. Since WoW animations are multiple timelines instead of one timeline like in WC3, you can see things like

   array_ref<array_ref<Integer>> timestamps;
   array_ref<array_ref<T>> values;

And of course I would like to read recursively the inner elements, without knowing the type. I could during the loop read each element but I would need to call a given method on them. Let's say I define unmarshal() as the method, fine for the second layer, but the third will be standard Integer and that will be a problem. In C this is solved by the sizeOf() function, but I obviously can't use it here. That seems very tricky to me.
Without that I won't ever be able to convert an MD20 in an MDX.

 

[Dr Super Good]

There is a fixed size header then the typical way to read other data is revealed via a number and an offset.
- The seek() method I'm using doesn't exist. I can't see any equivalent to move to a given offset in the file.

Such a file would probably need a different I/O system then since it is not an object input stream. An object input stream does not support seeking and returns objects in the order they were written.

One work around could be to abstract seeking to above the object input stream. The header object is then used to seek the other objects and then you can read them with a special object input stream. In the case of M3 (SC2/HotS) this should work since each section is headed with a magic number.

This applies to C/C++ as well since you need to read in the data element by element. You cannot do the wrong approach of reading into a buffer and type casting to a struct due to structs not having a well defined memory model (only the elements in them do). Specifically structs are prone to padding (bitfields conflict with multi-thread memory models) and memory alignment.

Any file with seek support (random access) becomes more of an archive than a normal file. As such interaction with it should be more like a file system rather than a stream. For example MPQ/CASC would be implemented as a FileSystem service provider instead of as an object input stream since although it contains objects they are not sequential (a stream) and cannot be accessed sequentially (need some form of path).

Without that I won't ever be able to convert an MD20 in an MDX.

Conversion needs a converter anyway. Basically taking one or more objects of one games format and outputting one or more objects of the other games format which you can then bundle in the appropriate file.

 

[Koward]

Okay, I guess I'll have to rewrite the IO part of WC3Utilities. I know nothing about FileSystem at the moment but there is always a first time I suppose.

That will be tricky but I could learn a lot. I just need to, some day, fully understand I/O streams and be able to code something efficient and clean. It's a bit hard to find authoritative documentation on the subject but I guess sometimes you just have to experiment yourself.

EDIT : What could be wrong with http://docs.oracle.com/javase/8/docs/api/java/io/RandomAccessFile.html ? Seems closer to what I already used before.

 

[Dr Super Good]

Okay, I guess I'll have to rewrite the IO part of WC3Utilities.

Seeing how that was intended for WC3 and SC2 map files there is no reason to rewrite it?

That will be tricky but I could learn a lot. I just need to, some day, fully understand I/O streams and be able to code something efficient and clean. It's a bit hard to find authoritative documentation on the subject but I guess sometimes you just have to experiment yourself.

There is none and that is the problem. Ultimately you want to do something intuitive for I/O that matches closely what proper examples do. In the case of Java I came up with an OjectInputStream so it mimics normal deserialization.

A stream is basically a non-seekable sequence of bytes. You can get seekable streams for sources like files but that is additional stream functionality not part of a common stream.

EDIT : What could be wrong with http://docs.oracle.com/javase/8/docs...ccessFile.html ? Seems closer to what I already used before.

RandomAccessFile is intended for JAVA use only. It returns in big endian which will likely not work with WoW files seeing how Blizzard has a tendency for using little endian files. Hence why I wrote my own DataInput implementation with endian support.

Streams can skip bytes but are not required to be seekable as that implies buffering and ordering which might not exist. What one could do is have a higher level class do the seeking (object deals with Path objects instead of streams) and uses the header object to feed the seek operations and then reads in objects from a derived stream at that location.

In the case of MDL/MDX it would be most simple to treat it as an object stream and unmarshal all objects in order and link at the end. If this applies to WoW formats I do not know.

 

[GhostWolf]

What I/O? You need to parse a binary buffer (both M2 and M3 use references, but I also don't see how it's any nicer to do otherwise for MDX which is sequential)...yes, RandomAccessFile is what you want, except it's big endian, plus Java doesn't support unsigned integers, truly a lovely language!

Here's something that might help you.

The amusing part is that if you write your own class that simply reads bytes, and you operate on them, it might actually come out simpler than using RandomAccessFile.
It's a simple matter of reading the file into a binary buffer (an array of bytes), have an index into this array, and make your own read/write methods that move the index around and operate on the bytes.

Luckily for you, I don't think there is any instance in any of the Blizzard file formats where unsigned integers are actually required, I only remember them using 0xFFFFFFFF for invalid IDs and the like, which you can equivalently use -1 with signed integers.
I don't think any bitmasks actually used all of the bits, so they should be safe.

Writing this in pretty much anything other than Java would be easier, but to everyone their choices of terrible languages.

This applies to C/C++ as well since you need to read in the data element by element. You cannot do the wrong approach of reading into a buffer and type casting to a struct due to structs not having a well defined memory model (only the elements in them do). Specifically structs are prone to padding (bitfields conflict with multi-thread memory models) and memory alignment.

You can keep writing that as you always do, and yet everyone in the world will still use a simple pragma pack and everything will work as intended. But you know, so what if all of the C++ code bases ever use it for I/O and networking, it must not work properly because you said so again and again and again and again...

 

[Dr Super Good]

yes, RandomAccessFile is what you want, except it's big endian, plus Java doesn't support unsigned integers, truly a lovely language!

Which is why I wrote a custom endian based input stream. Like an image input stream except an actual stream! I plan to create a proper OrderedDataInputStream and its output equivalent at some stage.

Lack of unsigned integer support being a problem is a fallacy. Most integer operations work the same on both signed and unsigned. The only real exception are multiply/divide and comparisons of which you can quickly hack a typecast to long (via a 32bit mask) to fix.

The amusing part is that if you write your own class that simply reads bytes, and you operate on them, it might actually come out simpler than using RandomAccessFile.
It's a simple matter of reading the file into a binary buffer (an array of bytes), have an index into this array, and make your own read/write methods that move the index around and operate on the bytes.

I implemented it by a constant 8 byte ByteBuffer which I write the appropriate number of bytes to (via its backing byte array) and read out the appropriate type. The byte buffer handles endian conversion and should do so very efficiently after JIT optimizations.

I don't think any bitmasks actually used all of the bits, so they should be safe.

Bitmasks do note care if signed or unsigned as they work on bits. Only care has to be taken to use >>> (unsigned bitwise right shift) instead of >> (signed bitwise right shift) but even then you can fix any problems with proper use of masks.

Writing this in pretty much anything other than Java would be easier, but to everyone their choices of terrible languages.

C/C++ is harder because there is no well defined endian flags. You also cannot do the stupid buffer typecast to struct approach because that is not portable so in the end you are forced to read in from a stream just like Java.

You can keep writing that as you always do, and yet everyone in the world will still use a simple pragma pack and everything will work as intended. But you know, so what if all of the C++ code bases ever use it for I/O and networking, it must not work properly because you said so again and again and again and again...

Which is still compiler dependent. One can get away with it in some compilers, especially aimed at microcontrollers, since they have very strict struct packing memory model. You cannot get away with it on many high performance compilers because packing often conflicts with the multi-thread memory model requirements. One day C/C++ might add proper well-defined byte packing as it is a heavily requested however the people requesting it usually are driver developers as generally software engineers do not care.

 

[Retera]

I have been using this Java MDX Lib extensively in conjunction with my Matrix Eater 3D editing software, and there appears to be some bugs in how it parses Cameras. While it might load and save them accurately (my tests weren't focused on that) the object oriented representation is wrong. It splits Portrait model cameras into multiple Camera objects with names like "??????" and other garbage values when they have Rotation/Scaling chunks or some other unexpected formatting of some kind (it's been a little while since I deeply investigated the reasoning behind why it happens).

So, while I'm not 100% sure that this creates an immediately evident bug, and some of the bugs might be in my translator code, I wrote Java code to convert the MdxModel class to an MDL class used in the Matrix Eater and this in-memory swap produces numerous broken camera chunks in the MDL.

Here is an example (Footman's Portrait):

MDLX Converter 1.04 Interpretation of Camera

Camera "Camera02" {
	Position { 40.179901, 194.434006, 88.611298 },
	Translation 2 {
		Hermite,
		117033: { 0.000000, 0.000000, 0.000000 },
			InTan { 0.000000, 0.000000, 0.000000 },
			OutTan { 0.000000, 0.000000, 0.000000 },
		119967: { 0.000000, 0.000000, 0.000000 },
			InTan { 0.000000, 0.000000, 0.000000 },
			OutTan { 0.000000, 0.000000, 0.000000 },
	}
	Rotation 2 {
		Linear,
		117033: 0.000000,
		119967: 0.000000,
	}
	FieldOfView 0.785398,
	FarClip 1000.000000,
	NearClip 8.000000,
	Target {
		Position { 8.254510, 214.873993, 90.070602 },
		Translation 3 {
			Hermite,
			0: { 0.000000, 0.000000, 0.000000 },
				InTan { 0.000000, 0.000000, 0.000000 },
				OutTan { 0.000000, 0.000000, 0.000000 },
			117033: { 0.000000, 0.000000, 0.000000 },
				InTan { 0.000000, 0.000000, 0.000000 },
				OutTan { 0.000000, 0.000000, 0.000000 },
			119967: { 0.000000, 0.000000, 0.000000 },
				InTan { 0.000000, 0.000000, 0.000000 },
				OutTan { 0.000000, 0.000000, 0.000000 },
		}
	}
}

YobGul's File Converter Interpretation (uses hacked Game.dll, Blizzard's Code, I think?)

Camera "Camera02" {
	Position { 40.1799, 194.434, 88.6113 },
	Translation 3 {
		Hermite,
		0: { 0, 0, 0 },
			InTan { 0, 0, 0 },
			OutTan { 0, 0, 0 },
		117033: { 0, 0, 0 },
			InTan { 0, 0, 0 },
			OutTan { 0, 0, 0 },
		119967: { 0, 0, 0 },
			InTan { 0, 0, 0 },
			OutTan { 0, 0, 0 },
	}
	Rotation 2 {
		Linear,
		117033: 0,
		119967: 0,
	}
	FieldOfView 0.785398,
	FarClip 1000,
	NearClip 8,
	Target {
		Position { 8.25451, 214.874, 90.0706 },
		Translation 2 {
			Hermite,
			117033: { 0, 0, 0 },
				InTan { 0, 0, 0 },
				OutTan { 0, 0, 0 },
			119967: { 0, 0, 0 },
				InTan { 0, 0, 0 },
				OutTan { 0, 0, 0 },
		}
	}
}

Java MDX Lib converted to Matrix Eater MDL Output

Camera "Camera02" {
	Position { 40.179901123046875, 194.4340057373047, 88.61129760742188 },
	Translation 3 {
		Hermite,
		0: { 0, 0, 0 },
			InTan { 0, 0, 0 },
			OutTan { 0, 0, 0 },
		117033: { 0, 0, 0 },
			InTan { 0, 0, 0 },
			OutTan { 0, 0, 0 },
		119967: { 0, 0, 0 },
			InTan { 0, 0, 0 },
			OutTan { 0, 0, 0 },
	}
	Rotation 2 {
		Linear,
		117033: 0,
		119967: 0,
	}
	FieldOfView 1.061752792e+009,
	FarClip 1.14884608e+009,
	NearClip 1.09051904e+009,
	Target {
		Position { 8.254509925842285, 214.87399291992188, 90.07060241699219 },
	}
}
Camera "" {
	Position { 0, 0, 0 },
	FieldOfView 0,
	FarClip 0,
	NearClip 0,
	Target {
		Position { 0, 0, 0 },
	}
}
Camera "????????????????????????????????????????????????????????????????????????????????" {
	Position { 0, 0, 0 },
	FieldOfView 0,
	FarClip 0,
	NearClip 0,
	Target {
		Position { 0, 0, 0 },
	}
}

What do you think it would take for us to get this bug fixed? Thanks!

 

[GhostWolf]

There is a class for camera rotation keyframes, but it isn't actually used, as far as I can see.
Aside from that, there are also type mismatches, e.g. the field of view and clipping planes are read as integers instead of floats.

This should fix the above, assuming the rest of the code is working:

Hidden

package de.wc3data.mdx;

import de.wc3data.stream.StreamUtils;
import java.io.IOException;
import java.util.*;

public class CameraChunk {
	public Camera[] camera = new Camera[0];

	public static final String key = "CAMS";

	public void load(BlizzardDataInputStream in) throws IOException {
		StreamUtils.checkId(in, "CAMS");
		int chunkSize = in.readInt();
		List<Camera> cameraList = new ArrayList();
		int cameraCounter = chunkSize;
		while (cameraCounter > 0) {
			Camera tempcamera = new Camera();
			cameraList.add(tempcamera);
			tempcamera.load(in);
			cameraCounter -= tempcamera.getSize();
		}
		camera = cameraList.toArray(new Camera[cameraList.size()]);
	}

	public void save(BlizzardDataOutputStream out) throws IOException {
		int nrOfCameras = camera.length;
		out.writeNByteString("CAMS", 4);
		out.writeInt(getSize() - 8);// ChunkSize
		for (int i = 0; i < camera.length; i++) {
			camera[i].save(out);
		}

	}

	public int getSize() {
		int a = 0;
		a += 4;
		a += 4;
		for (int i = 0; i < camera.length; i++) {
			a += camera[i].getSize();
		}

		return a;
	}

	public class Camera {
		public String name = "";
		public float[] position = new float[3];
		public float fieldOfView;
		public float farClippingPlane;
		public float nearClippingPlane;
		public float[] targetPosition = new float[3];
		public CameraPositionTranslation cameraPositionTranslation;
		public CameraTargetTranslation cameraTargetTranslation;
		public CameraRotation cameraRotation;
		
		public void load(BlizzardDataInputStream in) throws IOException {
			int inclusiveSize = in.readInt();
			name = in.readCharsAsString(80);
			position = StreamUtils.loadFloatArray(in, 3);
			fieldOfView = in.readFloat();
			farClippingPlane = in.readFloat();
			nearClippingPlane = in.readFloat();
			targetPosition = StreamUtils.loadFloatArray(in, 3);
			for (int i = 0; i < 3; i++) {
				if (StreamUtils.checkOptionalId(in, cameraPositionTranslation.key)) {
					cameraPositionTranslation = new CameraPositionTranslation();
					cameraPositionTranslation.load(in);
				} else if (StreamUtils.checkOptionalId(in,
						cameraTargetTranslation.key)) {
					cameraTargetTranslation = new CameraTargetTranslation();
					cameraTargetTranslation.load(in);
				} else if (StreamUtils.checkOptionalId(in, cameraRotation.key)) {
					cameraRotation = new CameraRotation();
					cameraRotation.load(in);
				}

			}
		}

		public void save(BlizzardDataOutputStream out) throws IOException {
			out.writeInt(getSize());// InclusiveSize
			out.writeNByteString(name, 80);
			if (position.length % 3 != 0) {
				throw new IllegalArgumentException(
						"The array position needs either the length 3 or a multiple of this number. (got "
								+ position.length + ")");
			}
			StreamUtils.saveFloatArray(out, position);
			out.writeFloat(fieldOfView);
			out.writeFloat(farClippingPlane);
			out.writeFloat(nearClippingPlane);
			if (targetPosition.length % 3 != 0) {
				throw new IllegalArgumentException(
						"The array targetPosition needs either the length 3 or a multiple of this number. (got "
								+ targetPosition.length + ")");
			}
			StreamUtils.saveFloatArray(out, targetPosition);
			if (cameraPositionTranslation != null) {
				cameraPositionTranslation.save(out);
			}
			if (cameraTargetTranslation != null) {
				cameraTargetTranslation.save(out);
			}
			if (cameraRotation != null) {
				cameraRotation.save(out);
			}

		}

		public int getSize() {
			int a = 0;
			a += 4;
			a += 80;
			a += 12;
			a += 4;
			a += 4;
			a += 4;
			a += 12;
			if (cameraPositionTranslation != null) {
				a += cameraPositionTranslation.getSize();
			}
			if (cameraTargetTranslation != null) {
				a += cameraTargetTranslation.getSize();
			}
			if (cameraRotation != null) {
				a += cameraRotation.getSize();
			}

			return a;
		}
	}
}

 

[edo494]

Which is why I wrote a custom endian based input stream. Like an image input stream except an actual stream! I plan to create a proper OrderedDataInputStream and its output equivalent at some stage.

Lack of unsigned integer support being a problem is a fallacy. Most integer operations work the same on both signed and unsigned. The only real exception are multiply/divide and comparisons of which you can quickly hack a typecast to long (via a 32bit mask) to fix.


I implemented it by a constant 8 byte ByteBuffer which I write the appropriate number of bytes to (via its backing byte array) and read out the appropriate type. The byte buffer handles endian conversion and should do so very efficiently after JIT optimizations.


Bitmasks do note care if signed or unsigned as they work on bits. Only care has to be taken to use >>> (unsigned bitwise right shift) instead of >> (signed bitwise right shift) but even then you can fix any problems with proper use of masks.


C/C++ is harder because there is no well defined endian flags. You also cannot do the stupid buffer typecast to struct approach because that is not portable so in the end you are forced to read in from a stream just like Java.


Which is still compiler dependent. One can get away with it in some compilers, especially aimed at microcontrollers, since they have very strict struct packing memory model. You cannot get away with it on many high performance compilers because packing often conflicts with the multi-thread memory model requirements. One day C/C++ might add proper well-defined byte packing as it is a heavily requested however the people requesting it usually are driver developers as generally software engineers do not care.

knock knock, alignas

 

[looking_for_help]

C/C++ is harder because there is no well defined endian flags. You also cannot do the stupid buffer typecast to struct approach because that is not portable so in the end you are forced to read in from a stream just like Java.

Which is still compiler dependent. One can get away with it in some compilers, especially aimed at microcontrollers, since they have very strict struct packing memory model. You cannot get away with it on many high performance compilers because packing often conflicts with the multi-thread memory model requirements. One day C/C++ might add proper well-defined byte packing as it is a heavily requested however the people requesting it usually are driver developers as generally software engineers do not care.

This stuff can also be easily made portable with some basic template metaprogramming: Writing Quick Code in C++, Quickly (by Andrei Alexandrescu).

 

[Dr Super Good]

knock knock, alignas

Does not help. Individual packing of structure members is still implementation dependant on the compiler. I do not even think it is required that they be in the order declared, only that all the declared members have storage guarantees as specified.

This stuff can also be easily made portable with some basic template metaprogramming: Writing Quick Code in C++, Quickly (by Andrei Alexandrescu).

Where is the transcript?

 

[looking_for_help]

Where is the transcript?

Around min. 14 (its a video). A similar idea is described in this blog. With this you can even have overlapping "meta" fields which take zero size.

 

[Dr Super Good]

Around min. 15 of the video. A similar idea is described in this blog. With this you can even have overlapping "meta" fields which take zero size.

Again that needs C++11 which is still not yet universally adopted. Meanwhile a stream view does not suffer from readability problems and is still quite fast.

 

[looking_for_help]

Again that needs C++11 which is still not yet universally adopted.

Maybe, but you spoke about C++ in general:

One day C/C++ might add proper well-defined byte packing as it is a heavily requested however the people requesting it usually are driver developers as generally software engineers do not care.

Also this stuff can also be done in C++03 with some more effort (e.g. use template specializations instead of std::conditional). And that is really old already.

 

[edo494]

Again that needs C++11 which is still not yet universally adopted. Meanwhile a stream view does not suffer from readability problems and is still quite fast.

There are only 4 compilers that you can actually call compiler for C++ nowadays.

Visual studio, gcc, clang and Intel's compiler, all of them have almost full C++11 confrontance, so no, C++11 is not sparsely adopted, it is fully adopted by every modern compiler.

Hell, some of them are implementing C++17 stuff, and some stuff that probs will not even get to C++17 at this point.

And if you mean ancient gcc 3.4 then I advise stop using ancient technology, you dont wear 20 years old shoes, why should you use 20 years old compiler?(not literally 20)

 

[Dr Super Good]

Also this stuff can also be done in C++03 with some more effort (e.g. use template specializations instead of std::conditional). And that is really old already.

Old yet still not that well used (although it really should be).

In any case using streams to read in members is neat, readable and allows the compiler to optimize memory usage how it sees fit. Reason packed bitfields are avoided by compilers is because it results in an unclear memory model in multi-threaded environments which can lead to strange and hard to diagnose bugs.

Visual studio, gcc, clang and Intel's compiler

You forgot ARM and IBM compilers, which are also pretty well used.

It has nothing to do with compiler support, but rather people wanting to migrate. Code can be incompatible after migrating.

you dont wear 20 years old shoes

Shoes were nicer 20 years ago than today and preformed just as well, if not better than they do today.

why should you use 20 years old compiler?

You might be required to in order to keep minimum system requirement obligations.

 

[edo494]

If you think C++98 performed better than C++11 or C++14, you must not know C++11 and/or C++14.

Your adoption arguments are kind of silly and wrong.

I dont give a fuck if facebook uses C++98 or C++14 compatible compilers, I, as end user, only care for what the compilers support, and all of them have full C++11(visual misses two phase lookup) standard compilance.

This is offtopic regardless, since the resource at hand is made in Java.

You can compile with visual studio 2015 for Xp still, if someone uses something older than Xp, he doesnt deserve to get to use "my" application anyways, so fuck him

 

[looking_for_help]

Old yet still not that well used (although it really should be).

I don't get your point, sorry.

This might be true if your job is to maintain 20 year old legacy code. But this is not the case here, we are talking about a new application in 2015. If someone starts today writing some new program, he chooses of course modern tools, right?

In any case using streams to read in members is neat, readable and allows the compiler to optimize memory usage how it sees fit.

Maybe, but you are comparing Java streams with C++ bitfields and this is comparing apples and pears.

Reason packed bitfields are avoided by compilers is because it results in an unclear memory model in multi-threaded environments which can lead to strange and hard to diagnose bugs.

Not really. Of course you have to protect the entire memory location with some synchronization primitive like a mutex. But thats just like with any other variable. Bitfields just look like they are different variables (and with that located at different memory locations, which is not the case) and are therefore often missused. If you understand how they work (and a compiler does know these kind of things) there is really nothing special about using bitfields in a multi-threading environment.

You forgot ARM and IBM compilers, which are also pretty well used.

Both have C++11 support so the proposed method can be easily implemented on those.

It has nothing to do with compiler support, but rather people wanting to migrate. Code can be incompatible after migrating.

If you don't want to migrate you can't really complain about not having full access to new features.

I know this is a real problem in industry, but if you are bound to some specific compiler due to that reasons, you can as well just assume its memory model as "fixed". The main reason after all is that native bitfields are problematic when changing the compiler due to unportability. If you can't change the compiler because you must stay compatible to legacy code, this problem doesn't exist anyway.

 

[GhostWolf]

Please, enough of this DSG says something wrong, everyone corrects him, loop back...
Especially when it has nothing to do with the topic.

 

[Dr Super Good]

Maybe, but you are comparing Java streams with C++ bitfields and this is comparing apples and pears.

No I was comparing the buffer typecast to struct approach to IO which is not portable due to poorly defined struct packing as well as not supporting some file protocols very well.

Not really. Of course you have to protect the entire memory location with some synchronization primitive like a mutex. But thats just like with any other variable. Bitfields just look like they are different variables (and with that located at different memory locations, which is not the case) and are therefore often missused. If you understand how they work (and a compiler does know these kind of things) there is really nothing special about using bitfields in a multi-threading environment.

Most processors lack operators to atomically modify bits of a byte. As such bitfields can produce strange bugs in multi threaded environments which are hard to diagnose. For that reason many compilers like VC do not pack bitfields. Sure if one knew what they were doing with bitfields it is safe, but maybe the next guy to maintain the source will not be aware they are bitfields.

If you don't want to migrate you can't really complain about not having full access to new features.

I was saying that the standard migration can cause incompatibility with existing code. For example in the Simutrans open source project, the VisualC++ compiler of VS2015 was incompatible with the source code. The compiler has to be run in a special compiler mode to disable "sized destructors" functionality for the source to compile which makes it no longer standard compliant.

 

[edo494]

if you work on code and dont notice members are bitfields you are bad programmer, thats like not noticing return value of function, it is in the syntax itself

 

[Retera]

Instead of arguing about code.... Let's write some!
My model editing software is still hitting bugs in cameras, so I figured I would upload my entire Eclipse workspace for anyone interested to take a look.

A good example is the Fetch -> Unit command, and then if you go to Neutral and click on Ogre Warrior as shown in the picture below.
Once you open it, even after GhostWolf's suggestion, this library is still splitting the Ogre's camera into 2!

So if we could find a way to fix this that would be great!

Edit: As shown in the pictures below, the 'Edit/delete model components' is a good way to see a summary of what the Matrix Eater thinks is in the model file, and we can see it is broken for the Ogre Warrior Portrait!
Also, to run the program, go to the JWC3-MatrixEater project and locate com.matrixeater.src.MainFrame for the main method if you don't already have it available.

 

[Dr Super Good]

The MDX code is bugged. It is encountering a camera part which it has no logic to interpret. Not only is this causing data loss (parts of the camera are not parsed), it also causes the camera chunk to become corrupted as it incorrectly decrements the chunk size remaining (it thinks there are more cameras to parse).

I am looking into a fix suggestion now.

EDIT:
More correctly, it can encounter the following when parsing cameras.

KCTR: float[3] translation
KCRL: uint32 rotation
KTTR: float[3] targetTranslation

Currently it only has logic to parse KCTR and KTTR. If it encounters the KCRL part of a camera (which it does) it breaks.

EDIT 2:
Looks like the code you are using is different from the GitHub above. I am guessing you changed it?

The problem is in the CameraChunk.Camera.getSize method since you did not update the size to reflect the new part size.

It should be...

		public int getSize() {
			int a = 0;
			a += 4;
			a += 80;
			a += 12;
			a += 4;
			a += 4;
			a += 4;
			a += 12;
			if (cameraPositionTranslation != null) {
				a += cameraPositionTranslation.getSize();
			}
			if (cameraTargetTranslation != null) {
				a += cameraTargetTranslation.getSize();
			}
			if (cameraRotation != null) {
				a += cameraRotation.getSize();
			}

			return a;
		}

EDIT 3:
Still not fixed, I am guessing there is something else wrong with size metering. Sigh I wish people would write proper code.
EDIT 4:
Solved... Turns out the above does fix the problem. The reason it was still not fixed for me was I broke some other part by mistake.

 

[Retera]

Hmmm...
Well, thanks! That fix made SOMETHING better, I haven't tested the models in game to see if they're still bugged, but the Ogre Warrior Portrait now shows as having only one camera.

As for the differences between my code and the github, mine forked from it several months ago and has added functionality to translate into MDL in RAM, to another set of classes for representing the same data that I wrote in high school (and which also have their own bugs).

Anyway, this is some cool progress! Thanks!

 

[Koward]

Excuse me, I have a hard time trying to follow the thread.
Who maintains the most recent code here ? Retera ? Dr SuperGood ?
Is there a GitHub somewhere ? I would like to interface my beginning project of a java M2 (WoW) library (https://github.com/Koward/jM2lib) with an MDX one to promote easy conversion.

 

[Retera]

I am just some guy who is using the library written by oger-lord. I think what you are looking for is his GitHub:
https://github.com/OgerLord/WcDataLibrary

But, in a previous post, I noted that I have copied this code into a project of my own, and been using it. This was mostly because I added a lot into it, because I wrote an MDL library similar to his MDX library.
My own version was a copy from several months ago, and is not the official version. It is in the com.hiveworkshop.wc3.mdx package, which is different than the de.wc3data.mdx package used on the official version.

Perhaps I should offer to contribute the MDL to to oger-lord's repo, but I was convinced it was more of a hack than a solution. If the official MDX API wants to adopt MDL support, it would seem to me best to work it into the same classes, into the same API, than to have two separate APIs with a hack conversion between them.

tl;dr
Use the one on the OgerLord's GitHub, not my stuff.

Or use my stuff if you want hacky MDL support, but knowing full well that all kinds of stuff could be broken, out of date, or unsupported.

 

[Dr Super Good]

OgerLord's is buggy however. It will not parse CameraChunks correctly as we discussed above. It lacks the capability to parse rotation parts.

 

[Koward]

Okay. Then I should probably make my own mash-up with your WC3Utils as a basis, OgerLord classes for MDX parsing, fixing Cameras, and modifying WCUtils as using a NIO2 FSP if it's not already done (I don't remember).