MDX Specifications

Level 27
Joined
Jul 29, 2007
Messages
5,100
This is mostly based on Magos' MDX specifications, but is more accurate.
Many of the keyframe track types where found by BlinkBoy.

The notation of this specifications is as follows:

Chunk names will always be given in their ASCII representation.
For example, every MDX starts with the bytes 'M', 'D', 'L' and 'X', or the string "MDLX".
The MDX format uses this concept extensively, as a way to give chunks meaningfull names.
In your code, you will probably want to define constants for this, for example MDLX_CHUNK = 0x4d444c58, and read the tags as integers, but this is up to you.
The (X) notation means X is optional, and may or may not exist.
Flag variables hold different values in their bits with special meanings.
The notation for them would be the hexadecimal number representing the correct bit.
If for example there is 0x4, it means that the third bit holds specific information which you can get with bitwise operators.
That is, (Flag & 0x4) == 0x4 will show you if the value in the third bit is true or false.
Type variables hold only one value, and will simply be written using normal decimal numbers.


MDX is made out of many chunks with no predefined order, especially since they are all optional.

This is the main file structure:
C++:
MDX {
  char[4] "MDLX"
  (VERS)
  (MODL)
  (SEQS)
  (GLBS)
  (TEXS)
  (SNDS)
  (MTLS)
  (TXAN)
  (GEOS)
  (GEOA)
  (BONE)
  (LITE)
  (HELP)
  (ATCH)
  (PIVT)
  (PREM)
  (PRE2)
  (RIBB)
  (EVTS)
  (CAMS)
  (CLID)
  // The following chunks are for version > 800
  (BPOS)
  (FAFX)
  (CORN)
}

Each chunk starts with a header that consists of the chunk tag and the chunk size in bytes, not including the size of the header itself.
C++:
Header {
  char[4] tag
  uint32 size
}

To parse the file you would generally have a loop which keeps reading headers. If you recognize the header's tag and know how to parse it, do it, otherwise just skip the header's size bytes.
C++:
Reader reader = ... // Some sort of binary reader

if (reader.read(4) == "MDLX") {
  while (reader.remaining() > 0) {
    Header header = ... // read 8 bytes and create a header
    if (canHandleTag(header.tag)) {
      // handle it
    } else {
      // skip it
      reader.skip(header.size)
    }
  }
}

Most of the chunks hold arrays of objects.
In many of the cases you can't know how many objects there are before parsing them, in which case the size will be written as [?] below.
For example, each sequence is known to be 132 bytes. Since you know the size of the sequence chunk from the header, you therefore know that you have size/132 sequences.
On the other hand, materials can have variable sizes, so you must resize your array as you parse the chunk.
To allow you to parse chunks with variable sized objects, these chunks give you another size for each object they hold.
For example, every material object first starts with a uint32 size, which is the size of the material itself including this size variable, and so we will call it inclusiveSize.
So whenever you have a chunk with variable sized objects, you do something along these lines:
C++:
Something[] somethings
uint32 totalSize = 0

while (totalSize < size) {
  uint32 inclusiveSize = ... // read a uint32
  somethings.push(new Something(...)) // construct a new object and add it to the array
  totalSize += inclusiveSize
}

There are a couple of exceptions that have no inclusive size variable, and so are a bit more troublesome to handle, more on those later on.

With no further ado, the main chunks:
C++:
VERS {
  // 800 for Warcraft 3: RoC and TFT
  // >800 for Warcraft 3: Reforged
  uint32 version
}

MODL {
  char[80] name
  char[260] animationFileName
  Extent extent
  uint32 blendTime
}

SEQS {
  Sequence[size / 132] sequences
}

GLBS {
  uint32[size / 4] globalSequences
}

TEXS {
  Texture[size / 268] textures
}

// Note that this is here for completeness' sake.
// These objects were only used at some point before Warcraft 3 released.
SNDS {
  SoundTrack[size / 272] soundTracks
}

PIVT {
  float[size / 12][3] points
}

MTLS {
  Material[?] materials
}

TXAN {
  TextureAnimation[?] animations
}

GEOS {
  Geoset[?] geosets
}

GEOA {
  GeosetAnimation[?] animations
}

BONE {
  Bone[?] bones
}

LITE {
  Light[?] lights
}

HELP {
  Helper[?] helpers
}

ATCH {
  Attachment[?] attachments
}

// Emitters that emit models.
PREM {
  ParticleEmitter[?] emitters
}

// Emitters that emit quads.
PRE2 {
  ParticleEmitter2[?] emitters
}

// Emitters that emit lines, which are all connected together to form a ribbon of quads.
RIBB {
  RibbonEmitter[?] emitters
}

// A group of emitters that emit models, quads, and sounds.
EVTS {
  EventObject[?] objects
}

CAMS {
  Camera[?] cameras
}

CLID {
  CollisionShape[?] shapes
}

BPOS {
  uint32 count
  float[count][12] bindPose
}

// Face animations using the FaceFX runtime.
FAFX {
  char[80] target
  char[260] path
}

// Emitters that use the PopcornFX runtime.
CORN {
  CornEmitter[?] emitters
}

And now to the actual meat, the objects themselves.
C++:
Extent {
  float boundsRadius
  float[3] minimum
  float[3] maximum
}

Node {
  uint32 inclusiveSize
  char[80] name
  uint32 objectId
  uint32 parentId
  uint32 flags // 0x0: helper
               // 0x1: dont inherit translation
               // 0x2: dont inherit rotation
               // 0x4: dont inherit scaling
               // 0x8: billboarded
               // 0x10: billboarded lock x
               // 0x20: billboarded lock y
               // 0x40: billboarded lock z
               // 0x80: camera anchored
               // 0x100: bone
               // 0x200: light
               // 0x400 event object
               // 0x800: attachment
               // 0x1000 particle emitter
               // 0x2000: collision shape
               // 0x4000: ribbon emitter
               // 0x8000: if particle emitter: emitter uses mdl, if particle emitter 2: unshaded
               // 0x10000: if particle emitter: emitter uses tga, if particle emitter 2: sort primitives far z
               // 0x20000: line emitter
               // 0x40000: unfogged
               // 0x80000: model space
               // 0x100000: xy quad
  (KGTR)
  (KGRT)
  (KGSC)
}

Sequence {
  char[80] name
  uint32[2] interval
  float moveSpeed
  uint32 flags // 0: looping
               // 1: non looping
  float rarity
  uint32 syncPoint
  Extent extent
}

Texture {
  uint32 replaceableId
  char[260] fileName
  uint32 flags
}

SoundTrack {
  char[260] fileName
  float volume
  float pitch
  uint32 flags
}

Material {
  uint32 inclusiveSize
  uint32 priorityPlane
  uint32 flags

  if (version > 800) {
    char[80] shader
  }

  char[4] "LAYS"
  uint32 layersCount
  Layer[layersCount] layers
}

Layer {
  uint32 inclusiveSize
  uint32 filterMode // 0: none
                    // 1: transparent
                    // 2: blend
                    // 3: additive
                    // 4: add alpha
                    // 5: modulate
                    // 6: modulate 2x
  uint32 shadingFlags // 0x1: unshaded
                      // 0x2: sphere environment map
                      // 0x4: ?
                      // 0x8: ?
                      // 0x10: two sided
                      // 0x20: unfogged
                      // 0x40: no depth test
                      // 0x80: no depth set
  uint32 textureId
  uint32 textureAnimationId
  uint32 coordId
  float alpha

  if (version > 800) {
    float emissiveGain
    float[3] fresnelColor
    float fresnelOpacity
    float fresnelTeamColor
  }

  (KMTF)
  (KMTA)
  if (version > 800) {
    (KMTE)
  }
  if (version > 900) {
    (KFC3)
    (KFCA)
    (KFTC)
  }
}

TextureAnimation {
  uint32 inclusiveSize
  (KTAT)
  (KTAR)
  (KTAS)
}

Geoset {
  uint32 inclusiveSize
  char[4] "VRTX"
  uint32 vertexCount
  float[vertexCount * 3] vertexPositions
  char[4] "NRMS"
  uint32 normalCount
  float[normalCount * 3] vertexNormals
  char[4] "PTYP"
  uint32 faceTypeGroupsCount
  uint32[faceTypeGroupsCount] faceTypeGroups // 0: points
                                             // 1: lines
                                             // 2: line loop
                                             // 3: line strip
                                             // 4: triangles
                                             // 5: triangle strip
                                             // 6: triangle fan
                                             // 7: quads
                                             // 8: quad strip
                                             // 9: polygons
  char[4] "PCNT"
  uint32 faceGroupsCount
  uint32[faceGroupsCount] faceGroups
  char[4] "PVTX"
  uint32 facesCount
  uint16[facesCount] faces
  char[4] "GNDX"
  uint32 vertexGroupsCount
  uint8[vertexGroupsCount] vertexGroups
  char[4] "MTGC"
  uint32 matrixGroupsCount
  uint32[matrixGroupsCount] matrixGroups
  char[4] "MATS"
  uint32 matrixIndicesCount
  uint32[matrixIndicesCount] matrixIndices
  uint32 materialId
  uint32 selectionGroup
  uint32 selectionFlags

  if (version > 800) {
    uint32 lod
    char[80] lodName
  }

  Extent extent
  uint32 extentsCount
  Extent[extentsCount] sequenceExtents

  if (version > 800) {
    (Tangents)
    (Skin)
  }

  char[4] "UVAS"
  uint32 textureCoordinateSetsCount
  TextureCoordinateSet[textureCoordinateSetsCount] textureCoordinateSets
}

Tangents {
  char[4] "TANG"
  uint32 count
  float[count * 4] tangents
}

Skin {
  char[4] "SKIN"
  uint32 count
  uint8[count] skin
}

TextureCoordinateSet {
  char[4] "UVBS"
  uint32 count
  float[count * 2] texutreCoordinates
}

GeosetAnimation {
  uint32 inclusiveSize
  float alpha
  uint32 flags
  float[3] color
  uint32 geosetId
  (KGAO)
  (KGAC)
}

Bone {
  Node node
  uint32 geosetId
  uint32 geosetAnimationId
}

Light {
  uint32 inclusiveSize
  Node node
  uint32 type // 0: omni light
              // 1: directional light
              // 2: ambient light
  float attenuationStart
  float attenuationEnd
  float[3] color
  float intensity
  float[3] ambientColor
  float ambientIntensity
  (KLAS)
  (KLAE)
  (KLAC)
  (KLAI)
  (KLBI)
  (KLBC)
  (KLAV)
}

Helper {
  Node node
}

Attachment {
  uint32 inclusiveSize
  Node node
  char[260] path
  uint32 attachmentId
  (KATV)
}

ParticleEmitter {
  uint32 inclusiveSize
  Node node
  float emissionRate
  float gravity
  float longitude
  float latitude
  char[260] spawnModelFileName
  float lifespan
  float initialiVelocity
  (KPEE)
  (KPEG)
  (KPLN)
  (KPLT)
  (KPEL)
  (KPES)
  (KPEV)
}

ParticleEmitter2 {
  uint32 inclusiveSize
  Node node
  float speed
  float variation
  float latitude
  float gravity
  float lifespan
  float emissionRate
  float length
  float width
  uint32 filterMode // 0: blend
                    // 1: additive
                    // 2: modulate
                    // 3: modulate 2x
                    // 4: alpha key
  uint32 rows
  uint32 columns
  uint32 headOrTail // 0: head
                    // 1: tail
                    // 2: both
  float tailLength
  float time
  float[3][3] segmentColor
  uint8[3] segmentAlpha
  float[3] segmentScaling
  uint32[3] headInterval
  uint32[3] headDecayInterval
  uint32[3] tailInterval
  uint32[3] tailDecayInterval
  uint32 textureId
  uint32 squirt
  uint32 priorityPlane
  uint32 replaceableId
  (KP2S)
  (KP2R)
  (KP2L)
  (KP2G)
  (KP2E)
  (KP2N)
  (KP2W)
  (KP2V)
}

RibbonEmitter {
  uint32 inclusiveSize
  Node node
  float heightAbove
  float heightBelow
  float alpha
  float[3] color
  float lifespan
  uint32 textureSlot
  uint32 emissionRate
  uint32 rows
  uint32 columns
  uint32 materialId
  float gravity
  (KRHA)
  (KRHB)
  (KRAL)
  (KRCO)
  (KRTX)
  (KRVS)
}

EventObject {
  Node node
  char[4] "KEVT"
  uint32 tracksCount
  uint32 globalSequenceId
  uint32[tracksCount] tracks
}

Camera {
  uint32 inclusiveSize
  char[80] name
  float[3] position
  float filedOfView
  float farClippingPlane
  float nearClippingPlane
  float[3] targetPosition
  (KCTR)
  (KTTR)
  (KCRL)
}

CollisionShape {
  Node node
  uint32 type // 0: cube
              // 1: plane
              // 2: sphere
              // 3: cylinder
  float[?][3] vertices // type 0: 2
                       // type 1: 2
                       // type 2: 1
                       // type 3: 2
  if (type == 2 || type == 3) {
    float radius
  }
}

CornEmitter {
  uint32 inclusiveSize
  Node node
  float lifeSpan
  float emissionRate
  float speed
  float[4] color
  uint32 replaceableId
  char[260] path
  char[260] flags
  (KPPA)
  (KPPC)
  (KPPE)
  (KPPL)
  (KPPS)
  (KPPV)
}

As you can see, bones, event objects and collision shapes have no inclusive size, so they need different handling than the rest.
For bones, we know that it starts with a Node object, followed by two uint32s, so we know that the size of each bone is the inclusive size of the node + 8.
Event objects also have a node, so we use its inclusive size, but we also need to get the size of the KEVT chunk if it exists.
For collision shapes, we need the inclusive size of the node, in addition to the shape's data, which is 28 bytes for cubes, and 16 bytes for spheres.

If you want a unified way to parse all the variable-sized chunks, then you can define the inclusive size by yourself for the chunks without it.
That is, for bones, define the inclusive size to be the node's inclusive size + 8, for event objects define it as the inclusive size of the node plus the size of the KEVT chunk if it exists, and so on.
Once everything has correct inclusive sizes, you can do something like this:
C++:
Something[] somethings
uint32 totalSize = 0

while (totalSize < size) {
  Something something = new Something(...) // construct a new object
  totalSize += something.inclusiveSize
  somethings.push(something)
}

Almost all of the optional fields in Wacraft 3 models are keyframe tracks, or in other words, things that can change over time while the animation runs.
A Node for example can have KGTR, KGRT, and KGSC tracks, but it doesn't mean it has to have them. It can have neither of them, one of them, two of them or all three.
All tracks except for KEVT (event objects), follow the same structure, but with different types for the fields.
The structure looks like this:
C++:
Track {
  int32 frame // Probably should be uint32, but I saw a model with negative values
  X value
  if (interpolationType > 1) {
    X inTan
    X outTan
  }
}

Where X is a data type.
You get the interpolation type from the tracks chunk, which looks like this:
C++:
TracksChunk {
  uint32 tag
  uint32 tracksCount
  uint32 interpolationType // 0: none
                           // 1: linear
                           // 2: hermite
                           // 3: bezier
  uint32 globalSequenceId
  Track[tracksCount] tracks
}
The only difference between all the track types is tag, and the primitive data type of value/inTan/outTan.

Here is a mapping between track tags and data types, and also their meaning:
C++:
// Node
KGTR: float[3] translation
KGRT: float[4] rotation
KGSC: float[3] scaling
// Layer
KMTF: uint32 textureId
KMTA: float alpha
KMTE: float emissiveGain
KFC3: float[3] fresnelColor
KFCA: float fresnelAlpha
KFTC: float fresnelTeamColor
// Texture animation
KTAT: float[3] translation
KTAR: float[4] rotation
KTAS: float[3] scaling
//Geoset animation
KGAO: float alpha
KGAC: float[3] color
// Light
KLAS: float attenuationStart
KLAE: float attenuationStartEnd
KLAC: float[3] color
KLAI: float intensity
KLBI: float ambientIntensity
KLBC: float[3] ambientColor
KLAV: float visibility
// Attachment
KATV: float visibility
// Particle emitter
KPEE: float emissionRate
KPEG: float gravity
KPLN: float longitude
KPLT: float latitude
KPEL: float lifespan
KPES: float speed
KPEV: float visibility
// Particle emitter 2
KP2E: float emissionRate
KP2G: float gravity
KP2L: float latitude
KP2S: float speed
KP2V: float visibility
KP2R: float variation
KP2N: float length
KP2W: float width
// Ribbon emitter
KRVS: float visibility
KRHA: float heightAbove
KRHB: float heightBelow
KRAL: float alpha
KRCO: float[3] color
KRTX: uint32 textureSlot
// Camera
KCTR: float[3] translation
KCRL: float rotation
KTTR: float[3] targetTranslation
// Corn emitter
KPPA: float alpha
KPPC: float[3] color
KPPE: float emissionRate
KPPL: float lifespan
KPPS: float speed
KPPV: float visibility

So if you are parsing a Node, for example, and you see it has a KGRT chunk in it, you will parse it as a tracks chunk with the data type of the tracks being float[4].

In order to know if there are indeed keyframe tracks, you can either track the size of your chunks, and if it's smaller than the inclusive size, it means there are tracks.
A second approach is to peek ahead with your reader, and see if you find a matching tag. This is a little easier, as you don't actually have to check sizes this way.
The down side to the second approach is that you must know beforehand what all the possible track types that are allowed for every kind of object.
This is usually OK, since I've listed them above, but what if there are actually more we don't know about, and you suddenly get a model with one of them? The parser will fail with weird errors.
However, it is unlikely there are more common track types you will actually encounter, so choose which ever way you like better.

Some notes:

All rotations are expressed with quaternions, except for KCRL (cameras).
While there are 10 face types, generally only type 4 - triangles - is used.
Warcraft 3 animations are specified in milliseconds. That is, if your code is running at the normal 60 frames per second, then you add 1000/60 to your animation counters every frame.


What do I do with this?

Let's say we parsed the whole file. Now what?
There is too much information, so I'll just go over the basics for now.

Geosets

The main goal is to draw the geosets.
This is a standard operation - take the vertices, texture coordinates, normals, face indices, and call an indexed draw function in your rendering API.
The primitive type can be any primitive type supported by the rendering API.
That being said, it will generally always be 4, or in other words triangles.

However...

Materials

...each geoset references a material.
Materials are groups of layers, where each layer gives information such as the texture used, the layer alpha for translucency, the texture coordinate set to be used, and much more.
As far as graphics code, every layer is a draw call.
In pseudo code it could be something along the lines of:
C++:
foreach geoset of geosets:
    foreach layer of materials[geoset.materialId]:
        applyLayer(layer)
        drawGeoset(geoset)

An example before we move on - to get team colors working, models generally have two layers - the base layer is marked as team color and is completely opaque, and the layer that is rendered above it uses blending with the desired diffuse texture, such that places in the diffuse texture that are not opaque will show the team color layer below.

Layers

So what should applyLayer above do?

textureId
The textureId is an index into the model's textures list, which you need to bind into your rendering API.
C++:
bindTexture(textures[layer.textureId])


filterMode
The filterMode refers to what kind of graphics operation this layer has, and usually involves blending.
Conceptually it's very similar to how you'd combine layers in a 2D image editing software like Photoshop.
Each layer has its own mode, like "add", "blend", etc., and the result is the combination of all layers.

If the filter mode is 0, this is a normal opaque draw call.

If the filter mode is 1, this is an alpha-tested opaque draw call with alpha=0.75.
This means that any pixel resulting from this draw call with alpha<0.75 will not be drawn.
This can be achieved either directly via the rendering API, or inside GPU shaders.

Filter modes 2-6 are for blended draw calls, with different blending operations.
Here's code that selects them for WebGL:
JavaScript:
switch (filterMode) {
    // Blended
    case 2:
        blendSrc = gl.SRC_ALPHA;
        blendDst = gl.ONE_MINUS_SRC_ALPHA;
        break;
    // Additive.
    // Note that this isn't pure additive where two colors are added as-is.
    // Warcrft 3 takes also the source alpha into consideration!
    case 3:
        blendSrc = gl.SRC_ALPHA;
        blendDst = gl.ONE;
        break;
    // Referred to as "Add Alpha" by Magos et al., however doesn't seem to be different than Additive.
    case 4:
        blendSrc = gl.SRC_ALPHA;
        blendDst = gl.ONE;
        break;
    // Modulate
    case 5:
        blendSrc = gl.ZERO;
        blendDst = gl.SRC_COLOR;
        break;
    // Modulate 2X
    case 6:
        blendSrc = gl.DST_COLOR;
        blendDst = gl.SRC_COLOR;
        break;
}

coordId
The coordId field selects a specific coordinate set in the geoset.
Generally speaking this will always be 0, and every geoset will always have exactly one coordinate set.
This is due to the fact that no Warcraft 3-related model editing tools support more than one coordinate set.
There are a few models made over the years with multiple coordinate sets. If you know of one I'd be happy to get a copy :)

flags
The flags field holds more graphics state information, like whether the draw should be double sided, do depth checks, etc.

The rest...
...are related to animations.
 
Last edited:

Dr Super Good

Spell Reviewer
Level 57
Joined
Jan 18, 2005
Messages
26,448
MDX is built out of a hierarchy of chunks.
This is the main file structure:
Do the chunks have to come in that order?

To parse the file you would generally have a loop which keeps reading headers. If you recognize the header's tag and know how to parse it, do it, otherwise just skip the header's size bytes.
So does that mean that you have 2 times the ascii chunk name followed by the size?.
 
Level 27
Joined
Jul 29, 2007
Messages
5,100
Do the chunks have to come in that order?

There is no order, this is further seen by the fact that _all_ chunks are optional (at least according to Magos).
That is, you can have an MDX file with the four bytes "MDLX" and it would be valid.


So does that mean that you have 2 times the ascii chunk name followed by the size?.

No. You read the 8 bytes of the header which is the header tag (e.g. "VERS") and the size.
If your code knows how to handle that tag (and wants to), you continue parsing the chunk.
If your code doesn't know how to handle it (or doesn't want to), you just skip <header's size> bytes.

This is important to implement, because of programs like MDLVIS which add their own chunk for their data. You probably don't want your parser to crash because you thought you know all the chunks out there.
It also allows you to ignore chunks you don't care about (e.g. currently the model viewer doesn't know that lights, attachments, event objects and collision shapes even exist, they are commented since they weren't used anyway).
 

Dr Super Good

Spell Reviewer
Level 57
Joined
Jan 18, 2005
Messages
26,448
There is no order, this is further seen by the fact that _all_ chunks are optional (at least according to Magos).
That points to the format being optimized for graph based game design maybe? Each model can be represented as a map to the various chunk types (hence the no ordering).

How I would recommend reading it is to use a stream based reader (since they are often in an archive which rules out memory mapping) and do two reads per chunk, one for header and other for chunk data. You could possibly recycle the read buffer between chunks if the next chunk is less than or equal to the buffer used for the current chunk.

A reasonable way for C/C++ to represent the file seems to be a map matching tags to a native interpretation of the array of structures. This means you can query it very easily to find pointers to structures of a certain type. In languages like Java a similar approach could be used except you should define methods to return the standard types.

What is good about the map approach is that you can parse tags you do not understand by simply keeping them as a raw buffer and adding them to the map under the name. This provides you with an interface that can still be used to read such chunks if required by using an extension that does know how to interpret them.

You could also use a map approach as a way to interpret the file as a whole. First mapping each chunk to a position and size in the file and then only reading the chunks required. Alternatively you could read all chunks and map a pointer to each raw chunk. You can then query this map and get the data for a chunk to be interpreted by a separate component. The advantage of this is you abstract the actual file I/O from the interpretation. Alternatively when writing back you would place each universal formatted chunk buffer into the map and then finally write the file using a single method that iterates over all the mapped tags and creates the appropriate file structure.
 
Level 27
Joined
Jul 29, 2007
Messages
5,100
You can test it all if you want. For the model viewer, there was only a single way - get the whole file and parse it variable by variable. Nothing else to do with JavaScript.
Not that it matters too much for a low amount of models, the time it takes to parse them is negligible (20-500KB...pretty much nothing today).

I don't understand why all the chunks don't have a uint32 defining the number of objects in them. It would have made parsing faster and the code much cleaner. The only reason I could fathom is that Blizzard artists edited MDL files manually...which sounds pretty stupid.
 
You know, the sc2 art tools are out now, we can start reingineering the M3 format.

On topic: You are lacking SND, SNEM, KSPE.
Colission Shapes are:
0: Box,
1: Plane,
2: Sphere,
3: Cylinder

in this line:
// #0x8000: if ribbon emitter: emitterUsesMdl, otherwise: unshaded
// #0x10000: if ribbon emitter: emitterUsesTga, otherwise: sortPrimitivesFarZ
should say if particle emitter else if particle emitter 2
(these are also used by sound emitters somehow).

In part2:
KP2R: float unknown // ?

that's variation animations.

Also KEVT uses another format. it just writes times, no float, point3 or quat data, just the time.

in TextureCoordinateSet, they use float, not uint32
 
Level 27
Joined
Jul 29, 2007
Messages
5,100
You know, the sc2 art tools are out now, we can start reingineering the M3 format.

On topic: You are lacking SND, SNEM, KSPE.
Colission Shapes are:
0: Box,
1: Plane,
2: Sphere,
3: Cylinder

in this line:
// #0x8000: if ribbon emitter: emitterUsesMdl, otherwise: unshaded
// #0x10000: if ribbon emitter: emitterUsesTga, otherwise: sortPrimitivesFarZ
should say if particle emitter else if particle emitter 2
(these are also used by sound emitters somehow).

In part2:
KP2R: float unknown // ?

that's variation animations.

Also KEVT uses another format. it just writes times, no float, point3 or quat data, just the time.

in TextureCoordinateSet, they use float, not uint32

How do plane and cylinder look data-wise?

0x8000/10000 - that makes sense, copied from Magos without actually looking at it. How do you know if the node belongs to type 2 emitter?

I saw KEVT being different in my parser, and I wondered if I have some weird error from the time I wrote it (two years ago?), guess I didn't, I'll fix it back.

Do you know how the structures of SND, SNEM and KSPE look? (KSPE is float / emissionRate?)

Also what are KLAS/KLAE? (I guess attenuation start/end respectively)

Figured you'll come here and find all my mistakes, thanks.
 

Dr Super Good

Spell Reviewer
Level 57
Joined
Jan 18, 2005
Messages
26,448
I don't understand why all the chunks don't have a uint32 defining the number of objects in them. It would have made parsing faster and the code much cleaner. The only reason I could fathom is that Blizzard artists edited MDL files manually...which sounds pretty stupid.
You clearly have not parsed .doo files. If you think this is bad, they are much worse. Every entry in them is of an unspecified size which varies based on a number of fields that are in variable locations based on the values of other fields, iteratively. Let us not forget that which fields are present vary depending on the version of the .doo file. Depending on the name of the .doo file the entire member structure is different (units or doodads). At the end of all doodad entries there is a different type of entry for terrain objects. The only real way to parse them efficiently is to either map the entire file into a buffer (single I/O op, high spike memory usage) or use some kind of buffered I/O which you query the amount you do know from.

Blizzard was not very logical with some of their formats at times. I strongly agree with you that they should have standardized sizes more and given more size fields for better I/O control.
 
GhostWolf said:
I don't understand why all the chunks don't have a uint32 defining the number of objects in them. It would have made parsing faster and the code much cleaner. The only reason I could fathom is that Blizzard artists edited MDL files manually...which sounds pretty stupid..

There's a very good reason to that; animations! because of animation tracks chunks may vary in size. Therefore, they used exclusive size and inclusive size.

Also remember this was the first time they made 3d format, for that time they did an impressive work. MDX files are very easy to read and write, except for a few little things

In the M3 format, which is an update of the MDX format. They fixed varying chunk sizes by using references. This makes writting an M3 file a lot harder, but makes reading a lot easier.

GhostWolf said:
How do plane and cylinder look data-wise?

0x8000/10000 - that makes sense, copied from Magos without actually looking at it. How do you know if the node belongs to type 2 emitter?

I saw KEVT being different in my parser, and I wondered if I have some weird error from the time I wrote it (two years ago?), guess I didn't, I'll fix it back.

Do you know how the structures of SND, SNEM and KSPE look? (KSPE is float / emissionRate?)

Also what are KLAS/KLAE? (I guess attenuation start/end respectively)

Figured you'll come here and find all my mistakes, thanks.

Plane uses 2 vertices, cylinder, asd far as I remember, used two vertices and bound radius.

of those 3, I've only figured one: SND.

SND Chunk (similar to Textures chunk):
C:
Sounds {
    SoundTracks[?];
}

SoundTrack {
     char fileName[260];
     float volume;
     float pitch;
     int flags;
}

Still need to figure out SNEM completely, but KSPE is looks like KEVT. It's hard to know since, there are no models with them, I've been guessing from reading the game?s assembly code and tried writting models with them.
 
Level 27
Joined
Jul 29, 2007
Messages
5,100
It doesn't matter if something has a variable size, the exporter still knows how big everything is.
The only reason why not to do it, then, is for work with no exporter, or in other words, working with MDL files.

SND is the actual name? where's the last byte?
 
It doesn't matter if something has a variable size, the exporter still knows how big everything is.
The only reason why not to do it, then, is for work with no exporter, or in other words, working with MDL files.

SND is the actual name? where's the last byte?
SNDS, sry

That's why it exports the inclusive size on the chunk, it says how big is that particular object in bytes. Even though most likely you don't need it.
 
Level 27
Joined
Jul 29, 2007
Messages
5,100
That's why it exports the inclusive size on the chunk, it says how big is that particular object in bytes. Even though most likely you don't need it.

Of course, but in addition every chunk should have a count, so you could efficiently allocate an array.

Not to mention some chunks arbitrarily don't even have inclusive size (bones, event objects, cameras and collision shapes)...what where they thinking when they made the format is beyond me.

MDX is obviously not as bad as things like OBJ (don't even get me started on that), but it could be so much better.
 
Level 9
Joined
Jul 11, 2010
Messages
422
C++:
uint32[faceTypeGroupsCount] faceTypeGroups // Should always be 0x4 for triangles
W3 should be abled to read any kind of face-type. I personnaly tested "lines" and it worked fine (I guess I can retrieve the model if there are sceptics :angry:). Ayane said they all work with the exceptions of "quads", "quads_strip" and "polygon" for some graphics boards.

Code:
POINTS                               0x0000
LINES                                0x0001
LINE_LOOP                            0x0002
LINE_STRIP                           0x0003
TRIANGLES                            0x0004
TRIANGLE_STRIP                       0x0005
TRIANGLE_FAN                         0x0006
QUADS                                0x0007
QUAD_STRIP                           0x0008
POLYGON                              0x0009
 
Blinkboy, I think I can help out with that :p

Sry I kinda got lost in the conversation. With what would you be willing to help. If it is about reverse engineering the M3 format. I'll make a whole thread about it, so don't worry. If you are indeed willing to help, I suggest you learn completely how MDX and M2(Wow models) work.

GhostWolf said:
Of course, but in addition every chunk should have a count, so you could efficiently allocate an array.

Not to mention some chunks arbitrarily don't even have inclusive size (bones, event objects, cameras and collision shapes)...what where they thinking when they made the format is beyond me.

MDX is obviously not as bad as things like OBJ (don't even get me started on that), but it could be so much better.

yeah, you are right that would have been nice. But is not a big deal if you use vectors or resizeable arrays, lists or trees. (I think they build up everything in a hierarchy tree).

About the non inclusive size objects, well these objects have known size except for the inner node, (Event objects must always have the KEVT). I don't think there's much to worry.

There are formats like .x, .3ds and .obj which are a bigger pain to use. Mainly because they aren't optimized for game engines.
 

Dr Super Good

Spell Reviewer
Level 57
Joined
Jan 18, 2005
Messages
26,448
Blizzard hates simple reading algorithms. I wrote a reader for their .doo format and it was a huge mess where sizes were based on values of fields. Basically most of their file formats depend on sequential reading of the entire file to extract anything from them.

At least they give you chunk sizes so you can skip sections you do not need. Like I said, you can model the entire file as a hashtable (map) of buffers for purposes of easily finding and processing stuff. This would also prevent information loss for chunks you cannot process.
 
Level 12
Joined
Feb 2, 2006
Messages
813
Hey,
I found this thread by chance. If someone needs to access MDX files I wrote this class which is able to read and write MDX files.
For MDL files at least writing is supported. Maybe this helps people who want to develop tools using the format.
 

Dr Super Good

Spell Reviewer
Level 57
Joined
Jan 18, 2005
Messages
26,448
Possible defect above?
Code:
KCRL: uint32 rotation
A track with no interpolation and 3 entries takes up 24 bytes (can confirm as it was last part of a camera and so runs from the track declaration to end of camera chunk). This means 8 bytes per entry. Since uint32 is only 4 bytes per entry this means the file does not match the above specification.

4B 43 52 4C 03 00 00 00 00 00 00 00 FF FF FF FF 21 00 00 00 00 00 00 00 87 16 00 00 00 00 00 00 93 46 00 00 00 00 00 00
Looking at the pattern of trailing "00" bytes it would mean that rotation is type "uint64"?
 
Level 2
Joined
Oct 17, 2019
Messages
4
Sorry to necro but this seems to be the most relevant post for the version 900 (Reforged) structure changes.

Structure-wise this should be pretty close but the field names are lacking - I've matched MDL tokens where possibly however the popcorn particle emitter has me stumped. Also to note; all new fields are wrapped in version checks to preserve backwards compatibility.

I also have this in C# here if that is easier to read.

Code:
//-- new chunks (CORN is after PRE2. FAFX and BPOS are at the end)
CORN { // PopcornFX
  ParticleEmitterPopcorn[?] particleEmitterPopcorn
}

FAFX { // FaceFX
  FaceFX[?] facialAnimationFiles
}

BPOS { // BindPose
  uint32 numBindPositions
  C34Matrix[numBindPositions] bindPositions
}

//-- changed objects
Geoset {
  ...
  uint32 selectionFlags
  uint32 levelOfDetail // v900+
  char[80] geosetName  // v900+, possibly filePath
  ...
  Extent[extentsCount] extents
  char[4] "TANG" // v900+
  uint32 tangentsCount
  C4Vector[tangentsCount] tangets // tangets.w stores handedness
  char[4] "SKIN" // v900+
  uint32 skinSize // data size in bytes
  C2iVector[skinSize / 8] boneInfo // (uint32 index, uint32 weight)
  ...
}

Material {
  ...
  uint32 flags
  char[80] shader // v900+
  ...
}

Layer {
  ...
  float alpha
  float emissiveGain // v900+
  ...
  (KMTE) // v900+
}

//-- new objects
FaceFX {
  char[80] type? // faceFX file descriptor, always 'Node'
  char[260] filePath // *.facefx
}

ParticleEmitterPopcorn {
  uint32 inclusiveSize
  Node node
  float float1 // static track values, haven't checked order
  float float2
  float float3
  float float4
  float float5
  float float6
  float float7
  uint32 uint1 // boolean? seen as '1' in 'popcorntemplate.mdx'
  char[260] filePath // *.pkfx
  char[260] animVisibilityGuide // e.g. 'Always=on, Death=off, Decay=off'
  (KPPS)
  (KPPV)
  (KPPL)
  (KPPA)
  (KPPE)
  (KPPC)
}

//-- new track types
KMTE: float emissiveGain
KPPS: float speed
KPPV: float visibility
KPPL: float lifespan
KPPA: float alpha
KPPE: float emissionRate
KPPC: float[3] color
 
Your SKIN is incorrect they represet 8 bytes, first 4 bytes are bone Id's, 4 next bytes are the weights but you need to devide them by 255.0, make sure the total is 1.0 else the vertexes are not weighted correctly

BPOS, if you apply the binding pose, only use the position of the matrix, else the animation breaks.

For FAFX, you are correct, it points to an external file
 
Level 27
Joined
Jul 29, 2007
Messages
5,100
Hello. I am aware of the changes, just didn't update the specs. Thanks either way :)

Can't say I saw some of the CORN animations you mentioned, but I also didn't check too many models.

The sound track objects are in the specs just for the sake of completeness.

Updated the specs and some of the text and comments.
 
Last edited:
Level 2
Joined
Oct 17, 2019
Messages
4
Had some spare time and have RE'd the next lot of changes and fixed my previous field names. Everything is document as how the client reads it and named by it's MDL token so will need tweaking.

Code:
//-- updated field names
ParticleEmitterPopcorn {
  uint32 inclusiveSize
  Node node
  float lifeSpan
  float emissionRate
  float speed
  float[3] color
  float alpha
  uint32 replaceableId
  char[260] path // *.pkfx
  ...
}

//-- new structure + field names
Layer {
  ...
  // this is the actual client (13769) check so no backwards compatibility for the previous beta files
  if (version >= 900) {
    float emissiveGain
    float[3] fresnelColor
    float fresnelOpacity
    float fresnelTeamColor
  }

  (KMTF)
  (KMTA)
  if (version > 800) {
    (KMTE)
  }
  if(version > 900) {
    (KFC3)
    (KFCA)
    (KFTC)
  }
}

//-- new track types
KFC3: float[3] fresnelColor
KFCA: float fresnelAlpha
KFTC: float fresnelTeamColour

Edit: I've just noticed there is a new layer shadingflag, 0x100: Unlit
 
Last edited:
PopcornFX effects files. I don't think they've actually been shipped so may be obsolete, part of the build pipeline or just not yet implemented.
maybe they are using them for in-game cinematics in the campaign. Who knows, seems odd they would just leave that there even though it's not the first time they have done that (ahem sound effect emitters)
 

Retera

Tool Reviewer
Level 27
Joined
Apr 19, 2008
Messages
1,182
Hey, I just wanted to post on here in case anybody else runs into the same problem that I had 10 years from now. Hopefully they will not run into the same problem because I will get my code published online somewhere.

But anyway, what I discovered is a crazy quirk in BillboardedLockX (or "BillboardLockX" as my old repo mistakenly called it, or Billboarded Lock X or whatever you want to call it). This quirk is really unlikely to show up when testing in any straightforward way, but essentially what I think I was seeing from some test models in the World Editor is that this setting (only X, not locking any other dimension) also will causes that particular node in the model to be rendered with scale.z *= -1. It will be inside out.

upload_2021-3-7_13-2-23.png

So, as far as I know this unusual quirk in BillboardedLockX was not published anywhere else. I'm hoping if anybody else gets as deep in this stuff as I have tried to get by reading Ghostwolf's code and applying what I learn, that they will be able to Google this now because of my post here and will not face this issue how I had to. I was stuck asking myself, "How does the arrow of the Night Elf Archer have its normals pointed in the correct direction??" And for this whole weekend it has been unclear to me, given other test models with one-sided arrows or other billboards that clearly indicated that despite the direction of the normals for lighting, the mesh itself was pointed opposite what you would expect.
 
Top