- Joined
- Jul 29, 2007
- Messages
- 5,174
This specification is mostly based on https://github.com/flo/m3addon/blob/master/structures.xml.
More values have been reverse engineered by BlinkBoy, and a couple of fixes by me.
Many things are still unknown, feel free to help in reverse engineering the format.
The notation of this specifications is as follows:
Unlike the MDX format, M3 doesn't store its data in linear chunks, but rather batches most of it together in large data structures, and then gives indices to access the correct data structure.
It starts (after the uint32 "43DM") with the MD34 header chunk:
Since M3 is largely based around referencing other chunks and data structures, you will see many reference types. I will write them as "Reference name // type". You can see above "Reference modelHeader // MODL", which is a reference that points to the MODL chunk.
All the data structures - the parts that actually hold all the reptitive data like vertices, faces, animation data, materials, etc. - are given by an index.
This index is an array of
The IndexEntry chunk looks like this:
Reading the index:
The
In addition to the index, the MD34 header also contains a direct reference to the MODL chunk. This is the actual main M3 chunk that will contain the references to all the rest of the data.
This is given in the header so you wont have to loop over all the index entries, in order to find it, every time you want to load a model.
The reference structure looks like this:
To use a reference, you use its
As an example, the following code reads the MODL chunk.
Do note, however, that the code is not reading multiple entries, that's because there is always exactly one MODL chunk.
A more general reference reading function will look something like this:
As previously stated, MODL is the main M3 chunk. It will reference all the data used by the model.
This is how it looks.
Note: remember the version variable of the index entries, this is where it starts to play a role.
As you can see, many references reference to primitive data types. For example, U16_ is a chunk of unsigned shorts.
The confusing thing is that the vertices chunk uses the name U8__, while it has nothing to do with unsigned bytes. Read this post for an explanation.
This is a list of the primitive types referenced throughout the format:
When you first start to make a parser, you can read all of the references as
All animation referencing follows the same structure called an AnimationReference, this is how it looks:
X detonates the data type of the animation reference, e.g.: 3D vector (translations and scales), quaternion (rotations), and so on.
Whenever an animation reference is used, I will write it as "AnimationReference name // data type", for example, "AnimationReference translation // float[3]".
Animation references, after some work, will end up referencing into data of sequence data chunks, or SD.
The SD chunk looks as follows:
X detonates the data type a SD contains.
There are 13 different SD chunks, each with its own type, and they are listed below the STC_ chunk, which holds them.
For example, when you see SD2V, it means it is the data type of the
Now to all the other chunks.
They will be listed in the order they show up in MODL.
If a chunk uses subchunks or objects, they will be listed right after it.
Some of the types (noteably vertices) need utility functions to transform their data correctly, these functions will be discussed later on.
Handling divisions
While the DIV_ reference is a normal reference, there is actually only one DIV_ chunk.
This chunk holds all the chunks that will bridge between the vertices and triangle indices to form regions.
The list of triangles is right there in the triangles unsigned short reference.
Each chunk in the regions reference of type REGN, will describe a geometry mesh, much like MDX's geoset.
The batch reference holds BAT_ chunks. Each one maps between regions and the materials they are using.
Finally, the boundingShapes reference holds MSEC objects which are the bounding shapes of all the different regions.
Handling regions
To construct the real regions, you need to take into account their offsets.
The firstVertexIndex variable, for example, is an index into the global vertex chunk (the U8__ reference, remember?).
That is, if we have a triangle index 5, it in fact points to
The same goes for firstTriangleIndex, and firstBoneLookupIndex.
Handling the bone lookup table
The bone lookup indices do not actually point to bones at all.
They are indices into the bone lookup reference belonging to MODL.
To get the first actual bone index of a vertex, you need the following:
The
Handling vertices
Vertices in particular need transformations for their data.
Specifically, the normal, uvs and tangent.
For the normal and tangent, you must apply the following equation for each axis:
For the uvs, each axis needs to be divided by 2048:
Handling triangles
After all of the above data, this is the easiest. Just remember adding the region offset and you are fine.
Handling materials
Each batch chunk, or BAT_, holds the index of the region to use for the batch, and the material index.
This material index doesn't actually point to an actual material, but rather into the material map reference, or MATM.
Each material map has a material type, and material index.
The material type starts with 1, being standard materials, 2 being displacement materials, and so on, in the same order as they are read in the MODL chunk.
The material index, is the index into the appropriate material reference.
That is, if a batch references material index 1, and we see that MODL.materialMap[1] has materialType 1 and materialIndex 3, then the correct material is MODL.standardMaterials[3].
Sequences, STC, STG and STS chunks
The SEQS chunk holds mostly sequence metadata.
The STC chunks are the chunks that actually hold all the animation data, more about using them in the following section.
STS chunks give an easy way to check if we have valid animation data, given an animation reference and a frame of animation. More about them after STC chunks.
STG chunks are the chunks that define how the animation should look - it has a list of indices into the STC array, which you will use for animation. More about them after STS.
Handling animation references - STC
Every animation reference has an animId variable.
This variable holds a random unique number, which lets us figure where the actual data is.
For each STC object, there is a list of animIds and animRefs.
First, you take your animId value, and search for the equivalent value in the STC animIds list.
If you found it, then save the index in the list where you found it.
For example, if STC.animIds[2] == animationReference.animId, then save the index 2.
In case the STC doesn't have the animId, return the animation reference's initValue variable.
Now, I wrote that STC.animRefs is a U32_ reference, because it is, but each element is in fact two separate unsigned short numbers.
Because of this, you might want to simply read them as a pair of unsigned shorts instead of an unsigned long.
Remember the 2 you saved from animIds?
Now index STC.animRefs with that, so save STC.animRefs[2].
The first part of this animRef is yet another index, into another array, so save it.
Let's say it was 5.
The second part of the animRef is the index of the array you want to index.
Let's say it was 2.
There are 13 different sequence data chunks, that hold all the actual animation data for the STC.
The first one is sdev, the second is sd2v, the third is sd3v, and so on.
Now, since our saved animRef was [5, 2], the correct animation data is from the third (0-based indexing) SD, like so:
STC.sd3v[5]
Finally, once you've got the correct block inside the correct sequence data chunk, you can use its keys (which form the animation's timeline) and values to get the actual animation data.
In code:
Handling animation references - STS
The only purpose of STS is to do a fast check whether some STC chunk will actually hold any animation data for an animation reference.
Each STC chunk has a reference index to a STS chunk.
If you want to check if a STC has animation data for an animation reference, simply select the relevant STS, and see if it has the animation reference's animId in its animIds list.
Handling animation references - STG
STG chunks are the last part of handling animation references.
They hold a list of indices into the STC array. Usually this list will only have one element, but sometimes it will have two, or more (for example, splitbody animations).
When you have an animation reference and want to get data from it, you don't directly access STC chunks.
Instead you do the following:
In code:
Note: STC chunks have a
Handling bones
The bone chunk reference gives us the skeletal structure of the model.
Generally speaking, to animate a skeleton, you first want to start with the root, transform it, then move on to its children, and to their children, and so on.
So far every model that I checked came sorted, such that a child bone will always come after its parent bone, in the array.
This helps, because there is no need to sort them on every run, instead you just loop over all the bones and transform each one as you loop over it.
To transfrom the bones for some chosen sequence i:
Before rendering, all the bones will need to have another transformation on them.
For each bone i, transform its world matrix with the i'th matrix from MODL.initialReference.
The purpose of the initial reference matrices is to move the model into bind pose, to which the bone animation is then applied.
Optimizations
More values have been reverse engineered by BlinkBoy, and a couple of fixes by me.
Many things are still unknown, feel free to help in reverse engineering the format.
The notation of this specifications is as follows:
- Chunk names will always be given in their ASCII representation.
For example, every M3 starts with a uint32 with the hexadecimel value 0x3433444d. If you break that to bytes, and remember how small endian works, you will get the bytes '4', '3', 'D' and 'M', or if you turn it around, the string "MD34".
The M3 format uses this concept extensively, as a way to give chunks meaningfull names.
In your code, you will probably want to define constants for this, for exampleMD34_CHUNK = 0x3433444d, but this is up to you. - Flag variables hold different values in their bits with special meanings.
The notation for them would be the hexadecimal number representing the correct bit.
If for example there is 0x4, it means that the third bit holds specific information which you can get with bitwise operators.
That is, (Flag & 0x4) == 0x4 will show you if the value in the third bit is true or false. - Type variables hold only one value, and will simply be written using normal decimal numbers.
- All the code examples will assume
readeris some kind of binary reader.
Unlike the MDX format, M3 doesn't store its data in linear chunks, but rather batches most of it together in large data structures, and then gives indices to access the correct data structure.
It starts (after the uint32 "43DM") with the MD34 header chunk:
C++:
MD34 {
uint32 indexOffset
uint32 entries
Reference modelHeader // MODL
}Since M3 is largely based around referencing other chunks and data structures, you will see many reference types. I will write them as "Reference name // type". You can see above "Reference modelHeader // MODL", which is a reference that points to the MODL chunk.
All the data structures - the parts that actually hold all the reptitive data like vertices, faces, animation data, materials, etc. - are given by an index.
This index is an array of
entries IndexEntry chunks, and starts at the byte offset indexOffset.The IndexEntry chunk looks like this:
C++:
IndexEntry {
uint32 tag
uint32 offset
uint32 entries
uint32 version
}Reading the index:
C++:
if (reader.read(4) == "43DM") {
MD34 header = new MD34(reader)
reader.seek(header.indexOffset)
IndexEntry[header.entries] indexEntries
for (uint32 i = 0; i < header.entries; i++) {
indexEntries[i] = new IndexEntry(reader)
}
}The
version variable used by index entries is important - some chunks have multiple known versions, which change their structure.In addition to the index, the MD34 header also contains a direct reference to the MODL chunk. This is the actual main M3 chunk that will contain the references to all the rest of the data.
This is given in the header so you wont have to loop over all the index entries, in order to find it, every time you want to load a model.
The reference structure looks like this:
C++:
Reference {
uint32 entries
uint32 index
uint32 flags
}To use a reference, you use its
index to access the correct index entry, offset reader to it, and read entries entries of the data type that the index entry holds.As an example, the following code reads the MODL chunk.
Do note, however, that the code is not reading multiple entries, that's because there is always exactly one MODL chunk.
C++:
IndexEntry modelHeader = indexEntries[header.modelHeader.index]
reader.seek(modelHeader.offset)
MODL model = new MODL(reader)A more general reference reading function will look something like this:
C++:
// Parses reference of some type X
X[] parseXReference(reader, indexEntries) {
Reference reference = new Reference(reader)
IndexEntry indexEntry = indexEntries[reference.index]
uint32 offset = reader.tell()
X[reference.entries] entries
// Go to the location of the entries
reader.seek(indexEntry.offset)
// Read them
for (uint32 i = 0; i < reference.entries; i++) {
// Pass the index entries if X also has references
// Always pass the entry version, even if X doesn't have more than 1 version now, maybe more will appear in the future
entries[i] = new X(reader, indexEntries, indexEntry.version)
}
// Go back to where we were
reader.seek(offset)
return entries
}As previously stated, MODL is the main M3 chunk. It will reference all the data used by the model.
This is how it looks.
Note: remember the version variable of the index entries, this is where it starts to play a role.
C++:
MODL {
Reference name // CHAR
uint32 flags // 0x100000: has mesh
Reference sequences // SEQS
Reference stc // STC_
Reference stg // STG_
float unknown0
float unknown1
float unknown2
float unknown3
Reference sts // STS_
Reference bones // BONE
uint skinBones
uint32 vertexFlags // 0x20000: 1 texture coordinate
// 0x40000: 2 texture coordinates
// 0x80000: 3 texture coordinates
// 0x100000: 4 texture coordinates
Reference vertices // U8__
Reference divisions // DIV_
Reference boneLookup // U16_
BoundingSphere boundings
uint32[16] unknown4To19
Reference attachmentPoints // ATT_
Reference attachmentPointAddons // U16_
Reference lights // LITE
Reference shbx // SHBX
Reference cameras // CAM_
Reference unknown20 // U16_
Reference materialMaps // MATM
Reference standardMaterials // MAT_
Reference displacementMaterials // DIS_
Reference compositeMaterials // CMP_
Reference terrainMaterials // TER_
Reference volumeMaterials // VOL_
Reference volumeNoiseMaterials // VON_
Reference creepMaterials // CREP
if (version > 24) {
Reference unknown21 // ?
}
if (version > 25) {
Reference unknown22 // ?
}
Reference particleEmitters // PAR_
Reference particleEmitterCopies // PARC
Reference ribbonEmitters // RIB_
Reference projections // PROJ
Reference forces // FOR_
Reference warps // WRP_
Reference unknown23 // ?
Reference rigidBodies // PHRB
Reference unknown24 // ?
Reference physicsJoints // PHYJ
Reference unknown25 // ?
Reference ikjt // IKJT
Reference unknown26 // ?
if (version > 24) {
Reference unknown27 // ?
}
Reference patu // PATU
Reference trgd // TRGD
Reference initialReference // IREF
SSGS tightHitTest
Reference fuzzyHitTestObjects // SSGS
Reference attachmentVolums // ATVL
Reference attachmentVolumesAddon0 // U16_
Reference attachmentVolumesAddon1 // U16_
Reference bbsc // BBSC
Reference tmd // TMD_
uint32 unknown28
Reference unknown29 // U32_
}As you can see, many references reference to primitive data types. For example, U16_ is a chunk of unsigned shorts.
This is a list of the primitive types referenced throughout the format:
C++:
CHAR {
uint8 x
}
U8__ {
uint8 x
}
U16_ {
uint16 x
}
U32_ {
uint32 x
}
I32_ {
int32 x
}When you first start to make a parser, you can read all of the references as
Reference objects. When you want to actually implement a chunk, you can then use the reference to construct the actual data.All animation referencing follows the same structure called an AnimationReference, this is how it looks:
C++:
AnimationReference {
uint16 interpolationType
uint32 animId
X initValue
X nullValue
int32 unknown0
}Whenever an animation reference is used, I will write it as "AnimationReference name // data type", for example, "AnimationReference translation // float[3]".
Animation references, after some work, will end up referencing into data of sequence data chunks, or SD.
The SD chunk looks as follows:
C++:
SD {
Reference keys // I32_
uint32 flags
uint32 biggestKey
Reference values // X
}There are 13 different SD chunks, each with its own type, and they are listed below the STC_ chunk, which holds them.
For example, when you see SD2V, it means it is the data type of the
values reference of the SD chunk.Now to all the other chunks.
They will be listed in the order they show up in MODL.
If a chunk uses subchunks or objects, they will be listed right after it.
Some of the types (noteably vertices) need utility functions to transform their data correctly, these functions will be discussed later on.
C++:
SEQS {
int32 unknown0
int32 unknown1
Reference name // CHAR
uint32 animationStart
uint32 animationEnd
float movementSpeed
uint32 flags // 0x1: not looping
// 0x2: always global
// 0x8: global in previewer
uint32 frequency
uint32 unknown2
uint32 unknown3
uint32 unknown4
if (version < 2) {
uint32 unknown5
}
BNDS boundingSphere
uint32 unknown6
uint32 unknown7
uint32 unknown8
}
BNDS {
float[3] minBorder
float[3] maxBorder
float radius
}
STC_ {
Reference name // CHAR
uint16 runsConcurrent
uint16 priority
uint16 stsIndex
uint16 stsIndexCopy
Reference animIds // U32_
Reference animRefs // U32_
uint32 unknown0
SD sdev // SDEV
SD sd2v // SD2V
SD sd3v // SD3V
SD sd4q // SD4Q
SD sdcc // SDCC
SD sdr3 // SDR3
SD unknown1 // ?
SD sds6 // SDS6
SD sdu6 // SDU6
SD unknown2 // ?
SD unknown3 // ?
SD sdfg // SDFG
SD sdmb // SDMB
}
SDEV {
Reference name // CHAR
int32 unknown0
int16 unknown1
uint16 unknown2
Matrix matrix
int32 unknown3
int32 unknown4
int32 unknown5
if (version > 0) {
int32 unknown6
int32 unknown7
}
if (version > 1) {
int32 unknown8
}
}
SD2V {
float x
float y
}
SD3V {
float x
float y
float z
}
SD4Q {
float x
float y
float z
float w
}
SDCC {
uint8 r
uint8 g
uint8 b
uint8 a
}
SDR3 {
float x
}
SDS6 {
int16 x
}
SDU6 {
uint16 x
}
SDFG {
uint32 x
}
SDMB {
BNDS boundingSphere
}
STG_ {
Reference name // CHAR
Reference stcIndices // U32_
}
STS_ {
Reference animIds // U32_
int32 unknown0
int32 unknown1
int32 unknown2
int16 unknown3
uint16 unknown4
}
BONE {
int32 unknown0
Reference name // CHAR
uint32 flags // 0x1: inherit translation
// 0x2: inherit scale
// 0x4: inherit rotation
// 0x10: billboard 1
// 0x40: billboard 2
// 0x100: 2D projection
// 0x200: animated
// 0x400: inverse kinematics
// 0x800: skinned
// 0x2000: real
int16 parent
uint16 unknown1
AnimationReference location // float[3]
AnimationReference rotation // float[4]
AnimationReference scale // float[3]
AnimationReference visibility // uint32
}
// The real U8__ is unsigned byte
U8__ {
float[3] position
uint8[4] boneWeights
uint8[4] boneLookupIndices
uint8[4] normal
int16[?][2] uvs // based on vertexFlags
uint8[4] tangent
}
DIV_ {
Reference triangles // U16_
Reference regions // REGN
Reference batches // BAT_
Reference boundingShapes // MSEC
uint32 unknown0
}
REGN {
uint32 unknown0
uint32 unknown1
uint32 firstVertexIndex
uint32 verticesCount
uint32 firstTriangleIndex
uint32 triangleIndicesCount
uint16 bonesCount
uint16 firstBoneLookupIndex
uint16 boneLookupIndicesCount
uint16 unknown2
uint8 boneWeightPairsCount
uint8 unknown3
uint16 rootBoneIndex
}
BAT_ {
uint32 unknown0
uint16 regionIndex
uint32 unknown1
uint16 materialReferenceIndex
uint16 unknown2
}
MSEC {
uint32 unknown0
AnimationReference boundings // BNDS
}
ATT_ {
int32 unknown0
Reference name // CHAR
uint32 bone
}
LITE {
uint8 type
uint8 unknown0
int16 bone
uint32 flags // 0x1: cast shadows
// 0x2: specular
// 0x4: ?
// 0x8: turn on
uint32 unknown1
int32 unknown2
AnimationReference lightColor // float[3]
AnimationReference lightIntensity // float
AnimationReference specularColor // float[3]
AnimationReference specularIntensity // float
AnimationReference attenuationFar // float
float unknown3
AnimationReference attenuationNear // float
AnimationReference hotSpot // float
AnimationReference falloff // float
}
SHBX {
?
}
CAM_ {
uint32 bone
Reference name // CHAR
AnimationReference filedOfView // float
uint32 unknown0
AnimationReference farClip // float
AnimationReference nearClip // float
AnimationReference clip2 // float
AnimationReference focalDepth // float
AnimationReference falloffStart // float
AnimationReference falloffEnd // float
AnimationReference depthOfField // float
}
MATM {
uint32 materialType
uint32 materialIndex
}
LAYR {
uint32 unknown0
Reference imathPath // CHAR
AnimationReference color // float[3]
uint32 flags // 0x4: texture wrap x
// 0x8: texture wrap y
// 0x10: invert color
// 0x20: clamp
// 0x100: use particle flipbook
// 0x400: use color from animation
uint32 uvSource // 0: explicit unwrap 0
// 1: explicit unwrap 1
// 2: reflective cube env
// 3: reflective sphere env
// 4: planar local z
// 5: planar world z
// 6: particle flipbook
// 7: cubic env
// 8: sphere env
// 9: explicit unwrap 3
// 10: explicit unwrap 4
// 11: planar local x
// 12: planar local y
// 13: planar world x
// 14: planar world y
// 15: screen space
// 16: tri planar local
// 17: tri planar world
// 18: tri planar world local z
uint32 colorChannels // 0: RGB
// 1: RGBA
// 2: A
// 3: R
// 4: G
// 5: B
AnimationReference rgbMultiply // float
AnimationReference rgbAdd // float
uint32 unknown1
float perlinNoiseAmplifier
float perlinNoiseFrequency
uint32 replaceableChannel // -1 no replaceable
// 0: channel 1
// 1: channel 2
// 2: channel 3
// 3: channel 4
// 4: channel 5
// 5: channel 6
// 6: channel 7
uint32 unknown4
uint32 unknown5
uint32 unknown6
AnimationReference unknown7 // uint32
AnimationReference unknown8 // float
uint32 flipbookRows
uint32 flipbookColumns
AnimationReference flipbookFrame // uint16
AnimationReference uvOffset // float[2]
AnimationReference uvAngle // float[3]
AnimationReference uvTiling // float[2]
AnimationReference unknown9 // uint32
AnimationReference unknown10 // uint32
AnimationReference brightness // float
if (version > 23) {
AnimationReference triPlannarOffset // float[3]
AnimationReference triPlannarScale // float[3]
}
uint32 unknown11
uint32 fresnelType
float fresnelExponent
float fresnelMin
float fresnelMax
float unknown12
float unknown13
float unknown14
float fresnelMask
float fresnelRotation
}
MAT_ {
Reference name // CHAR
uint32 specialFlags // 0x1: use depth blend
// 0x4: use vertex color
// 0x8: use vertex alpha
// 0x200: use transparent shadows
uint flags // 0x1: use vertex color
// 0x2: use vertex alpha
// 0x4: unfogged
// 0x8: two sided
// 0x10: unshaded
// 0x20: no shadows cast
// 0x40: no hit test
// 0x80: no shadows recieved
// 0x100: depth prepass
// 0x200: use terrain HDR
// 0x800: splat UV fix
// 0x1000: soft blending
// 0x4000: transparent shadows
// 0x8000: decal lighting
// 0x10000: transparency affects depth
// 0x20000: local lights on transparencies
// 0x40000: disable soft depth blend
// 0x80000: double lambert
// 0x100000: hair layer sorting
// 0x200000: accepts splats
// 0x400000: decal required on low end
// 0x800000: emissive required on low end
// 0x1000000: specular required on low end
// 0x2000000: accepts splats only
// 0x4000000: is background object
// 0x10000000: zfill required on low end
// 0x20000000: exclude from highlighting
// 0x40000000: clamp output
// 0x80000000: visible
uint32 blendMode // 0: opaque
// 1: blend
// 2: additive
// 3: addAlpha
// 4: mod
// 5: mod2x
int32 priority
uint32 unknown0
float specularity
float depthBlend
uint8 alphaTestThreshold
uint8 unknown1
uint8 unknown2
uint8 unknown3
float specMult
float emisMult
Reference diffuseLayer // LAYR
Reference decalLayer // LAYR
Reference specularLayer // LAYR
if (version > 15) {
Reference glossLayer // LAYR
}
Reference emissiveLayer // LAYR
Reference emissive2Layer // LAYR
Reference diffuseLayer // LAYR
Reference evioLayer // LAYR
Reference evioMaskLayer // LAYR
Reference alphaMaskLayer // LAYR
Reference alphaMask2Layer // LAYR
Reference normalLayer // LAYR
Reference heightLayer // LAYR
Reference lightMapLayer // LAYR
Reference ambientOcclusionLayer // LAYR
uint32 unknown4
uint32 layerBlendType
uint32 emisBlendType
uint32 emisMode
uint32 specrType
AnimationReference parallaxHeight // float
AnimationReference unknown6 // float
}
DIS_ {
Reference name // CHAR
uint32 unknown0
AnimationReference strengthFactor // float
Reference normalLayer // LAYR
Reference strengthLayer // LAYR
uint32 flags
int32 priority
}
CMP_ {
Reference name // CHAR
uint32 unknown0
Reference sections // CMS_
}
CMS_ {
uint32 materialReferenceIndex
AnimationReference alphaFactor // float
}
TER_ {
Reference name // CHAR
Reference terrainLayer // LAYR
if (version > 0) {
uint32 unknown0
}
}
VOL_ {
Reference name // CHAR
uint32 blendMode
uint32 falloffType
AnimationReference volumeDensity // float
Reference colorDefiningLayer // LAYR
Reference noiseLayer1 // LAYR
Reference noiseLayer2 // LAYR
uint32 unknown0
uint32 unknown1
}
VON_ {
Reference name // CHAR
uint32 falloffType // 0: linear
// 1: exponential
uint32 flags // 0x1: draw before transparency
AnimationReference volumeDensity // foat
AnimationReference nearPlane // float
AnimationReference falloff // float
Reference colorDefiningLayer // ?
Reference noiseLayer1 // ?
Reference noiseLayer2 // ?
AnimationReference scrollRate // float[3]
AnimationReference translation // float[3]
AnimationReference scale // float[3]
AnimationReference rotation // float[3]
uint32 alphaThreshold
uint32 unknown0
}
CREP {
Reference name // CHAR
Reference creepLayer // LAYR
}
PAR_ {
uint32 bone
uint32 materialReferenceIndex
if (version > 16) {
uint32 additionalFlags // 0x1: randomize with emission speed 2
// 0x2: randomize with lifespan 2
// 0x4: randomize with mass 2
// 0x8: trailing enabled
}
AnimationReference emissionSpeed1 // float
AnimationReference emisisonSpeed2 // float
if (version < 13) {
uint32 randomizeWithEmissionSpeed2
}
AnimationReference emissionAnbleX // float
AnimationReference emissionAnbleY // float
AnimationReference emissionSpreadX // float
AnimationReference emissionSpreadY // float
AnimationReference lifespan1 // float
AnimationReference lifespan2 // float
if (version < 13) {
uint32 randomizeWithLifespan2
}
Reference unknown0 // ?
float zAcceleration
float sizeAnimationMiddle
float colorAnimationMiddle
float rotationAnimationMiddle
if (version > 16) {
float sizeHoldTime
float colorHoldTime
float alphaHoldTime
float rotationHoldTime
}
AnimationReference particleSizes1 // float[3]
AnimationReference rotationValues1 // float[3]
AnimationReference initialColor1 // uint8[4]
AnimationReference middleColor1 // uint8[4]
AnimationReference finalColor1 // uint8[4]
float slowdown
float mass
float mass2
float unknown1
if (version < 13) {
uint32 unknown2
uint32 trailingEnabled
}
uint16 localForceChannels
uint16 worldForceChannels
uint16 forceChannelsFillerBytes
uint16 worldForceChannelsCopy
float noiseAmplitude
float noiseFrequency
float noiseCohesion
float noiseEdge
uint32 indexPlusHighestIndex
uint32 maxParticles
AnimationReference emissionRate // float
uint32 emissionAreaType
AnimationReference emissionAreaSize // float[3]
AnimationReference emissionAreaCutoutSize // float[3]
AnimationReference emissionAreaRadius // float
AnimationReference emissionAreaCutoutRadius // float
if (version > 16) {
Reference unknown3 // U32_
}
uint32 emissionType
uint32 randomizeWithParticleSizes2
AnimationReference particleSizes2 // float[3]
uint32 randomizeWithRotationValues2
AnimationReference rotationValues2 // float[3]
uint32 randomizeWithColor2
AnimationReference initialColor2 // uint8[4]
AnimationReference middleColor2 // uint8[4]
AnimationReference finalColor2 // uint8[4]
uint32 unknown4
AnimationReference partEmit // int16
uint8 phase1StartImageIndex
uint8 phase1EndImageIndex
uint8 phase2StartImageIndex
uint8 phase2EndImageIndex
float relativePhase1Length
uint16 numberOfColumns
uint16 numberOfRows
float columnWidth
float rowHeight
float unknown5
float unknown6
float unknown7
byte[20] unknown8
uint32 particleType // ?
float lengthWidthRatio
byte[8] unknown9
float unknown10
if (version > 16) {
float unknown11
}
uint32 unknown12
AnimationReference unknown13 // float
AnimationReference unknown14 // float
uint32 unknown15
AnimationReference unknown16 // float
AnimationReference unknown17 // float
uint32 unknown18
AnimationReference unknown19 // float
AnimationReference unknown20 // float
uint32 unknown21
AnimationReference unknown22 // float
AnimationReference unknown23 // float
uint32 unknown24
AnimationReference unknown25 // float
AnimationReference unknown26 // float
uint32 unknown27
AnimationReference unknown28 // uint32
AnimationReference unknown29 // uint32
uint32 unknown30
AnimationReference unknown31 // uint32
AnimationReference unknown32 // uint32
uint32 unknown33
AnimationReference unknown34 // uint32
AnimationReference unknown35 // uint32
uint32 unknown36
AnimationReference unknown37 // uint32
AnimationReference unknown38 // uint32
AnimationReference unknown39 // float
if (version > 21) {
AnimationReference unknown40 // uint32
}
uint32 flags // 0x1: sort
// 0x2: collide terrain
// 0x4: collide objects
// 0x8: spawn on bounce
// 0x10: cutout emission area
// 0x20: inherit emission params
// 0x40: inherit parent velocity
// 0x80: sort by z height
// 0x100: reverse iteration
// 0x200: smooth rotation
// 0x400: bezier smooth rotation
// 0x2000: smooth color
// 0x4000: bezier smooth color
// 0x8000: lit parts
// 0x10000: random flipboo start
// 0x20000: multiply by gravity
// 0x40000: clamp tail parts
// 0x80000: spawn trailing parts
// 0x100000: fixed length tail parts
// 0x200000: use vertex alpha
// 0x400000: model parts
// 0x800000: swap y zon model parts
// 0x1000000: scale time by parent
// 0x2000000: use local time
// 0x4000000: simulate on init
// 0x8000000: copy
if (version > 17) {
uint32 rotationFlags // 0x2: relative rotation
// 0x4: always set
}
if (version > 16) {
uint32 colorSmoothingType // 0: linear
uint32 sizeSmoothingType // 1: smooth
uint32 rotationSmoothingType // 2: bezier
// 3: linear hold
// 4: bezier hold
AnimationReference unknown41 // float
AnimationReference unknown42 // float[2]
AnimationReference unknown43 // float[3]
AnimationReference unknown44 // float[2]
}
Reference spawnPoints // SVC3
float unknown45
uint32 unknown46
uint32 unknown47
AnimationReference unknown48 // uint32
AnimationReference unknown49 // float
int32 trailingParticleIndex
float trailingParticleChance
AnimationReference trailingParticlesRate // float
byte[8] unknown50
Reference usedModel // SCHR
Reference copyIndices // U32_
}
SVC3 {
float x
float y
float z
}
SCHR {
Reference path // CHAR
}
PARC {
AnimationReference emissionRate // float
AnimationReference partEmit // int16
uint32 bone
}
RIB_ {
uint8 bone
uint8 short1
uint8 short2
uint8 short3
uint32 materialReferenceIndex
if (version > 7) {
uint32 unknown0
}
AnimationReference waveLength // float
AnimationReference unknown1 // uint32
if (version < 7) {
int32 unknown2
}
AnimationReference unknown3 // float
AnimationReference unknown4 // float
AnimationReference unknown5 // float
AnimationReference unknown6 // float
AnimationReference unknown7 // float
AnimationReference unknown8 // float
uint32 unknown9
uint32 unknown10
float unknown11
float unknown12
float tipOffsetZ
float centerBias
float unknown13
float unknown14
float unknown15
if (version > 7) {
Reference unknown16 // ?
}
AnimationReference radiusScale // float[3]
AnimationReference twist // float
uint32 unknown17
uint32 unknown18
uint32 unknown19
uint32 unknown20
AnimationReference baseColoring // uint8[4]
AnimationReference centerColoring // uint8[4]
AnimationReference tipColoring // uint8[4]
float stretchAmount
if (version < 7) {
float unknown21
}
if (version > 7) {
float unknown22
}
float stretchLimit
float unknown23
float unknown24
uint32 unknown25
if (version < 7) {
uint32 unknown26
uint32 unknown27
}
float surfaceNoiseAmplitude
float surfaceNoiseNumberOfWaves
float surfaceNoiseFrequency
float surfaceNoiseScale
uint32 unknown28
uint32 type
uint32 unknown29
float ribbonDivisions
uint32 ribbonSides
float unknown30
AnimationReference ribbonLength // float
if (version < 7) {
int32 filler1
}
Reference srib // SRIB
AnimationReference unknown31 // uint32
uint32 flags // 0x2: collide with terrain
// 0x4: collide with objects
// 0x8: edge falloff
// 0x10: inherit parent velocity
// 0x20: smooth size
// 0x40: bezier smooth size
// 0x80: use vertex alpha
// 0x100: scale time by parent
// 0x200: force legacy
// 0x400: use locale time
// 0x800: simulate on initialization
// 0x1000: use length and time
uint32 unknown32
float unknown33
uint32 unknown34
uint32 unknown35
if (version > 7) {
byte[8] unknown36
}
uint32 directionVariationBool
AnimationReference directionVariationAmount // float
AnimationReference directionVariationFrequency // float
uint32 amplitudeVariationBool
AnimationReference amplitudeVariationAmount // float
AnimationReference amplitudeVariationFrequency // float
uint32 lengthVariationBool
AnimationReference lengthVariationAmount // float
AnimationReference lengthVariationFrequency // float
uint32 radiusVariationBool
AnimationReference radiusVariationAmount // float
AnimationReference radiusVariationFrequency // float
int32 unknown37
AnimationReference unknown38 // float
AnimationReference unknown39 // float
AnimationReference unknown40 // float
AnimationReference unknown41 // float
}
SRIB {
?
}
PROJ {
uint32 unknown0
uint32 bone
uint32 materialReferenceIndex // Maybe
AnimationReference unknown1 // float[3]
AnimationReference unknown2 // uint32
AnimationReference unknown3 // uint32
AnimationReference unknown4 // uint32
AnimationReference unknown5 // uint32
AnimationReference unknown6 // uint32
AnimationReference unknown7 // uint32
AnimationReference unknown8 // uint32
AnimationReference unknown9 // float
AnimationReference unknown10 // float
AnimationReference unknown11 // float
AnimationReference unknown12 // float
AnimationReference unknown13 // float
AnimationReference unknown14 // float
uint32 unknown15
float unknown16
float unknown17
float unknown18
float unknown19
uint32 unknown20
float unknown21
uint32 unknown22
float unknown23
uint32 unknown24
float unknown25
AnimationReference unknown26 // uint32
uint32 unknown27
uint32 unknown28
uint32 unknown29
uint32 unknown30
}
FOR_ {
uint32 type
uint32 unknown0
uint32 unknown1
uint32 bone
uint32 unknown2
uint32 forceChannels
AnimationReference forceStrength // float
AnimationReference forceRange // float
AnimationReference unknown3 // float
AnimationReference unknown4 // float
}
WRP_ {
uint32 unknown0
uint32 bone
uint32 unknown1
AnimationReference radius // float
AnimationReference unknown2 // float
AnimationReference compressionStrength // float
AnimationReference unknown3 // float
AnimationReference unknown4 // float
AnimationReference unknown5 // float
}
PHRB {
if (version < 3) {
float[15] unknown0To14
uint16 bone
uint16 boneUnused
byte[15] unknown15
}
if (version > 3) {
uint16 unknown16
uint16 unknown17
uint32 unknown18
uint32 bone
float unknown19
float unknown20
float unknown21
float unknown22
float unknown23
float unknown24
AnimationReference unknown25 // uint32
float unknown26
}
Reference physicsShapes // PHSH
uint32 flags // 0x1: collideable
// 0x2: walkable
// 0x4: stackable
// 0x8: simulate on collision
// 0x10: ignore local bodies
// 0x20: always exists
// 0x80: do not simulate
uint16 localForces // 0x1: 1
// 0x2: 2
// 0x4: 3
// 0x8: 4
// 0x10: 5
// 0x20: 6
// 0x40: 7
// 0x80: 8
// 0x100: 9
// 0x200: 10
// 0x400: 11
// 0x800: 12
// 0x1000: 13
// 0x2000: 14
// 0x4000: 15
// 0x8000: 16
uint16 worldForces // 0x1: wind
// 0x2: explosion
// 0x4: energy
// 0x8: blood
// 0x10: magnetic
// 0x20: grass
// 0x40: brush
// 0x80: trees
uint32 priority
}
PHSH {
float[16] matrix
if (version < 2) {
uint32 unknown0
}
uint8 shape // 0: box
// 1: sphere
// 2: capsule
// 3: cylinder
// 4: convex hull
// 5: mesh
uint8 unknown1
uint16 unknown2
if (version < 2) {
Reference vertices // VEC3
Reference unknown3 // U8__
Reference triangles // U16_
Reference planeEquations // VEC4
}
if (version > 2) {
byte[24] unknown4
}
float size0
float size1
float size2
if (version > 2) {
Reference unknown5 // VEC3
Reference unknown6 // VEC4
Reference unknown7 // DMSE
Reference unknown8 // U8__
Reference unknown9 // ?
uint32 unknown10
uint32 unknown11
uint32 unknown12
uint32 unknown13
uint32 unknown14
byte[84] unknown15
uint32 unknown16
uint32 unknown17
uint32 unknown18
uint32 unknown19
uint32 unknown20
uint32 unknown21
uint32 unknown22
uint32 unknown23
}
}
VEC3 {
float x
float y
float z
}
VEC4 {
float x
float y
float z
float w
}
DMSE {
uint8 unknown0
uint8 i0
uint8 i1
uint8 i2
}
PHYJ {
?
}
IKJT {
?
}
PATU {
?
}
TRGD {
Reference unknown0 // U32_
Reference name // CHAR
}
IREF {
float[16] matrix
}
SSGS {
uint32 shape // 0: cube
// 1: sphere
// 2: cylinder
int16 bone
uint16 unknown0
float[16] matrix
uint32 unknown1
uint32 unknown2
uint32 unknown3
uint32 unknown4
uint32 unknown5
uint32 unknown6
float size0
float size1
float size2
}
ATVL {
uint32 bone0
uint32 bone1
uint32 type
uint32 bone2
float[16] matrix
Reference unknown0 // VEC3
Reference unknown1 // U16_
float size0
float size1
float size2
}
BBSC {
?
}
TMD_ {
?
}Handling divisions
While the DIV_ reference is a normal reference, there is actually only one DIV_ chunk.
This chunk holds all the chunks that will bridge between the vertices and triangle indices to form regions.
The list of triangles is right there in the triangles unsigned short reference.
Each chunk in the regions reference of type REGN, will describe a geometry mesh, much like MDX's geoset.
The batch reference holds BAT_ chunks. Each one maps between regions and the materials they are using.
Finally, the boundingShapes reference holds MSEC objects which are the bounding shapes of all the different regions.
Handling regions
To construct the real regions, you need to take into account their offsets.
The firstVertexIndex variable, for example, is an index into the global vertex chunk (the U8__ reference, remember?).
That is, if we have a triangle index 5, it in fact points to
vertices[region.firstVertexIndex + 5].The same goes for firstTriangleIndex, and firstBoneLookupIndex.
Handling the bone lookup table
The bone lookup indices do not actually point to bones at all.
They are indices into the bone lookup reference belonging to MODL.
To get the first actual bone index of a vertex, you need the following:
C++:
boneIndex = MODL.boneLookup[region.firstBoneLookupIndex + vertex.boneLookupIndices[0]]The
region.boneWeightPairsCount variable tells you how many bones (up to 4 of course) any specific vertex actually uses.Handling vertices
Vertices in particular need transformations for their data.
Specifically, the normal, uvs and tangent.
For the normal and tangent, you must apply the following equation for each axis:
C++:
x = (x / 255.0) * 2.0) - 1.0For the uvs, each axis needs to be divided by 2048:
C++:
x = x / 2048.0Handling triangles
After all of the above data, this is the easiest. Just remember adding the region offset and you are fine.
C++:
triangleIndex[i] = division.triangles[region.firstTriangleIndex + i]Handling materials
Each batch chunk, or BAT_, holds the index of the region to use for the batch, and the material index.
This material index doesn't actually point to an actual material, but rather into the material map reference, or MATM.
Each material map has a material type, and material index.
The material type starts with 1, being standard materials, 2 being displacement materials, and so on, in the same order as they are read in the MODL chunk.
The material index, is the index into the appropriate material reference.
That is, if a batch references material index 1, and we see that MODL.materialMap[1] has materialType 1 and materialIndex 3, then the correct material is MODL.standardMaterials[3].
Sequences, STC, STG and STS chunks
The SEQS chunk holds mostly sequence metadata.
The STC chunks are the chunks that actually hold all the animation data, more about using them in the following section.
STS chunks give an easy way to check if we have valid animation data, given an animation reference and a frame of animation. More about them after STC chunks.
STG chunks are the chunks that define how the animation should look - it has a list of indices into the STC array, which you will use for animation. More about them after STS.
Handling animation references - STC
Every animation reference has an animId variable.
This variable holds a random unique number, which lets us figure where the actual data is.
For each STC object, there is a list of animIds and animRefs.
First, you take your animId value, and search for the equivalent value in the STC animIds list.
If you found it, then save the index in the list where you found it.
For example, if STC.animIds[2] == animationReference.animId, then save the index 2.
In case the STC doesn't have the animId, return the animation reference's initValue variable.
Now, I wrote that STC.animRefs is a U32_ reference, because it is, but each element is in fact two separate unsigned short numbers.
Because of this, you might want to simply read them as a pair of unsigned shorts instead of an unsigned long.
Remember the 2 you saved from animIds?
Now index STC.animRefs with that, so save STC.animRefs[2].
The first part of this animRef is yet another index, into another array, so save it.
Let's say it was 5.
The second part of the animRef is the index of the array you want to index.
Let's say it was 2.
There are 13 different sequence data chunks, that hold all the actual animation data for the STC.
The first one is sdev, the second is sd2v, the third is sd3v, and so on.
Now, since our saved animRef was [5, 2], the correct animation data is from the third (0-based indexing) SD, like so:
STC.sd3v[5]
Finally, once you've got the correct block inside the correct sequence data chunk, you can use its keys (which form the animation's timeline) and values to get the actual animation data.
In code:
C++:
animIds = STC.animIds
animRefId = -1
for (uint32 i = 0; i < animIds.length; i++) {
if (animIds[i] == animationReference.animId) {
animRefId = i
}
}
if (animRefId != -1) {
animRef = STC.animRefs[animRefId]
// This part assumes you read all the sequence data chunks into an array, e.g. [sdev, sd2v, sd3v, sd4q, ...]
sd = STC.SD[animRef[1]]
sdblock = sd[animRef[0]]
// Now do something with the sequence data block
} else {
return animationReference.initValue
}Handling animation references - STS
The only purpose of STS is to do a fast check whether some STC chunk will actually hold any animation data for an animation reference.
Each STC chunk has a reference index to a STS chunk.
If you want to check if a STC has animation data for an animation reference, simply select the relevant STS, and see if it has the animation reference's animId in its animIds list.
C++:
animIds = STS[STC.stsIndex].animIds
for (uint32 i = 0; i < animIds.length; i++) {
if (animIds[i] == animationReference.animId) {
// The relevant STC has animation data
}
}Handling animation references - STG
STG chunks are the last part of handling animation references.
They hold a list of indices into the STC array. Usually this list will only have one element, but sometimes it will have two, or more (for example, splitbody animations).
When you have an animation reference and want to get data from it, you don't directly access STC chunks.
Instead you do the following:
- Get the STG you want.
- Loop over all the STC indices it holds.
- If a STC chunk has animation data for this animation reference, return it, else continue looping.
- If no STC had animation data, return the animation reference's initValue variable.
In code:
C++:
stcIndices = STG.stcIndices
for (uint32 i = 0; i < stcIndices.length; i++) {
stc = STC[stcIndices[i]]
sts = STS[stc.stsIndex]
// First check the STS to see if there is animation data
if (hasAnimationData(sts, animationReference)) {
// And since there is, get it
return getAnimationData(stc, animationReference)
}
}
// Non of the STCs in the list had animation data, so return initValue
return animationReference.initValueNote: STC chunks have a
priority variable. You must always loop over the STC chunks according to priority, highest goes first. So far, however, every model that I tested had STG.stcIndices already sorted by priorities.Handling bones
The bone chunk reference gives us the skeletal structure of the model.
Generally speaking, to animate a skeleton, you first want to start with the root, transform it, then move on to its children, and to their children, and so on.
So far every model that I checked came sorted, such that a child bone will always come after its parent bone, in the array.
This helps, because there is no need to sort them on every run, instead you just loop over all the bones and transform each one as you loop over it.
To transfrom the bones for some chosen sequence i:
- Grab STG i.
- Use it as explained in the previous section to get the animation data of the bone's translation, scale, and rotation animation references.
- Apply them all in that order to an identity matrix, let's call the resulting matrix the local matrix.
- If the bone has a parent, transform it by its parent's world matrix.
- Finally the result (if there is no parent, the result is the local matrix) is the bone's world matrix.
C++:
for (uint32 i = 0; i < bones.length; i++) {
bone = bones[i]
// Start with an identity matrix
localMatrix = [1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1]
// Assuming getAnimationData() gets correctly animation data from an STG as described above, for some known frame of animation
translation = getAnimationData(stg, bone.translation)
scale = getAnimationData(stg, bone.scale)
rotation = getAnimationData(stg, bone.rotation)
// Transform the local matrix by the above animation data
// This section is highly dependant on your application code and handiness
// For the sake of simplicity, let's assume that all the animation data is stored in matrices, in which case we can throw the identity matrix to the garbage
localMatrix = translation * scale * rotation
if (bone.parent != -1) {
// If there is a parent, transform the local matrix by it
bone.worldMatrix = bones[bone.parent].worldMatrix * localMatrix
} else {
bone.worldMatrix = localMatrix
}
}Before rendering, all the bones will need to have another transformation on them.
For each bone i, transform its world matrix with the i'th matrix from MODL.initialReference.
The purpose of the initial reference matrices is to move the model into bind pose, to which the bone animation is then applied.
C++:
for (uint32 i = 0; i < bones.length; i++) {
bones[i].finalMatrix = bones[i].worldMatrix * MODL.initialReference[i]
}Optimizations
- While I wrote all the material types and all the SD chunks as their own variables, it will make life easier if they were read into an array.
This way you can directly index that array when you need to get the correct SD or material type.
For example, instead of STC.sd3v[5], have STC.SD[2][5]. - When you read in the STC and STS chunks, it is a good idea to transform their animIds arrays into maps.
Simply take each element of the animIds array into a map, and set it to its index:
With this map, when you want to figure out if a STC/STS object has a desired animId from an animation reference, you can get it directly withC++:animIds = new Map() // Allows direct checks instead of loops for (uint32 i = 0; i < STC.animIds.length; i++) { animIds[STC.animIds[i]] = i; }animIds[animationReference.animId]. - VEC3 and SVC3 both describe a 3D float vector, use one structure for the both of them.
- VEC4 and QUAT both describe a 4D float vector, use one structure for the both of them.
- SCHR holds only a CHAR reference, so you can read it directly as a CHAR reference.
Last edited:
