OBD:M3GM: Difference between revisions

From OniGalore
Jump to navigation Jump to search
mNo edit summary
(objective complete... ish)
Line 15: Line 15:
{{OBDtr|0x18|link    |FFFFC8|01 ED 00 00|237  | link to [[OBD:TXCA|00237-.TXCA]] (vertex UVs) }}
{{OBDtr|0x18|link    |FFFFC8|01 ED 00 00|237  | link to [[OBD:TXCA|00237-.TXCA]] (vertex UVs) }}
{{OBDtr|0x1C|link    |FFFFC8|01 0C 01 00|268  | link to [[OBD:IDXA_M3GM_1|00268-.IDXA]] (triangle strips) }}
{{OBDtr|0x1C|link    |FFFFC8|01 0C 01 00|268  | link to [[OBD:IDXA_M3GM_1|00268-.IDXA]] (triangle strips) }}
{{OBDtr|0x20|link    |FFFFC8|01 11 01 00|273  | link to [[OBD:IDXA_M3GM_2|00273-.IDXA]] (face normal assignment) }}
{{OBDtr|0x20|link    |FFFFC8|01 11 01 00|273  | link to [[OBD:IDXA_M3GM_2|00273-.IDXA]] (face grouping by normal) }}
{{OBDtr|0x24|link    |FFFFC8|01 D8 00 00|216  | link to 00216-.[[OBD:TXMP|TXMP]] (texture) }}
{{OBDtr|0x24|link    |FFFFC8|01 D8 00 00|216  | link to 00216-.[[OBD:TXMP|TXMP]] (texture) }}
{{OBDtr|0x28|link    |C8FFC8|00 00 00 00|unused| obsolete GMAN (geometry animation) link; never used in Oni}}
{{OBDtr|0x28|link    |C8FFC8|00 00 00 00|unused| obsolete GMAN (geometry animation) link; never used in Oni}}
Line 41: Line 41:
----
----
;PS2 implementation
;PS2 implementation
:PS2 M3GMs have the same data size as PC and Mac ones (0x24 bytes of actual data, 0x2C if counting the instance and level IDs), but the layout is different. Normals aren't stored at all, hence both VCRA and the second IDXA are simply missing. Instead the M3GM explicitly stores the number of vertices (i.e., the array size common to PNTA and TXCA), as well as another vertex count (not fully understood). There is also a mysterious .raw part (also not documented yet).
:PS2 M3GMs have the same data size as PC and Mac ones (0x24 bytes of actual data, 0x2C if counting the instance and level IDs), but the layout is different.
:*Face normals aren't stored at all, and vertex normals aren't stored as VCRA either, therefore both VCRA and the second IDXA are simply missing, which frees up 12 bytes worth of space for data fields unique to the PS2 format.
:*8 of the 12 freed-up bytes are used by explicitly storing the number of vertices (i.e., the array size common to PNTA and TXCA) and the number of faces (same count as the 2nd VCRA on PC and Mac), as two int32's.
:*The other 4 bytes are used to reference the compressed storage of vertex normals in the.raw file. This storage is very compact/lossy, using only 1 byte per normal (instead of 12 bytes per normal in the case of VCRA storage).
:An an example, consider M3GMaxes of level0_Final.
:An an example, consider M3GMaxes of level0_Final.
{{Table}}
{{Table}}
Line 48: Line 51:
{{OBDtr|0x04|lev_id  |FFFF00|01 00 00 00|0    | level 0}}
{{OBDtr|0x04|lev_id  |FFFF00|01 00 00 00|0    | level 0}}
{{OBDtr|0x08|int32  |FFC8C8|00 00 00 00|0    | runtime geometry flags (supposedly the same as for PC and Mac) }}
{{OBDtr|0x08|int32  |FFC8C8|00 00 00 00|0    | runtime geometry flags (supposedly the same as for PC and Mac) }}
{{OBDtr|0x0C|int32  |FFC8C8|60 00 00 00|96    | number of actual vertices? (see discussion below) }}
{{OBDtr|0x0C|int32  |FFC8C8|60 00 00 00|96    | number of faces (same as in second VCRA on PC and Mac) }}
{{OBDtr|0x10|int32  |FFFFC8|01 64 00 00|100  | link to [[OBD:PNTA|00100-.PNTA]] (vertex XYZs) }}
{{OBDtr|0x10|int32  |FFFFC8|01 64 00 00|100  | link to [[OBD:PNTA|00100-.PNTA]] (vertex XYZs) }}
{{OBDtr|0x14|link    |FFFFC8|01 65 00 00|101  | link to [[OBD:TXCA|00101-.TXCA]] (vertex UVs) }}
{{OBDtr|0x14|link    |FFFFC8|01 65 00 00|101  | link to [[OBD:TXCA|00101-.TXCA]] (vertex UVs) }}
{{OBDtr|0x18|link    |FFFFC8|01 66 00 00|102  | link to [[OBD:IDXA_M3GM_1|00102-.IDXA]] (triangle strips) }}
{{OBDtr|0x18|link    |FFFFC8|01 66 00 00|102  | link to [[OBD:IDXA_M3GM_1|00102-.IDXA]] (triangle strips) }}
{{OBDtr|0x1C|int32  |FFC8C8|6C 00 00 00|108  | number of vertices (same as in PNTA and TXCA); also size of .raw part in bytes }}
{{OBDtr|0x1C|int32  |FFC8C8|6C 00 00 00|108  | number of vertices (same as in PNTA and TXCA); also size of .raw part in bytes }}
{{OBDtr|0x20|offset  |FFFFC8|A0 8E 01 00| 0x00018EA0 | offset into the .raw file, purpose unknown (first 108 bytes shown in bold)
{{OBDtr|0x20|offset  |FFFFC8|A0 8E 01 00| 0x00018EA0 | offset into the .raw file where the the 108 compressed vertex normals are stored:
  '''2D 2D 2D 2D 2D 2D 2D 2D 2D 2D 2D 2D F8 F8 C2 55'''
  '''2D 2D 2D 2D 2D 2D 2D 2D 2D 2D 2D 2D F8 F8 C2 55'''
  '''40 40 10 D4 D4 09 C2 5F 6B 6B 6B 6B 6B 6B 6B 6B'''
  '''40 40 10 D4 D4 09 C2 5F 6B 6B 6B 6B 6B 6B 6B 6B'''
Line 60: Line 63:
  '''EA EA A7 A7 24 EA EA EA EA 24 02 02 02 A7 A7 A7'''
  '''EA EA A7 A7 24 EA EA EA EA 24 02 02 02 A7 A7 A7'''
  '''A7 EF EA EA EA EA 02 02 02 02 24 24 24 24 F8 2D'''
  '''A7 EF EA EA EA EA 02 02 02 02 24 24 24 24 F8 2D'''
  '''2D 2D 2D A7 A7 A7 A7 EA EA EA EA 16''' 02 02 02 02
  '''2D 2D 2D A7 A7 A7 A7 EA EA EA EA 16'''
After looking at PS2's implementation of TXMP, it is a fair assumption that they are color indices, but then it is unclear what palette is referenced (grayscale?)
See below for a overview of the compression/decompression rule, i.e., how the 256 values of a compressed normal are mapped to the unit sphere.
:It could also be that they're packed normal vectors, although compressing one normal direction into a single byte is rather lossy and coarse.
}}
}}
{{OBDtr|0x24|link    |FFFFC8|01 67 00 00|103  | texture link (to the empty 00103-_AXIS.[[OBD:TXMP|TXMP]]; the actually relevant 00104-_axis.TXMP is orphaned) }}
{{OBDtr|0x24|link    |FFFFC8|01 67 00 00|103  | texture link (to the empty 00103-_AXIS.[[OBD:TXMP|TXMP]]; the actually relevant 00104-_axis.TXMP is orphaned) }}
Line 68: Line 70:
{{OBDtr|0x2C|char[20]|C8FFFF|AD DE      |dead  | unused}}
{{OBDtr|0x2C|char[20]|C8FFFF|AD DE      |dead  | unused}}
|}
|}
Even without a full understanding of the .raw part, the handling of normals and the vertex count at 0x0C, the current knowledge is sufficient to extract M3GM data from PS2 resources and even to export it to other formats (with missing or autogenerated normals).
;OniBrowser being able to load PS2 M3GMs (a mystery of the modern times)
;OniBrowser being able to load PS2 M3GMs (a mystery of the modern times)
:Somehow OniBrowser is able to display M3GMs from PS2 instance files (except those with textures), even though all the links (PTNA, TXCA, IDXA) are in the wrong positions. Apparently, in the case of M3GM, OniBrowser reads in all the fields past 0x0C as a bunch of non-typed instance links, then identifies their types and populates the PNTA, VCRA, TXCA and IDXA references as soon (or as late) as a matching type is encountered.
:Somehow OniBrowser is able to display M3GMs from PS2 instance files (except those with textures), even though all the links (PTNA, TXCA, IDXA) are in the wrong positions. Apparently, in the case of M3GM, OniBrowser reads in all the fields past 0x0C as a bunch of non-typed instance links, then identifies their types and populates the PNTA, VCRA, TXCA and IDXA references with the first encountered instance of the matching type.
;Number of actual vertices (0x0C)
;Compressed vertex normals
:For low-poly meshes this vertex count corresponds to the number or individual XYZ coordinate meshes, i.e., the number of "actual" vertices or "control points", regardless of texture coordinates on adjacent faces. Thus this number is systematically 8 for cube-like meshes, 12, for a two-segment square tube, etc. However, for higher-poly meshes (e.g., character body parts), the vertex count at 0x0C actually exceeds the PNTA/TXCA count, possibly indicating that vertices with split normals (at sharp edges) are counted several times. This number isn't directly tied to other resources, so its meaning can only be deduced from a more careful/massive analysis of meshes.
:It is somewhat uncommon to pack a normal vector into a single byte, but it turns out that there is a relatively straightforward way to subdivide the unit sphere into 256 sectors. One starts by splitting the sphere into four large quadrants (based on tetrahedral symmetry), and then each of the big "triangles" is subdivided into 4, then 16, then 64 smaller triangles, like [https://www.researchgate.net/publication/338662028/figure/fig8/AS:848495074349060@1579308398315/The-4-fold-tetrahedron-version-Discrete-Global-Grid-Systems-DGGSs-based-on-SACs-level_Q320.jpg this]. There is some distorsion near the poles
;Packed normals?
:There isn't much of a point in giving a detailed description of the mapping here, but the basic idea would be to use the lowest 64 values for the first quadrant, the next 64 values for the second quadrant, etc. In actuality the PS2 implementation uses 63 for the first quadrant, 62 for the second, 63 for the third, and 68 for the last - for whatever reason. Within a quadrant the distribution isn't entirely regular, either. To be documented later if at all.
:Regarding the bytes in the .raw file and how they could be packed normals (e.g., in spherical coordinates) - these days normals are packed into no less that 2 bytes, but Oni's lighting is lackluster overall, so maybe drastically compressed normals were deemed sufficient. The analysis of box-like door meshes M3GMsphere (and maybe some pipes from a level's "furniture") should be able to clarify the format completely.
:When looking at the .raw part of the M3GM for some doors (level geometry), a striking observation is that there are only three distinct normal values (0xDE, 0x6A and 0x82), rather than the 6 normals that you'd expect for an axis-aligned box. Apparently half of those are bad, inward-pointing normals that no one ever bothered fixing. The regular VCRA normals for the same doors (on PC and Mac) are messed up in exactly the same way. Possibly this is related to the BSL variable '''door_pop_lighting''' ("uses bad door lighting").





Revision as of 21:26, 20 November 2022

ONI BINARY DATA
M3GA << Other file types >> Mtrl
M3GM : Geometry
switch to XML:M3GM page
Overview @ Oni Stuff
OBD.png


M3gm all.gif


Offset Type Raw Hex Value Description
0x00 res_id 01 D7 00 00 215 00215-door_1_0.M3GM
0x04 lev_id 01 00 00 06 3 level 3
0x08 int32 00 00 00 00 0 runtime only
0x0C link 01 DA 00 00 218 link to 00218-.PNTA (vertex XYZs)
0x10 link 01 F7 00 00 247 link to 00247-.VCRA (vertex normals)
0x14 link 01 F0 00 00 240 link to 00240-.VCRA (face normals)
0x18 link 01 ED 00 00 237 link to 00237-.TXCA (vertex UVs)
0x1C link 01 0C 01 00 268 link to 00268-.IDXA (triangle strips)
0x20 link 01 11 01 00 273 link to 00273-.IDXA (face grouping by normal)
0x24 link 01 D8 00 00 216 link to 00216-.TXMP (texture)
0x28 link 00 00 00 00 unused obsolete GMAN (geometry animation) link; never used in Oni
0x2C char[20] AD DE dead unused


Vertices
XYZ and UV coordinates are stored in parallel (same number of entries in PNTA and TXCA).
Vertex normals
The first VCRA stores the normals for every vertex (same entries as in PNTA and TXCA).
Vertex normals are used by Gouraud shading (directional lighting).
Face normals
The second VCRA stores the normals for every face (groups defined by the second IDXA).
Face normals are used for backface culling.
Triangle strips
The first IDXA lists the triangles as strips. The IDs are the ones in PNTA and IDXA.
The start of a new strip is signaled by a high bit in the ID of its first vertex.
Strips are more optimal for rendering, they are generated when authoring an M3GM.
Face groupings
The second IDXA groups the triangles into faces (oriented by the second VCRA).
Geometry animation
Geometry animation worked in a way similar to texture animation, by picking M3GMs from an array (GMAN) and displaying them at a specified frame rate, optionally with randomization.

PS2 implementation
PS2 M3GMs have the same data size as PC and Mac ones (0x24 bytes of actual data, 0x2C if counting the instance and level IDs), but the layout is different.
  • Face normals aren't stored at all, and vertex normals aren't stored as VCRA either, therefore both VCRA and the second IDXA are simply missing, which frees up 12 bytes worth of space for data fields unique to the PS2 format.
  • 8 of the 12 freed-up bytes are used by explicitly storing the number of vertices (i.e., the array size common to PNTA and TXCA) and the number of faces (same count as the 2nd VCRA on PC and Mac), as two int32's.
  • The other 4 bytes are used to reference the compressed storage of vertex normals in the.raw file. This storage is very compact/lossy, using only 1 byte per normal (instead of 12 bytes per normal in the case of VCRA storage).
An an example, consider M3GMaxes of level0_Final.
Offset Type Raw Hex Value Description
0x00 res_id 01 63 00 00 99 00099-axes.M3GM
0x04 lev_id 01 00 00 00 0 level 0
0x08 int32 00 00 00 00 0 runtime geometry flags (supposedly the same as for PC and Mac)
0x0C int32 60 00 00 00 96 number of faces (same as in second VCRA on PC and Mac)
0x10 int32 01 64 00 00 100 link to 00100-.PNTA (vertex XYZs)
0x14 link 01 65 00 00 101 link to 00101-.TXCA (vertex UVs)
0x18 link 01 66 00 00 102 link to 00102-.IDXA (triangle strips)
0x1C int32 6C 00 00 00 108 number of vertices (same as in PNTA and TXCA); also size of .raw part in bytes
0x20 offset A0 8E 01 00 0x00018EA0 offset into the .raw file where the the 108 compressed vertex normals are stored:
2D 2D 2D 2D 2D 2D 2D 2D 2D 2D 2D 2D F8 F8 C2 55
40 40 10 D4 D4 09 C2 5F 6B 6B 6B 6B 6B 6B 6B 6B
6B 82 82 D9 16 E3 E3 00 EB EB 6D 6D 6D 6D 6D 6D
6D 6D 6D 6D AB AB EB 80 52 FC 95 30 30 EA EA EA
EA EA A7 A7 24 EA EA EA EA 24 02 02 02 A7 A7 A7
A7 EF EA EA EA EA 02 02 02 02 24 24 24 24 F8 2D
2D 2D 2D A7 A7 A7 A7 EA EA EA EA 16

See below for a overview of the compression/decompression rule, i.e., how the 256 values of a compressed normal are mapped to the unit sphere.

0x24 link 01 67 00 00 103 texture link (to the empty 00103-_AXIS.TXMP; the actually relevant 00104-_axis.TXMP is orphaned)
0x28 link 00 00 00 00 unused supposedly the same GMAN link (geometry animation) as for PC and Mac
0x2C char[20] AD DE dead unused
OniBrowser being able to load PS2 M3GMs (a mystery of the modern times)
Somehow OniBrowser is able to display M3GMs from PS2 instance files (except those with textures), even though all the links (PTNA, TXCA, IDXA) are in the wrong positions. Apparently, in the case of M3GM, OniBrowser reads in all the fields past 0x0C as a bunch of non-typed instance links, then identifies their types and populates the PNTA, VCRA, TXCA and IDXA references with the first encountered instance of the matching type.
Compressed vertex normals
It is somewhat uncommon to pack a normal vector into a single byte, but it turns out that there is a relatively straightforward way to subdivide the unit sphere into 256 sectors. One starts by splitting the sphere into four large quadrants (based on tetrahedral symmetry), and then each of the big "triangles" is subdivided into 4, then 16, then 64 smaller triangles, like this. There is some distorsion near the poles
There isn't much of a point in giving a detailed description of the mapping here, but the basic idea would be to use the lowest 64 values for the first quadrant, the next 64 values for the second quadrant, etc. In actuality the PS2 implementation uses 63 for the first quadrant, 62 for the second, 63 for the third, and 68 for the last - for whatever reason. Within a quadrant the distribution isn't entirely regular, either. To be documented later if at all.
When looking at the .raw part of the M3GM for some doors (level geometry), a striking observation is that there are only three distinct normal values (0xDE, 0x6A and 0x82), rather than the 6 normals that you'd expect for an axis-aligned box. Apparently half of those are bad, inward-pointing normals that no one ever bothered fixing. The regular VCRA normals for the same doors (on PC and Mac) are messed up in exactly the same way. Possibly this is related to the BSL variable door_pop_lighting ("uses bad door lighting").


ONI BINARY DATA
M3GA << Other file types >> Mtrl
M3GM : Geometry
Generic file