OBD:Instance file format: Difference between revisions

{{UpdatedForOniX|1.0.0}}

{{Hatnote|".dat" redirects here; for other files ending in ".dat", see [[Oni (folder)]].}}

{{Hatnote|You should read the [[Game data terminology]] page before this one.}}

{{Hatnote|The [[Raw|Raw and separate file formats]] page should be read after this one.}}

Files in GameDataFolder/ named "level[0-19]_Final.dat", together with ".raw" and sometimes ".sep" counterparts, contain the game data for Oni.

 

The level 0 files do not actually contain a level, but instances (resources) shared across all levels. Level 0 is loaded when the game starts, and never unloaded. All other level files, 1-19, are only loaded when their corresponding level starts, and unloaded when it ends. Since Oni can only hold these two levels in memory concurrently, resources have to be duplicated on disk whenever a character class, sound effect, etc. occurs in more than one level. For instance, although there are only 2,380 unique sounds in the game, there are 7,386 sounds stored across all level data files.

During development, Oni had an [[level0_Tools|in-game editor]] which presented a GUI for manipulating AIs, particles, etc. in a level. When a developer saved his work, the contents of the level, stored in his PC's RAM, were flushed directly to disk. Thus the structure of the .dat/.raw/.sep files reflects the way in which Bungie West chose to store levels in memory. So when we read the data in the files with a hex editor, we can see eccentricities such as blank space (coming from unused fields and byte-alignment padding) and garbage data (such as now-meaningless pointer values). [[OBD:Raw_and_separate_file_formats#Gaps|These gaps]] between data chunks add up to about 25 MB for the whole game.

Additionally, because the levels were built on Intel-based machines, which use a little-endian architecture, sequences of bytes which represent numbers were written from least-significant to most-significant byte. [[wikipedia:FourCC|FourCCs]] in the data are stored "backwards", such as "13RV" which is meant to be read "VR31", because Bungie defined those four bytes as a 32-bit integer, not a string, causing them to be written to disk in little-endian order.

Here is a walkthrough of an instance file using the level0_Final.dat in English Windows Oni. Follow along in a hex editor for maximum learnage. Each term will be explained in-depth when we fully consider the related data. First, here is how the file begins:

{{OBDtr| 0x00 | int64   | | 1F 27 DC 33 DF BC 03 00 | 0x0003BCDF33DC271F | Total template checksum (main indicator of engine compatibility):

*0x0003BCDF33DC271F ("PC") - templates compatible with Windows retail engine(s)

*0x0003BCDF23C13061 ("Mac") - templates compatible with Windows demo and Mac engines

*0x0003BA70A8D8AE11 ("PS2") - templates compatible with PlayStation 2 engine(s)

*0x0000000000000000 (blank) - for use with [[OniX]] engine(s) (VR33), which handle data versioning using the 0x3C field below

OniSplit's .oni files (VR32) use PC checksum by default and Mac/PS2 checksums where required (SNDD, TXMP, AGQG, M3GM, IGSt, TSFT/TSGA, TRAM/TREX) }}

{{OBDtr| 0x08 | int32   | | 31 33 52 56             | '13RV'             | .dat version (meant to be read as "VR31")<br>OniSplit's .oni files use '23RV' ("VR32") instead<br>OniX's [[Oni (folder)|GDFX]] uses '33RV' ("VR33") to signify that the new data versioning system is in use }}

{{OBDtr| 0x0C | int16   | | 40 00 | 64 | size of this header }}

{{OBDtr| 0x0E | int16   | | 14 00 | 20 | size of instance descriptor }}

{{OBDtr| 0x10 | int16   | | 10 00 | 16 | size of template descriptor }}

{{OBDtr| 0x12 | int16   | | 08 00 | 8 | size of name descriptor }}

{{OBDtr| 0x14 | int32   | | 83 24 00 00 | 9347      | instance descriptor count  }}

{{OBDtr| 0x18 | int32   | | D4 1B 00 00 | 7124      | name descriptor count }}

{{OBDtr| 0x1C | int32   | | 38 00 00 00 | 56        | template descriptor count }}

{{OBDtr| 0x20 | int32   | | A0 BC 03 00 | 0x03BCA0  | data table offset }}

{{OBDtr| 0x24 | int32   | | A0 35 25 00 | 2438560  | data table size }}

{{OBDtr| 0x28 | int32   | | 40 F2 28 00 | 0x28F240  | name table offset }}

{{OBDtr| 0x2C | int32   | | 04 4F 02 00 | 151300    | name table size }}

{{OBDtr| 0x30 | int32   | | 99 CF 40 00 | (garbage) | used by OniSplit for raw table offset }}

{{OBDtr| 0x34 | int32   | | 90 4F 63 00 | (garbage) | used by OniSplit for raw table size }}

{{OBDtr| 0x38 | int32   | | F4 55 5F 00 | (garbage) | unused }}

{{OBDtr| 0x3C | int32   | | 90 4F 63 00 | (garbage) | used by OniX (three high bytes) for data versioning; contains the highest data type version found in any instance in this .dat }}

This frees up the three higher bytes for the data versioning timestamp:

:0x00 '''00''' 00 00 - versioning timestamp – year

:0x00 00 00 '''00''' - versioning timestamp – day

*:In vanilla Oni .dats there are some rare occurrences of unnamed non-empty ''orphan'' instances (e.g., [[OBD:File_types/Named#TRCM|TRCM]]). These are a form of garbage and are discarded by OniSplit when unpacking a level.  

The template descriptor array contains information about all templates (that is, resource types, AKA tags), used in the file (56 in this case, as we learned from the file header). Any resource occurring in this instance file has to have its type listed here. Here is the template descriptor at 0x3B9FC:

{{OBDtr| 0x0C | int32  | | 01 00 00 00            | 1          | number of resources in file that use this template }}

The '''template checksum''' is used to prevent loading of instance files that are not compatible with the current engine version. The '''tag''' is the same kind of number-written-as-backwards-ASCII that we discussed in the "Backwards and garbage data" section; in this case, 'EGRT' means [[TRGE]]. The '''number of resources''' is self-explanatory.