OBD:Instance file format: Difference between revisions

{{Hatnote|".dat" redirects here; for other files ending in ".dat", see [[Oni (folder)]].<br>

:''The [[Raw|Raw and separate file formats]] page should be read after this one.''}}

Files in GameDataFolder/ named "level[0-19]_Final.dat", together with ".raw" and sometimes ".sep" counterparts, contain the game data for Oni.

The same format was used for the tool files, named level0_Tools.dat/.raw/.sep, however the retail Oni game application does not load tool files; for the story behind the tool files, see [[level0_Tools]].

During development, Oni had an [[level0_Tools|in-game editor]] which presented a GUI for manipulating AIs, particles, etc. in a level. When a developer saved his work, the contents of the level, stored in his PC's RAM, were flushed directly to disk. Thus the structure of the .dat/.raw/.sep files reflects the way in which Bungie West chose to store levels in memory. So when we read the data in the files with a hex editor, we can see eccentricities such as blank space (coming from unused fields and byte-alignment padding) and garbage data (such as now-meaningless pointer values). [[OBD:Raw and separate file formats#Gaps|Further gaps]], mostly representing orphaned obsolete resources, add up to about 25 MB for the whole game.

Additionally, because the levels were built on Intel-based machines, which use a little-endian architecture, sequences of bytes which represent numbers were written from least-significant to most-significant byte. [[wp:FourCC|FourCCs]] in the data are stored "backwards", such as "13RV" which is meant to be read "VR31", because Bungie defined those four bytes as a 32-bit integer, not a string, causing them to be written to disk in little-endian order.

{{OBDtr| 0x00 | int64  | | 1F 27 DC 33 DF BC 03 00 | 0x0003BCDF33DC271F | Total template checksum (main indicator of engine compatibility):

*0x0003BCDF23C13061 ("Mac") - templates compatible with Windows demo and Mac engines

OniSplit's .oni files (VR32) use PC checksum by default and Mac/PS2 checksums where required (SNDD, TXMP, AGQG, M3GM, IGSt, TSFT/TSGA, TRAM/TREX) }}

{{OBDtr| 0x08 | int32  | | 31 33 52 56            | '13RV'            | .dat version (meant to be read as "VR31")<br>OniSplit's .oni files use '23RV' ("VR32") instead<br>OniX's [[Oni (folder)|GDFX]] uses '33RV' ("VR33") to signify that the new data versioning system is in use }}

{{OBDtr| 0x0C | int16   | | 40 00 | 64 | size of this header }}

{{OBDtr| 0x0E | int16  | | 14 00 | 20 | size of instance descriptor }}

{{OBDtr| 0x10 | int16  | | 10 00 | 16 | size of template descriptor }}

{{OBDtr| 0x12 | int16  | | 08 00 | 8  | size of name descriptor }}

{{OBDtr| 0x30 | int32  | | 99 CF 40 00 | (garbage) | used by OniSplit for raw table offset }}

{{OBDtr| 0x34 | int32  | | 90 4F 63 00 | (garbage) | used by OniSplit for raw table size }}

{{OBDtr| 0x38 | int32  | | F4 55 5F 00 | (garbage) | unused }}

{{OBDtr| 0x3C | int32  | | 90 4F 63 00 | (garbage) | used by OniX (three high bytes) for data versioning; contains the highest data version (timestamp) found in any instance in this .dat; see instance descriptor table's 0x10 for format }}

The file's '''total template checksum''' is the sum of all the template checksums (see "Template descriptors" below). Oni looks at this number in order to validate that it can read this version of the game data format. In practical terms, the total checksum value given for Windows above tells us that this level data is in the .dat/.raw file scheme, and the value given for Mac Oni and the Windows demo tells us that the level data uses the .dat/.raw/.sep file scheme.

The '''version''' of the instance file is the format version. Reading it backwards, as discussed under the "Backwards and garbage data" section, we get "VR31", which probably means "version 3.1". This is the format version of all instance files in all releases of Oni, regardless of file scheme.

The '''descriptor sizes''' are the sizes of the instance, template, and name descriptors which are coming up in this file (see breakdowns in later sections). For instance, each instance descriptor will be 0x14, or 20 bytes, in length.

The '''descriptor counts''' are the sizes of arrays which are coming up in this file: the instance, name and template descriptors. For instance, the size of the instance descriptor array will be 0x2483, or 9,347 items, in length.

After this comes four "int"s of '''garbage'''. Space occupied by random values like this is common in the data files, and indicates that something stored in memory at this relative position was written to disk even though it wouldn't be meaningful on disk (probably pointers or uninitialized memory in a space that was being reserved for possible future use). The first two 32-bit fields are, however, used in .oni files generated by OniSplit, and the last 32-bit field is partly used by OniX for a new form of template versioning. Future usage of these fields by OniSplit and/or OniX may change (hopefully not too much).

{| class="wikitable"

| 0x00

| tag

| 54 42 55 53

| 'SUBT'

|colspan="2" align=left | template tag

| 0x04

| int32

| C8 30 22 00

| 0x2230C8

|colspan="2" align=left | data offset (relative to data table)

| 0x08

| int32

| 01 CB 00 00

| 0xCB01

|colspan="2" align=left | name offset (relative to name table)

| 0x0C

| int32

| C0 09 00 00

| 2496

|colspan="2" align=left | data size

| 0x10

| int32

| align=left | flags; possible values:

:0x'''01''' 00 00 00 - unnamed

:0x'''08''' 00 00 00 - instance's data is being used by duplicate instances as a source

| align=left | flags; possible values:

:0x'''04''' 00 00 00 - never used; intended to mark instance as pointing to duplicate data rather than its own data

:0x'''20''' 00 00 00 - "in batch file"

:0x'''40''' 00 00 00 - delete upon next save

This frees up the three higher bytes for the data versioning timestamp which is in YY/MM/DD format, stored thusly:

:0x00 00 '''00''' 00 - versioning timestamp – month

:0x00 00 00 '''00''' - versioning timestamp – year

You'll notice that the level file header lists fewer names (7,124) than instances (9,347). That's because there are 3 types of instance:

*Named and empty - "empty" instances are used in level-specific instance files (i.e. not in level0_Final.dat) to associate an instance ID with a name. For every empty resource, there's another one with a matching name in level0_Final.dat that has data in it. The empty resource in the instance file is (usually) looked up by ID, then the engine searches all the loaded files for a non-empty instance with the same name, causing it to find the actual file in the global data in level0_Final.dat.

Before we talk about the name table in depth, we can peek ahead at the name of this resource using the offset we've just been given. Let's add the offset 0xCB01 to 0x28F240, the file header's address for the name table. This gives us the address 0x29BD41. There we find the string "SUBTsubtitles".

The actual subtitle data should be found by adding the offset 0x2230C8 to 0x03BCA0, the file header's address for the data table, to get 0x25ED68. We're going to leave the full details of the data table for later, but below is the data you should actually see for the English Oni SUBT file at this address. You have to consult the [[SUBT]] page to know how to read this data.

After '''padding''' of 16 unused bytes, we find that, instead of data, there's an address of the actual data: it's in the level's raw file. Open level0_Final.raw and jump to address 0x01444480, and you should see "01_01_01 Griffin: Give me another reading.", and the rest of some very familiar dialogue continuing from there.

The '''array size''' of 609 tells the part of the engine that reads SUBT data to expect a chunk of 609 subtitled lines of dialogue.

The name descriptor array starts immediately after the instance descriptors array. To find the end of the instance descriptors, we can simply take the size of an instance descriptor, 20 bytes, and multiply it by the number of instance descriptors in the file header. In this case, that means 20 * 9347 = 186940, or 0x02DA3C. Adding that to 0x40 (the start of the instance descriptors) takes us to address 0x02DA7C. Voila, the start of the name descriptors.

Likewise, the template descriptor array starts directly after the name descriptors. Since name descriptors are 8 bytes, 8 * 7124 (taken from the header) = 56992, or 0xDEA0, and adding that to the name descriptor array's start address (0x02DA7C) gives us 0x03B91C as the start of the template descriptors.

The template descriptor array contains information about all templates (that is, resource types, aka tags), used in the file (56 in this case, as we learned from the file header). Any resource occurring in this instance file has to have its type listed here. Here is the template descriptor at 0x3B9FC:

{{OBDtr| 0x00 | int64  | | 3C B9 A6 71 08 00 00 00 | 0x871A6B93C | template checksum }}

{{OBDtr| 0x08 | tag    | | 45 47 52 54             | 'EGRT'      | template tag }}

{{OBDtr| 0x0C | int32  | | 01 00 00 00            | 1          | unused: number of resources in file that use this template }}

The '''template checksum''' is used to prevent loading of instance files that are not compatible with the current engine version. The '''tag''' is the same kind of number-written-as-backwards-ASCII that we discussed in the "Backwards and garbage data" section; in this case, 'EGRT' means [[TRGE]]. The field for the '''number of resources''' using this template is unused. The number should be correct for each template, but Oni never uses it for anything.

 

You might wonder how Oni knows how to read each type of data, such as a SUBT or an ABNA. The simple answer is that this information is hard-coded into Oni. In fact, the information on each instance type, as stored in Oni's code, is actually the real "template". The file data only gives the tag and checksum that refer to a certain template. Which types of data fields are encountered in which order is already known by Oni. These hardcoded templates also tell Oni which parts of the file data are reserved for pointers.

 

That's because an instance may have pointers to other related instances, but pointers are only valid in memory; they cannot be stored on disk. They must be set when the level data is loaded into memory and the address in RAM has been determined. So one type of data field in Oni's templates is a raw pointer; on Macs and the Windows demo, there is an additional "separate offset" type. The pointer and offset are 32 bits in length, as one must expect since Oni was compiled for 32-bit PCs.

 

Incidentally, the templates in Oni's code have not just the familiar four-character tags attached to them, but also a descriptive string, e.g. "BSP Tree Node Array". This is the source of the names on the [[OBD:File types|File types]] page.

The data table stores all the instance data (or points to its actual location in a raw/separate file). We peeked at this before when we looked at the instance descriptor for SUBTsubtitles.

The instance ID and file ID are not actually part of the instance data, but are considered to be the resource header. The engine always keeps pointers to the start of the type-specific data itself; we saw this before when we jumped to 0x25ED68 and saw the data for the SUBT rather than the header for this data. The instance ID and file ID are accessed using negative offsets when needed (usually to find the name or template tag of an instance, given a pointer to it).

The '''instance's ID''' is computed as:

  (instance_descriptor_index << 8) <nowiki>|</nowiki> 1

The 1 allows the engine to know which IDs have already been converted to pointers (an instance pointer will always be 8 byte-aligned, so it will never have the zero bit already set).

The '''file ID''' is computed from the name of the instance file. For "_Final" files the file ID is computed as:

Again, the 1 allows the engine to know which file IDs have already been converted to pointers.

As you can see, after the header, the size of the actual instance data can be almost anything. Thus, we cannot compute the end of the data table in any simple way. That's why the instance file header explicitly gives us the address of the name table that comes after this.

By the way, how do we know which resource's data we're looking at in the data table? Let's look at the very first data, at 0x03BCA0. Noting that the first two numbers, the instance and file ID, do not count as data, there must be a resource with a data offset of 0x08, the lowest offset possible into the table. We can find this offset listed right at the start of the instance descriptor array:

{{OBDtr| 0x10 | int32  | | 00 00 00 00 | 0      | flags }}

These names can be up to 63 characters long, counting the tag. The instance file concludes with the end of the name table.