OBD:Instance file format
- For other files ending in ".dat", see Oni (folder).
Files in GameDataFolder/ named "level[0-19]_Final.dat", together with ".raw" and sometimes ".sep" counterparts, contain the game data for Oni. The documentation for raw and separate files can be read after this page.
The same format was used for the tools files, named level0_Tools.dat/.raw/.sep, however the retail Oni game application does not load tools files; for the story behind the tools files, see HERE.
The level 0 files do not actually contain a level, but instances (resources) shared across all levels. Level 0 is loaded when the game starts, and never unloaded. All other level files are only loaded when the corresponding level starts and unloaded when it ends.
Terminology
- Level data files
The data for each level is found in two files in Windows retail Oni, ending in ".dat" and ".raw". On Macs and in the Windows demo, a third file type ending in ".sep" is used. These two/three files are collectively called level data files.
- Instance file
Any file ending in ".dat" found in GameDataFolder/. Instance files are the main type of data file in the sense that, when loading a level, Oni reads the instance file first, and this file serves as an index that allows it to find resources which are packed into the level data files.
- Instance, resource, file
This one can be confusing. An instance is often called a file on this wiki, but an instance is not the same as an "instance file". An instance is an individual resource, such as a texture. They are called files because they are individual pieces of data stored and cataloged in a "file system" composed of a set of .dat/.raw[/.sep] files. In at least some cases they were indeed actual, separate files on the developer's computer before being packed into the level data files.
When OniSplit is used to split level data files into individual files for each resource, in effect it is creating thousands of "single-instance instance files", at which point "instance", "file", and "instance file" become fully synonymous.
- Template, file type
A template represents a type of resource. Templates are identified with four-letter codes, often called tags, such as "SUBT" for subtitle files. Just as instances are often called files on the wiki, templates are often called file types.
- .oni
Generated by OniSplit, these files are Windows-format .dat files which basically contain a single instance extracted from an instance file, with all the instance data that was contained by the .raw/.sep files appended at the end.
Backwards and garbage data
During development, Oni had in-game editing tools. These tools presented a GUI for things like placing AIs and setting their attributes, editing particles, etc. When a developer saved his work, the contents of the level, stored in RAM, were written directly to disk. The structure of the .dat/.raw/.sep files reflects the way in which Bungie West chose to store levels in memory, and thus when when we read the data in the files with a hex editor, we can see various eccentricities such as blank space and garbage data that represented various RAM contents from the developer's PC.
Additionally, because the levels were built on Intel-based machines, which use a little-endian architecture, sequences of bytes which represent numbers were written from least-significant to most-significant byte, which looks "backwards" from the standpoint of a culture that reads left-to-right. When Macs, which were big-endian at the time due to their PowerPC architecture, read these files, they then had to flip each sequence of bytes in memory before they could be understood.
An exception to this backwards-writing rule is when strings of ASCII characters were written to disk. These are not numbers and thus are not subject to endianness, so they retain their left-to-right order. Now, this may not seem to be the case as you continue reading below. The first two strings of characters which you'll see are "13RV" and "TBUS", which are meant to be read "VR31" and "SUBT". The reason these four-character strings are backwards is that Oni stored them as a 32-bit integer. Any sequence of four characters can be represented as such a number. Writing the integer 1,448,227,633 to disk results in the bytes 0x31, 0x33, 0x52, and 0x56, which produce the ASCII codes for '1', '3', 'R' and 'V' (the computer would have had to be big-endian to be able to naturally write them in the left-to-right order we would prefer to see). This practice of Bungie's provided a combination of convenient storage of a tag in memory as a number, and human-readability when organizing game assets by tag.
File limits
- Max level number: 127
- Max number of instance files in GameDataFolder: 512 (Windows), 16 (Windows demo, Mac)
- Max number of simultaneously loaded instance files: 64
- Max number of instances in a file: 131071
- Max length of an instance file name: 31
- Max length of an instance name: 63 (including the 4 character template tag)
Header
Here is a walkthrough of an instance file using the level0_Final.dat in English Windows Oni. Follow along in a hex editor for maximum learnage. Each term will be explained in-depth when we fully consider the related data. First, here is how the file begins:
| Offset | Type | Raw Hex | Value | Description |
|---|---|---|---|---|
| 0x00 | int64 | 1F 27 DC 33 DF BC 03 00 | 0x0003BCDF33DC271F | Windows level file template checksum; Windows demo and Mac retail/demo use 0x0003BCDF23C13061 instead |
| 0x08 | int32 | 31 33 52 56 | 'VR31' | .dat version; .oni files use 'VR32' instead |
| 0x0C | int64 | 40 00 14 00 10 00 08 00 | 0x0008001000140040 | signature |
| 0x14 | int32 | 83 24 00 00 | 9347 | instance descriptor count |
| 0x18 | int32 | D4 1B 00 00 | 7124 | name descriptor count |
| 0x1C | int32 | 38 00 00 00 | 56 | template descriptor count |
| 0x20 | int32 | A0 BC 03 00 | 0x03BCA0 | data table offset |
| 0x24 | int32 | A0 35 25 00 | 2438560 | data table size |
| 0x28 | int32 | 40 F2 28 00 | 0x28F240 | name table offset |
| 0x2C | int32 | 04 4F 02 00 | 151300 | name table size |
| 0x30 | int32 | 00 00 00 00 | used by OniSplit only: raw table offset | |
| 0x34 | int32 | 00 00 00 00 | used by OniSplit only: raw table size | |
| 0x38 | int32 | 00 00 00 00 | unused | |
| 0x3C | int32 | 00 00 00 00 | unused |
The file's template checksum tells us that this level data is in the .dat/.raw file scheme, as opposed to the .dat/.raw/.sep file scheme used by Mac Oni and the Windows demo of Oni.
The version of the instance file is the format version. Reading it backwards, as discussed under "Introduction", we get "VR31", which is probably "version 31". This is the format version of all instance files in all releases of Oni.
The signature is identical in all instance files.
The descriptor counts are the sizes of some arrays which are coming up soon in this file: the instance, name and template descriptors. For instance, the size of the instance descriptor array will be 0x2483, or 9,347 items, in length.
Next we are told the addresses and sizes of the data and name tables in the instance file. The name table simply follows the data table, as you'll see if you add the data table offset plus the data table size, but that doesn't mean the name table offset is redundant; if its start was not 32-bit-aligned, it probably would be moved down to start at the next 32-bit word, but this is unnecessary because it happens to be aligned already.
After this comes four "int"s of zeroes. Empty space like this is common in the data files, and indicates that something stored in memory at this relative position was not written to disk (probably pointers, or sometimes a space reserved for possible future use in a resource type).
That concludes the header of the instance file. Immediately after this header, we find the instance descriptors array.
Instance descriptors
The instance descriptor array tells Oni where to find the data and the name of every instance (resource) indexed by the .dat file. The descriptors start at 0x40 in the .dat file, but below is a descriptor found at 0x017B50 in the file which makes a better example. In the table below, we use offsets relative to the start of this descriptor.
| Offset | Type | Raw Hex | Value | Description |
|---|---|---|---|---|
| 0x00 | tag | 54 42 55 53 | 'SUBT' | template tag |
| 0x04 | int32 | C8 30 22 00 | 0x2230C8 | data offset (relative to data table) |
| 0x08 | int32 | 01 CB 00 00 | 0xCB01 | name offset (relative to name table) |
| 0x0C | int32 | C0 09 00 00 | 2496 | data size |
| 0x10 | int32 | 00 00 00 00 | 0 | flags; possible values:
|
This descriptor tells us that a resource of type SUBT (a subtitle file for Oni; there are only two of these, one containing all speech subtitles, and one for help messages) has data that can be found 0x2230C8 bytes into the data table, which we learned from the file header starts at 0x03BCA0. Its name can be found 0xCB01 bytes into the name table that starts, according to the file header, at 0x28F240. The data's size is 0x09C0, or 2,496, bytes. The flags "unnamed" and "empty" require special explanation.
Unnamed and empty resources
You'll notice that the level file header lists fewer names (7,124) than instances (9,347). That's because there are 3 types of instance descriptors:
- Unnamed and not empty - they are only referenced by other instances in the same file, generally as child data (e.g., 3D geometry elements like ABNA are "contained" by AKEV, a level's environment).
- Named and not empty - they can be referenced by other instances in any file and the engine can use their name or template tag to find them.
- Named and empty - "empty" instances are used in level-specific instance files (i.e. not in level0_Final.dat) to associate an instance ID with a name. For every empty resource, there's another one with a matching name in level0_Final.dat that has data in it. The empty resource in the instance file is (usually) looked up by ID, then the engine searches all the loaded files for a non-empty instance with the same name, causing it to find the actual file in the global data in level0_Final.dat.
Peeking at instance name
If you want to see the name of this resource, let's look at address 0xCB01 + 0x28F240 (the file header's address for the name table) = 0x29BD41. There we find the string "SUBTsubtitles". The actual subtitle data should be found at the address 0x2230C8 + 0x03BCA0 (the file header's address for the data table) = 0x25ED68. Let's go there now....
Peeking at instance data
For some reason, the addresses we calculate from the descriptor data offsets are all off by eight bytes, so we need to subtract 8 from 0x25ED68 and go to 0x25ED60. Compare what you see here to the documentation for the SUBT type. Below is the data you should actually see for the English Oni SUBT file at this address. Note that we still haven't found the actual subtitle data, because SUBT stores its data in the raw file. The princess is in another castle:
| Offset | Type | Raw Hex | Value | Description |
|---|---|---|---|---|
| 0x00 | res_id | 01 F4 12 00 | 4852 | 04852-subtitles.SUBT |
| 0x04 | lev_id | 01 00 00 00 | 0 | level 0 |
| 0x08 | char[16] | AD DE | dead | unused |
| 0x18 | offset | 80 44 44 01 | 0x01444480 | raw file data address |
| 0x1C | int32 | 61 02 00 00 | 609 | array size |
The first two words, or 32-bit sequences, are the standard resource header. The second and third bytes of the first word are the resource ID. The second and third bytes of the second word are the level number where this resource is found.
After a buffer of 16 unused bytes, we find the address of the actual data: it's in the level's raw file. Open level0_Final.raw and jump to address 0x01444480, and you should see "01_01_01 Griffin: Give me another reading.", and the rest of some very familiar dialogue continuing from there.
The array size of 609 tells us that this is a chunk of 609 subtitled lines of dialogue.
Name descriptors
The name descriptor array starts immediately after the instance descriptors array. To find the end of the instance descriptors, we can simply take the size of an instance descriptor, 20 bytes, and multiply it by the number of instance descriptors in the file header. In level 0 there are 9347 instance descriptors, so 20 * 9347 = 186940. In hex, that's 0x02DA3C. Adding that to 0x40 (the start of the instance descriptors) takes us to address 0x02DA7C. Voila, the start of the name descriptors.
The name descriptor array stores the numbers of all named instances in alphabetical order. This allows the engine to do a binary search to quickly find instances by name. It is also used when finding instances by type. However the addresses of these instances in memory cannot be known until the file is loaded into RAM, so a space of 32 bits is reserved for that runtime pointer.
| Offset | Type | Raw Hex | Value | Description |
|---|---|---|---|---|
| 0x00 | int32 | 15 16 00 00 | 5653 | instance number |
| 0x04 | int32 | 60 2C 1C 0E | (garbage) | runtime: pointer to instance name |
Template descriptors
Likewise, the template descriptor array starts directly after the name descriptors. Since name descriptors are 8 bytes, 8 * 7124 (taken from the header) = 56992, or 0xDEA0, and adding that to the name descriptor array's start address (0x02DA7C) gives us 0x03B91C: the start of the template descriptors.
The template descriptor array contains information about all templates (that is, resource types, AKA tags), used in the file (56 in this case). So any resource occurring in this instance file has to have its type listed here.
The template checksum is used to prevent loading of instance files that are not compatible with the current engine version. The number of resources is self-explanatory. Note that "TBUS" has a usage number of 2, which corresponds to what we learned earlier about Oni having only two subtitle files, "SUBTsubtitles" and "SUBTmessages".
| Offset | Type | Raw Hex | Value | Description |
|---|---|---|---|---|
| 0x00 | int64 | A0 6D 12 00 00 00 00 00 | 0x126DA0 | template checksum |
| 0x0C | tag | 41 4E 42 41 | 'ABNA' | template tag |
| 0x08 | int32 | 01 00 00 00 | 1 | number of resources in file that use this template |
- Template checksum
An instance can have pointers to other related instances, but since pointers are only valid in memory, they cannot be stored on disk. They must be set when the level data is loaded into memory. To be able to do this, the engine must know where pointers are kept in an instance's data, and this is done using "templates". This template info is hard-coded into the game:
- a checksum of the data contained by the template (the checksum algorithm is unknown, but the checksum stored in Oni's code for a given tag must match the one in the template descriptors array for that tag)
- a 4-letter tag used to identify the template (ABNA, ONCC, WMDD, etc.)
- a short description of the data structure, e.g. "BSP Tree Node Array"
- a list of all the instance's data fields and their types (see OBD:Data types)
- other data that appears to be unused, like the size of the fixed part and the size of an array element for data structures that contain variable-length arrays
Data table
The data table stores all the instance data. We peeked at this before when we looked at the instance descriptor for SUBTsubtitles.
The start of each instance's record, the ID number, is always 32 byte-aligned. Thus, even though the template descriptors ended at 0x03BC9C, there are four empty bytes here so that the data table can begin at 0x03BCA0, which divides evenly by 32. This alignment also means that the instance-specific data will always be found at an offset like 0x0008, 0x0028, 0x0148 etc.
The instance ID and file ID are not actually part of the instance data. The engine always keeps pointers to the start of the type-specific data, and the instance ID and file ID are accessed using negative offsets when needed (usually to find the name or template tag of an instance, given a pointer to it).
| Offset | Type | Raw Hex | Value | Description |
|---|---|---|---|---|
| 0x00 | res_id | 01 00 00 00 | 0 | instance ID |
| 0x04 | lev_id | 01 00 00 00 | 0 | file ID |
| 0x08 | ... | ... | ... | type-specific data... |
- Instance ID
The ID of an instance is computed as:
(instance_descriptor_index << 8) | 1.
The 1 allows the engine to know which IDs have already been converted to pointers (a instance pointer will always be 8 byte aligned so it can never have the bit 0 set).
- File ID
The file ID is computed from the name of the instance file. For "_Final" files the file ID is computed as:
(level_number << 25) | 1
Again the 1 allows the engine to know which file IDs have already been converted to pointers.
As you can see, the size of a given instance's data can be almost anything. Thus, we cannot compute the end of this table in any simple way. That's why the instance file header explicitly gives us the address of the name table.
By the way, how do we know which resource's data we're looking at in the table? Let's look at the very first data, at 0x03BCA0. Noting that the first two numbers, the instance and file ID, do not count as data, and knowing that the instance descriptor gives the offset into the data table for the start of each instance's data, that means that there must be a resource with a data offset of 0x08, the lowest offset possible into the table. We can find this right at the start of the instance descriptor array:
| Offset | Type | Raw Hex | Value | Description |
|---|---|---|---|---|
| 0x00 | tag | 53 47 4E 4F | 'ONGS' | template tag |
| 0x04 | int32 | 08 00 00 00 | 0x08 | data offset (relative to data table) |
| 0x08 | int32 | 00 00 00 00 | 0x00 | name offset (relative to name table) |
| 0x0C | int32 | 60 0F 00 00 | 3936 | data size
{{OBDtr| 0x10 | int32 | | 00 00 00 00 | 0 | flags |
So this tells us that the first data in the data table belongs to the ONGS resource, and that it extends for 3,936 bytes.
Name table
The name table stores all the instance names as C-style strings (terminated by a zero byte). We peeked at this before when we looked at the instance descriptor for SUBTsubtitles.
| Offset | Type | Raw Hex | Value | Description |
|---|---|---|---|---|
| 0x00 | string | 53 55 ... 00 | "SUBTsubtitles" | name string (zero-terminated) |
These names can be up to 63 characters long, counting the tag.