OBD:Instance file format: Difference between revisions

:You should read the [[Game data terminology]] page before this one.<br>

:The [[Raw|Raw and separate file formats]] page should be read after this one.}}

The level 0 files do not contain resources for a specific level but rather resources (instances) shared across all levels. Level 0 is loaded when the game starts and is never unloaded. All other level files, 1-19, are only loaded when their corresponding level starts and then unloaded when it ends. Oni can only hold two level files in memory concurrently. Thus, resources have to be duplicated on disk whenever a character class, sound effect, etc. occurs in more than one level. For instance, although there are only 2,380 unique sounds in the game, there are 7,386 sounds stored across all level data files.

==Backwards and garbage data==

As mentioned, the game's developers used the in-game editor to create AIs, particles, etc. in a level. When one of these developers saved his work, the contents of the level, stored in his PC's RAM, were flushed directly to disk. Thus the structure of the .dat/.raw/.sep files reflects the way in which Bungie West chose to store levels in memory. So when we read the data in the files with a hex editor, we can see eccentricities such as blank space (coming from unused fields and byte-alignment padding) and garbage data (such as now-meaningless pointer values). [[OBD:Raw and separate file formats#Gaps|Further gaps]], mostly representing orphaned obsolete resources, add up to about 25 MB for the whole game.

Additionally, because the levels were built on Intel-based machines, which use a little-endian architecture, sequences of bytes which represent numbers were written from least-significant to most-significant byte. [[wp:FourCC|FourCCs]] in the data are stored "backwards", such as "13RV" which is meant to be read "VR31", because Bungie defined those four bytes as a 32-bit integer, not a string, causing them to be written to disk in little-endian order.

*Max number of instance files in GameDataFolder: 512 (Windows), 16 (Windows demo, Mac)

*0x0003BCDF33DC271F (v1.0) - templates compatible with Windows retail engine

*0x0003BCDF23C13061 (v1.1) - templates compatible with Windows demo and Mac engines

*0x0003BA70A8DBAE11 (PS2) - templates compatible with PlayStation 2 engine

<!--*0x0000000000000000 (blank) - for use with [[OniX]] engine, which instead handle data versioning using the 0x3C field below-->

{{OBDtr| 0x20 | uint32  | | A0 BC 03 00 | 0x03BCA0  | data table offset }}

{{OBDtr| 0x24 | uint32  | | A0 35 25 00 | 2438560  | data table size }}

{{OBDtr| 0x28 | uint32  | | 40 F2 28 00 | 0x28F240  | name table offset }}

{{OBDtr| 0x2C | uint32  | | 04 4F 02 00 | 151300    | name table size }}

Next we are told the addresses and sizes of the '''data and name tables''' in the instance file. The name table simply follows the data table, as you'll see if you add the data table offset plus the data table size, so the name table offset is technically redundant. The name table offset plus the name table size equals the total size of the file since it's the last segment of the file.

After the name table's size comes four "int"s of '''garbage'''; this is padding in order to align the start of the next segment of the file on a 32-byte boundary. The first two 32-bit fields in this space are, however, used in .oni files generated by OniSplit<!--, and the last 32-bit field is partly used by OniX for a new form of template versioning. Future usage of these fields by OniSplit and/or OniX may change (hopefully not too much)-->.

|align=left | data offset (relative to data table)

|align=left | name offset (relative to name table)

|align=left | data offset (relative to data table)

|align=left | name offset (relative to name table)

The retail version of this instance descriptor tells us that a resource of '''type''' SUBT (a subtitle file for Oni; there are only two of these, one containing all speech subtitles, and one for help messages) has '''data''' that can be found 0x2230C8 bytes into the data table, which we learned from the file header starts at 0x03BCA0. Its '''name''' can be found 0xCB01 bytes into the name table that starts, according to the file header, at 0x28F240.

Before we talk about the name table in depth, we can peek ahead at the name of this resource using the offset we've just been given. Let's add the offset 0xCB01 to 0x28F240, the file header's address for the name table. This gives us the address 0x29BD41. There we find the string "SUBTsubtitles".

The actual subtitle data should be found by adding the offset 0x2230C8 to 0x03BCA0, the file header's address for the data table, to get 0x25ED68. We're going to leave the full details of the data table for later, but below is the data you should actually see for the English Oni SUBT file at this address. You have to consult the [[SUBT]] page to know how to read this data.

The name descriptor array stores the numbers of all named instances in alphabetical order. This allows the engine to do a binary search to quickly find instances by name. It is also used when finding instances by type. However the addresses of these instances in memory cannot be known until the file is loaded into RAM, so a space of 32 bits is reserved for each runtime pointer.

{{OBDtr| 0x00 | int32  | | 15 16 00 00 | 5653      | instance number }}

==Data table==

The data table occupies the majority of the file and stores all the instance data (though this data sometimes points to the location of more data in a raw/separate file). We peeked at this table before when we looked at the instance descriptor for SUBTsubtitles. The table's starting point is found at the offset given in the header, in this case 0x03BCA0, saving us the trouble of adding up the size of the four preceding segments of the file and then aligning to the next 32-byte boundary.

The reason we'd need to align to 32 bytes is that the start of each instance's record (the ID number) is always 32 byte-aligned. Thus, even though the template descriptors ended at 0x03BC9C, there are four empty bytes here so that the data table can begin at 0x03BCA0, which divides evenly by 32. This alignment rule also means that the instance-specific data will always start at an offset like 0x0008, 0x0028, 0x0148, etc.  

{{OBDtr| 0x00 | res_id  | | 01 00 00 00 | 0  | instance ID }}

{{OBDtr| 0x04 | lev_id  | | 01 00 00 00 | 0  | file ID }}

{{OBDtr| 0x08 | ...    | | ...        | ... | [[OBD:File types|type-specific data]]... }}

The '''instance's ID''' is computed as:

(instance descriptor index << 8) <nowiki>|</nowiki> 1

The 1 allows the engine to know which IDs have already been converted to pointers (an instance pointer will always be 8-byte aligned, so it will never have the zero bit already set). These '1's were retained when the file was written to disk but are meaningless now. At level-load time the '1's are cleared and then set again when Oni allocates memory for each instance.

The '''file ID''' is computed from the number found in the name of the instance file:

(level number << 25) <nowiki>|</nowiki> 1

Again, the 1 is used by the engine to know which file IDs have been converted to pointers at runtime, but on disk this is a relic which has no meaning to us.

By the way, if you pick a random place in the data table to look at with a hex editor, how do you know which resource you're looking at? You would look for the highest data offset in the instance descriptor array that is less than your position in the file. Let's say that the string at 0x3BD40 caught our eye: "powerup_ammo". Subtracting the start of the data table, 0x3BCA0, gives us 0xA0 as the position of this string. Now looking back at the instance descriptor array, the instances' data offsets occur every 20 bytes and come directly after the tags. We can see that the first data offset is 0x8 and the next one is 0xF68, thus our offset into the data table of 0xA0 means we are looking at the instance which starts at 0x8. It's the very first instance listed at the start of the instance descriptor array:

{{OBDtr| 0x04 | int32  | | 08 00 00 00 | 0x08    | data offset (relative to data table) }}

{{OBDtr| 0x08 | int32  | | 00 00 00 00 | 0x00    | name offset (relative to name table) }}

So this tells us that the first data in the data table belongs to the solitary [[ONGS]] resource, and that it extends for 3,936 bytes. Since its name offset is 0x0, it's the first string in the name table, which we can see below is SUBTsubtitles.

==Name table==

This final segment of the file stores all the instance names as C-style ASCII strings (terminated by a zero byte). We peeked at this before when we looked at the instance descriptor for SUBTsubtitles. The start of this table is 32-byte aligned but after that the strings are simply packed end to end, separated only by their null terminator. As with the data table, the name table's starting point is given in the header, in this case 0x28F240.

These names can be up to 63 characters long, counting the tag. The instance file concludes with the end of the name table.