18,700
edits
OBD:Instance file format: Difference between revisions
more explaining
(more explaining) |
(more explaining) |
||
| Line 1: | Line 1: | ||
==Terminology== | ==Terminology== | ||
Oni's level data is broken into two kinds of files in Windows retail Oni. One type ends in ".dat" and is called an instance file. The | Oni's level data is broken into two kinds of files in Windows retail Oni. One type ends in ".dat" and is called an instance file. An "instance" is essentially a resource, in plain English, such as a texture. Before raw and separate files existed, all resources would have been stored in the levelX_Final.dat file, so it was rightfully called an "instance file". The second type of file ends in ".raw" and is simply called a raw file. Windows demo Oni and Mac retail/demo Oni use a third type which ends in ".sep", short for "separate". You can read about raw and separate files [[Raw|HERE]]. | ||
Note that ".dat" is a generic suffix originally used by Oni for all kinds of data, including [[persist.dat]]. The only reason that any other suffixes exist at all is that raw and separate files were created later in development and given unique suffixes to distinguish them from the .dat files in the same folder. Therefore, the proper, specific name for the <u>level data format</u>, as opposed to the save-game format, film format, etc. is not ".dat file" or "DAT file", but "instance file". That being said, ".dat" has only been used by the community historically to refer to instance files, so you can reasonably assume that's what is meant when you see the suffix. | Note that ".dat" is a generic suffix originally used by Oni for all kinds of data, including [[persist.dat]]. The only reason that any other suffixes exist at all is that raw and separate files were created later in development and given unique suffixes to distinguish them from the .dat files in the same folder. Therefore, the proper, specific name for the <u>level data format</u>, as opposed to the save-game format, film format, etc. is not ".dat file" or "DAT file", but "instance file". That being said, ".dat" has only been used by the community historically to refer to instance files, so you can reasonably assume that's what is meant when you see the suffix. | ||
| Line 7: | Line 7: | ||
Instance files are the "main" type of data file in the sense that, when loading a level, Oni reads the instance file first, and this file serves as an index that allows it to find resources which are packed back-to-back into the raw and separate files. All instance files begin with a 64 byte header followed by 3 "descriptor" arrays, a data table and a name table. Among other things, the header contains the number of descriptors in each of the 3 arrays and the offset of the data and name tables (relative to the start of the file). | Instance files are the "main" type of data file in the sense that, when loading a level, Oni reads the instance file first, and this file serves as an index that allows it to find resources which are packed back-to-back into the raw and separate files. All instance files begin with a 64 byte header followed by 3 "descriptor" arrays, a data table and a name table. Among other things, the header contains the number of descriptors in each of the 3 arrays and the offset of the data and name tables (relative to the start of the file). | ||
During development, Oni had in-game editing tools. These tools presented a GUI for things like placing AIs and setting their attributes, editing particles, etc. When a developer saved his work, the contents of the level, stored in RAM, were written directly to disk. The structure of the .dat/.raw/.sep files reflects the way in which Bungie West chose to | During development, Oni had in-game editing tools. These tools presented a GUI for things like placing AIs and setting their attributes, editing particles, etc. When a developer saved his work, the contents of the level, stored in RAM, were written directly to disk. The structure of the .dat/.raw/.sep files reflects the way in which Bungie West chose to store levels in memory, and thus when when we read the data in the files with a hex editor, we can see various eccentricities such as blank space and garbage data that represented unused memory on the development machine. | ||
Additionally, because the levels were built on Intel-based machines, which use a little-endian architecture, | Additionally, because the levels were built on Intel-based machines, which use a little-endian architecture, sequences of bytes which represent numbers were written from least-significant to most-significant byte, which looks "backwards" from the standpoint of a culture that reads left-to-right. When Macs, which were big-endian at the time due to their PowerPC architecture, read these files, they then had to flip each sequence of bytes in memory before they could be understood. | ||
An exception to this backwards-writing rule is when strings of ASCII characters were written to disk. These are not numbers and thus are not subject to endianness, thus they are written left-to-right. Now, this may not seem to be the case as you continue reading below. The first two strings of characters which you'll see are "13RV" and "TBUS", which are meant to be read "VR31" and "SUBT". The reason these four-character strings are backwards is that Oni stored them as a number. For instance, writing the number 1,448,227,633 to disk in little-endian order results in the bytes 0x31, 0x33, 0x52, and 0x56, which happens to produce the ASCII codes for '1', '3', 'R' and 'V'. This provided a combination of more convenient storage in memory as a number, and human-readability when on disk. | |||
==Walkthrough== | ==Walkthrough== | ||
| Line 34: | Line 36: | ||
The '''template checksum''' tells us that this level data is in the .dat/.raw file scheme, as opposed to the .dat/.raw/.sep file scheme. | The '''template checksum''' tells us that this level data is in the .dat/.raw file scheme, as opposed to the .dat/.raw/.sep file scheme. | ||
The '''version''' of the instance file is the format version. | The '''version''' of the instance file is the format version. Reading it backwards, as discussed under "Introduction", we get "VR31", which is probably "version 31". This is the format version of all instance files in all releases of Oni. | ||
The '''signature''' is identical in all instance files. | The '''signature''' is identical in all instance files. | ||
| Line 42: | Line 44: | ||
Next we are told the addresses and sizes of the '''data and name tables''' in this file. The name table simply follows the data table, as you'll see if you add the data table offset plus the data table size, but that doesn't mean the name table offset is redundant; if its start was not 32-bit-aligned, it probably would be moved down to start at the next 32-bit word, but this is unnecessary because it just happens to fall on such an even number already. | Next we are told the addresses and sizes of the '''data and name tables''' in this file. The name table simply follows the data table, as you'll see if you add the data table offset plus the data table size, but that doesn't mean the name table offset is redundant; if its start was not 32-bit-aligned, it probably would be moved down to start at the next 32-bit word, but this is unnecessary because it just happens to fall on such an even number already. | ||
After this comes four "int"s of '''zeroes'''. Empty space like this is common in the data files, and indicates that something stored in memory at this relative position was not written to disk (probably pointers). | After this comes four "int"s of '''zeroes'''. Empty space like this is common in the data files, and indicates that something stored in memory at this relative position was not written to disk (probably pointers, sometimes a buffer reserved for possible future use). | ||
That concludes the header of the instance file. Immediately after this header, we find the instance descriptors, starting with.... | That concludes the header of the instance file. Immediately after this header, we find the instance descriptors, starting with.... | ||
===Instance descriptors=== | ===Instance descriptors=== | ||
The "instance descriptors" array | The "instance descriptors" array tells Oni where to find the data and the name of every instance (resource) indexed by the .dat file. It starts at 0x40 in the .dat file, but below is a descriptor found at 0x17B50 in the file which makes a good example. In the table below, we use offsets relative to the start of this descriptor. | ||
{{Table}} | {{Table}} | ||
{{OBD_Table_Header}} | {{OBD_Table_Header}} | ||
{{OBDtr| 0x00 | tag | | | {{OBDtr| 0x00 | tag | | 54 42 55 53 | 'SUBT' | template tag }} | ||
{{OBDtr| 0x04 | int32 | | | {{OBDtr| 0x04 | int32 | | C8 30 22 00 | 0x2230C8 | data offset (relative to data table) }} | ||
{{OBDtr| 0x08 | int32 | | | {{OBDtr| 0x08 | int32 | | 01 CB 00 00 | 0xCB01 | name offset (relative to name table) }} | ||
{{OBDtr| 0x0C | int32 | | | {{OBDtr| 0x0C | int32 | | C0 09 00 00 | 0x09C0 | data size }} | ||
{{OBDtr| 0x10 | int32 | | 00 00 00 00 | 0 | flags; possible values: | {{OBDtr| 0x10 | int32 | | 00 00 00 00 | 0 | flags; possible values: | ||
:0x'''01''' 00 00 00 - unnamed | :0x'''01''' 00 00 00 - unnamed | ||
:0x'''02''' 00 00 00 - empty | :0x'''02''' 00 00 00 - empty | ||
:0x'''04''' 00 00 00 - never used; appears to mean "big endian" data | :0x'''04''' 00 00 00 - never used; appears to mean "big-endian" data | ||
:0x'''08''' 00 00 00 - shared }} | :0x'''08''' 00 00 00 - shared }} | ||
|} | |} | ||
This descriptor tells us that a resource of type SUBT (a subtitle file for Oni; there are only two in the game) has data that can be found 0x2230C8 bytes into the data table, which we learned from the file header starts at 0x03BCA0. Its name can be found 0xCB01 bytes into the name table that starts, according to the file header, at 0x28F240. The data is 0x09C0, or 2,496 bytes. | |||
If you want to see the name of this resource, let's look at address 0xCB01 + 0x28F240 = 0x29BD41. There we find the string "SUBTsubtitles". | |||
The "name descriptors" array stores the numbers of all named instances in | ===Name descriptors=== | ||
The "name descriptors" array stores the numbers of all named instances in alphabetical order. This allows the engine to do a binary search to faster find instances by name. It is also used when finding instances by type. | |||
{{Table}} | {{Table}} | ||
| Line 71: | Line 76: | ||
{{OBDtr| 0x04 | int32 | | 00 00 00 00 | 0 | runtime: pointer to instance name }} | {{OBDtr| 0x04 | int32 | | 00 00 00 00 | 0 | runtime: pointer to instance name }} | ||
|} | |} | ||
The "template descriptor" array contains information about all templates used in the file. The template checksum is used to prevent loading of instance files that are not compatible with the current engine version. | The "template descriptor" array contains information about all templates used in the file. The template checksum is used to prevent loading of instance files that are not compatible with the current engine version. | ||