18,739
edits
OBD:Instance file format: Difference between revisions
more explaining
(Neo doesn't know where he got this sample data, so I'm replacing it with the data from Windows level 0) |
(more explaining) |
||
| Line 1: | Line 1: | ||
==Terminology== | |||
Oni's level data is broken into two kinds of files in Windows retail Oni. One type ends in ".dat" and is called an instance file. The other type ends in ".raw" and is simply called a raw file. Windows demo Oni and Mac retail/demo Oni use a third type which ends in ".sep", short for "separate". You can read about raw and separate files [[Raw|HERE]]. | Oni's level data is broken into two kinds of files in Windows retail Oni. One type ends in ".dat" and is called an instance file. The other type ends in ".raw" and is simply called a raw file. Windows demo Oni and Mac retail/demo Oni use a third type which ends in ".sep", short for "separate". You can read about raw and separate files [[Raw|HERE]]. | ||
Note that ".dat" is a generic suffix originally used by Oni for all kinds of data, including [[persist.dat]]. The only reason that any other suffixes exist at all is that raw and separate files were created later in development and given unique suffixes to distinguish them from the .dat files in the same folder. Therefore, the proper, specific name for the <u>level data format</u>, as opposed to the save-game format, film format, etc. is not ".dat file" or "DAT file", but "instance file". That being said, ".dat" has only been used by the community historically to refer to instance files, so you can reasonably assume that's what is meant when you see the suffix. | Note that ".dat" is a generic suffix originally used by Oni for all kinds of data, including [[persist.dat]]. The only reason that any other suffixes exist at all is that raw and separate files were created later in development and given unique suffixes to distinguish them from the .dat files in the same folder. Therefore, the proper, specific name for the <u>level data format</u>, as opposed to the save-game format, film format, etc. is not ".dat file" or "DAT file", but "instance file". That being said, ".dat" has only been used by the community historically to refer to instance files, so you can reasonably assume that's what is meant when you see the suffix. | ||
==Introduction== | |||
Instance files are the "main" type of data file in the sense that, when loading a level, Oni reads the instance file first, and this file serves as an index that allows it to find resources which are packed back-to-back into the raw and separate files. All instance files begin with a 64 byte header followed by 3 "descriptor" arrays, a data table and a name table. Among other things, the header contains the number of descriptors in each of the 3 arrays and the offset of the data and name tables (relative to the start of the file). | Instance files are the "main" type of data file in the sense that, when loading a level, Oni reads the instance file first, and this file serves as an index that allows it to find resources which are packed back-to-back into the raw and separate files. All instance files begin with a 64 byte header followed by 3 "descriptor" arrays, a data table and a name table. Among other things, the header contains the number of descriptors in each of the 3 arrays and the offset of the data and name tables (relative to the start of the file). | ||
Here is a walkthrough of an instance file using the level0_Final.dat in English Windows Oni. Follow along in a hex editor for maximum learnage. | During development, Oni had in-game editing tools. These tools presented a GUI for things like placing AIs and setting their attributes, editing particles, etc. When a developer saved his work, the contents of the level, stored in RAM, were written directly to disk. The structure of the .dat/.raw/.sep files reflects the way in which Bungie West chose to load levels into memory, and thus when when we read the data in the files with a hex editor, we can see various eccentricities such as blank space and garbage data that resulted from unused memory. | ||
Additionally, because the levels were built on Intel-based machines, which use a little-endian architecture, every sequence of bytes was written "backwards" from the standpoint of a left-to-right reading culture. See the "VR31" string below for an example; it is actually saved to disk as "13RV". Likewise, every number is written from least-significant to most-significant byte. When Macs, which were big-endian at the time due to their PowerPC architecture, read these files, they then had to flip each sequence of bytes in memory before they could be understood. | |||
==Walkthrough== | |||
===Header=== | |||
Here is a walkthrough of an instance file using the level0_Final.dat in English Windows Oni. Follow along in a hex editor for maximum learnage. Each term will be explained in-depth when we fully consider the related data. First, here is how the file begins: | |||
{{Table}} | {{Table}} | ||
{{OBD_Table_Header}} | {{OBD_Table_Header}} | ||
| Line 18: | Line 26: | ||
{{OBDtr| 0x28 | int32 | | 40 F2 28 00 | 0x28F240 | name table offset }} | {{OBDtr| 0x28 | int32 | | 40 F2 28 00 | 0x28F240 | name table offset }} | ||
{{OBDtr| 0x2C | int32 | | 04 4F 02 00 | 0x024F04 | name table size }} | {{OBDtr| 0x2C | int32 | | 04 4F 02 00 | 0x024F04 | name table size }} | ||
{{OBDtr| 0x30 | int32 | | 00 00 00 00 | | OniSplit only: raw table offset }} | {{OBDtr| 0x30 | int32 | | 00 00 00 00 | | used by OniSplit only: raw table offset }} | ||
{{OBDtr| 0x34 | int32 | | 00 00 00 00 | | OniSplit only: raw table size }} | {{OBDtr| 0x34 | int32 | | 00 00 00 00 | | used by OniSplit only: raw table size }} | ||
{{OBDtr| 0x38 | int32 | | 00 00 00 00 | | unused }} | {{OBDtr| 0x38 | int32 | | 00 00 00 00 | | unused }} | ||
{{OBDtr| 0x3C | int32 | | 00 00 00 00 | | unused }} | {{OBDtr| 0x3C | int32 | | 00 00 00 00 | | unused }} | ||
|} | |} | ||
The template checksum tells us that this level data is in the .dat/.raw file scheme, as opposed to the .dat/.raw/.sep file scheme. | The '''template checksum''' tells us that this level data is in the .dat/.raw file scheme, as opposed to the .dat/.raw/.sep file scheme. | ||
The '''version''' of the instance file is the format version. This is written in ASCII characters, but you have to read it backwards because it's been written to disk in small-endian. Get used to this, because you'll be doing a lot of backwards-ASCII-reading. Thus we get "VR31", which is probably "version 31". This is the format version of all instance files in all releases of Oni. | |||
The '''signature''' is identical in all instance files. | |||
The '''descriptor counts''' are the sizes of some arrays which are coming up soon: the instance, name and template descriptors. For instance, the size of the instance descriptor array will be 0x2483, or 9,347 items, in length. | |||
Next we are told the addresses and sizes of the '''data and name tables''' in this file. The name table simply follows the data table, as you'll see if you add the data table offset plus the data table size, but that doesn't mean the name table offset is redundant; if its start was not 32-bit-aligned, it probably would be moved down to start at the next 32-bit word, but this is unnecessary because it just happens to fall on such an even number already. | |||
After this comes four "int"s of '''zeroes'''. Empty space like this is common in the data files, and indicates that something stored in memory at this relative position was not written to disk (probably pointers). | |||
That concludes the header of the instance file. Immediately after this header, we find the instance descriptors, starting with.... | |||
The "instance descriptors" array stores information about every instance contained in the file. | ===Instance descriptors=== | ||
The "instance descriptors" array stores information about every instance contained in the file. | |||
{{Table}} | {{Table}} | ||