OBD:Instance file format: Difference between revisions

added note about unused field in template descriptor
(corrected OniX data versioning timestamp format)
(added note about unused field in template descriptor)
(2 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{UpdatedForOniX|1.0.0}}
{{UpdatedForOniX|1.0.0}}
{{OBD Home}}
{{OBD Home}}
{{Hatnote|".dat" redirects here; for other files ending in ".dat", see [[Oni (folder)]].}}
{{Hatnote|".dat" redirects here; for other files ending in ".dat", see [[Oni (folder)]].<br>
{{Hatnote|You should read the [[Game data terminology]] page before this one.}}
:''You should read the [[Game data terminology]] page before this one.''<br>
{{Hatnote|The [[Raw|Raw and separate file formats]] page should be read after this one.}}
:''The [[Raw|Raw and separate file formats]] page should be read after this one.''}}
Files in GameDataFolder/ named "level[0-19]_Final.dat", together with ".raw" and sometimes ".sep" counterparts, contain the game data for Oni.
Files in GameDataFolder/ named "level[0-19]_Final.dat", together with ".raw" and sometimes ".sep" counterparts, contain the game data for Oni.


Line 11: Line 11:
{{TOClimit}}
{{TOClimit}}
==Backwards and garbage data==
==Backwards and garbage data==
During development, Oni had an [[level0_Tools|in-game editor]] which presented a GUI for manipulating AIs, particles, etc. in a level. When a developer saved his work, the contents of the level, stored in his PC's RAM, were flushed directly to disk. Thus the structure of the .dat/.raw/.sep files reflects the way in which Bungie West chose to store levels in memory. So when we read the data in the files with a hex editor, we can see eccentricities such as blank space (coming from unused fields and byte-alignment padding) and garbage data (such as now-meaningless pointer values). [[OBD:Raw_and_separate_file_formats#Gaps|These gaps]] between data chunks add up to about 25 MB for the whole game.
During development, Oni had an [[level0_Tools|in-game editor]] which presented a GUI for manipulating AIs, particles, etc. in a level. When a developer saved his work, the contents of the level, stored in his PC's RAM, were flushed directly to disk. Thus the structure of the .dat/.raw/.sep files reflects the way in which Bungie West chose to store levels in memory. So when we read the data in the files with a hex editor, we can see eccentricities such as blank space (coming from unused fields and byte-alignment padding) and garbage data (such as now-meaningless pointer values). [[OBD:Raw and separate file formats#Gaps|Further gaps]], mostly representing orphaned obsolete resources, add up to about 25 MB for the whole game.


Additionally, because the levels were built on Intel-based machines, which use a little-endian architecture, sequences of bytes which represent numbers were written from least-significant to most-significant byte. [[wikipedia:FourCC|FourCCs]] in the data are stored "backwards", such as "13RV" which is meant to be read "VR31", because Bungie defined those four bytes as a 32-bit integer, not a string, causing them to be written to disk in little-endian order.
Additionally, because the levels were built on Intel-based machines, which use a little-endian architecture, sequences of bytes which represent numbers were written from least-significant to most-significant byte. [[wp:FourCC|FourCCs]] in the data are stored "backwards", such as "13RV" which is meant to be read "VR31", because Bungie defined those four bytes as a 32-bit integer, not a string, causing them to be written to disk in little-endian order.


==File limits==
==File limits==
Line 169: Line 169:
Likewise, the template descriptor array starts directly after the name descriptors. Since name descriptors are 8 bytes, 8 * 7124 (taken from the header) = 56992, or 0xDEA0, and adding that to the name descriptor array's start address (0x02DA7C) gives us 0x03B91C as the start of the template descriptors.
Likewise, the template descriptor array starts directly after the name descriptors. Since name descriptors are 8 bytes, 8 * 7124 (taken from the header) = 56992, or 0xDEA0, and adding that to the name descriptor array's start address (0x02DA7C) gives us 0x03B91C as the start of the template descriptors.


The template descriptor array contains information about all templates (that is, resource types, AKA tags), used in the file (56 in this case, as we learned from the file header). Any resource occurring in this instance file has to have its type listed here. Here is the template descriptor at 0x3B9FC:
The template descriptor array contains information about all templates (that is, resource types, aka tags), used in the file (56 in this case, as we learned from the file header). Any resource occurring in this instance file has to have its type listed here. Here is the template descriptor at 0x3B9FC:


{{Table}}
{{Table}}
Line 175: Line 175:
{{OBDtr| 0x00 | int64  | | 3C B9 A6 71 08 00 00 00 | 0x871A6B93C | template checksum }}
{{OBDtr| 0x00 | int64  | | 3C B9 A6 71 08 00 00 00 | 0x871A6B93C | template checksum }}
{{OBDtr| 0x08 | tag    | | 45 47 52 54            | 'EGRT'      | template tag }}
{{OBDtr| 0x08 | tag    | | 45 47 52 54            | 'EGRT'      | template tag }}
{{OBDtr| 0x0C | int32  | | 01 00 00 00            | 1          | number of resources in file that use this template }}
{{OBDtr| 0x0C | int32  | | 01 00 00 00            | 1          | unused: number of resources in file that use this template }}
|}
|}


The '''template checksum''' is used to prevent loading of instance files that are not compatible with the current engine version. The '''tag''' is the same kind of number-written-as-backwards-ASCII that we discussed in the "Backwards and garbage data" section; in this case, 'EGRT' means [[TRGE]]. The '''number of resources''' is self-explanatory.
The '''template checksum''' is used to prevent loading of instance files that are not compatible with the current engine version. The '''tag''' is the same kind of number-written-as-backwards-ASCII that we discussed in the "Backwards and garbage data" section; in this case, 'EGRT' means [[TRGE]]. The field for the '''number of resources''' using this template is unused. The number should be correct for each template, but Oni never uses it for anything.


You might wonder how Oni knows how to read each type of data, such as a SUBT or an ABNA. The simple answer is that this information is hard-coded into Oni. In fact, the information on each instance type, as stored in Oni's code, is actually the real "template". The file data only gives the tag and checksum that refer to a certain template. Which types of data fields are encountered in which order is already known by Oni. These hardcoded templates also tell Oni which parts of the file data are reserved for pointers.
You might wonder how Oni knows how to read each type of data, such as a SUBT or an ABNA. The simple answer is that this information is hard-coded into Oni. In fact, the information on each instance type, as stored in Oni's code, is actually the real "template". The file data only gives the tag and checksum that refer to a certain template. Which types of data fields are encountered in which order is already known by Oni. These hardcoded templates also tell Oni which parts of the file data are reserved for pointers.