18,700
edits
OBD:Instance file format: Difference between revisions
rewriting part of intro as a glossary (this will probably be moved somewhere else more appropriate for such an overview); other wording improvements
(integrating misc. info from bottom of article) |
(rewriting part of intro as a glossary (this will probably be moved somewhere else more appropriate for such an overview); other wording improvements) |
||
| Line 1: | Line 1: | ||
:''For other files ending in ".dat", see [[Oni (folder)]].'' | :''For other files ending in ".dat", see [[Oni (folder)]].'' | ||
Files named "level[0-19]_Final.dat", together with ".raw" and sometimes ".sep" counterparts, contain the game data for Oni. The same format was used for the tools files, named level0_Tools.dat/.raw/.sep, however the retail Oni game application does not load tools files; for the story behind the tools files, see [[Big Blue Box|HERE]]. | Files in GameDataFolder/ named "level[0-19]_Final.dat", together with ".raw" and sometimes ".sep" counterparts, contain the game data for Oni. The same format was used for the tools files, named level0_Tools.dat/.raw/.sep, however the retail Oni game application does not load tools files; for the story behind the tools files, see [[Big Blue Box|HERE]]. | ||
The level 0 files do not actually contain a level, but instances (resources) shared across all levels. Level 0 is loaded when the game starts, and never unloaded. All other level files are only loaded when the corresponding level starts and unloaded when it ends. | The level 0 files do not actually contain a level, but instances (resources) shared across all levels. Level 0 is loaded when the game starts, and never unloaded. All other level files are only loaded when the corresponding level starts and unloaded when it ends. | ||
==Terminology== | ==Terminology== | ||
;Level data files | |||
The data for each level is found in two files in Windows retail Oni, ending in ".dat" and ".raw". On Macs and in the Windows demo, a third file type ending in ".sep" is used. These two/three files are collectively called level data files. | |||
;Instance file | |||
Any file ending in ".dat" found in GameDataFolder/. Instance files are the "main" type of data file in the sense that, when loading a level, Oni reads the instance file first, and this file serves as an index that allows it to find resources which are packed into the binary files. | |||
;.dat file | |||
".dat" is a generic suffix originally used by Oni for all kinds of data, including [[persist.dat]]. The proper, specific name for the .dat files containing <u>level data</u>, as opposed to the .dat files containing the save-game data, films, etc. is "instance file". That being said, ".dat" has only been used by the community historically to refer to instance files, so you can reasonably assume that's what is meant when you see the suffix. On occasion, ".dat" also refers to the complete set of level data files, .dat/.raw[/.sep]. | |||
;Binary file | |||
The binary files are basically used for large and unstructured data like textures and sounds. They have no file header, since the instance file serves as the table of contents for them. The only rule about binary files is that all data parts are stored 32 byte-aligned and the first 32 bytes of the file are always zero (reserved to represent null pointers). At load-time, the offsets given in the instance file are converted to pointers to the data in the binary files. | The binary files are basically used for large and unstructured data like textures and sounds. They have no file header, since the instance file serves as the table of contents for them. The only rule about binary files is that all data parts are stored 32 byte-aligned and the first 32 bytes of the file are always zero (reserved to represent null pointers). At load-time, the offsets given in the instance file are converted to pointers to the data in the binary files. | ||
During development, Oni had in-game editing tools. These tools presented a GUI for things like placing AIs and setting their attributes, editing particles, etc. When a developer saved his work, the contents of the level, stored in RAM, were written directly to disk. The structure of the .dat/.raw/.sep files reflects the way in which Bungie West chose to store levels in memory, and thus when when we read the data in the files with a hex editor, we can see various eccentricities such as blank space and garbage data that represented | ;Raw file, .raw | ||
A type of binary file found in both Windows and Mac Oni. On Macs and in the Windows demo these files are much smaller because several resource types store their data in .sep files instead. | |||
;Separate file, .sep | |||
By the time that Oni for the Mac was finalized for release, some of the raw data was moved to a third file type which ends in ".sep", short for "separate". You can read about raw and separate files [[Raw|HERE]]. | |||
;Instance, resource | |||
An "instance" is not the same as an "instance file". An instance is an individual resource, such as a texture. This can get confusing when OniSplit is used to split level data into one file per resource, in effect creating thousands of "single-instance instance files". | |||
;Template | |||
A template represents a type of resource. Templates are identified with four-letter codes, often called tags, such as "SUBT" for subtitle files. | |||
;.oni | |||
Generated by [[OniSplit]], these files are Windows-format .dat files which basically contain a single instance extracted from an instance file, with all the instance data that was contained by the .raw/.sep files appended at the end. | |||
==Backwards and garbage data== | |||
During development, Oni had in-game editing tools. These tools presented a GUI for things like placing AIs and setting their attributes, editing particles, etc. When a developer saved his work, the contents of the level, stored in RAM, were written directly to disk. The structure of the .dat/.raw/.sep files reflects the way in which Bungie West chose to store levels in memory, and thus when when we read the data in the files with a hex editor, we can see various eccentricities such as blank space and garbage data that represented various RAM contents from the developer's PC. | |||
Additionally, because the levels were built on Intel-based machines, which use a little-endian architecture, sequences of bytes which represent numbers were written from least-significant to most-significant byte, which looks "backwards" from the standpoint of a culture that reads left-to-right. When Macs, which were big-endian at the time due to their PowerPC architecture, read these files, they then had to flip each sequence of bytes in memory before they could be understood. | Additionally, because the levels were built on Intel-based machines, which use a little-endian architecture, sequences of bytes which represent numbers were written from least-significant to most-significant byte, which looks "backwards" from the standpoint of a culture that reads left-to-right. When Macs, which were big-endian at the time due to their PowerPC architecture, read these files, they then had to flip each sequence of bytes in memory before they could be understood. | ||
| Line 132: | Line 151: | ||
;Template checksum | ;Template checksum | ||
An instance can have pointers to other instances but since pointers are only valid in memory they | An instance can have pointers to other related instances, but since pointers are only valid in memory, they cannot be stored on disk. They must be set when the level data is loaded into memory. To be able to do this, the engine must know where pointers are kept in an instance's data, and this is done using "templates". This template info is hard-coded into the game: | ||
*a checksum of the data contained by the template (the checksum algorithm is unknown) | *a checksum of the data contained by the template (the checksum algorithm is unknown, but the checksum stored in Oni's code for a given tag must match the one in the template descriptors array for that tag) | ||
*a 4-letter tag used to identify the template (ABNA, ONCC, WMDD etc.) | *a 4-letter tag used to identify the template (ABNA, ONCC, WMDD, etc.) | ||
*a short description of the data structure | *a short description of the data structure, e.g. "BSP Tree Node Array" | ||
*a list of all | *a list of all the instance's data fields and their types (see [[OBD:Data types]]) | ||
*other data that appears to be unused like size of the fixed part and size of an array element for data structures that contain variable length arrays | *other data that appears to be unused, like the size of the fixed part and the size of an array element for data structures that contain variable-length arrays | ||
==Data table== | ==Data table== | ||