OBD:Instance file format: Difference between revisions

m
clarified and corrected OniSplit checksum note that has been confusing me – still not sure that all the listed types are stored in 1.1 format since the statement about TXMP was incorrect (unless this was written and true for OS 1.0?)
(added section to directly address the different engine versions, though I feel like this might be best on another page; improved wording of blank space/garbage data section)
m (clarified and corrected OniSplit checksum note that has been confusing me – still not sure that all the listed types are stored in 1.1 format since the statement about TXMP was incorrect (unless this was written and true for OS 1.0?))
 
(5 intermediate revisions by the same user not shown)
Line 2: Line 2:
{{OBD Home}}
{{OBD Home}}
{{Hatnote|".dat" redirects here; for other files ending in ".dat", see [[Oni (folder)]].<br>
{{Hatnote|".dat" redirects here; for other files ending in ".dat", see [[Oni (folder)]].<br>
:You should read the [[Game data terminology]] page before this one.<br>
:You should read [[Engine terminology]] and [[Game data terminology]] before this page.<br>
:The [[Raw|Raw and separate file formats]] page should be read after this one.}}
:The [[raw|Raw and separate file formats]] page should be read after this one.}}
Files in GameDataFolder/ named "level[0-19]_Final.dat", together with ".raw" and sometimes ".sep" counterparts, contain the game data for Oni. These are called "instance files" internally, but a more common-sense name for them is level data files. The format described below was also used for the tool files which supplied the GUI for the in-game editor, however the retail Oni game application refuses to load tool files; for the story behind the tool files, see [[level0_Tools]].
Files in GameDataFolder/ named "level[0-19]_Final.dat", together with ".raw" and sometimes ".sep" counterparts, contain the game data for Oni. These are called "instance files" internally, but a more common-sense name for them is level data files. The format described below was also used for the tool files which supplied the GUI for the in-game editor, however the retail Oni game application refuses to load tool files; for the story behind the tool files, see [[level0_Tools]].


The level 0 files do not contain resources for a specific level but rather resources (instances) shared across all levels. Level 0 is loaded when the game starts and is never unloaded. All other level files, 1-19, are only loaded when their corresponding level starts and then unloaded when it ends. Oni can only hold two level files in memory concurrently, thus resources have to be duplicated on disk whenever a character class, sound effect, etc. occurs in more than one level. For instance, although there are only 2,380 unique sounds in the game, there are 7,386 sounds stored across all level data files.
The level 0 files do not contain resources for a specific level but rather resources (instances) shared across all levels. Level 0 is loaded when the game starts and is never unloaded. All other level files, 1-19, are only loaded when their corresponding level starts and then unloaded when it ends. Oni can only hold two level files in memory concurrently, thus resources have to be duplicated on disk whenever a character class, sound effect, etc. occurs in more than one level. For instance, although there are only 2,380 unique sounds in the game, there are 7,386 sounds stored across all level data files.
{{TOClimit}}
{{TOClimit}}
==Engine versions==
There are three versions of Oni's engine found in released versions of the game. They are very similar but contain small differences which we sometimes need to note on this page and on the individual [[OBD:File types|OBD file type pages]]:
*Windows retail: This version of the engine can be identified as '''v1.0''' from [[Oni (folder)/readme.txt|its read-me]] but is often referred to as the "Windows" or "PC" engine. This version of the engine was used for all localized Windows releases translated into non-English languages as well.
*Mac retail/demo, Windows demo: This version can be identified as '''v1.1''' from [[Oni (folder)/Oni ReadMe|the Mac read-me]], and underwent another month of development after v1.0. It was also used for the Windows demo, but is sometimes referred to as the "Mac" engine. <!--[[OniX]] is built on this version of the engine.--> Its most obvious difference is the addition of .sep files to the GameDataFolder, but some templates were changed as well.
*PlayStation 2: This version also uses .sep files but makes additional changes to templates. It went through another 1-2 months of development after the Mac version and could possibly be thought of as v1.2, but is probably best understood as a fork of the v1.1 engine made for PS2 compatibility.
==Blank space and garbage==
==Blank space and garbage==
The structure of the level data files files reflects the way in which Bungie West chose to store levels in memory. So when we read the data in the files with a hex editor we can see eccentricities such as blank space coming from unused fields and byte-alignment padding, and garbage data such as now-meaningless pointer values and random RAM contents from uninitialized memory. There are also [[OBD:Raw and separate file formats#Gaps|gaps in the .raw/.sep files]], mostly representing orphaned obsolete resources, which add up to about 25 MB for the whole game.
The structure of the level data files files reflects the way in which Bungie West chose to store levels in memory. So when we read the data in the files with a hex editor we can see eccentricities such as blank space coming from unused fields and byte-alignment padding, and garbage data such as now-meaningless pointer values and random RAM contents from uninitialized memory. There are also [[OBD:Raw and separate file formats#Gaps|gaps in the .raw/.sep files]], mostly representing orphaned obsolete resources, which add up to about 25 MB for the whole game.
Line 37: Line 31:
*0x0003BA70A8DBAE11 - templates compatible with PlayStation 2 engine
*0x0003BA70A8DBAE11 - templates compatible with PlayStation 2 engine
<!--*0x0000000000000000 (blank) - for use with [[OniX]] engine, which instead handles data versioning using the 0x3C field below-->
<!--*0x0000000000000000 (blank) - for use with [[OniX]] engine, which instead handles data versioning using the 0x3C field below-->
OniSplit's .oni files use PC 1.0 checksum by default and 1.1 checksums when holding data that is stored differently in the 1.1 format (SNDD, TXMP, AGQG, M3GM, IGSt, TSFT/TSGA, TRAM/TREX) }}
OniSplit's .oni files use PC 1.0 checksum by default and 1.1 checksum when storing data that comes from the 1.1 engine and which perceptibly differs in format for that engine (AGQG, IGSt, M3GM, SNDD, TRAM, TSFT, TSGA, but not TXMP) }}
{{OBDtr| 0x08 | uint32  | | 31 33 52 56 | '13RV'    | .dat version (meant to be read as "VR31")<br>OniSplit's .oni files use '23RV' ("VR32") instead<!--<br>OniX's [[Oni (folder)|GDFX]] uses '33RV' ("VR33") to signify that the new data versioning system is in use--> }}
{{OBDtr| 0x08 | uint32  | | 31 33 52 56 | '13RV'    | .dat version (meant to be read as "VR31")<br>OniSplit's .oni files use '23RV' ("VR32") instead<!--<br>OniX's [[Oni (folder)|GDFX]] uses '33RV' ("VR33") to signify that the new data versioning system is in use--> }}
{{OBDtr| 0x0C | uint16  | | 40 00      | 64        | size of this header }}
{{OBDtr| 0x0C | uint16  | | 40 00      | 64        | size of this header }}
Line 44: Line 38:
{{OBDtr| 0x12 | uint16  | | 08 00      | 8        | size of name descriptor }}
{{OBDtr| 0x12 | uint16  | | 08 00      | 8        | size of name descriptor }}
{{OBDtr| 0x14 | uint32  | | 83 24 00 00 | 9347      | instance descriptor count  }}
{{OBDtr| 0x14 | uint32  | | 83 24 00 00 | 9347      | instance descriptor count  }}
{{OBDtr| 0x18 | uint32  | | D4 1B 00 00 | 7124      | name descriptor count }}
{{OBDtr| 0x18 | uint32  | | D4 1B 00 00 | 7124      | name descriptor count; 0 for .oni files because there is no name desc. table }}
{{OBDtr| 0x1C | uint32  | | 38 00 00 00 | 56        | template descriptor count }}
{{OBDtr| 0x1C | uint32  | | 38 00 00 00 | 56        | template descriptor count; 0 for .oni files because there is no template desc. table }}
{{OBDtr| 0x20 | uint32  | | A0 BC 03 00 | 0x03BCA0  | data block offset }}
{{OBDtr| 0x20 | uint32  | | A0 BC 03 00 | 0x03BCA0  | data block offset }}
{{OBDtr| 0x24 | uint32  | | A0 35 25 00 | 2438560  | data block size }}
{{OBDtr| 0x24 | uint32  | | A0 35 25 00 | 2438560  | data block size }}
{{OBDtr| 0x28 | uint32  | | 40 F2 28 00 | 0x28F240  | name block offset }}
{{OBDtr| 0x28 | uint32  | | 40 F2 28 00 | 0x28F240  | name block offset; will be lower than data block offset for .oni files }}
{{OBDtr| 0x2C | uint32  | | 04 4F 02 00 | 151300    | name block size }}
{{OBDtr| 0x2C | uint32  | | 04 4F 02 00 | 151300    | name block size }}
{{OBDtr| 0x30 | uint32  | | 99 CF 40 00 | (garbage) | used by OniSplit for raw table offset }}
{{OBDtr| 0x30 | uint32  | | 99 CF 40 00 | (garbage) | used by OniSplit for raw/sep block offset }}
{{OBDtr| 0x34 | uint32  | | 90 4F 63 00 | (garbage) | used by OniSplit for raw table size }}
{{OBDtr| 0x34 | uint32  | | 90 4F 63 00 | (garbage) | used by OniSplit for raw/sep block size }}
{{OBDtr| 0x38 | uint32  | | F4 55 5F 00 | (garbage) | unused<!--used by OniX for data versioning; the three high bytes contains the highest data version (timestamp) found in any instance in this .dat; see instance descriptor table's 0x10 for format--> }}
{{OBDtr| 0x38 | uint32  | | F4 55 5F 00 | (garbage) | unused<!--used by OniX for data versioning; the three high bytes contains the highest data version (timestamp) found in any instance in this .dat; see instance descriptor table's 0x10 for format--> }}
{{OBDtr| 0x3C | uint32  | | 90 4F 63 00 | (garbage) | unused<!--used by OniX for content versioning; the three high bytes contain the highest content version (timestamp) found in any instance in this .dat; see instance descriptor table's 0x10 for format--> }}
{{OBDtr| 0x3C | uint32  | | 90 4F 63 00 | (garbage) | unused<!--used by OniX for content versioning; the three high bytes contain the highest content version (timestamp) found in any instance in this .dat; see instance descriptor table's 0x10 for format--> }}
Line 64: Line 58:
The '''descriptor counts''' are the sizes of arrays which are coming up in this file: the instance, name and template descriptors. For instance, the size of the instance descriptor array will be 0x2483, or 9,347 items, in length.
The '''descriptor counts''' are the sizes of arrays which are coming up in this file: the instance, name and template descriptors. For instance, the size of the instance descriptor array will be 0x2483, or 9,347 items, in length.


Next we are told the addresses and sizes of the '''data and name tables''' in the instance file. The name block simply follows the data block, as you'll see if you add the data block offset plus the data block size, so the name block offset is technically redundant. The name block offset plus the name block size equals the total size of the file since it's the last segment of the file.
Next we are told the addresses and sizes of the '''data and name block''' in the instance file. The name block simply follows the data block, as you'll see if you add the data block offset plus the data block size, so the name block offset is technically redundant. The name block offset plus the name block size equals the total size of the file since it's the last segment of the file. (However in a .oni file, the order of these blocks is reversed.)


After the name block's size comes four "int"s of '''garbage'''; this is padding in order to align the start of the next segment of the file on a 32-byte boundary. The first two 32-bit fields in this space are, however, used in .oni files generated by OniSplit<!--, and the last 32-bit field is partly used by OniX for a new form of template versioning. Future usage of these fields by OniSplit and/or OniX may change (hopefully not too much)-->.
After the name block's size comes four "int"s of '''garbage'''; this is padding in order to align the start of the next segment of the file on a 32-byte boundary. The first two 32-bit fields in this space are, however, used in .oni files generated by OniSplit<!--, and the next two 32-bit fields are used by OniX for a new form of template versioning-->.


That concludes the header of the instance file. Immediately after this header we find the instance descriptors array.
That concludes the header of the instance file. Immediately after this header we find the instance descriptors array.
Line 253: Line 247:


==Data block==
==Data block==
The data block occupies the majority of the file and stores all the instance data (though this data sometimes points to the location of more data in a raw/separate file). We peeked at this table before when we looked at the instance descriptor for SUBTsubtitles. The table's starting point is found at the offset given in the header, in this case 0x03BCA0, saving us the trouble of adding up the size of the four preceding segments of the file and then aligning to the next 32-byte boundary.
The data block occupies the majority of the file and stores all the instance data (though this data sometimes points to the location of more data in a raw/separate file). We peeked at this block before when we looked at the instance descriptor for SUBTsubtitles. The block's starting point is found at the offset given in the header, in this case 0x03BCA0, saving us the trouble of adding up the size of the four preceding segments of the file and then aligning to the next 32-byte boundary.


The reason we'd need to align to 32 bytes is that the start of each instance's record (the instance ID) is always 32 byte-aligned. Thus, even though the template descriptors ended at 0x03BC9C, there are four empty bytes here so that the data block can begin at 0x03BCA0, which divides evenly by 32. This alignment rule also means that the instance-specific data will always start at an offset like 0x0008, 0x0028, 0x0148, etc.  
The reason we'd need to align to 32 bytes is that the start of each instance's record (the instance ID) is always 32 byte-aligned. Thus, even though the template descriptors ended at 0x03BC9C, there are four empty bytes here so that the data block can begin at 0x03BCA0, which divides evenly by 32. This alignment rule also means that the instance-specific data will always start at an offset like 0x0008, 0x0028, 0x0148, etc.  
Line 289: Line 283:


==Name block==
==Name block==
This final segment of the file stores all the instance names as C-style ASCII strings (terminated by a zero byte). We peeked at this before when we looked at the instance descriptor for SUBTsubtitles. The start of this table is 32-byte aligned but after that the strings are simply packed end to end, separated only by their null terminator. As with the data block, the name block's starting point is given in the header, in this case 0x28F240.
This final segment of the file (third-to-last in a .oni, followed by the data and raw/sep blocks) stores all the instance names as C-style ASCII strings (terminated by a zero byte). We peeked at this before when we looked at the instance descriptor for SUBTsubtitles. The start of this block is 32-byte aligned but after that the strings are simply packed end to end, separated only by their null terminator. As with the data block, the name block's starting point is given in the header, in this case 0x28F240.


{{Table}}
{{Table}}
Line 296: Line 290:
|}
|}


These names can be up to 63 characters long, counting the tag. The instance file concludes with the end of the name block.
These names can be up to 63 characters long, counting the tag. The instance file concludes with the end of the name block, or the raw/sep block in a .oni file.


{{OBD}}
{{OBD}}