19,991
edits
(the engine refers to these sections as blocks, not tables, and I think that term is more accurate) |
(moved engine version section to new page Engine terminology) |
||
| (2 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
{{OBD Home}} | {{OBD Home}} | ||
{{Hatnote|".dat" redirects here; for other files ending in ".dat", see [[Oni (folder)]].<br> | {{Hatnote|".dat" redirects here; for other files ending in ".dat", see [[Oni (folder)]].<br> | ||
:You should read | :You should read [[Engine terminology]] and [[Game data terminology]] before this page.<br> | ||
: | :[[OBD:Raw and separate file formats]] should be read after this page.}} | ||
Files in GameDataFolder/ named "level[0-19]_Final.dat", together with ".raw" and sometimes ".sep" counterparts, contain the game data for Oni. These are called "instance files" internally, but a more common-sense name for them is level data files. The format described below was also used for the tool files which supplied the GUI for the in-game editor, however the retail Oni game application refuses to load tool files; for the story behind the tool files, see [[level0_Tools]]. | Files in GameDataFolder/ named "level[0-19]_Final.dat", together with ".raw" and sometimes ".sep" counterparts, contain the game data for Oni. These are called "instance files" internally, but a more common-sense name for them is level data files. The format described below was also used for the tool files which supplied the GUI for the in-game editor, however the retail Oni game application refuses to load tool files; for the story behind the tool files, see [[level0_Tools]]. | ||
The level 0 files do not contain resources for a specific level but rather resources (instances) shared across all levels. Level 0 is loaded when the game starts and is never unloaded. All other level files, 1-19, are only loaded when their corresponding level starts and then unloaded when it ends. Oni can only hold two level files in memory concurrently | The level 0 files do not contain resources for a specific level but rather resources (instances) shared across all levels. Level 0 is loaded when the game starts and is never unloaded. All other level files, 1-19, are only loaded when their corresponding level starts and then unloaded when it ends. Oni can only hold two level files in memory concurrently, thus resources have to be duplicated on disk whenever a character class, sound effect, etc. occurs in more than one level. For instance, although there are only 2,380 unique sounds in the game, there are 7,386 sounds stored across all level data files. | ||
{{TOClimit}} | {{TOClimit}} | ||
== | ==Blank space and garbage== | ||
The structure of the level data files files reflects the way in which Bungie West chose to store levels in memory. So when we read the data in the files with a hex editor we can see eccentricities such as blank space coming from unused fields and byte-alignment padding, and garbage data such as now-meaningless pointer values and random RAM contents from uninitialized memory. There are also [[OBD:Raw and separate file formats#Gaps|gaps in the .raw/.sep files]], mostly representing orphaned obsolete resources, which add up to about 25 MB for the whole game. | |||
==Backwards tags== | |||
Because the levels were built on Intel-based machines, which use a little-endian architecture, sequences of bytes which represent numbers were written from least-significant to most-significant byte. [[wp:FourCC|FourCCs]] in the data are stored "backwards" (e.g. "13RV" which is meant to be read "VR31") because Bungie defined those four bytes as a 32-bit integer, not a string, causing them to be written to disk in little-endian order. | |||
==File limits== | ==File limits== | ||
*Max level number: 127 | *Max level number: 127 | ||
*Max number of instance files in GameDataFolder: 512 (Windows), 16 (Windows demo, Mac) | *Max number of instance files in GameDataFolder: 512 (Windows retail), 16 (Windows demo, Mac) | ||
*Max number of simultaneously loaded instance files: 64 | *Max number of simultaneously loaded instance files: 64 | ||
*Max number of instances in a file: 131071 | *Max number of instances in a file: 131071 | ||
| Line 26: | Line 27: | ||
{{OBD_Table_Header}} | {{OBD_Table_Header}} | ||
{{OBDtr| 0x00 | uint64 | | 1F 27 DC 33 DF BC 03 00 | 0x0003BCDF33DC271F | Total template checksum (main indicator of engine compatibility): | {{OBDtr| 0x00 | uint64 | | 1F 27 DC 33 DF BC 03 00 | 0x0003BCDF33DC271F | Total template checksum (main indicator of engine compatibility): | ||
*0x0003BCDF33DC271F | *0x0003BCDF33DC271F - templates compatible with v1.0 engine (Windows retail) | ||
*0x0003BCDF23C13061 | *0x0003BCDF23C13061 - templates compatible with v1.1 engine (Windows demo and Mac) | ||
*0x0003BA70A8DBAE11 | *0x0003BA70A8DBAE11 - templates compatible with PlayStation 2 engine | ||
<!--*0x0000000000000000 (blank) - for use with [[OniX]] engine, which instead | <!--*0x0000000000000000 (blank) - for use with [[OniX]] engine, which instead handles data versioning using the 0x3C field below--> | ||
OniSplit's .oni files use PC 1.0 checksum by default and 1.1 checksums when holding data that is stored differently in the 1.1 format (SNDD, TXMP, AGQG, M3GM, IGSt, TSFT/TSGA, TRAM/TREX) }} | OniSplit's .oni files use PC 1.0 checksum by default and 1.1 checksums when holding data that is stored differently in the 1.1 format (SNDD, TXMP, AGQG, M3GM, IGSt, TSFT/TSGA, TRAM/TREX) }} | ||
{{OBDtr| 0x08 | uint32 | | 31 33 52 56 | '13RV' | .dat version (meant to be read as "VR31")<br>OniSplit's .oni files use '23RV' ("VR32") instead<!--<br>OniX's [[Oni (folder)|GDFX]] uses '33RV' ("VR33") to signify that the new data versioning system is in use--> }} | {{OBDtr| 0x08 | uint32 | | 31 33 52 56 | '13RV' | .dat version (meant to be read as "VR31")<br>OniSplit's .oni files use '23RV' ("VR32") instead<!--<br>OniX's [[Oni (folder)|GDFX]] uses '33RV' ("VR33") to signify that the new data versioning system is in use--> }} | ||
| Line 256: | Line 257: | ||
{{OBDtr| 0x00 | res_id | | 01 0B 04 00 | 1035 | instance descriptor index }} | {{OBDtr| 0x00 | res_id | | 01 0B 04 00 | 1035 | instance descriptor index }} | ||
{{OBDtr| 0x04 | lev_id | | 01 00 00 06 | 3 | level number }} | {{OBDtr| 0x04 | lev_id | | 01 00 00 06 | 3 | level number }} | ||
{{OBDtr| 0x08 | ... | | ... | ... | {{OBDtr| 0x08 | ... | | ... | ... | type-specific data... }} | ||
|} | |} | ||
This example is taken from level 3 so that the file ID is more instructive. In the OBD documentation, these fields are called res_id and lev_id as seen above. | This example is taken from level 3 so that the file ID is more instructive. In the OBD documentation, these fields are called res_id and lev_id as seen above. | ||
The '''instance's ID''' is stored as "(instance descriptor index << 8) | 1". Thus the 1, | The '''instance's ID''' is stored as "(instance descriptor index << 8) | 1". Thus the ID for item #1,035 (0x40B) in the instance descriptor index will be encoded as 0x40B01. The '1' is used by the engine to mark which IDs have already been converted to pointers (an instance pointer will always be 8-byte aligned, so it will never have the zero bit already set). These pointer flags were retained when the file was written to disk but are meaningless now. At level-load time the flags are cleared and then set again when Oni allocates memory for each instance. The purpose of left-shifting the index number is simply to leave the lowest byte open for the pointer flag. | ||
The '''file ID''' is computed from the number found in the name of the instance file: "(level number << 25) | 1". Thus instances found in level3_Final.dat will have the file ID encoded as 0x6000001. Again, the '1' is used by the engine to know which file IDs have been converted to pointers at runtime, but on disk this is a relic which has no meaning to us. The reason for left-shifting the level number might have originally been to store it alongside the instance ID and the pointer flag in a single int32, but they are separate numbers now, perhaps so that both IDs can have their own pointer flag. | The '''file ID''' is computed from the number found in the name of the instance file: "(level number << 25) | 1". Thus instances found in level3_Final.dat will have the file ID encoded as 0x6000001. Again, the '1' is used by the engine to know which file IDs have been converted to pointers at runtime, but on disk this is a relic which has no meaning to us. The reason for left-shifting the level number might have originally been to store it alongside the instance ID and the pointer flag in a single int32, but they are separate numbers now, perhaps so that both IDs can have their own pointer flag. | ||