Jump to content

OBD:Instance file format: Difference between revisions

OBD:Instance file format (view source)

Revision as of 21:01, 18 July 2014

214 bytes removed ,  18 July 2014
wording; I may be keeping the intro here after all, so I removed the unneeded definitions; merged the tiny content from OBD:File types/Empty so I can delete /Empty
(rewriting part of intro as a glossary (this will probably be moved somewhere else more appropriate for such an overview); other wording improvements)
(wording; I may be keeping the intro here after all, so I removed the unneeded definitions; merged the tiny content from OBD:File types/Empty so I can delete /Empty)
Line 1: Line 1:
:''For other files ending in ".dat", see [[Oni (folder)]].''
:''For other files ending in ".dat", see [[Oni (folder)]].''
Files in GameDataFolder/ named "level[0-19]_Final.dat", together with ".raw" and sometimes ".sep" counterparts, contain the game data for Oni. The same format was used for the tools files, named level0_Tools.dat/.raw/.sep, however the retail Oni game application does not load tools files; for the story behind the tools files, see [[Big Blue Box|HERE]].
Files in GameDataFolder/ named "level[0-19]_Final.dat", together with ".raw" and sometimes ".sep" counterparts, contain the game data for Oni. The [[Raw|documentation for raw and separate files]] can be read after this page.
 
The same format was used for the tools files, named level0_Tools.dat/.raw/.sep, however the retail Oni game application does not load tools files; for the story behind the tools files, see [[Big Blue Box|HERE]].


The level 0 files do not actually contain a level, but instances (resources) shared across all levels. Level 0 is loaded when the game starts, and never unloaded. All other level files are only loaded when the corresponding level starts and unloaded when it ends.
The level 0 files do not actually contain a level, but instances (resources) shared across all levels. Level 0 is loaded when the game starts, and never unloaded. All other level files are only loaded when the corresponding level starts and unloaded when it ends.
Line 9: Line 11:


;Instance file
;Instance file
Any file ending in ".dat" found in GameDataFolder/. Instance files are the "main" type of data file in the sense that, when loading a level, Oni reads the instance file first, and this file serves as an index that allows it to find resources which are packed into the binary files.
Any file ending in ".dat" found in GameDataFolder/. Instance files are the main type of data file in the sense that, when loading a level, Oni reads the instance file first, and this file serves as an index that allows it to find resources which are packed into the level data files.
 
;.dat file
".dat" is a generic suffix originally used by Oni for all kinds of data, including [[persist.dat]]. The proper, specific name for the .dat files containing <u>level data</u>, as opposed to the .dat files containing the save-game data, films, etc. is "instance file". That being said, ".dat" has only been used by the community historically to refer to instance files, so you can reasonably assume that's what is meant when you see the suffix. On occasion, ".dat" also refers to the complete set of level data files, .dat/.raw[/.sep].
 
;Binary file
The binary files are basically used for large and unstructured data like textures and sounds. They have no file header, since the instance file serves as the table of contents for them. The only rule about binary files is that all data parts are stored 32 byte-aligned and the first 32 bytes of the file are always zero (reserved to represent null pointers). At load-time, the offsets given in the instance file are converted to pointers to the data in the binary files.
 
;Raw file, .raw
A type of binary file found in both Windows and Mac Oni. On Macs and in the Windows demo these files are much smaller because several resource types store their data in .sep files instead.


;Separate file, .sep
;Instance, resource, file
By the time that Oni for the Mac was finalized for release, some of the raw data was moved to a third file type which ends in ".sep", short for "separate". You can read about raw and separate files [[Raw|HERE]].
This one can be confusing. An instance is often called a file on this wiki, but an instance is not the same as an "instance file". An instance is an individual resource, such as a texture. They are called files because they are individual pieces of data stored and cataloged in a "file system" composed of a set of .dat/.raw[/.sep] files. In at least some cases they were indeed actual, separate files on the developer's computer before being packed into the level data files.


;Instance, resource
When OniSplit is used to split level data files into individual files for each resource, in effect it is creating thousands of "single-instance instance files", at which point "instance", "file", and "instance file" become fully synonymous.
An "instance" is not the same as an "instance file". An instance is an individual resource, such as a texture. This can get confusing when OniSplit is used to split level data into one file per resource, in effect creating thousands of "single-instance instance files".


;Template
;Template, file type
A template represents a type of resource. Templates are identified with four-letter codes, often called tags, such as "SUBT" for subtitle files.
A template represents a type of resource. Templates are identified with four-letter codes, often called tags, such as "SUBT" for subtitle files. Just as instances are often called files on the wiki, templates are often called file types.


;.oni
;.oni
Line 51: Line 43:
{{Table}}
{{Table}}
{{OBD_Table_Header}}
{{OBD_Table_Header}}
{{OBDtr| 0x00 | int64  | | 1F 27 DC 33 DF BC 03 00 | 0x0003BCDF33DC271F | Windows template checksum; Windows demo and Mac retail/demo use 0x0003BCDF23C13061 instead }}
{{OBDtr| 0x00 | int64  | | 1F 27 DC 33 DF BC 03 00 | 0x0003BCDF33DC271F | Windows level file template checksum; Windows demo and Mac retail/demo use 0x0003BCDF23C13061 instead }}
{{OBDtr| 0x08 | int32  | | 31 33 52 56            | 'VR31'            | .dat version; .oni files use 'VR32' instead }}
{{OBDtr| 0x08 | int32  | | 31 33 52 56            | 'VR31'            | .dat version; .oni files use 'VR32' instead }}
{{OBDtr| 0x0C | int64  | | 40 00 14 00 10 00 08 00 | 0x0008001000140040 | signature }}
{{OBDtr| 0x0C | int64  | | 40 00 14 00 10 00 08 00 | 0x0008001000140040 | signature }}
Line 67: Line 59:
|}
|}


The '''template checksum''' tells us that this level data is in the .dat/.raw file scheme, as opposed to the .dat/.raw/.sep file scheme used by Mac Oni and the Windows demo of Oni.
The file's '''template checksum''' tells us that this level data is in the .dat/.raw file scheme, as opposed to the .dat/.raw/.sep file scheme used by Mac Oni and the Windows demo of Oni.


The '''version''' of the instance file is the format version. Reading it backwards, as discussed under "Introduction", we get "VR31", which is probably "version 31". This is the format version of all instance files in all releases of Oni.
The '''version''' of the instance file is the format version. Reading it backwards, as discussed under "Introduction", we get "VR31", which is probably "version 31". This is the format version of all instance files in all releases of Oni.
Line 75: Line 67:
The '''descriptor counts''' are the sizes of some arrays which are coming up soon in this file: the instance, name and template descriptors. For instance, the size of the instance descriptor array will be 0x2483, or 9,347 items, in length.
The '''descriptor counts''' are the sizes of some arrays which are coming up soon in this file: the instance, name and template descriptors. For instance, the size of the instance descriptor array will be 0x2483, or 9,347 items, in length.


Next we are told the addresses and sizes of the '''data and name tables''' in this file. The name table simply follows the data table, as you'll see if you add the data table offset plus the data table size, but that doesn't mean the name table offset is redundant; if its start was not 32-bit-aligned, it probably would be moved down to start at the next 32-bit word, but this is unnecessary because it happens to be aligned already.
Next we are told the addresses and sizes of the '''data and name tables''' in the instance file. The name table simply follows the data table, as you'll see if you add the data table offset plus the data table size, but that doesn't mean the name table offset is redundant; if its start was not 32-bit-aligned, it probably would be moved down to start at the next 32-bit word, but this is unnecessary because it happens to be aligned already.


After this comes four "int"s of '''zeroes'''. Empty space like this is common in the data files, and indicates that something stored in memory at this relative position was not written to disk (probably pointers, or sometimes a space reserved for possible future use in a resource type).
After this comes four "int"s of '''zeroes'''. Empty space like this is common in the data files, and indicates that something stored in memory at this relative position was not written to disk (probably pointers, or sometimes a space reserved for possible future use in a resource type).
Line 83: Line 75:
==Instance descriptors==
==Instance descriptors==
The instance descriptor array tells Oni where to find the data and the name of every instance (resource) indexed by the .dat file. The descriptors start at 0x40 in the .dat file, but below is a descriptor found at 0x017B50 in the file which makes a better example. In the table below, we use offsets relative to the start of this descriptor.
The instance descriptor array tells Oni where to find the data and the name of every instance (resource) indexed by the .dat file. The descriptors start at 0x40 in the .dat file, but below is a descriptor found at 0x017B50 in the file which makes a better example. In the table below, we use offsets relative to the start of this descriptor.
There are 3 types of instance descriptors:
*unnamed - they are referenced by other instances in the same file and the engine never reaches them directly
*named and not empty - they can be referenced by other instances in any file and the engine can use their name or template tag to find them
*named and empty - the instance data is stored in a different file and they exist only to associate an instance id with a name; when an instance references such an instance id the engine searches all the loaded files for a non empty instance with the same name


{{Table}}
{{Table}}
Line 94: Line 81:
{{OBDtr| 0x04 | int32  | | C8 30 22 00 | 0x2230C8  | data offset (relative to data table) }}
{{OBDtr| 0x04 | int32  | | C8 30 22 00 | 0x2230C8  | data offset (relative to data table) }}
{{OBDtr| 0x08 | int32  | | 01 CB 00 00 | 0xCB01    | name offset (relative to name table) }}
{{OBDtr| 0x08 | int32  | | 01 CB 00 00 | 0xCB01    | name offset (relative to name table) }}
{{OBDtr| 0x0C | int32  | | C0 09 00 00 | 0x09C0    | data size }}
{{OBDtr| 0x0C | int32  | | C0 09 00 00 | 2496      | data size }}
{{OBDtr| 0x10 | int32  | | 00 00 00 00 | 0        | flags; possible values:
{{OBDtr| 0x10 | int32  | | 00 00 00 00 | 0        | flags; possible values:
:0x'''01''' 00 00 00 - unnamed
:0x'''01''' 00 00 00 - unnamed
Line 102: Line 89:
|}
|}


This descriptor tells us that a resource of type SUBT (a subtitle file for Oni; there are only two of these, one for speech subtitles and one for help messages) has data that can be found 0x2230C8 bytes into the data table, which we learned from the file header starts at 0x03BCA0. Its name can be found 0xCB01 bytes into the name table that starts, according to the file header, at 0x28F240. The data is 0x09C0, or 2,496, bytes long.
This descriptor tells us that a resource of '''type''' SUBT (a subtitle file for Oni; there are only two of these, one containing all speech subtitles, and one for help messages) has '''data''' that can be found 0x2230C8 bytes into the data table, which we learned from the file header starts at 0x03BCA0. Its '''name''' can be found 0xCB01 bytes into the name table that starts, according to the file header, at 0x28F240. The '''data's size''' is 0x09C0, or 2,496, bytes.  The '''flags''' "unnamed" and "empty" require special explanation.
 
===Unnamed and empty resources===
You'll notice that the level file header lists fewer names (7,124) than instances (9,347). That's because there are 3 types of instance descriptors:
*Unnamed and not empty - they are only referenced by other instances in the same file, generally as child data (e.g., 3D geometry elements like ABNA are "contained" by AKEV, a level's environment).
*Named and not empty - they can be referenced by other instances in any file and the engine can use their name or template tag to find them.
*Named and empty - "empty" instances are used in level-specific instance files (i.e. not in '''level0_Final.dat''') to associate an instance ID with a name. For every empty resource, there's another one with a matching name in '''level0_Final.dat''' that has data in it. The empty resource in the instance file is (usually) looked up by ID, then the engine searches all the loaded files for a non-empty instance with the same name, causing it to find the actual file in the global data in '''level0_Final.dat'''.


===Peeking at instance name===
===Peeking at instance name===
Retrieved from "https://wiki.oni2.net/Special:MobileDiff/23742"