20,517
edits
OBD:Instance file format: Difference between revisions
added more explanation on the purpose of "garbage" data; wikilinked a couple section names, which if I had done this earlier would have alerted me to the fact that the sections had been renamed
m (using the engine term "placeholder" instead of "empty" because it's clearer) |
(added more explanation on the purpose of "garbage" data; wikilinked a couple section names, which if I had done this earlier would have alerted me to the fact that the sections had been renamed) |
||
| Line 9: | Line 9: | ||
{{TOClimit}} | {{TOClimit}} | ||
==Blank space and garbage== | ==Blank space and garbage== | ||
The structure of the level data | The structure of the level data files reflects the way in which the engine stores levels in memory. When the level data was authored, it was generally written directly from memory to disk rather than undergoing any special serialization. So when we read the data in the files with a hex editor we can see eccentricities such as blank space coming from unused fields and byte-alignment padding fields, and garbage data such as now-meaningless pointer values and random RAM contents from uninitialized memory. (These fields are garbage to us, but they still serve a purpose from an engine standpoint because the on-disk data is loaded back into memory as-is and then the fields with values from a past runtime session are overwritten with new runtime values; thus the garbage fields are needed as placeholders.) There are also [[OBD:Raw and separate file formats#Gaps|gaps in the .raw/.sep files]], mostly representing orphaned obsolete resources, which add up to about 25 MB for the whole game. | ||
==Backwards tags== | ==Backwards tags== | ||
| Line 52: | Line 52: | ||
The file's '''total template checksum''' is the sum of all the template checksums (see "Template descriptors" below). Oni looks at this number in order to validate that it can read this version of the game data format. In practical terms, the total checksum value given for Windows above tells us that this level data is in the .dat/.raw file scheme, and the value given for Mac Oni and the Windows demo tells us that the level data uses the .dat/.raw/.sep file scheme. | The file's '''total template checksum''' is the sum of all the template checksums (see "Template descriptors" below). Oni looks at this number in order to validate that it can read this version of the game data format. In practical terms, the total checksum value given for Windows above tells us that this level data is in the .dat/.raw file scheme, and the value given for Mac Oni and the Windows demo tells us that the level data uses the .dat/.raw/.sep file scheme. | ||
The '''version''' of the instance file is the format version. Reading it backwards, as discussed under | The '''version''' of the instance file is the format version. Reading it backwards, as discussed under {{SectionLink||Backwards tags}}, we get "VR31" (which probably means "version 3.1" because the engine subsystem that reads template data was in its third iteration when the game shipped). This is the format version of all instance files in all releases of Oni. | ||
The '''descriptor sizes''' are the sizes of the instance, template, and name descriptors which are coming up in this file (see breakdowns in later sections). For instance, each instance descriptor will be 0x14, or 20 bytes, in length. | The '''descriptor sizes''' are the sizes of the instance, template, and name descriptors which are coming up in this file (see breakdowns in later sections). For instance, each instance descriptor will be 0x14, or 20 bytes, in length. | ||
| Line 190: | Line 190: | ||
You'll notice that the level file header lists fewer names (7,124) than instances (9,347). That's because there are 3 types of instance: | You'll notice that the level file header lists fewer names (7,124) than instances (9,347). That's because there are 3 types of instance: | ||
*Unnamed and not placeholder - they are only referenced by other instances in the same file via instance ID; these are generally what can be considered child data of a named instance (e.g., 3D geometry elements like ABNA are "contained" by AKEV, a level's environment). | *Unnamed and not placeholder - they are only referenced by other instances in the same file via instance ID; these are generally what can be considered child data of a named instance (e.g., 3D geometry elements like ABNA are "contained" by AKEV, a level's environment). | ||
*:In vanilla Oni .dats there are some rare occurrences of unnamed non-placeholder ''orphan'' instances (e.g., [[OBD:File types/Naming#TRCM|TRCM]]). These are a form of garbage and are discarded by OniSplit when unpacking a level. | *:In vanilla Oni .dats there are some rare occurrences of unnamed non-placeholder ''orphan'' instances (e.g., [[OBD:File types/Naming#TRCM|TRCM]]). These are a form of garbage data and are discarded by OniSplit when unpacking a level. | ||
*Named and not placeholder - they can be referenced by other instances in any file and the engine can use their name or template tag to find them. | *Named and not placeholder - they can be referenced by other instances in any file and the engine can use their name or template tag to find them. | ||
*Named and placeholder - "placeholder" instances are used in level-specific instance files (i.e. not in level0_Final.dat) to associate an instance ID with a name. For every placeholder resource, there's another one with a matching name in level0_Final.dat that has data in it. The placeholder resource in the instance file is looked up by ID or name and then the engine searches all the loaded files for a non-placeholder instance with the same name, causing it to find the actual file in the global data in level0_Final.dat. | *Named and placeholder - "placeholder" instances are used in level-specific instance files (i.e. not in level0_Final.dat) to associate an instance ID with a name. For every placeholder resource, there's another one with a matching name in level0_Final.dat that has data in it. The placeholder resource in the instance file is looked up by ID or name and then the engine searches all the loaded files for a non-placeholder instance with the same name, causing it to find the actual file in the global data in level0_Final.dat. | ||
| Line 238: | Line 238: | ||
|} | |} | ||
The '''template checksum''' is used to prevent loading of instance files that are not compatible with the current engine version. The '''tag''' is the same kind of number-written-as-backwards-ASCII that we discussed | The '''template checksum''' is used to prevent loading of instance files that are not compatible with the current engine version. The '''tag''' is the same kind of number-written-as-backwards-ASCII that we discussed under {{SectionLink|Backwards tags}}; in this case, 'EGRT' means [[TRGE]]. The field for the '''number of resources''' using this template is unused. The number should be correct for each template, but Oni never uses it for anything. | ||
You might wonder how Oni knows how to read each type of data, such as a SUBT or an ABNA. The simple answer is that this information is hard-coded into Oni. In fact, the information on each instance type, as stored in Oni's code, is actually the real "template". The file data merely gives the tag and checksum that identify the template in use so that Oni knows how to read the following data fields. These hardcoded templates also tell Oni which parts of the file data are reserved for pointers. | You might wonder how Oni knows how to read each type of data, such as a SUBT or an ABNA. The simple answer is that this information is hard-coded into Oni. In fact, the information on each instance type, as stored in Oni's code, is actually the real "template". The file data merely gives the tag and checksum that identify the template in use so that Oni knows how to read the following data fields. These hardcoded templates also tell Oni which parts of the file data are reserved for pointers. | ||