OBD:Raw and separate file formats: Difference between revisions
(just moving this in from "Windows Oni vs. Mac Oni", to be fixed up later when I rewrite the page) |
(general copy-edit) |
||
| Line 1: | Line 1: | ||
{{OBD Home}} | |||
:''You should read the [[dat|Instance file format]] page before this one.'' | |||
In Windows retail Oni, raw files (ending in ".raw") contain chunks of data which are indexed by instance files and loaded on demand. In Mac Oni and the Windows demo, separate files (ending in ".sep") contain some of the data formerly contained by raw files. Thus, data offsets given in instance files' instance descriptors may apply to a raw file in Windows and a separate file on Macs. | |||
==Alignment== | |||
:Although this may not be strictly required by the engine, the offsets of all original chunks start at 32-byte multiples. This implies a slight loss of space, but the storage is somewhat tidier that way. However, the data is not actually packed as closely as possible while maintaining alignment, as detailed under the "Gaps" section below. | |||
==Null offset== | |||
:At the start of the .raw or .sep file, there are 32 bytes of garbage. That way, all the data chunks start at offsets greater than 0. If that wasn't the case, the null value in memory of a pointer to raw/separate data would be ambiguous. | |||
==Differences across versions== | ==Differences across versions== | ||
===Language versions=== | |||
Most localized (non-English) versions of Oni have new speech dubbed in. This will result in large differences in the SNDD files and SUBT for level 0. | |||
===Extra level 0 textures=== | |||
The level0_Final.raw of older versions stores a few megabytes worth of "obsolete" textures. | Not all releases of Oni have the same data in them. The level0_Final.raw of older versions stores a few megabytes worth of "obsolete" textures. | ||
===Texture compression=== | |||
Textures ([[TXMP]]s) can be stored with compression, and the amount of textures which are compressed in a given version ranges from "most" to "almost none". This leads to dramatic variations in size for the .raw (over 10% of the total amount of game content). | |||
===Mac vs. Windows=== | |||
==== | ====Instance files==== | ||
These files are mostly identical between Mac and Windows, but | These files are mostly identical between Mac and Windows, but the SNDD resources (which are just wrappers for the actual sound data in the raw/separate files) are 32 bytes long on the Mac, and 96 bytes long in Windows. This is because of the difference in SNDD format noted below under ".raw files". | ||
==== | ====Raw files==== | ||
In Windows, these files contain the core data of the [[AGDB]], [[AKVA]], [[BINA]], [[OSBD]], [[SNDD]], [[SUBT]], [[TRAM]] and [[TXMP]] resources. | In Windows, these files contain the core data of the [[AGDB]], [[AKVA]], [[BINA]], [[OSBD]], [[SNDD]], [[SUBT]], [[TRAM]] and [[TXMP]] resources. | ||
On Mac and in the Windows demo, these files only contain the core data of the AKVA, SNDD, SUBT and TRAM resources; the rest have been moved to the .sep files (except for AGDB resources, which were omitted from Macs and the Windows demo because this data is unused by Oni). | On Mac and in the Windows demo, these files only contain the core data of the AKVA, SNDD, SUBT and TRAM resources; the rest have been moved to the .sep files (except for AGDB resources, which were omitted from Macs and the Windows demo because this data is unused by Oni). | ||
In terms of differences in the actual data formats, the most notable change is that Macs | In terms of differences in the actual data formats, the most notable change is that Macs store sound data (the core data of an SNDD) in the AIFF format, and Windows uses WAV sound data. | ||
====Separate files==== | |||
Windows retail Oni lacks these files, all the relevant data being in the .raw files. On Macs and in the Windows demo, .sep files contain the BINA, OSBD and TXMP resources. For TXMPs in particular, only the "wrapper" for the texture data is stored in a Mac's .raw file, with a link to the actual image data in the level's .sep file. | |||
==Gaps== | ==Gaps== | ||
The original .raw are not packed as a compact sequence of 32-byte-aligned chunks. | The original .raw are not packed as a compact sequence of 32-byte-aligned chunks. There are gaps of various sizes in every level, totalling up 25 megabytes for the whole game. Those gaps seem to be the same across version, even if other content differs. The content of those gaps is detailed below. | ||
===Skybox mipmaps=== | ===Skybox mipmaps=== | ||
Those gaps occur in every level after each of the 5 skybox textures. They're filled with zeroes. | Those gaps occur in every level after each of the 5 skybox textures. They're filled with zeroes. Their size matches that required to store MIP map generations of the preceding [[TXMP]] chunk. | ||
===Orphaned chunks=== | ===Orphaned chunks=== | ||
Those seem to be mostly .raw parts of existing resources, orphaned at some point by the appending of a new version to the .raw. | Those seem to be mostly .raw parts of existing resources, orphaned at some point by the appending of a new version to the .raw. The pointer in the .dat got redirected to the new version, and the old version was lost track of. So, these are either duplicates of existing resources, or obsolete versions reflecting development. | ||
So, either duplicates of existing resources, or obsolete versions reflecting development. | ====OSBD==== | ||
==== | A few of those in level 0. Reauthored duplicates of existing OSAm/OSGr pairs: for every pair, there are (minor) changes either in the OSAm or in the OSGr. | ||
A few of those in level 0. Reauthored duplicates of existing OSAm/OSGr pairs: for every pair, there are (minor) changes either in the OSAm or in the OSGr | |||
==== | ====SNDD==== | ||
Those make the most of the gap space. Some of the sounds were actually | Those make up the most of the gap space. Some of the sounds were actually re-authored (higher sample rates, shorter durations...). Most of them, however, are duplicates, apparently due to a bug in Bungie's packing system. | ||
{{OBD}} | {{OBD}} | ||
Revision as of 21:57, 18 July 2014
- You should read the Instance file format page before this one.
In Windows retail Oni, raw files (ending in ".raw") contain chunks of data which are indexed by instance files and loaded on demand. In Mac Oni and the Windows demo, separate files (ending in ".sep") contain some of the data formerly contained by raw files. Thus, data offsets given in instance files' instance descriptors may apply to a raw file in Windows and a separate file on Macs.
Alignment
- Although this may not be strictly required by the engine, the offsets of all original chunks start at 32-byte multiples. This implies a slight loss of space, but the storage is somewhat tidier that way. However, the data is not actually packed as closely as possible while maintaining alignment, as detailed under the "Gaps" section below.
Null offset
- At the start of the .raw or .sep file, there are 32 bytes of garbage. That way, all the data chunks start at offsets greater than 0. If that wasn't the case, the null value in memory of a pointer to raw/separate data would be ambiguous.
Differences across versions
Language versions
Most localized (non-English) versions of Oni have new speech dubbed in. This will result in large differences in the SNDD files and SUBT for level 0.
Extra level 0 textures
Not all releases of Oni have the same data in them. The level0_Final.raw of older versions stores a few megabytes worth of "obsolete" textures.
Texture compression
Textures (TXMPs) can be stored with compression, and the amount of textures which are compressed in a given version ranges from "most" to "almost none". This leads to dramatic variations in size for the .raw (over 10% of the total amount of game content).
Mac vs. Windows
Instance files
These files are mostly identical between Mac and Windows, but the SNDD resources (which are just wrappers for the actual sound data in the raw/separate files) are 32 bytes long on the Mac, and 96 bytes long in Windows. This is because of the difference in SNDD format noted below under ".raw files".
Raw files
In Windows, these files contain the core data of the AGDB, AKVA, BINA, OSBD, SNDD, SUBT, TRAM and TXMP resources.
On Mac and in the Windows demo, these files only contain the core data of the AKVA, SNDD, SUBT and TRAM resources; the rest have been moved to the .sep files (except for AGDB resources, which were omitted from Macs and the Windows demo because this data is unused by Oni).
In terms of differences in the actual data formats, the most notable change is that Macs store sound data (the core data of an SNDD) in the AIFF format, and Windows uses WAV sound data.
Separate files
Windows retail Oni lacks these files, all the relevant data being in the .raw files. On Macs and in the Windows demo, .sep files contain the BINA, OSBD and TXMP resources. For TXMPs in particular, only the "wrapper" for the texture data is stored in a Mac's .raw file, with a link to the actual image data in the level's .sep file.
Gaps
The original .raw are not packed as a compact sequence of 32-byte-aligned chunks. There are gaps of various sizes in every level, totalling up 25 megabytes for the whole game. Those gaps seem to be the same across version, even if other content differs. The content of those gaps is detailed below.
Skybox mipmaps
Those gaps occur in every level after each of the 5 skybox textures. They're filled with zeroes. Their size matches that required to store MIP map generations of the preceding TXMP chunk.
Orphaned chunks
Those seem to be mostly .raw parts of existing resources, orphaned at some point by the appending of a new version to the .raw. The pointer in the .dat got redirected to the new version, and the old version was lost track of. So, these are either duplicates of existing resources, or obsolete versions reflecting development.
OSBD
A few of those in level 0. Reauthored duplicates of existing OSAm/OSGr pairs: for every pair, there are (minor) changes either in the OSAm or in the OSGr.
SNDD
Those make up the most of the gap space. Some of the sounds were actually re-authored (higher sample rates, shorter durations...). Most of them, however, are duplicates, apparently due to a bug in Bungie's packing system.