OBD:Raw and separate file formats: Difference between revisions

From OniGalore
Jump to navigation Jump to search
m (gaps: more, later)
 
m (looks better as two separate hatnotes)
 
(12 intermediate revisions by 2 users not shown)
Line 1: Line 1:
Unless mentioned otherwise, this is about the PC retail .raw files.
{{OBD Home}}
:When mentioned otherwise, this can be about the .sep files as well.
{{Hatnote|You should read the [[dat|Instance file format]] page before this one.}}
==Chunk offsets==
{{Hatnote|You can read about the [[OBD:File types|specific types of resources]] in the binary data files after this page.}}
This applies for .sep files as well
In Windows retail Oni, raw files (ending in ".raw") contain chunks of data which are indexed by instance files and loaded on demand. In Mac Oni and the Windows demo, separate files (ending in ".sep") contain some of the data formerly contained by raw files (see "Mac vs. Windows" below for details). Thus, data offsets given in instance files' instance descriptors may apply to a raw file in Windows and a separate file on Macs.
===Alignment===
:Although this may not be strictly required by the engine, the offsets of all original chunks start at 32-byte multiples.
:This implies a slight loss of space, but the storage is somewhat tidier that way. Easier to manipulate in hex editors ^^
:[[OUP]] will probably continue to store data that way. Storing things the way Bungie did is a way to avoid surprises.
===Null offset===
:At the start of the .raw or .sep file, there are 32-bytes of garbage
:That way, all meaningful raw/sep chunks start at offsets strictly greater than 0.
:If that wasn't the case, the null value of a raw/sep link would be ambiguous.


==Null offset==
At the start of the raw or separate file, there are 32 bytes of garbage. That way, all the data chunks start at offsets greater than 0. If that wasn't the case, the null value in memory of a pointer to raw/separate data would be ambiguous.


----
==Alignment==
==Differences across versions==
Although this may not be strictly required by the engine, the offsets of all original chunks start at 32-byte multiples. This implies a slight loss of space, but the storage is somewhat tidier that way. However, the data is not actually packed as closely as possible while maintaining alignment, as detailed under the "Gaps" section.
===Different content===
====Language versions====
In this case the main difference in the .raw is due to the SNDD files.
====Extra level0 textures====
The level0_Final.raw of older versions stores a few megabytes worth of "obsolete" textures.
====Texture compression====
Depending on the version, the amount of textures stored in a lossy compressed format can be "most" to "almost none".
:This leads to dramatic variations in size for the .raw (over 10% of the total amount of game content).


==Gaps==
The raw data is not packed as a compact sequence of 32-byte-aligned chunks. There are gaps of various sizes in every level, totalling 25 megabytes for the whole game. Those gaps seem to be the same across all versions, even if other content differs. The content of those gaps is detailed below.


----
==Gaps==
The original .raw are not packed as a compact sequence of 32-byte-aligned chunks.
:There are gaps of various sizes in every level, totalling up to 2 megabytes per level.
Interestingly, those gaps seem to be the same across version, even if other content differs.
:The content of those gaps is detailed below.
===Skybox mipmaps===
===Skybox mipmaps===
Those gaps occur in every level after each of the 5 skybox textures. They're filled with zeroes.
Zero-filled gaps occur in every level after each of the 5 skybox textures. Their size matches that required to store [[wikipedia:Mipmap|mipmaps]] of the preceding texture.
:Their size matches that required to store MIP map generations of the preceding [[TXMP]] chunk.
 
===Orphaned chunks===
===Orphaned chunks===
Those seem to be mostly .raw parts of existing resources, orphaned at some point by the appending of a new version to the .raw.
Those seem to be mostly .raw parts of existing resources, orphaned at some point by the appending of a new version to the .raw. The pointer in the .dat got redirected to the new version, and the old version was lost track of. So, these are either duplicates of existing resources, or obsolete versions reflecting development.
:The pointer in the .dat got redirected to the new version, and the old version was lost.
 
So, either duplicates of existing resources, or obsolete versions reflecting development.
====OSBD====
====[[TRAM]]====
A few of those in level 0. Re-authored duplicates of existing OSAm/OSGr pairs: for every pair, there are (minor) changes either in the OSAm or in the OSGr.
Not too many of those. Always 32 bytes long. Probably always the same type. Content checking should tell.
 
====[[OSBD]]====
====SNDD====
A few of those in level 0. Nothing but duplicates?
Those make up the most of the gap space. Some of the sounds were actually re-authored (higher sample rates, shorter durations...). Most of them, however, are duplicates, apparently due to a bug in Bungie's packing system.
====[[SNDD]]====
 
Those make the most of the gap space. There seems to be ore than duplicates in the case of the [[SNDD]].
==Differences across versions==
===Language versions===
Most localized (non-English) versions of Oni have new speech dubbed in. This will result in large differences in the SNDD and SUBT files.
 
===Extra level 0 textures===
Not all releases of Oni have the same data in them. The level0_Final.raw of older versions stores a few megabytes worth of "obsolete" textures.
 
===Texture compression===
Textures can be stored with compression, and the amount of textures which are compressed in a given version ranges from "most" to "almost none". This leads to dramatic variations in size for the .raw (over 10% of the total amount of game content).
 
===Mac vs. Windows===
====Storage====
The following table illustrates how some data was moved from raw files to separate files after the Windows version was finalized and before the Mac version was finalized. See [[History of Mac Oni]] for the reason why the Windows demo uses separate files like Mac Oni does.
 
{|class="wikitable" style="margin-left:auto; margin-right:auto;"
!
!Windows retail
!Mac demo, Mac retail,<br />and Windows demo
|-
|'''.raw'''
|<center>[[AGDB]], [[AKVA]], [[BINA]], [[OSBD]],<br />[[SNDD]], [[SUBT]], [[TRAM]], [[TXMP]]</center>
|<center>AKVA, SNDD, SUBT, TRAM</center>
|-
|'''.sep'''
|style="background:black;color:white"|
|<center>BINA, OSBD, TXMP</center>
|}
 
AGDB resources are omitted from Macs and the Windows demo because this data is unused by Oni.
 
====Sounds====
The SNDD resources in the instance files (which are just wrappers for the actual sound data in the raw files) are 32 bytes long on the Mac, and 96 bytes long in Windows. This is because of the difference in the format of the core data: Macs store sound data in the AIFF format, and Windows uses the WAV format.
 
{{OBD}}

Latest revision as of 01:05, 3 July 2022

Click to return to the main OBD page.
You should read the Instance file format page before this one.
You can read about the specific types of resources in the binary data files after this page.

In Windows retail Oni, raw files (ending in ".raw") contain chunks of data which are indexed by instance files and loaded on demand. In Mac Oni and the Windows demo, separate files (ending in ".sep") contain some of the data formerly contained by raw files (see "Mac vs. Windows" below for details). Thus, data offsets given in instance files' instance descriptors may apply to a raw file in Windows and a separate file on Macs.

Null offset

At the start of the raw or separate file, there are 32 bytes of garbage. That way, all the data chunks start at offsets greater than 0. If that wasn't the case, the null value in memory of a pointer to raw/separate data would be ambiguous.

Alignment

Although this may not be strictly required by the engine, the offsets of all original chunks start at 32-byte multiples. This implies a slight loss of space, but the storage is somewhat tidier that way. However, the data is not actually packed as closely as possible while maintaining alignment, as detailed under the "Gaps" section.

Gaps

The raw data is not packed as a compact sequence of 32-byte-aligned chunks. There are gaps of various sizes in every level, totalling 25 megabytes for the whole game. Those gaps seem to be the same across all versions, even if other content differs. The content of those gaps is detailed below.

Skybox mipmaps

Zero-filled gaps occur in every level after each of the 5 skybox textures. Their size matches that required to store mipmaps of the preceding texture.

Orphaned chunks

Those seem to be mostly .raw parts of existing resources, orphaned at some point by the appending of a new version to the .raw. The pointer in the .dat got redirected to the new version, and the old version was lost track of. So, these are either duplicates of existing resources, or obsolete versions reflecting development.

OSBD

A few of those in level 0. Re-authored duplicates of existing OSAm/OSGr pairs: for every pair, there are (minor) changes either in the OSAm or in the OSGr.

SNDD

Those make up the most of the gap space. Some of the sounds were actually re-authored (higher sample rates, shorter durations...). Most of them, however, are duplicates, apparently due to a bug in Bungie's packing system.

Differences across versions

Language versions

Most localized (non-English) versions of Oni have new speech dubbed in. This will result in large differences in the SNDD and SUBT files.

Extra level 0 textures

Not all releases of Oni have the same data in them. The level0_Final.raw of older versions stores a few megabytes worth of "obsolete" textures.

Texture compression

Textures can be stored with compression, and the amount of textures which are compressed in a given version ranges from "most" to "almost none". This leads to dramatic variations in size for the .raw (over 10% of the total amount of game content).

Mac vs. Windows

Storage

The following table illustrates how some data was moved from raw files to separate files after the Windows version was finalized and before the Mac version was finalized. See History of Mac Oni for the reason why the Windows demo uses separate files like Mac Oni does.

Windows retail Mac demo, Mac retail,
and Windows demo
.raw
AGDB, AKVA, BINA, OSBD,
SNDD, SUBT, TRAM, TXMP
AKVA, SNDD, SUBT, TRAM
.sep
BINA, OSBD, TXMP

AGDB resources are omitted from Macs and the Windows demo because this data is unused by Oni.

Sounds

The SNDD resources in the instance files (which are just wrappers for the actual sound data in the raw files) are 32 bytes long on the Mac, and 96 bytes long in Windows. This is because of the difference in the format of the core data: Macs store sound data in the AIFF format, and Windows uses the WAV format.