OBD:Raw and separate file formats
- You should read the Instance file format page before this one.
- You can read about the specific types of resources in the binary data files after this page.
In Windows retail Oni, raw files (ending in ".raw") contain chunks of data which are indexed by instance files and loaded on demand. In Mac Oni, the Windows demo, and OniX, separate files (ending in ".sep") contain some of the data formerly contained by raw files (see § File storage for details on what was moved and § Why separate files were created for the reason). Thus, data offsets given in instance files' instance descriptors may apply to a raw file in Windows and a separate file on Macs.
Null offset
At the start of the raw or separate file, there are 32 bytes of garbage. That way, all the data chunks start at offsets greater than 0. If that wasn't the case, the null value in memory of a pointer to raw/separate data would be ambiguous.
Alignment
Although this may not be strictly required by the engine, the offsets of all original chunks start at 32-byte multiples. This implies a slight loss of space, but the storage is somewhat tidier that way. However, the data is not actually packed as closely as possible while maintaining alignment, as detailed under the "Gaps" section.
Gaps
The raw data is not packed as a compact sequence of 32-byte-aligned chunks. There are gaps of various sizes in every level, totaling 25 megabytes for the whole game. Those gaps seem to be the same across all versions, even if other content differs. The content of those gaps is detailed below.
Skybox mipmaps
Zero-filled gaps occur in every level after each of the 5 skybox textures. Their size matches that required to store mipmaps of the preceding texture.
Orphaned chunks
Those seem to be mostly .raw parts of existing resources, orphaned at some point by the appending of a new version to the .raw. The pointer in the .dat got redirected to the new version, and the old version was lost track of. So, these are either duplicates of existing resources, or obsolete versions reflecting development.
OSBD
A few of those in level 0. Re-authored duplicates of existing OSAm/OSGr pairs: for every pair, there are (minor) changes either in the OSAm or in the OSGr.
SNDD
Those make up the most of the gap space. Some of the sounds were actually re-authored (higher sample rates, shorter durations...). Most of them, however, are duplicates, apparently due to a bug in Bungie's packing system.
Differences across versions
Language versions
Most localized (non-English) versions of Oni have new speech dubbed in. This will result in large differences in the SNDD and SUBT files.
Extra level 0 textures
Not all releases of Oni have the same data in them. The level0_Final.raw of older versions stores a few megabytes worth of "obsolete" textures.
Texture compression
Textures can be stored with compression, and the amount of textures which are compressed in a given version ranges from "most" to "almost none". This leads to dramatic variations in size for the .raw (over 10% of the total amount of game content).
Sound format
The format of the samples (actual sound data) stored in the raw file differs: Mac Oni stores its sound data in the AIFF format (technically AIF-C), and Windows Oni uses the WAV format (technically MS ADPCM). The portion of the SNDD resource in the instance file (which is just a wrapper for the sound data in the raw file) is 32 bytes long on the Mac, and 96 bytes long in Windows retail Oni, but this is somewhat unrelated to the difference in sound data format and instead relates to the period in the game's development when the engine was built: the Windows demo uses the same short wrappers as the Mac does.
File storage
The following table illustrates how some of the data was moved from raw files to separate files after the Windows version was finalized and before the Mac version was finalized. This version of the engine was also used for the Windows and Mac demos.
| Windows retail | Mac demo, Mac retail, and Windows demo | |
|---|---|---|
| .raw | SNDD, SUBT, TRAM, TXMP |
|
| .sep |
AGDB resources were omitted from Macs and the Windows demo because this data is unused by Oni. See the next section for why BINA, OSBD and TXMP were moved to .sep files.
Why separate files were created
A long-standing mystery in the community was why Bungie West created .sep files at all when the game already worked fine without them. It was logical to assume that something specific about the Mac OS necessitated them (or at least made them a good idea), since they were only added in the final month of development between the gold mastering of the Windows version and the Mac version. The reason is only apparent if someone with experience developing for both Classic Mac OS and Windows inspects the source code.
When Oni loads a level starting from the .dat file, it also reads that level's .raw file into memory in one go. However on Windows this is performed through a feature called file mapping which costs almost nothing in disk access or memory usage – Windows merely "maps" the file into a program's address space initially, and then it quietly fetches from disk only the parts of the file that the code actually attempts to read or might read in the near future. When a page (section of the file) has not been read for a while it can be quietly unloaded, keeping RAM usage to a minimum.
As Bungie West discovered, the Classic Mac OS had no such trick available: it simply allocated a block of memory into which it read the whole file. Also, Classic Mac OS programs ran inside a fixed memory allotment which was set in advance via the Get Info window. The level1_Final.raw file is 37 MB in Windows Oni, so on a Mac that would have meant 37 MB of RAM going out the window the instant that the level started, and with a memory ceiling for the application that could be anything since it was determined by the user. The solution below allowed Bungie West to move 12 MB of level1's data out of the .raw file and lazy-load it when it was needed; across all level files, 38% of the game's binary data was moved into .sep files.
Bungie West's fix was to take everything that the engine converts into some other form, and then has no further use for, and move it into a companion .sep file that is not loaded into memory up front. This term "converts into some other form" applies to two very different kinds of data. First, textures are the largest kind of data that could be moved to the separate files. After a video game loads a texture into memory, it is passed to the driver for the GPU which holds another copy of it (typically in VRAM). To take advantage of the fact that there is no need to hold the texture in RAM too, Mac Oni reads each texture out of the .sep one at a time, on demand, into a 384 KB scratch buffer (large enough to fit any one texture and its mipmaps), hands it to the graphics card, then immediately unloads it from RAM – so Oni's own memory never holds more than one texture at a time.[1]
The second kind of data that could be moved were binary objects, that is, BINA and OSBD, but for a very different reason: unlike most of the resource types, these are serialized to disk at a field level instead of a struct level and have to be specially unpacked into memory.[2] Once these binary objects are unpacked into the game's internal structures, Bungie West's .sep solution allows the packed copy to be deleted from memory, whereas if these resources had been saved in the .raw file then the packed copy would continue occupying RAM for the entire level. In a level-specific data file, these objects occupy less than 1 MB, with textures making up the remainder of the .sep file, but level 0 contains 3 MB of binary objects for global use; thus, discarding the original copy of the BINA and OSBD objects after loading them into memory would have saved up to 4 MB of RAM at any given time during the game session.
By contrast, any data (besides textures) that the engine could read directly into memory – animations, sounds, level geometry – stayed in the .raw.
- ↑ This of course had a performance penalty, as the player turning the corner in a level and seeing a lot of new textures would cause the game to wait on disk I/O, and several frames might be dropped as a consequence. However the use of file mapping in Windows essentially had the same effect, only loading data from disk when the engine attempted to read the section of the file with the newly encountered textures. The .sep mechanism introduced on Macs was simply a replacement for this feature, and a more precise one, because Windows file mapping reads the file in multiples of 4 KB, and Bungie West's reimplementation of this concept on the Mac side would load exactly the needed data from disk.
Today, the lazy-loading approaches used by both Windows and Mac Oni have no measurable performance penalty because of the speed of SSDs (particularly their superior seek speeds) and the small size of Oni's textures. Even back in 2001, the cost was alleviated somewhat by Windows and Mac OS performing file caching and predictive read-ahead; if a texture had been read before, or was sequentially after textures that had been read, it was likely that the OS had already read those file contents into a buffer in the background, and thus could supply them to Oni without waiting on a disk read. - ↑ In other words, most resources in Oni were authored by dumping the whole C struct to disk, but BINA and OSBD objects were serialized selectively on a field-by-field level, only saving some fields to disk, and doing so in a different order from how the fields were stored in memory. Thus, most of Oni's resources can be loaded in a single read by loading the on-disk data directly into a struct (and then updating any pointers in the struct). BINA and OSBD subtypes require special handlers which load the data field by field into memory.