OniSplit/WIP

v1.0 (WIP)

Really? Yes, really. Get the nightly HERE (1.0a7, 2022/02/08)

  0. Possible introduction of aliases for the notoriously confusing -export and -import (easily mistaken for the -extract/-create functionality).

Historical synonyms for -export and -import are "unpack" and "pack", but I am leaning towards new aliases, most probably -split and -link.
(OniSplit itself may be rebranded as GameDataTool, and a lightweight app (limited to the split/link functionality) may be dubbed (Oni)SpLink, although this is likely a bad idea.)

1. The required .NET framework is changed back to .NET 2.0 (a few past versions of OniSplit were built for .NET 4.0 for some reason, but it appears that 2.0 is sufficient).
2. A new packing format called -import:onix was added, which is the same as -import:sep (PC demo-like) but suppresses template checksums and replaces them with a new versioning system.
   The produced .dat/.raw/.sep files are marked VR33 instead of VR31, only work with the OniX engine, and allow for new features in the game data format - provided that the OniX engine implements their support as well.
3. The misguided implementation of 32-bit transparent textures has been remastered, with a distinction between "rgba" and "bgra32" import formats: "rgba" corresponds to TXMP type 11 (directly supported by all Oni engines); "bgra32" is TXMP type 7, mistakenly used by previous versions of OniSplit (and Daodan/Mac patches) as a substitute for the overlooked type 11.
   Old instances of type 7 TXMPs are automatically detected and repaired (replaced with type 11) at the first opportunity.
   New TXMPs created with -format:bgra32 are stored as legitimate type 7, which will only be handled properly by the new OniX engine (or by an amended Daodan DLL).
4. Sound conversion is now (almost) fully featured, although with a strong bias towards the Windows platform (a.k.a. "PC") and MS ADPCM compression.
   In addition to the previously available stream copying, you can now -extract:wav from a Mac SNDD (-extract:aif from a PC SNDD may be added later).
   When using -extract:wav, an additional option -pcm decompresses the waveform and produces a .wav file with raw PCM storage.
   Compressed ADPCM .wav-files (exported through -extract:wav) are now standard-compliant, and the importing of a .wav preserves custom (AD)PCM settings fully as well, meaning that third-party WAV files can now be roundtripped in and out of Oni (PC retail) without any issues. (However, the validation of corrupt third-party WAVs has not been thoroughly implemented or tested.)
   In addition to the automatic creation of PC or Mac SNDDs through the -create command (depending on whether the input is a .wav or an .aif file), the -demo option can be used to generate a PC demo-like SNDD (short header, no custom sample rate or compression) both from a .wav and from an .aif file (useful when mass-converting sounds for VR33/OniX).
   By default all SNDDs made with -create are ADPCM-compressed SNDDs even if the input is an uncompressed .wav file. The -pcm option allows uncompressed/decompressed storage for PC SNDDs, but this is not recommended: PCM sounds take up 4 times as much space as ADPCM, and their playback is currently broken for the PC demo and OniX engines.
   Considering that, as of now, 18:55, 14 December 2021 (CET), all Oni engines play back SNDDs as 22.05 kHz waveforms (the PC retail engine only pretends to support custom sample rates, see HERE), it is now OniSplit's duty to report any waveforms that are not sampled at 22.05 kHz and to suggest the custom playback speed that should be used at OSGr level. 44.1 kHz sounds will automatically be downsampled to 22.05 kHz except when doing so would involve an additional round of decoding and reencoding; this potentially lossy operation must be explicitly requested with -forcestd.
   When creating short-header SNDDs for the PC demo/OniX (using the -demo tag), incoming MS ADPCM files are ultimately required to have a standard block size of 512 bytes per channel. Here too, a potentially lossy reencoding must be explicitly requested through -forcestd, otherwise importing will fail with a report of the non-conformant block size.
   Uncompressed PCM sounds can be imported from .wav, with support for the following bit depths: 16-bit (CD-quality), 24-bit (overkill) and 32-bit (super-overkill). Bit depth reduction is straightforward, practically non-lossy and therefore automatic, with merely a warning printed after the conversion. 8-bit linear PCM input (unsigned) is also supported, but don't tell anyone.
5. Subtitle files now have a custom XML format as requested by Script10k. You go from SUBT*.oni to XML using -extract:xml, and you import using -create (both the old .txt subtitles and the new XML format are accepted as input). Vanilla English SUBTs roundtrip exactly both through TXT and through XML. Non-Vanilla SUBTs and other language versions not tested.
6. The -fullname option is available for -extract:dae operations, prepending the 4-character template tag to the filename (TRBS, ONCC, AKEV, etc).
7. Level creation works again. The <Import> feature of Physics.xml has been repaired (it is the feature that lets you import several OBOA entries and their OBANs from the same Collada file) and enhanced with a couple of options, like explicit script IDs for each animated object and custom prefixes/suffixes for animations. As a minor convenience, all _marker textures are now imported automatically. Support of geometry split into several .dae files has been repaired as well.
8. Interaction with Blender has been consolidated through the -blender option, available both when -extracting and -createing.
   The primary use of -blender is to control the transition between Oni's and Blender's axis conventions, both in terms of scene orientation (Y-up for Oni, Z-up for Blender) and rotation order (X-then-Y-then-Z for Oni, Z-then-Y-then-X for Blender). A non-optional Blender-friendly change is that materials are now set up with <technique sid="common"> rather than <technique> in the -extracted Collada files. In contexts other than materials, <technique_common> is still used.
9. When -createing a TRBS, non-standard skeleton hierarchy is still allowed (controlled by the node tree in the Collada file), but standard sibling order is enforced for "thigh" and "shoulder" bones (if detected).
   (At this point there is still no proper skeletal setup, meaning it is still the same "nested" approach, with rigid body parts hinged to one another, and the rest pose is still the infamous "folded umbrella". A more intuitive rest pose may be implemented later, either as an option or as a new default.)
10. Characters/animations have been consolidated. Quaternion math is more accurate (preventing "upside-down" pelvis animation and other artifacts), as well as the "filtering" a.k.a. "smoothing" of rotation curves (Euler angles). The identification of animation curves in Collada input (when -createing) is more robust. For long animations, warnings are now given for any data that is out of range for TRAM storage, most importantly when/if the rotation keys do not fit into 65535 bytes.
11. New options pertaining to rotation keyframes are available for TRAM operations:
    • When -extracting a TRAM to .xml/.dae, Oni's sparse rotation keys are now preserved by default (with Euler smoothing/filtering applied); dense Euler curves (with 60 keyframes per second) can be forced through -dense.
    • For TRAMs that are -created from .xml/.dae, the default is to generate a dense set of keys (through quaternion interpolation) and then regenerate a sparse set of rotation keys for each bone with default/optimal trimming settings (a.k.a. -tolerance:0). If the rotation data is too large with the optimal trimming settings (i.e. if the data for one or more bones ends up out of 16-bit indexing range), OniSplit will report an error, suggesting the use of a looser tolerance: -tolerance:0.1, -tolerance:0.2, etc.
    • Through the -keepkeys option it is possible to exactly preserve the rotation timeline, e.g., if roundtripping a Vanilla animation, or if importing an authored animation that is known to be optimal and Oni-compatible. If the rotation data in the .dae is not directly compatible with Oni (e.g., if keyframes occur between game ticks, or if intervals between keys exceed 255 ticks, or if there is no rotation key at the next-to-last tick), then -keepkeys will fail with a report of the non-conformant data.
    • Dense keys (not recommended) can be produced with the -dense option. If -keepkeys and -dense are present simultaneously, -keepkeys is ignored.
12. A new -norle switch affects the conversion of Oni textures to TGA (in any context, such as ONCC or AKEV extraction, not just -extract:tga for a TXMP). The effect is that RLE encoding is not attempted, and the TGA files are typically larger.
13. A new -nocolor switch affects -extract:dae for AKEV. The effect is that vertex color information is omitted from the .dae scene (useful, e.g., if you work on lightmaps and don't need vertex colors at all).

At the time of writing, the status of the above features is as follows:

• items (1) through (13) are done and ready for testing as described;
• item (0) is readily feasible, but not yet implemented (subject to debate?):
  • the -link and -split aliases will likely be added (and their use instead of the ambiguous -import and -export will be encouraged);
  • the "rebranding" part (GameDataTool, SpLink, Navarre/Picasso, etc) will likely be dismissed as a not-so-good idea.

For planned/requested features (in future versions), see below. --geyser (talk) 13:11, 17 December 2021 (CET)

Future features

Here is a list of planned and/or requested features for future versions of OniSplit. You can add feature requests here: numbered through #, signed through --~~~~; line breaks within a # can done with <br />, subitems with ## or #* (for clarity, please sign each subitem if any).

1. TSFF (font family) and its TSFT (fonts) can be -extracted (to an XML file referencing several BDF files) and -created, allowing the customization of Oni's fonts/encodings. --geyser (talk) 12:16, 12 January 2022 (CET)
2. Unified approach to text encoding, favoring Unicode (Basic Multilingual Plane) for internal code points and UTF-8 (up to three bytes per character) for text input (from XML or TXT). Legacy encodings should be handled too (autodetected or specified from the command-line when -createing text consoles, menus, subtitles, etc). --geyser (talk) 12:16, 12 January 2022 (CET)
3. Full support of sound conversion, including IMA4 encoding, and/or support for hybrid formats that can store IMA4 data as well as MS ADPCM and PCM: the PC retail SNDD format already supports all three, and it sounds reasonable (pun intended?) to implement a hybrid flavor for "short" SNDDs as well, exclusive to new engines (OniX/Daodan). --geyser (talk) 13:11, 8 February 2022 (CET)
4. Lightmapping, normal/specular mapping, light sources (OBLS)... --geyser (talk) 13:11, 8 February 2022 (CET)
5. Automatic generation of FLAG entities from Collada scenes ("empty" scene nodes identified by name). geyser (talk) 12:57, 10 February 2022 (CET)
6. Support for multiple <Import> tags in an animation XML (to assemble a TRAM from several sources). geyser (talk) 12:57, 10 February 2022 (CET)
7. It would be great if OniSplit could advise the user when a supplied option does not exist, or preferably even refuse to execute the command when it contains an unknown option, since the odds are that the user does not want that command to execute as-is. Right now OniSplit v0.9̄ will let you supply any made-up option which doesn't exist, and won't even print an "Unknown option" warning on the command line. --Iritscen (talk) 01:25, 16 May 2022 (CEST)