OniSplit/WIP

v1.0 (WIP)

Really? Yes, really. Get the nightly HERE (1.0a5, 2022/01/09)

  0. Possible introduction of aliases for the notoriously confusing -export and -import (easily mistaken for the -extract/-create functionality).

Historical synonyms for -export and -import are "unpack" and "pack", but I am leaning towards new aliases, most probably -split and -link.
(OniSplit itself may be rebranded as GameDataTool, and a lightweight app (limited to the split/link functionality) may be dubbed (Oni)SpLink, although this is likely a bad idea.)

1. The required .NET framework is changed back to .NET 2.0 (a few past versions of OniSplit were built for .NET 4.0 for some reason, but it appears that 2.0 is sufficient).
2. A new packing format called -import:onix was added, which is the same as -import:sep (PC demo-like) but suppresses template checksums and replaces them with a new versioning system.
   The produced .dat/.raw/.sep files are marked VR33 instead of VR31, only work with the OniX engine, and allow for new features in the game data format - provided that the OniX engine implements their support as well.
3. The misguided implementation of 32-bit transparent textures has been remastered, with a distinction between "rgba" and "bgra32" import formats: "rgba" corresponds to TXMP type 11 (directly supported by all Oni engines); "bgra32" is TXMP type 7, mistakenly used by previous versions of OniSplit (and Daodan/Mac patches) as a substitute for the overlooked type 11.
   Old instances of type 7 TXMPs are automatically detected and repaired (replaced with type 11) at the first opportunity.
   New TXMPs created with -format:bgra32 are stored as legitimate type 7, which will only be handled properly by the new OniX engine (or by an amended Daodan DLL).
4. Sound conversion is now (almost) fully featured, although with a strong bias towards the Windows platform (a.k.a. "PC") and MS ADPCM compression.
   In addition to the previously available stream copying, you can now -extract:wav from a Mac SNDD (-extract:aif from a PC SNDD may be added later).
   When using -extract:wav, an additional option -pcm decompresses the waveform and produces a .wav file with raw PCM storage.
   Compressed ADPCM .wav-files (exported through -extract:wav) are now standard-compliant, and the importing of a .wav preserves custom (AD)PCM settings fully as well, meaning that third-party WAV files can now be roundtripped in and out of Oni (PC retail) without any issues. (However, the validation of corrupt third-party WAVs has not been thoroughly implemented or tested.)
   In addition to the automatic creation of PC or Mac SNDDs through the -create command (depending on whether the input is a .wav or an .aif file), the -demo option can be used to generate a PC demo-like SNDD (short header, no custom sample rate or compression) both from a .wav and from an .aif file (useful when mass-converting sounds for VR33/OniX).
   By default all SNDDs made with -create are ADPCM-compressed SNDDs even if the input is an uncompressed .wav file. The -pcm option allows uncompressed/decompressed storage for PC SNDDs, but this is not recommended: PCM sounds take up 4 times as much space as ADPCM, and their playback is currently broken for the PC demo and OniX engines.
   Considering that, as of now, 18:55, 14 December 2021 (CET), all Oni engines play back SNDDs as 22.05 kHz waveforms (the PC retail engine only pretends to support custom sample rates, see HERE), it is now OniSplit's duty to report any waveforms that are not sampled at 22.05 kHz and to suggest the custom playback speed that should be used at OSGr level. 44.1 kHz sounds will automatically be downsampled to 22.05 kHz except when doing so would involve an additional round of decoding and reencoding; this potentially lossy operation must be explicitly requested with -forcestd.
   When creating short-header SNDDs for the PC demo/OniX (using the -demo tag), incoming MS ADPCM files are ultimately required to have a standard block size of 512 bytes per channel. Here too, a potentially lossy reencoding must be explicitly requested through -forcestd, otherwise importing will fail with a report of the non-conformant block size.
   Uncompressed PCM sounds can be imported from .wav, with support for the following bit depths: 16-bit (CD-quality), 24-bit (overkill) and 32-bit (super-overkill). Bit depth reduction is straightforward, practically non-lossy and therefore automatic, with merely a warning printed after the conversion. 8-bit linear PCM input (unsigned) is also supported, but don't tell anyone.
5. Subtitle files now have a custom XML format as requested by Script10k. You go from SUBT*.oni to XML using -extract:xml, and you import using -create (both the old .txt subtitles and the new XML format are accepted as input). Vanilla English SUBTs roundtrip exactly both through TXT and through XML. Non-Vanilla SUBTs and other language versions not tested.
6. The -fullname option is available for -extract:dae operations, prepending the 4-character template tag to the filename (TRBS, ONCC, AKEV, etc).
7. Level creation works again. The <Import> feature of Physics.xml has been repaired (it is the feature that lets you import several OBOA entries and their OBANs from the same Collada file) and enhanced with a couple of options, like explicit script IDs for each animated object and custom prefixes/suffixes for animations. As a minor convenience, all _marker textures are now imported automatically. Support of multiple geometry .dae has been repaired as well.
8. Interaction with Blender has been consolidated through the -blender option, available both when -extracting and -createing.
   The primary use of -blender is to control the transition between Oni's and Blender's axis conventions, both in terms of scene orientation (Y-up for Oni, Z-up for Blender) and rotation order (X-then-Y-then-Z for Oni, Z-then-Y-then-X for Blender).
   Another effect of -blender, exclusive to -extract, is to set up materials with <technique sid="common"> rather than <technique_common> in the exported Collada files. In contexts other than materials, <technique_common> is still used regardless of the -blender option. When importing a Collada file through -create, both <technique_common> and <technique sid="common"> are accepted.
9. When -createing a TRBS, non-standard skeleton hierarchy is still allowed (controlled by the node tree in the Collada file), but standard sibling order is enforced for "thigh" and "shoulder" bones (if detected).
   (At this point there is still no proper skeletal setup, meaning it's still the same "nested" approach, with rigid body parts hinged to one another, and the rest pose is still the infamous "folded umbrella". A more intuitive rest pose may be implemented later.)
10. Characters/animations have been consolidated. Quaternion math is more accurate (preventing "upside-down" pelvis animation and other artifacts), as well as the "filtering" a.k.a. "smoothing" of rotation curves (Euler angles). The identification of animation curves in Collada input (when -createing) is more robust. For long animations, warnings are now given for any data that is out of range for TRAM storage, most importantly when/if the rotation keys do not fit into 65535 bytes.
11. New options pertaining to dense rotation keyframes are available for TRAM operations:
    • When -extracting a TRAM to .xml/.dae, Oni's sparse rotation keys are now preserved by default (with Euler smoothing/filtering applied); re-keying can be forced through -rekey, with the following flavors: -rekey:-1 (or with any negative parameter) will produce "dense" Euler curves, with a key for every bone at every tick; -rekey:0 (or simply -rekey) will generate new curves with default/optimal trimming settings; a looser tolerance, resulting in sparser keyframes (and less accurate rotations) can be specified through -rekey:0.1, -rekey:1, etc.
    • For TRAMs -created from .xml/.dae, the new default is again to preserve the keys from the .dae, without any additional interpolation or trimming; -rekey (or -rekey:0) will generate new keys with default/optimal trimming settings; -rekey:-1 (or with any negative parameter) will produce dense keys at every tick (not recommended); a custom tolerance can be specified through -rekey:0.1, -rekey:1, etc.

At the time of writing, the status of the above features is as follows:

• items (1) through (10) are done and ready for testing as described;
• item (11) is being re-done (replacing the -dense and -trim options from alpha 6 with -rekey logic)
• item (0) is redily feasible, but not yet implemented (subject to debate?):
  • the -link and -split aliases will likely be added (and their use instead of the ambiguous -import and -export will be encouraged);
  • the "rebranding" part (GameDataTool, SpLink, Navarre/Picasso, etc) will likely be dismissed as a not-so-good idea.

For planned/requested features (in future versions), see below. --geyser (talk) 13:11, 17 December 2021 (CET)

Future features

Here is a list of planned and/or requested features for future versions of OniSplit. You can add feature requests here: numbered through #, signed through --~~~~; line breaks within a # can done with <br />, subitems with ## or #* (for clarity, please sign each subitem if any).

1. TSFF (font family) and its TSFT (fonts) can be -extracted (to an XML file referencing several BDF files) and -created, allowing the customization of Oni's fonts/encodings. --geyser (talk) 12:16, 12 January 2022 (CET)
2. Unified approach to text encoding, favoring Unicode (Basic Multilingual Plane) for internal code points and UTF-8 (up to three bytes per character) for text input (from XML or TXT). Legacy encodings should be handled too (autodetected or specified from the command-line when -createing text consoles, menus, subtitles, etc). --geyser (talk) 12:16, 12 January 2022 (CET)
3. ...
4. PROFIT!!!