OBD:Text encoding: Difference between revisions

Beyond the US English release of the game, Oni's text is known to have been translated into French, Italian, Spanish, German, Japanese and Chinese by various localization companies. Depending on the language version, vanilla Oni uses one of the following five encodings to render text:

Below is the code page implemented by [[TSFF]]Tahoma in the US English version of Oni. It is based on Mac OS Roman ("MacRoman"), but with two differences:

*Of the 223 printable glyphs provided by MacRoman, 42 were not created in TSFFTahoma (shown as grey-on-black).

*Control point 0x7F (a typically non-printable "delete" character) is visible as a box glyph (◻).

!<span style="color:silver;font-size:70%">SP</span>!!&#33;!!&#34;!!&#35;!!&#36;!!&#37;!!&#38;!!&#39;!!&#40;!!&#41;!!&#42;!!&#43;!!&#44;!!&#45;!!&#46;!!&#47;

|-bgcolor=yellow

|-bgcolor=yellow

|-bgcolor=yellow

|-bgcolor=yellow

|-bgcolor=yellow

!bgcolor=black|<span style="color:darkslategray;font-size:70%">NBSP</span>

|-bgcolor=yellow

|-bgcolor=yellow

|-bgcolor=yellow

*The layout was apparently "borrowed" before Mac OS 8.5 came out in 1998. How can we say that? Because 8.5 replaced MacRoman's glyph at 0xDB, which was the [[wp:Currency sign (typography)|international currency sign]] or scarab (¤), with the euro symbol (€), but TSFFTahoma still has the scarab.

*The Yen/Yuan symbol only has a single horizontal stroke, presumably due to lack of available detail at small font sizes.

*Some of the removed glyphs (most importantly ß, Ê, ù and û, but also Ú and ú) occur in [[wp:Languages of the European Union#Knowledge|common European languages]]. This made the US TSFFTahoma unsuitable for [[wikt:EFIGS|EFIGS]] localizations, requiring the creation of a new version (see below).  

*The US engine cannot interpret any code points beyond the US-ASCII range (first 6 rows, white background), such as "…" (see [[#Ellipsis_issue|"Ellipsis issue"]] below). This is because of a provision for Asian encoding systems (EUC-CN and Shift JIS), which use two-byte sequences starting with a high-bit byte.

*It tends to the needs of European localizations by adding back the following characters:

:German ß; French Ê and û; French/Italian ù; Spanish/Italian Ú and ú (relatively rare).

*Awkwardly enough, the six characters are not restored in their original positions (grey-on-black), but take the place of math symbols. Four more "math" positions are inexplicably filled with three duplicate characters (œ, ¡ and ª) and the truly enigmatic character ʖ̇ (which doesn't seem to be a character in any known language and is not in Unicode).

:'''N.B.''' The broken italic font variants do not fully implement the 10 new glyphs and use a regular question mark instead of the  ʖ̇.

!<span style="color:silver;font-size:70%">SP</span>!!&#33;!!&#34;!!&#35;!!&#36;!!&#37;!!&#38;!!&#39;!!&#40;!!&#41;!!&#42;!!&#43;!!&#44;!!&#45;!!&#46;!!&#47;

In the Russian version of Oni, TSFFTahoma fully implements the [[wp:Windows-1251|Windows-1251]] (Cyrillic) code page.

*All the Windows-1251 characters are present, although only 66 (purple) are used by Russian script.  

*The character 0x98 is normally non-printable, but in this font is visible as a box glyph (☐), like 0x7F.

*Apart from 0x20, there are two whitespace characters: the [[wp:Non-breaking space|non-breaking space]] and the [[wp:Soft hyphen|soft hyphen]].

 

!<span style="color:silver;font-size:70%">SP</span>!!&#33;!!&#34;!!&#35;!!&#36;!!&#37;!!&#38;!!&#39;!!&#40;!!&#41;!!&#42;!!&#43;!!&#44;!!&#45;!!&#46;!!&#47;

!Ђ||Ѓ||‚||ѓ||„||…||†||‡||€||‰||Љ||‹||Њ||Ќ||Ћ||Џ

!<span style="color:gray;font-size:70%">NBSP</span>||Ў||ў||Ј||¤||Ґ||¦||§||bgcolor=fuchsia|Ё||©||Є||«||¬||<span style="color:gray;font-size:70%">SHY</span>||®||Ї  

The Chinese version of Oni has the same TSFFTahoma as the original US version (trimmed-down Mac OS Roman), but the engine cannot interpret the extended ASCII range, and in fact does not use TSFFTahoma at all. Instead the user launches a wrapper mini-app called '''oni.exe''' which in turn executes the code in '''Oni.dat''' (the game binary itself – in fact, the original Oni.exe from the US version), injecting a custom text engine found in '''xfhsm_oni.dll''' and the font data in '''xf_font.dat'''. Text strings that Oni intends to display are then intercepted by xfhsm_oni.dll and the resulting pixel data from xf_font.dat is injected into Oni's OpenGL context.

Unlike other versions of Oni, the Chinese font doesn't have a table listing the valid code points along with their "glyph descriptors" (i.e., instructions on how to extract a glyph from the raw pixel data). Instead all the glyphs have a standard size of 16x16 pixels and there are exactly 94x94=8,836 glyphs, filling up a standard [[wp:GB_2312|GB 2312]] plane (''qūwèi''), indexed through a compact numbering scheme known as [[wp:Extended_Unix_Code#EUC-CN|EUC-CN]]: each of the 94x94 code points is indexed by a pair of bytes that are both in the 0xA1-0xFE range. Code points that are not assigned under GB 2312 (e.g. rows 10-15 and 90-94) simply have blank pixel data in the corresponding regions of xf_font.dat.

As for the first code page of the Japanese TSFFTahoma, it implements only the 0x20-0x7F range of characters, i.e., is limited to [[wp:US-ASCII|US-ASCII]]. This is consistent with the simplified logic used by the Japanese engine, where any high-bit byte (in the 0x80-0xFF range) is treated as the start of a two-byte sequence (in actual Shift JIS some high-bit bytes are interpreted as half-width kana).

It must be noted that, as compared to the separate .fnt files, TSFFTahoma provides a very rudimentary implementation of JIS X 0208 (only coding for 154 double-byte glyphs, whereas the .fnt files implement 1,357) and is essentially useless/unusable.

Possibly the incomplete Shift JIS code pages present in the Japanese TSFFTahoma represent an early attempt to implement all the glyphs within Oni's existing text system. As the number of kanji increased, supposedly, the TSFT grew prohibitively large due to the use of 8-bit grayscale storage for the pixel data, and the size taken up by the sparsely populated TSGA also increased out of proportion with the rest of the game data. Once the switch to separate .fnt files was made, no one bothered to clean up TSFFTahoma.

Unlike other Western versions (UK English, French, German, Italian, Spanish, Russian), the US engine treats high-bit characters as part of a two-byte control sequence (a provision for Asian encodings), and therefore fails to render any character from the extended ASCII range. This happens twice in English Oni, because the ellipsis (…), encoded at 0xC9, was accidentally used in [[Quotes/Consoles/level_19d|these]] [[Quotes/Consoles/level_19e|two]] text consoles in place of three consecutive dots (probably an auto-substitution by a text editor). The result is that the two lines using a "…" are cut off at the offending character.

The text strings in the the Chinese version mostly conform to the EUC-CN scheme, except for the rare occurrence of the (A1,A0) sequence. This is not a valid control sequence under any common extension of EUC-CN, and in any case it does not correspond to any pixel data within xf_font.dat, which only covers the standard 94x94 ''qūwèi'' plane, corresponding to a strict 0xA1-0xFE range for the two encoding bytes.

Although not strictly speaking a font issue, some of Oni's text fails to render because it doesn't fit vertically into a fixed-size frame (such as a [[:File:DATA_CONSOLE.png|text console]]). This is known to happen for [[Quotes/Consoles/level_1e|these]] [[Quotes/Consoles/level_8b|two]] consoles in the English version, and possibly for other screens in other language versions.

Chinese glyphs have a fixed size of 16x16 pixels and do not fit horizontally into the drop-down lists, causing a disruptive line wrap to take place (this is visible in the Options screen's Resolution and Difficulty menus).