OBD:Text encoding: Difference between revisions

*The original US version uses a trimmed-down [[wp:Mac_OS_Roman|Mac OS Roman]] code page that is effectively limited to US-ASCII (96 code points used, 256 available).

!bgcolor=black|[[File:Platform-Mac.png|12px]]

*The MacRoman layout was apparently "borrowed" before 1998, when Mac OS 8.5 came out and the [[wp:Currency sign (typography)|international currency sign]] a.k.a. scarab (¤), at 0xDB, was replaced with the euro symbol (€).

*The actual font (see [[/Fonts|HERE]]) has some unusual typographical features, such as a single-stroke Yen/Yuan symbol (Ұ) and a vertical-stroke cent symbol (￠).

*The US engine actually cannot interpret any code points beyond the US-ASCII range (first 6 rows, white background), notably failing on "…" (see [[#Ellipsis_issue|"Ellipsis issue"]] below). This is because of a provision for Asian encoding systems (EUC-CN and Shift JIS), which use two-byte sequences starting with a high-bit byte.

*Awkwardly enough, the six characters are not restored in their original positions (grey-on-black), but take the place of math symbols.<br/>Four more "math" positions are inexplicably filled with three duplicate characters (œ, ¡ and ª) and a truly enigmatic ʖ̇ , which doesn't seem to occur in any known language and has no dedicated code point in Unicode.

:'''N.B.''' The broken italic font variants (see [[/Fonts#Italic|HERE]]) do not fully implement the 10 new glyphs and use a regular question mark instead of the  ʖ̇.

:The Russian version only provides an implementation of Windows-1251 for regular and bold fonts. The five italic fonts (7pt, 9pt, 10pt, 12pt and 14pt) have exactly the same data (pixels and glyph descriptors) as for the European iteration of Mac OS Roman. This makes sense because italic fonts are inherently broken (see [[/Fonts#Italic|HERE]]) and thus not used by any text in vanilla Oni.  

;Bold 14 font

:Somewhat surprisingly, the size-14 TSFT in the Russian version of TSFFTahoma does not have a complete Windows-1251 code page either. Instead it is limited to the US-ASCII character set (including the "printable delete" box at code point 0x7F), i.e., the upper section of the above table (white background). This causes no issue in vanilla Oni, but only because there is no text that uses bold 14.   

:Last but not least, some fonts in the Russian TSFFTahoma have inconsistent vertical alignment, the most blatant example being 12 bold: some glyphs are one pixel shorter or taller than the full line height (ascender+descender), without a properly compensated vertical glyph offset; others simply have pixels that are not properly aligned within a glyph's rectangle. Besides, many glyphs have excessive padding to the left and/or right of a character, which affects readability.<br />'''N.B.''' There are other examples of poor alignment, e.g. for 12 bold, the character 0x9C (њ) has its right side cut off and is thus unusable (luckily it doesn't occur in Russian script).

Reading garbage pixel data shouldn't be causing memory corruption per se (merely nonsensical/garbled text), but if similar out-of-bounds pointers occur for glyph rendering, then xfhsm_oni.dll may occasionally overwrite its own memory or even Oni's. This has not been thoroughly investigated, but it seems advisable to ensure that all text consists of valid EUC-CN code points (which is unfortunately not the case, see [[#Invalid EUC-CN input|"Invalid EUC-CN input"]] below).

As for the first code page of the Japanese TSFFTahoma, it implements only the 0x20-0x7F range of characters, i.e., is limited to [[wp:US-ASCII|US-ASCII]]. This is consistent with the simplified logic used by the Japanese engine, where any high-bit byte (in the 0x80-0xFF range) is treated as the start of a two-byte sequence. (In actual Shift JIS some high-bit bytes are interpreted as half-width kana, a feature that isn't supported by Oni's engine.)

Unlike other Western versions (UK English, French, German, Italian, Spanish, Russian), the US engine treats high-bit characters as part of a two-byte control sequence (a provision for Asian encodings), and therefore fails to render any character from the extended ASCII range. This happens twice in English Oni, because the ellipsis character (…), encoded as 0xC9, was accidentally used in [[Quotes/Consoles/level_19d|These]] [[Quotes/Consoles/level_19e|Two]] text consoles in place of three consecutive dots (probably auto-substituted by a text editor). The result is that the two lines using a "…" are cut off at the offending character.

![[Quotes/Weapons#w6_vdg|WPgew6_vdg]]

![[Quotes/Weapons#w9_scr|WPgew9_scr]]

Being encoded as plain US-ASCII, this string is entirely illegal considering the limited implementation of EUC-CN by xfhsm_oni.dll, which does not detect US-ASCII as single-byte code points and keeps interpreting pairs of ASCII bytes as (invalid) quwei indices. Through lucky coincidence, the string has an even number of printable bytes, so that the null character is still in a suitable place for terminating the string (the EUN-CN parser will see it as a null lead-byte and will not keep reading further data). However, the string still consists of 31 invalid two-byte code points (not counting the null). As a further lucky coincidence, this string is never read by Oni's engine, because the subtitle's handle (02_05_05) is one of those that have been clobbered by the spurious double-null (see [[#Chinese_SUBT_issues|"Chinese_SUBT_issues"]] below). If it wasn't for the clobbering, the game would crash upon displaying this subtitle.

The "level definitions" ([[ONLD]]s) of [[Pre-beta_content#Cut_levels|pre-beta levels]] are never seen in vanilla Oni, but would appear in the "Load Game" dialog if a valid level#_Final.dat were to be supplied at startup (e.g. by a mod). Since xfhsm_oni.dll does not actually support US-ASCII, any untranslated ONLDs are potentially disruptive.

Although not strictly speaking a font issue, some of Oni's text fails to render because it doesn't fit vertically into a fixed-size frame (such as a [[:File:DATA_CONSOLE.png|text console]]). This is known to happen for [[Quotes/Consoles/level_1e|These]] [[Quotes/Consoles/level_8b|Two]] consoles in the English version, and possibly for other screens in other language versions.

Oni's ingame UI is stylized as a futuristic computer screen (it is supposed to be Konoko's "Data Comlink") and has fixed-width frames reserved for text display. Large amounts of text can appear in the text console frame, or in the upper and lower sections of the Help menu (F1). It turns out that these frames have enough width to accommodate 26.5 16x16 glyphs (text console frame) or 19.5 glyphs (Help menu frame), therefor one would expect the Chinese text renderer to wrap lines around at 26 or 19 characters, respectively. Unfortunately the lines are wrapped around at 27 and 20, so the right half of the last glyph on every long line is cut off.