OBD:Text encoding: Difference between revisions

*The original US version uses a trimmed-down [[wp:Mac_OS_Roman|Mac OS Roman]] code page that is effectively limited to US-ASCII (96 code points used, 256 available).

!bgcolor=black|[[File:Platform-Mac.png|12px]]

*The MacRoman layout was apparently "borrowed" before 1998, when Mac OS 8.5 came out and the [[wp:Currency sign (typography)|international currency sign]] a.k.a. scarab (¤), at 0xDB, was replaced with the euro symbol (€).

*The actual font (see [[/Fonts|HERE]]) has some unusual typographical features, such as a single-stroke Yen/Yuan symbol (Ұ) and a vertical-stroke cent symbol (￠).

*The US engine actually cannot interpret any code points beyond the US-ASCII range (first 6 rows, white background), notably failing on "…" (see [[#Ellipsis_issue|"Ellipsis issue"]] below). This is because of a provision for Asian encoding systems (EUC-CN and Shift JIS), which use two-byte sequences starting with a high-bit byte.

*Awkwardly enough, the six characters are not restored in their original positions (grey-on-black), but take the place of math symbols.<br/>Four more "math" positions are inexplicably filled with three duplicate characters (œ, ¡ and ª) and a truly enigmatic ʖ̇ , which doesn't seem to occur in any known language and has no dedicated code point in Unicode.

:'''N.B.''' The broken italic font variants (see [[/Fonts#Italic|HERE]]) do not fully implement the 10 new glyphs and use a regular question mark instead of the  ʖ̇.

:The Russian version only provides an implementation of Windows-1251 for regular and bold fonts. The five italic fonts (7pt, 9pt, 10pt, 12pt and 14pt) have exactly the same data (pixels and glyph descriptors) as for the European iteration of Mac OS Roman. This makes sense because italic fonts are inherently broken (see [[/Fonts#Italic|HERE]]) and thus not used by any text in vanilla Oni.  

;Bold 14 font

:Somewhat surprisingly, the size-14 TSFT in the Russian version of TSFFTahoma does not have a complete Windows-1251 code page either. Instead it is limited to the US-ASCII character set (including the "printable delete" box at code point 0x7F), i.e., the upper section of the above table (white background). This causes no issue in vanilla Oni, but only because there is no text that uses bold 14.   

:Last but not least, some fonts in the Russian TSFFTahoma have inconsistent vertical alignment, the most blatant example being 12 bold: some glyphs are one pixel shorter or taller than the full line height (ascender+descender), without a properly compensated vertical glyph offset; others simply have pixels that are not properly aligned within a glyph's rectangle. Besides, many glyphs have excessive padding to the left and/or right of a character, which affects readability.<br />'''N.B.''' There are other examples of poor alignment, e.g. for 12 bold, the character 0x9C (њ) has its right side cut off and is thus unusable (luckily it doesn't occur in Russian script).

The current understanding is that xfhsm_oni.dll simply turns any two-byte code point QQ WW into the offset [(QQ-A1)*5E + (WW-A1)]*0x20, relative either to the start of the xf_font.dat data (for the 16x16 font) or to the middle of the data (for the small 12x12 font). Both components of the offset can exceed the intended 0-93 range, with values as high as 94 and as low as -161, depending on the values of QQ and WW, and there doesn't seem to be any sanity check. The only special case is if QQ==00, in which case WW is ignored and the string is terminated.

A valid EUC-CN code point (with both bytes in the 0xA1-0xFE range) results in a valid offset pointing to an actual glyph for the relevant font, whereas illegal bytes or byte pairs may point to a different glyph within the same font, or to a glyph of the other font, or to a completely unrelated memory region. In the worst case scenario, pixel data will be read at 486,432 bytes (~475 kB) ahead of the actual pixel data (if displaying the code point 01,00 for the large font) or at 3008-3040 bytes (~3 kB) past the actual pixel data (if diplaying the code point FF,FF for the small font).

Reading garbage pixel data shouldn't be causing memory corruption per se (merely nonsensical/garbled text), but if similar out-of-bounds pointers occur for glyph rendering, then xfhsm_oni.dll may occasionally overwrite its own memory or even Oni's. This has not been thoroughly investigated, but it seems advisable to ensure that all text consists of valid EUC-CN code points (which is unfortunately not the case, see [[#Invalid EUC-CN input|"Invalid EUC-CN input"]] below).

As for the first code page of the Japanese TSFFTahoma, it implements only the 0x20-0x7F range of characters, i.e., is limited to [[wp:US-ASCII|US-ASCII]]. This is consistent with the simplified logic used by the Japanese engine, where any high-bit byte (in the 0x80-0xFF range) is treated as the start of a two-byte sequence. (In actual Shift JIS some high-bit bytes are interpreted as half-width kana, a feature that isn't supported by Oni's engine.)

It must be noted that, as compared to the separate .fnt files, the Japanese TSFFTahoma provides a very rudimentary implementation of JIS X 0208 (only coding for 154 double-byte glyphs, whereas the .fnt files implement 1,357) and is essentially useless/unusable.

*The Japanese engine requires all four .fnt files to be present (bails out if any of them are missing) and uses them for all of the vanilla text strings, which only contain double-byte control codes. Thus, under normal conditions, TSFFTahoma remains completely unused in the Japanese version, and would only be used for (artificially added) US-ASCII input.

*If the US engine is used on the Japanese game data, then the .fnt files are ignored (obviously), and the incomplete TSFFTahoma is used to render the Japanese text strings as well as the few English strings supplied by the EXE. Due to the limited character set, many strings end up broken.  

Unlike other Western versions (UK English, French, German, Italian, Spanish, Russian), the US engine treats high-bit characters as part of a two-byte control sequence (a provision for Asian encodings), and therefore fails to render any character from the extended ASCII range. This happens twice in English Oni, because the ellipsis character (…), encoded as 0xC9, was accidentally used in [[Quotes/Consoles/level_19d|These]] [[Quotes/Consoles/level_19e|Two]] text consoles in place of three consecutive dots (probably auto-substituted by a text editor). The result is that the two lines using a "…" are cut off at the offending character.

The text strings in the Chinese version mostly conform to the EUC-CN scheme, but there are two recurrent invalid characters, as well as one instance of non-translated US-ASCII (!!!).

 

 

 

 

 

 

Interestingly, there are ''some'' types of primitive debug printout that is not intercepted by xfhsm_oni.dll and thus is displayed normally using the smallest-sized TSFT from level0_Final's TSFFTahoma. All (most?) of the primitive printout is available without Dev mode. There is the Ctrl+Shift+Y hotkey (FPS display), some HUD-like overlays toggled by [BSL:Variables|script variables]] (e.g., chr_debug_characters), and finally some 3D sprites added to the game (e.g., health indicators or name labels displayed above a character's head).

The illegal sequence (A3,A0) is much more common than (A3,0x89). It occurs in SUBT entries (both in actual subtitles and in "messages"), as well as in the [[OBD:IGSt|IGSt]] resources of multiple [[OBD:TxtC|TxtC]] (text consoles) and one [[OBD:OPge|OPge]] (objective page). Lists of occurrences are provided below.

Although not strictly speaking a font issue, some of Oni's text fails to render because it doesn't fit vertically into a fixed-size frame (such as a [[:File:DATA_CONSOLE.png|text console]]). This is known to happen for [[Quotes/Consoles/level_1e|These]] [[Quotes/Consoles/level_8b|Two]] consoles in the English version, and possibly for other screens in other language versions.

This is only known to cause a problem for the "resolution" item in the Options menu (a WMM_ generated at runtime). The actual dropdown list is wide enough to accommodate even the longest resolution strings, but the currently selected resolution appears in a small window that is only 150 pixels wide, too narrow even for the shortest resolution string "640×480×16位" (which needs 176 pixels). As a result the active resolution is always displayed on two lines, no longer fitting into the frame vertically and thus unreadable.

====Long subtitles in Asian Oni====

Another "over-long" issue is for Japanese or Chinese subtitles that are too long to fit on the screen horizontally. In such situations the engine displays the subtitle on multiple lines, placing line breaks at arbitrary positions instead of at ideographic spaces or after punctuation (if any), and aligning the ''end'' of the over-long string to the right of the screen, rather than vice-versa. This looks especially awkward when a subtitle is only slightly over-long and the automatic line break occurs near the ''start'' of the string, in the middle of a speaker's name (subtitles practically always have a colon ''and'' space after the speaker's name, and an arbitrary word break intuitively makes no sense in the vicinity of a space). For Western languages, the space character is registered as a line-breaking character in [[TSFL]]Roman, and allows for sensible display of long text, seeing as spaces occur both between words and after punctuation. Asian languages should supposedly do the same by specifying the ideographic space (a two-byte code point) as a line-breaking character, and possibly by setting up some rules for ideographic punctuation as well. In practice there is no such thing, hence the over-long subtitle issue.

(For the Japanese Oni, a possible workaround would be to insert an ASCII space character, or a narrow variant thereof, after punctuation. Chinese Oni doesn't support ASCII characters in text at all.)