tools/font/FONT.md

@@ -0,0 +1,83 @@
+# `Graphics/font.ttf` — how it is made, and how the cabinet actually uses its fonts
+
+## What the cabinet ships
+
+The cabinet has **no vector font**. `Data\x64\font\` holds pre-rasterised BC7 glyph atlases,
+one family per language and one atlas per size/weight bucket, each with an XML sidecar
+(`codepoint → rect`; no bearings, no kerning):
+
+| language | buckets on disk | distinct codepoints | face (as drawn in the atlas) |
+|---|---|---|---|
+| `jp` | `30`, `32_B`, `32_DB`, `32_EB`, `64` | 7 532 | 30/32: rounded gothic in three weights; 64: brush/勘亭流-style display face |
+| `tw` | `30`, `32`, `64` | 17 678 | same split |
+| `cn` | `30`, `32`, `64` | 7 608 | same split (the 64 face is a heavier brush) |
+| `kr` | `30`, `32`, `64` | 8 407 | same split |
+| `en` | `30`, `32`, `64` | 299 | same split |
+
+(The JP 39.06 dump has the same files minus `cn_*`; the CN 05 dump adds `cn_*`. The
+`jp/tw/kr/en` XMLs are byte-identical between the two.)
+
+## How a text record picks its atlas
+
+* **Language** comes from the cabinet language setting: wordlist rows are looked up in the
+  language's column and rendered from that language's atlas family. Rows flagged
+  `fontType 0` (untranslated) always render from `jp`.
+* **Size/weight bucket** is authored per text record in the `.nulm` (`DefineFont` entry
+  symbol `"0".."4"` → `64`, `30`, `32_EB`, `32_B`, `32_DB`). It is not language dependent.
+  For `en/tw/kr/cn` the three weighted `32_*` buckets are registered onto the plain `32`
+  atlas, so a record that asks for `32_EB` on a `cn` cabinet is drawn from `cn_32`.
+* The only fallback is `jp_64` when the language itself is unknown; a codepoint missing from
+  the chosen atlas is simply not drawn. Of the 692 text records in the shipped movies, 449
+  use the 30/32 (rounded gothic) buckets and 168 use the 64 (brush) bucket.
+
+YataiDON draws every string through a single `.ttf`, so one face has to serve all of it.
+This skin's `font.ttf` is built from the **`_64` (brush) atlases**, which is the face used for
+headings, song titles and most of the large text.
+
+## Building `font.ttf`
+
+`tools/font/build_font.py` merges the five `_64` atlases into one TrueType font:
+
+1. Decode each `<lang>_64.nutexb` to a DDS (BC7, `nutexb_dump -o`) next to its XML.
+2. For each language in priority order `jp, tw, cn, kr, en`, every codepoint that does not
+   yet have an inked glyph takes its bitmap from that atlas (blank placeholder cells are
+   skipped so a later atlas can supply the character).
+3. Each bitmap is upscaled 4× (bicubic), thresholded at alpha ≥ 128, traced with potrace
+   and converted to quadratic outlines (`cu2qu`).
+4. Metrics: `unitsPerEm 1024`, `scale = 1024 / fontPoint` of the atlas (16 units per pixel
+   for `jp_64`), `advance = cell width × scale`, baseline = the atlas's median bottom ink row
+   of its Han/Hangul glyphs, so the five families sit on one baseline with equal Han height.
+   `ascender 880 / descender −144`, as the earlier build.
+
+```
+pip install potracer fonttools pillow numpy
+python build_font.py <dir with *_64.dds + *_64.xml> font.ttf --order jp,tw,cn,kr,en
+```
+
+Result (version 3.000): 22 382 glyphs — 7 499 from `jp`, 10 107 from `tw`, 1 956 from `cn`,
+2 737 from `kr`, 82 from `en`. Coverage of each atlas's repertoire: jp 7521/7532,
+tw 17651/17678, cn 7589/7608, kr 8070/8407, en 268/295 (the remainder are blank cells in
+the atlases themselves). Version 2.000 had no `cn` or `kr` glyphs, which is why Simplified
+Chinese and Korean text showed `?`.
+
+## Per-language fonts (`Graphics/font_<family>.ttf`)
+
+The engine loads `Graphics/font_<language code>.ttf` if present, else `Graphics/font_<cabinet family>.ttf`
+(zh → cn, ko → kr, ja → jp), else `font.ttf`. The settings screen reloads the skin, so switching
+language switches the font. This mirrors the cabinet, which draws everything from the language's own
+atlas family and only lacks what that family does not have.
+
+| file | atlas priority | used for |
+|---|---|---|
+| `font.ttf` | `jp, tw, cn, kr, en` | Japanese, and any language without its own file (unchanged 3.000) |
+| `font_cn.ttf` | `cn, tw, jp, en, kr` | Simplified Chinese (`zh`) |
+| `font_tw.ttf` | `tw, jp, cn, en, kr` | Traditional Chinese (`zh_tw` / `tw`) |
+| `font_en.ttf` | `en, jp, tw, cn, kr` | English |
+| `font_kr.ttf` | `kr, jp, en, tw, cn` | Korean (`ko`) |
+
+```
+python build_font.py <dir> font_cn.ttf --order cn,tw,jp,en,kr
+python build_font.py <dir> font_tw.ttf --order tw,jp,cn,en,kr
+python build_font.py <dir> font_en.ttf --order en,jp,tw,cn,kr
+python build_font.py <dir> font_kr.ttf --order kr,jp,en,tw,cn
+```

tools/font/build_font.py

@@ -0,0 +1,157 @@
+"""Build a TrueType font from the cabinet's bitmap glyph atlases (font/<lang>_64.{xml,nutexb}).
+
+The cabinet ships no vector font: each language has its own BC7 atlas + XML (codepoint ->
+rect, no bearings, no kerning), and RenderText picks the atlas family by the cabinet
+language (fontType 0 rows -> jp) and the size/weight bucket per text record.  YataiDON
+draws everything through one .ttf, so this merges the five 64pt atlases into one font:
+every codepoint takes its glyph from the first atlas (in --order) that has ink for it.
+
+Per atlas:  scale   = upm / fontPoint          (jp_64: 16 units per pixel)
+            advance = glyph width * scale     (A: 51 px -> 816, same as the earlier build)
+            baseline row = median bottom ink row of the Han/Hangul glyphs (+1), so the
+            different atlases sit on one baseline; Latin-only atlases use 'A'.
+Tracing: alpha >= 128 at 4x bicubic upscale -> potrace -> cubic -> quadratic (cu2qu).
+
+usage: python build_font.py <atlas_dir with *_64.dds/*_64.xml> <out.ttf> [--order jp,tw,cn,kr,en] [--jobs N]
+"""
+import sys, os, argparse, statistics, time
+import xml.etree.ElementTree as ET
+import numpy as np
+from PIL import Image
+import potrace
+from multiprocessing import Pool
+
+UPM = 1024
+UPSCALE = 4
+ASCENT, DESCENT = 880, -144
+
+def parse_xml(path):
+    r = ET.parse(path).getroot().find('font')
+    glyphs = {int(x.get('index')): (int(x.get('offsetU')), int(x.get('offsetV')),
+                                    int(x.get('width')), int(x.get('height'))) for x in r.findall('glyph')}
+    return glyphs, {k: int(v) for k, v in r.attrib.items()}
+
+_A = None
+def _init(alpha_path):
+    global _A
+    _A = np.load(alpha_path, mmap_mode='r')
+
+def _trace(job):
+    cp, u, v, w, h, scale, baseline_px = job
+    a = np.asarray(_A[v:v + h, u:u + w])
+    rows = np.where(a.max(axis=1) >= 128)[0]
+    if rows.size == 0:
+        return cp, int(round(w * scale)), None            # blank: advance only
+    big = np.array(Image.fromarray(a).resize((w * UPSCALE, h * UPSCALE), Image.BICUBIC)) < 128   # potracer: True = white
+    path = potrace.Bitmap(big).trace(turdsize=4,
+                                     alphamax=1.0, opticurve=True, opttolerance=0.2)
+    k = scale / UPSCALE
+    def P(p): return (p.x * k, (baseline_px * UPSCALE - p.y) * k)
+    contours = []
+    for curve in path.curves:
+        segs = [('M', P(curve.start_point))]
+        for s in curve.segments:
+            if s.is_corner:
+                segs.append(('L', P(s.c))); segs.append(('L', P(s.end_point)))
+            else:
+                segs.append(('C', P(s.c1), P(s.c2), P(s.end_point)))
+        contours.append(segs)
+    return cp, int(round(w * scale)), contours
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('atlas_dir'); ap.add_argument('out')
+    ap.add_argument('--order', default='jp,tw,cn,kr,en')
+    ap.add_argument('--han-order', default='', help='separate priority for CJK ideographs (U+2E80-9FFF, F900-FAFF, 20000+); --order then applies to everything else')
+    ap.add_argument('--jobs', type=int, default=max(1, os.cpu_count() - 4))
+    ap.add_argument('--limit', type=int, default=0, help='debug: glyphs per atlas')
+    ap.add_argument('--embolden', default='', help='lang=px: dilate that atlas alpha by px before tracing')
+    args = ap.parse_args()
+    langs = args.order.split(',')
+    han_langs = args.han_order.split(',') if args.han_order else langs
+    def is_han(cp): return 0x2E80 <= cp <= 0x9FFF or 0xF900 <= cp <= 0xFAFF or cp >= 0x20000
+    def rank(lang, cp): return (han_langs if is_han(cp) else langs).index(lang) if lang in (han_langs if is_han(cp) else langs) else 99
+
+    taken = {}      # cp -> (advance, contours or None, lang)
+    all_langs = list(dict.fromkeys(langs + han_langs))
+    for lang in all_langs:
+        t0 = time.time()
+        glyphs, attr = parse_xml(os.path.join(args.atlas_dir, f'{lang}_64.xml'))
+        im = Image.open(os.path.join(args.atlas_dir, f'{lang}_64.dds'))
+        A = np.array(im)[:, :, 3]
+        emb = dict(kv.split('=') for kv in args.embolden.split(',') if kv)
+        if lang in emb and int(emb[lang]) > 0:
+            from scipy.ndimage import grey_dilation
+            k = 2 * int(emb[lang]) + 1; A = grey_dilation(A, size=(k, k))
+        scale = UPM / attr['fontPoint']
+        # baseline: where the Han / Hangul glyphs sit (Latin-only atlas: 'A')
+        bots = []
+        for cp, (u, v, w, h) in glyphs.items():
+            if 0x4E00 <= cp <= 0x9FFF or 0xAC00 <= cp <= 0xD7A3:
+                r = np.where(A[v:v + h, u:u + w].max(axis=1) >= 128)[0]
+                if r.size: bots.append(int(r.max()))
+        if bots:
+            baseline_px = statistics.median(bots) + 1
+        else:
+            u, v, w, h = glyphs[0x41]
+            r = np.where(A[v:v + h, u:u + w].max(axis=1) >= 128)[0]
+            baseline_px = int(r.max()) + 1 + 2 * 64 / attr['fontPoint']   # 'A' sits ~2 jp-px above the Han bottom
+        alpha_path = os.path.join(args.atlas_dir, f'_{lang}_alpha.npy')
+        np.save(alpha_path, A)
+        jobs = [(cp, u, v, w, h, scale, baseline_px) for cp, (u, v, w, h) in sorted(glyphs.items())
+                if (cp not in taken or taken[cp][1] is None and cp not in (0x20, 0x3000, 0xA0) or rank(lang, cp) < rank(taken[cp][2], cp))]
+        if args.limit: jobs = jobs[:args.limit]
+        got = 0
+        with Pool(args.jobs, initializer=_init, initargs=(alpha_path,)) as pool:
+            for cp, adv, contours in pool.imap_unordered(_trace, jobs, chunksize=64):
+                if contours is None:
+                    if cp in (0x20, 0x3000, 0xA0) and cp not in taken:
+                        taken[cp] = (adv, None, lang)
+                    continue
+                if cp in taken and taken[cp][1] is not None and rank(lang, cp) >= rank(taken[cp][2], cp): continue
+                taken[cp] = (adv, contours, lang); got += 1
+        os.remove(alpha_path)
+        print(f'{lang}_64: fontPoint {attr["fontPoint"]} scale {scale:.2f} baseline {baseline_px:.1f}px '
+              f'-> {got} new glyphs ({time.time() - t0:.0f}s)', flush=True)
+
+    # ---- assemble
+    from fontTools.fontBuilder import FontBuilder
+    from fontTools.pens.ttGlyphPen import TTGlyphPen
+    from fontTools.pens.cu2quPen import Cu2QuPen
+    order = ['.notdef'] + [(f'u{cp:05X}' if cp > 0xFFFF else f'uni{cp:04X}') for cp in sorted(taken)]
+    fb = FontBuilder(UPM, isTTF=True)
+    fb.setupGlyphOrder(order)
+    cmap, glyf, hmtx = {}, {}, {}
+    pen = TTGlyphPen(None); glyf['.notdef'] = pen.glyph(); hmtx['.notdef'] = (512, 0)
+    for name, cp in zip(order[1:], sorted(taken)):
+        adv, contours, _ = taken[cp]
+        pen = TTGlyphPen(None)
+        if contours:
+            qpen = Cu2QuPen(pen, max_err=1.0, reverse_direction=False)
+            for segs in contours:
+                for s in segs:
+                    if s[0] == 'M': qpen.moveTo(s[1])
+                    elif s[0] == 'L': qpen.lineTo(s[1])
+                    else: qpen.curveTo(s[1], s[2], s[3])
+                qpen.closePath()
+        g = pen.glyph()
+        glyf[name] = g
+        g.recalcBounds(glyf) if contours else None
+        hmtx[name] = (adv, getattr(g, 'xMin', 0) if contours else 0)
+        cmap[cp] = name
+    fb.setupCharacterMap(cmap)
+    fb.setupGlyf(glyf)
+    fb.setupHorizontalMetrics(hmtx)
+    fb.setupHorizontalHeader(ascent=ASCENT, descent=DESCENT)
+    fb.setupNameTable({'familyName': 'YataiDON HSS Atlas', 'styleName': 'Regular',
+                       'uniqueFontIdentifier': 'YataiDON HSS Atlas 3.000', 'fullName': 'YataiDON HSS Atlas Regular',
+                       'version': 'Version 3.000', 'psName': 'YataiDONHSSAtlas-Regular'})
+    fb.setupOS2(sTypoAscender=ASCENT, sTypoDescender=DESCENT, sTypoLineGap=0,
+                usWinAscent=ASCENT, usWinDescent=-DESCENT, fsType=0)
+    fb.setupPost()
+    fb.save(args.out)
+    from collections import Counter
+    print('saved', args.out, len(order), 'glyphs;', dict(Counter(v[2] for v in taken.values())))
+
+if __name__ == '__main__':
+    main()

tools/font/build_font_lite.py

@@ -0,0 +1,155 @@
+"""Build a TrueType font from the cabinet's bitmap glyph atlases (font/<lang>_64.{xml,nutexb}).
+
+The cabinet ships no vector font: each language has its own BC7 atlas + XML (codepoint ->
+rect, no bearings, no kerning), and RenderText picks the atlas family by the cabinet
+language (fontType 0 rows -> jp) and the size/weight bucket per text record.  YataiDON
+draws everything through one .ttf, so this merges the five 64pt atlases into one font:
+every codepoint takes its glyph from the first atlas (in --order) that has ink for it.
+
+Per atlas:  scale   = upm / fontPoint          (jp_64: 16 units per pixel)
+            advance = glyph width * scale     (A: 51 px -> 816, same as the earlier build)
+            baseline row = median bottom ink row of the Han/Hangul glyphs (+1), so the
+            different atlases sit on one baseline; Latin-only atlases use 'A'.
+Tracing: alpha >= 128 at 4x bicubic upscale -> potrace -> cubic -> quadratic (cu2qu).
+
+usage: python build_font.py <atlas_dir with *_64.dds/*_64.xml> <out.ttf> [--order jp,tw,cn,kr,en] [--jobs N]
+"""
+import sys, os, argparse, statistics, time
+import xml.etree.ElementTree as ET
+import numpy as np
+from PIL import Image
+import potrace
+from multiprocessing import Pool
+
+UPM = 1024
+UPSCALE = 4
+ASCENT, DESCENT = 880, -144
+
+def parse_xml(path):
+    r = ET.parse(path).getroot().find('font')
+    glyphs = {int(x.get('index')): (int(x.get('offsetU')), int(x.get('offsetV')),
+                                    int(x.get('width')), int(x.get('height'))) for x in r.findall('glyph')}
+    return glyphs, {k: int(v) for k, v in r.attrib.items()}
+
+_A = None
+def _init(alpha_path):
+    global _A
+    _A = np.load(alpha_path, mmap_mode='r')
+
+def _trace(job):
+    cp, u, v, w, h, scale, baseline_px = job
+    a = np.asarray(_A[v:v + h, u:u + w])
+    rows = np.where(a.max(axis=1) >= 128)[0]
+    if rows.size == 0:
+        return cp, int(round(w * scale)), None            # blank: advance only
+    big = np.array(Image.fromarray(a).resize((w * UPSCALE, h * UPSCALE), Image.BICUBIC)) < 128   # potracer: True = white
+    path = potrace.Bitmap(big).trace(turdsize=4,
+                                     alphamax=1.0, opticurve=True, opttolerance=0.5)
+    k = scale / UPSCALE
+    def P(p): return (p.x * k, (baseline_px * UPSCALE - p.y) * k)
+    contours = []
+    for curve in path.curves:
+        segs = [('M', P(curve.start_point))]
+        for s in curve.segments:
+            if s.is_corner:
+                segs.append(('L', P(s.c))); segs.append(('L', P(s.end_point)))
+            else:
+                segs.append(('C', P(s.c1), P(s.c2), P(s.end_point)))
+        contours.append(segs)
+    return cp, int(round(w * scale)), contours
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('atlas_dir'); ap.add_argument('out')
+    ap.add_argument('--order', default='jp,tw,cn,kr,en')
+    ap.add_argument('--jobs', type=int, default=max(1, os.cpu_count() - 4))
+    ap.add_argument('--limit', type=int, default=0, help='debug: glyphs per atlas')
+    ap.add_argument('--files', default='', help='lang=atlas stem overrides, e.g. jp=jp_32_EB,tw=tw_32')
+    ap.add_argument('--embolden', default='', help='lang=px: dilate that atlas alpha by px before tracing')
+    args = ap.parse_args()
+    langs = args.order.split(',')
+    stems = dict(kv.split('=') for kv in args.files.split(',') if kv)
+
+    taken = {}      # cp -> (advance, contours or None, lang)
+    for lang in langs:
+        t0 = time.time()
+        stem = stems.get(lang, f'{lang}_64')
+        glyphs, attr = parse_xml(os.path.join(args.atlas_dir, f'{stem}.xml'))
+        im = Image.open(os.path.join(args.atlas_dir, f'{stem}.dds'))
+        A = np.array(im)[:, :, 3]
+        emb = dict(kv.split('=') for kv in args.embolden.split(',') if kv)
+        if lang in emb and int(emb[lang]) > 0:
+            from scipy.ndimage import grey_dilation
+            k = 2 * int(emb[lang]) + 1; A = grey_dilation(A, size=(k, k))
+        scale = UPM / attr['fontPoint']
+        # baseline: where the Han / Hangul glyphs sit (Latin-only atlas: 'A')
+        bots = []
+        for cp, (u, v, w, h) in glyphs.items():
+            if 0x4E00 <= cp <= 0x9FFF or 0xAC00 <= cp <= 0xD7A3:
+                r = np.where(A[v:v + h, u:u + w].max(axis=1) >= 128)[0]
+                if r.size: bots.append(int(r.max()))
+        if bots:
+            baseline_px = statistics.median(bots) + 1
+        else:
+            u, v, w, h = glyphs[0x41]
+            r = np.where(A[v:v + h, u:u + w].max(axis=1) >= 128)[0]
+            baseline_px = int(r.max()) + 1 + 2 * 64 / attr['fontPoint']   # 'A' sits ~2 jp-px above the Han bottom
+        alpha_path = os.path.join(args.atlas_dir, f'_{lang}_alpha.npy')
+        np.save(alpha_path, A)
+        jobs = [(cp, u, v, w, h, scale, baseline_px) for cp, (u, v, w, h) in sorted(glyphs.items())
+                if cp not in taken or taken[cp][1] is None and cp not in (0x20, 0x3000, 0xA0)]
+        if args.limit: jobs = jobs[:args.limit]
+        got = 0
+        with Pool(args.jobs, initializer=_init, initargs=(alpha_path,)) as pool:
+            for cp, adv, contours in pool.imap_unordered(_trace, jobs, chunksize=64):
+                if contours is None:
+                    if cp in (0x20, 0x3000, 0xA0) and cp not in taken:
+                        taken[cp] = (adv, None, lang)
+                    continue
+                if cp in taken and taken[cp][1] is not None: continue
+                taken[cp] = (adv, contours, lang); got += 1
+        os.remove(alpha_path)
+        print(f'{stem}: fontPoint {attr["fontPoint"]} scale {scale:.2f} baseline {baseline_px:.1f}px '
+              f'-> {got} new glyphs ({time.time() - t0:.0f}s)', flush=True)
+
+    # ---- assemble
+    from fontTools.fontBuilder import FontBuilder
+    from fontTools.pens.ttGlyphPen import TTGlyphPen
+    from fontTools.pens.cu2quPen import Cu2QuPen
+    order = ['.notdef'] + [(f'u{cp:05X}' if cp > 0xFFFF else f'uni{cp:04X}') for cp in sorted(taken)]
+    fb = FontBuilder(UPM, isTTF=True)
+    fb.setupGlyphOrder(order)
+    cmap, glyf, hmtx = {}, {}, {}
+    pen = TTGlyphPen(None); glyf['.notdef'] = pen.glyph(); hmtx['.notdef'] = (512, 0)
+    for name, cp in zip(order[1:], sorted(taken)):
+        adv, contours, _ = taken[cp]
+        pen = TTGlyphPen(None)
+        if contours:
+            qpen = Cu2QuPen(pen, max_err=2.0, reverse_direction=False)
+            for segs in contours:
+                for s in segs:
+                    if s[0] == 'M': qpen.moveTo(s[1])
+                    elif s[0] == 'L': qpen.lineTo(s[1])
+                    else: qpen.curveTo(s[1], s[2], s[3])
+                qpen.closePath()
+        g = pen.glyph()
+        glyf[name] = g
+        g.recalcBounds(glyf) if contours else None
+        hmtx[name] = (adv, getattr(g, 'xMin', 0) if contours else 0)
+        cmap[cp] = name
+    fb.setupCharacterMap(cmap)
+    fb.setupGlyf(glyf)
+    fb.setupHorizontalMetrics(hmtx)
+    fb.setupHorizontalHeader(ascent=ASCENT, descent=DESCENT)
+    fb.setupNameTable({'familyName': os.environ.get('FONT_FAMILY', 'YataiDON HSS Atlas'), 'styleName': 'Regular',
+                       'uniqueFontIdentifier': os.environ.get('FONT_FAMILY', 'YataiDON HSS Atlas') + ' 3.000', 'fullName': os.environ.get('FONT_FAMILY', 'YataiDON HSS Atlas') + ' Regular',
+                       'version': 'Version 3.000', 'psName': os.environ.get('FONT_FAMILY', 'YataiDON HSS Atlas').replace(' ', '') + '-Regular'})
+    fb.setupOS2(sTypoAscender=ASCENT, sTypoDescender=DESCENT, sTypoLineGap=0,
+                usWinAscent=ASCENT, usWinDescent=-DESCENT, fsType=0)
+    fb.setupPost()
+    fb.save(args.out)
+    from collections import Counter
+    print('saved', args.out, len(order), 'glyphs;', dict(Counter(v[2] for v in taken.values())))
+
+if __name__ == '__main__':
+    main()