petal/scripts/build_hunspell_dictionary.py

#!/usr/bin/env python3
"""Build one of Petal's browser spelling dictionaries from a Hunspell one.

Why this script exists at all
----------------------------
English is vendored the obvious way: `dictionary-en`'s `en.aff` + `en.dic` go
into web/public/dictionaries/en and nspell reads them in the browser. The plan
for Phase 21 said "Hunspell pt-PT vendored like en-US", and that turns out not to
work, for a measured reason.

nspell expands affixes **eagerly at load time** — it materialises every surface
form into a hash the moment you construct it. English gets away with this: ~50k
stems and a small rule set. European Portuguese does not. `pt_PT.aff` carries
1,340 affix rules (the full verb paradigm: six persons x a dozen tenses, plus
diminutives, plus productive prefixes) over 44,257 stems. Measured on this
machine, nspell needed ~340 MB of heap for the first 12,000 entries alone and had
not returned after three minutes on the whole file; extrapolated, it wants well
over a gigabyte. That is not something to hand a browser, still less a tablet.
French is larger again: 5,600 affix rules over 84,140 stems.

So the expansion happens **here**, once, at build time, and the browser gets a
flat word list it can load with no affix machinery at all. The runtime code path
is then *identical* to English — same nspell, same interface — which is the real
prize. The aff shipped alongside keeps only the suggestion-shaping directives
(TRY/KEY/REP/MAP), so corrections still know that "cao" wants "ção" and that a
missing acute accent is a near miss.

What Phase 24 had to add
------------------------
The build plan recorded that the pt-PT version of this script "generalizes" to
French. It did not. It handled single-character flags and plain PFX/SFX and
stopped on anything else — the right call, because `fr.aff` uses four of the
things it stopped on, and getting any of them wrong changes which words are
accepted:

  * **`FLAG long`** — French flags are *two characters* (`S.`, `L'`, `Um`). The
    pt-PT reader took `set(flagstr)`, one flag per character, which on a French
    entry yields a bag of unrelated single letters: every entry would have been
    expanded through the wrong paradigm. This is the one that fails silently.
  * **Continuation flags** — pt-PT's affixes append plain text, so that script
    dropped anything after a `/` and asserted the drop was safe. French really
    does affix an affixed form: `PFX Um 0 0/S.` says the prefixed form then takes
    the plural suffix, and the elision prefixes arrive the same way from the
    other side (`SFX ... ait/n'q'l'm't's'`).
  * **`NEEDAFFIX`** — French marks thousands of stems "not a word on its own"
    (`Allemagne/S.()`), the bare form arriving instead through a zero-append
    rule. Ignoring the flag accepts stems the real dictionary rejects.
  * **`FULLSTRIP`** — a rule may strip the whole stem.

`CIRCUMFIX` and `FORBIDDENWORD` are *declared* in `fr.aff` and used by nothing,
which this script asserts rather than assumes: an upstream release that started
using either would otherwise change what is accepted without changing this file.
`KEEPCASE` and `NOSUGGEST` are honoured by being ignored on purpose — they shape
casing and suggestions, not membership, and a NOSUGGEST word is still a word.

Elision is handled at lookup, not here — and that is the size decision
----------------------------------------------------------------------
Most of French's affix machinery by volume is elision: `l'`, `d'`, `qu'`, `j'`,
`n'`, `s'`, `jusqu'`, `puisqu'`. Hunspell treats `l'arbre` as one word, so a
faithful expansion carries much of the language thirty-four times over — and
Petal's tokenizer keeps internal apostrophes, so `l'arbre` really does arrive at
the dictionary as one token and really would be underlined if it were absent.

Both halves were built and measured. Keeping the elided forms: **3,159,832 forms,
8.25 MB gzipped**, ~45 MB of text for nspell to hash on a tablet. Dropping them:
**473,326 forms, 1.19 MB gzipped**. The elided seven-eighths are not new words —
they are thirteen little words glued to words already in the list — so the third
option is the one taken: rules whose append carries an apostrophe are skipped
here (the count is printed), and `withElision` in `useSpellChecker.ts` splits a
token at a *known clitic* and checks the remainder. `l'arbre` costs one extra
lookup instead of seven megabytes, and `zzz'arbre` is still flagged because
`zzz` is not one of the thirteen.

Stems that carry an apostrophe of their own — `aujourd'hui`, `quelqu'un`,
`presqu'île`, `prud'homme` — are dictionary entries rather than affixed forms,
so they are kept verbatim and matched directly. `entr'aide` and `grand'mère` are
absent for the same reason they are absent from Dicollecte: modern French spells
them `entraide` and `grand-mère`.

Choosing the source
-------------------
Both languages have a trap here, and they are different traps.

**pt-PT: the wrong country.** npm's `dictionary-pt` is not European Portuguese.
Both it and `dictionary-pt-br` package VERO ("Verificador Ortográfico Livre",
Brasil), so vendoring the obvious npm name would have shipped Brazilian spellings
under a pt-PT label — the pt-BR drift SUGGESTIONS.md §3 warns about, arriving
through the packaging rather than through the model. The authentic dictionary is
the Projecto Natura one (Universidade do Minho) that LibreOffice ships and Debian
packages as `hunspell-pt-pt`; its aff declares `LANG pt_PT`.

**es: the wrong country again, hidden one layer further down.** Spanish looked
like it would repeat the pt trap — `hunspell-es` installs twenty country codes,
`es_AR` through `es_VE` — and then looked like it did not, because every one of
them is a symlink to a single `es_ES.aff`/`es_ES.dic`. Both readings were wrong.
Debian collapses the twenty because it ships **one** of upstream's builds, and
the one it ships is the **peninsular** `es_ES`. RLA (Santiago Bosio's project,
`sbosio/rla-es`) publishes twenty-four dictionaries per release: one per country,
plus a **generic `es`** that is the union of all of them. Debian packages neither
the generic one nor a choice — it packages Spain, under a name that reads like
"Spanish".

Measured against the v2.9 release: Debian's file is 659,085 expanded forms and
upstream `es_ES` is 659,018; the generic `es` is **717,640**. The 58,622-form
difference is almost entirely **voseo** — `vení`, `tenés`, `querés`, `sabés`,
`andá` — the present tense of most of Latin America, which Debian's package
rejects as misspellings. Petal ships the **generic** build.

**Vocabulary cannot detect this and morphology can.** The first version of the es
profile asserted the pan-Hispanic lexicon — *computadora* and *ordenador*, *papa*
and *patata* — and passed happily on the peninsular file, because **every** RLA
variant carries the full pan-Hispanic vocabulary; only the verb paradigms are
localised. The `REP` table is no help either: its `ll`↔`y` and `ás`↔`az` entries
look like evidence of yeísmo and seseo, but they are shared by all twenty-four
builds. What separates them is exactly two things, and the profile now demands
both at once: **voseo** (absent from `es_ES`) and **vosotros** (largely absent
from `es_MX`). Only the generic build has both, so only the generic build passes.

This is the same decision fr made between `-classical` and `-revised`, arriving
by a different road. The only thing this dictionary can do is underline
something, and *tienes* and *tenés* are both correct Spanish taught in different
countries — so Petal takes the build that accepts every variety rather than one
that makes a writer wrong for where she is from. Nothing is generated to get
there: the forms come from a real upstream package, which is what lets the
MUST_ACCEPT list prove which package it was.

Licensing note: RLA is tri-licensed GPL-3+ / LGPL-3+ / MPL-1.1+; Petal
redistributes under the MPL. The upstream README and LICENSE are vendored beside
the output.

**fr: the wrong side of an argument the French have not settled.** The regional
question turns out to be a non-question — Debian's `fr_FR`, `fr_CA`, `fr_BE`,
`fr_CH`, `fr_LU` and `fr_MC` are all symlinks to one `fr.dic`, so unlike pt there
is no country here to get wrong. What there is instead is the 1990 spelling
reform, packaged three ways: `hunspell-fr-classical` (traditional), `-revised`
(reform only) and `-comprehensive` (both). Petal ships **comprehensive**, because
Petal never corrects her French — the only thing this dictionary can do is
underline something. *coût* and *cout* are both correct French, taught in
different decades to different people, and a writing companion has no business
underlining one of them to take a side. The `fr` MUST_ACCEPT list is written to
*prove* which package was used: classical rejects `cout`, revised rejects `coût`,
and only comprehensive accepts both.

Licensing: pt-PT is GPL-2 or LGPL-2.1 or MPL-1.1, (c) José João de Almeida, Rui
Vilela, Alberto Simões. fr is MPL-2.0, (c) 2007-2018 the Dicollecte contributors
(grammalecte.net). The upstream copyright file is vendored beside each output.

Usage
-----
    apt-get download hunspell-fr-comprehensive     # or hunspell-pt-pt
    dpkg-deb -x hunspell-fr-comprehensive_*.deb src
    python3 scripts/build_hunspell_dictionary.py fr \\
        src/usr/share/hunspell/fr.aff \\
        src/usr/share/hunspell/fr.dic \\
        web/public/dictionaries/fr

Spanish does not come from Debian — see below; `hunspell-es` is the peninsular
build. Take the generic dictionary from an upstream release instead:

    curl -LO https://github.com/sbosio/rla-es/releases/download/v2.9/es.oxt
    unzip -d src es.oxt                            # an .oxt is a zip
    python3 scripts/build_hunspell_dictionary.py es \\
        src/es.aff src/es.dic web/public/dictionaries/es
"""
import gzip
import os
import re
import sys
import unicodedata

# Directives worth keeping in the shipped aff. These shape *suggestions*, not
# membership: TRY orders the alphabet the corrector tries, KEY knows which keys
# are adjacent, REP holds the language's own confusions (cao/ção, ss/ç), and MAP
# says an accented vowel and its bare form are the same letter for scoring —
# which is most of what an ESL writer gets wrong in either language.
KEEP_DIRECTIVES = ("SET", "TRY", "KEY", "REP", "MAP", "WORDCHARS")

# Directives that would change which words are *accepted* and that this expander
# does not implement. If a future upstream release starts using one, the output
# would silently disagree with the real dictionary, so the build stops instead.
UNSUPPORTED = (
    "COMPOUNDFLAG", "COMPOUNDMIN", "COMPOUNDRULE", "COMPOUNDBEGIN",
    "ONLYINCOMPOUND", "PSEUDOROOT", "AF", "AM",
)

APOSTROPHES = "'’"


class Aff:
    """The parts of an .aff file that decide which words exist."""

    def __init__(self):
        self.pfx = {}      # flag -> [(strip, append, condition, continuation)]
        self.sfx = {}
        # Cross-product, per flag — and kept per table, because PFX and SFX are
        # separate flag namespaces in hunspell: the same flag may name a prefix
        # table and a suffix table, with different cross-product settings. One
        # shared dict let the second block silently overwrite the first.
        self.cross_pfx = {}   # flag -> bool
        self.cross_sfx = {}
        self.flag_kind = "char"
        self.needaffix = None
        self.circumfix = None
        self.forbidden = None
        self.dropped_apostrophe_rules = 0


def parse_flags(raw, kind):
    """Split a flag string into flags, per the aff's FLAG declaration."""
    raw = raw.strip()
    if not raw:
        return set()
    if kind == "long":
        # Two characters per flag, exactly. An odd length is a malformed flag
        # string, and silently dropping the trailing character would quietly
        # expand an entry through the wrong paradigm — the failure mode this
        # whole FLAG-aware rewrite exists to avoid.
        if len(raw) % 2:
            raise SystemExit(f"odd-length long flag string {raw!r}")
        return {raw[i:i + 2] for i in range(0, len(raw), 2)}
    if kind == "num":
        return {f for f in raw.split(",") if f}
    return set(raw)


def parse_aff(path):
    with open(path, encoding="utf-8") as fh:
        lines = fh.read().splitlines()

    aff = Aff()

    # FLAG has to be known before anything containing a flag is read, and it can
    # sit anywhere in the file. So: header pass first, rules second.
    for line in lines:
        parts = line.split()
        if not parts:
            continue
        head = parts[0]
        if head in UNSUPPORTED:
            raise SystemExit(
                f"{path}: unsupported directive {head!r} — this expander handles "
                "PFX/SFX affixation with continuation flags, and honouring "
                f"{head} would change which words are accepted. Extend the "
                "script before shipping."
            )
        if len(parts) < 2:
            continue
        if head == "FLAG":
            aff.flag_kind = parts[1]
            if aff.flag_kind not in ("long", "num", "UTF-8"):
                raise SystemExit(f"{path}: unknown FLAG type {aff.flag_kind!r}")
        elif head == "NEEDAFFIX":
            aff.needaffix = parts[1]
        elif head == "CIRCUMFIX":
            aff.circumfix = parts[1]
        elif head == "FORBIDDENWORD":
            aff.forbidden = parts[1]

    i = 0
    while i < len(lines):
        parts = lines[i].split()
        if parts and parts[0] in ("PFX", "SFX"):
            kind, flag, cross_flag, count = parts[0], parts[1], parts[2], int(parts[3])
            table = aff.pfx if kind == "PFX" else aff.sfx
            cross = aff.cross_pfx if kind == "PFX" else aff.cross_sfx
            cross[flag] = cross_flag == "Y"
            rules = table.setdefault(flag, [])
            for j in range(1, count + 1):
                p = lines[i + j].split()
                strip = "" if p[2] == "0" else p[2]
                append, _, cont_raw = p[3].partition("/")
                if append == "0":
                    append = ""
                # Elision. See the header: these forms are `l'` and its twelve
                # siblings glued to words already in the list, they multiply the
                # download by seven, and `withElision` reconstructs them at
                # lookup for the cost of one extra hash probe.
                if any(a in append for a in APOSTROPHES):
                    aff.dropped_apostrophe_rules += 1
                    continue
                cont = parse_flags(cont_raw, aff.flag_kind)
                if aff.circumfix and aff.circumfix in cont:
                    raise SystemExit(
                        f"{path}: CIRCUMFIX is used by a {kind} {flag} rule. It "
                        "was declared-but-unused when this expander was written "
                        "and is not implemented; honouring it would change which "
                        "words are accepted."
                    )
                cond = p[4] if len(p) > 4 else "."
                anchored = ("^" + cond) if kind == "PFX" else (cond + "$")
                rules.append((strip, append, re.compile(anchored), cont))
            i += count + 1
            continue
        i += 1
    return aff


def apply_suffix(word, rules):
    """Every (form, continuation flags) a suffix table yields for `word`."""
    out = []
    for strip, append, cond, cont in rules:
        if strip and not word.endswith(strip):
            continue
        if not cond.search(word):
            continue
        stem = word[: len(word) - len(strip)] if strip else word
        out.append((stem + append, cont))
    return out


def apply_prefix(word, rules):
    out = []
    for strip, append, cond, cont in rules:
        if strip and not word.startswith(strip):
            continue
        if not cond.search(word):
            continue
        stem = word[len(strip):] if strip else word
        out.append((append + stem, cont))
    return out


def expand_entry(word, flags, aff, out):
    """Add every surface form of one dictionary entry to `out`.

    Hunspell's model without compounding: a form is the stem plus at most one
    prefix and at most one suffix. A flag reaches an affix either from the stem's
    own flags or from the continuation flags of the affix applied on the other
    side; when both sides apply, both rules must be declared cross-product.

    NEEDAFFIX is why the bare form is not simply added: the flag says *this* form
    is not a word, only whatever can be built from it — and it arrives both on
    stems and on continuations.
    """
    def is_word(carried):
        return not (aff.needaffix and aff.needaffix in carried)

    if is_word(flags):
        out.add(word)

    suffixed = []   # (form, flag, continuation flags)
    for f in flags:
        if f in aff.sfx:
            for form, cont in apply_suffix(word, aff.sfx[f]):
                suffixed.append((form, f, cont))
                if is_word(cont):
                    out.add(form)

    prefixed = []
    for f in flags:
        if f in aff.pfx:
            for form, cont in apply_prefix(word, aff.pfx[f]):
                prefixed.append((form, f, cont))
                if is_word(cont):
                    out.add(form)

    # Prefix then suffix. The suffix flag may come from the stem or from the
    # prefix's own continuation (`PFX Um 0 0/S.`), and the suffix condition is
    # matched against the whole prefixed word, which is what hunspell does.
    # NEEDAFFIX is checked here too, exactly as on the single-affix paths above:
    # a doubly-affixed form whose last continuation still carries the flag is
    # "not a word on its own", and without compounding there is no third affix
    # left to make it one.
    for form, pf, pcont in prefixed:
        if not aff.cross_pfx.get(pf):
            continue
        for f in flags | pcont:
            if f in aff.sfx and aff.cross_sfx.get(f):
                for full, fcont in apply_suffix(form, aff.sfx[f]):
                    if is_word(fcont):
                        out.add(full)

    # Suffix then prefix — the same pair reached from the other side, which is
    # how the elision prefixes arrive in French. Only the flags the suffix hands
    # forward are new here; the stem's own were covered above.
    for form, sf, scont in suffixed:
        if not aff.cross_sfx.get(sf):
            continue
        for f in scont:
            if f in aff.pfx and aff.cross_pfx.get(f):
                for full, fcont in apply_prefix(form, aff.pfx[f]):
                    if is_word(fcont):
                        out.add(full)


def expand(aff_path, dic_path):
    aff = parse_aff(aff_path)
    forms = set()
    needaffix_stems = 0
    with open(dic_path, encoding="utf-8") as fh:
        fh.readline()  # leading entry count, not a word
        for raw in fh:
            # Morphological fields (po:nom is:fem) follow the entry, separated by
            # a tab in pt-PT and by a space in fr.
            entry = raw.strip().split("\t")[0].split(" ")[0]
            if not entry:
                continue
            word, _, flagstr = entry.partition("/")
            word = word.strip()
            if not word:
                continue
            flags = parse_flags(flagstr, aff.flag_kind)
            if aff.forbidden and aff.forbidden in flags:
                raise SystemExit(
                    f"{dic_path}: FORBIDDENWORD is in use ({word!r}). It was "
                    "declared-but-unused when this expander was written; the "
                    "forms it removes would be wrongly accepted."
                )
            if aff.needaffix and aff.needaffix in flags:
                needaffix_stems += 1
            expand_entry(word, flags, aff, forms)

    # NFC, because the aff's own ICONV table normalises decomposed accents on the
    # way in and the browser hands nspell whatever the keyboard produced.
    forms = {unicodedata.normalize("NFC", f) for f in forms}
    return forms, aff, needaffix_stems


def shipped_aff(aff_path):
    keep = []
    for line in open(aff_path, encoding="utf-8").read().splitlines():
        head = line.split()[0] if line.split() else ""
        if head in KEEP_DIRECTIVES:
            keep.append(line)
    return "\n".join(keep) + "\n"


# Words the built list must accept, and must reject, before it is written. Each
# set is chosen to fail loudly on the *specific* wrong source that language has a
# packaged, plausible way of reaching — not to spot-check spelling in general.
PROFILES = {
    # The pt-PT/pt-BR fault lines: post-Acordo spellings, the European lexicon,
    # and the first-person-plural preterite accent that only pt-PT writes.
    "pt-PT": {
        "accept": ("receção", "húmido", "telemóvel", "autocarro", "comboio",
                   "ótimo", "pensámos", "escrevêssemos", "jardim"),
        "reject": ("recepção", "úmido", "ônibus", "óptimo"),
        "wrong": "this does not look like European Portuguese",
    },
    # Which of the three 1990-reform packagings this is. `coût`/`cout` and
    # `paraître`/`paraitre` are each accepted by exactly one of classical and
    # revised, so a build accepting all four is comprehensive and one that drops
    # any of them is not. `Allemagne` is a NEEDAFFIX stem reachable only through
    # a zero-append rule and `km` only through a prefix continuation, so between
    # them they also check that this expander honoured the two features the
    # pt-PT one refused.
    "fr": {
        "accept": ("coût", "cout", "paraître", "paraitre", "nénuphar", "nénufar",
                   "oignon", "ognon", "événement", "évènement", "jardin",
                   "Allemagne", "écrivissions", "km"),
        "reject": ("jardinn", "écrivaitz", "xyzzyque"),
        "wrong": "this does not look like the comprehensive French dictionary",
    },
    # The generic RLA build, and the accept list is written to reject the four
    # neighbouring builds rather than to describe this one.
    #
    # The first version of this profile demanded *computadora* and *ordenador*,
    # *papa* and *patata*, and passed — on the peninsular file, because **every**
    # RLA variant carries the whole pan-Hispanic vocabulary. Vocabulary does not
    # discriminate here at all; only morphology does, and it discriminates
    # completely:
    #
    #   * **voseo** (`vení`, `tenés`, `querés`) is in `es` and `es_AR` and not in
    #     `es_ES` or Debian's package. Demanding it rejects the peninsular build.
    #   * **vosotros** (`tenéis`, `escribid`) is in `es`, `es_AR` and `es_ES`, and
    #     largely absent from `es_MX`. Demanding it rejects the Mexican build.
    #
    # Requiring both at once leaves exactly one package standing: the generic
    # `es`, which is the only one that accepts every variety of Spanish. That is
    # the same reason fr ships `-comprehensive` — the only thing this dictionary
    # can do is underline something, and *tienes* and *tenés* are both correct
    # Spanish taught in different countries.
    #
    # The rest are shape checks: `escribiésemos` is the -se imperfect subjunctive,
    # `dámelo` proves the enclitic pronoun rules ran, and `jardín`/`niño` prove
    # FLAG UTF-8 was read as characters rather than bytes.
    "es": {
        "accept": (
            # Rejects es_ES and Debian's hunspell-es.
            "vení", "tenés", "querés", "sabés", "andá",
            # Rejects es_MX.
            "tenéis", "escribid",
            # Rejects es_AR, which has both voseo and vosotros and would
            # otherwise pass. Caribbean and Andean everyday words: the generic
            # build is the union of all twenty-four, so it is the only one that
            # holds another region's vocabulary as well as its own.
            "arepa", "chévere", "bacán",
            # Pan-Hispanic vocabulary. These pass on every RLA build, so they
            # prove nothing on their own — kept because a source that stopped
            # being RLA at all would fail them.
            "computadora", "ordenador", "papa", "patata", "jugo", "zumo",
            # Morphology and encoding.
            "escribiéramos", "escribiésemos", "escríbeme", "dámelo",
            "jardín", "niño", "corazón",
        ),
        "reject": ("jardinn", "escribiz", "xyzzyque", "haiga"),
        "wrong": "this is not the generic RLA build (a per-country one accepts "
                 "only some of these)",
    },
}


def main(lang, aff_path, dic_path, out_dir):
    profile = PROFILES.get(lang)
    if profile is None:
        raise SystemExit(f"no profile for {lang!r}; known: {', '.join(PROFILES)}")

    forms, aff, needaffix_stems = expand(aff_path, dic_path)

    missing = [w for w in profile["accept"] if w not in forms]
    present = [w for w in profile["reject"] if w in forms]
    if missing or present:
        raise SystemExit(
            f"{profile['wrong']}: missing {missing}, unexpectedly present {present}"
        )

    os.makedirs(out_dir, exist_ok=True)
    ordered = sorted(forms)
    body = f"{len(ordered)}\n" + "\n".join(ordered) + "\n"

    dic_out = os.path.join(out_dir, f"{lang}.dic.gz")
    # mtime=0 so rebuilding identical input produces an identical file — a
    # vendored asset that changes on every build is noise in the diff.
    with gzip.GzipFile(dic_out, "wb", compresslevel=9, mtime=0) as fh:
        fh.write(body.encode("utf-8"))

    aff_out = os.path.join(out_dir, f"{lang}.aff")
    with open(aff_out, "w", encoding="utf-8") as fh:
        fh.write(shipped_aff(aff_path))

    print(f"{len(ordered)} forms -> {dic_out} "
          f"({os.path.getsize(dic_out) / 1e6:.2f} MB gzipped); "
          f"{needaffix_stems} NEEDAFFIX stems, "
          f"{aff.dropped_apostrophe_rules} elision rules skipped", file=sys.stderr)


if __name__ == "__main__":
    if len(sys.argv) != 5:
        raise SystemExit(
            "usage: build_hunspell_dictionary.py <lang> <aff> <dic> <out-dir>\n"
            f"       lang is one of: {', '.join(PROFILES)}"
        )
    main(*sys.argv[1:5])