#!/usr/bin/env python3 """Build one of Petal's browser spelling dictionaries from a Hunspell one. Why this script exists at all ---------------------------- English is vendored the obvious way: `dictionary-en`'s `en.aff` + `en.dic` go into web/public/dictionaries/en and nspell reads them in the browser. The plan for Phase 21 said "Hunspell pt-PT vendored like en-US", and that turns out not to work, for a measured reason. nspell expands affixes **eagerly at load time** — it materialises every surface form into a hash the moment you construct it. English gets away with this: ~50k stems and a small rule set. European Portuguese does not. `pt_PT.aff` carries 1,340 affix rules (the full verb paradigm: six persons x a dozen tenses, plus diminutives, plus productive prefixes) over 44,257 stems. Measured on this machine, nspell needed ~340 MB of heap for the first 12,000 entries alone and had not returned after three minutes on the whole file; extrapolated, it wants well over a gigabyte. That is not something to hand a browser, still less a tablet. French is larger again: 5,600 affix rules over 84,140 stems. So the expansion happens **here**, once, at build time, and the browser gets a flat word list it can load with no affix machinery at all. The runtime code path is then *identical* to English — same nspell, same interface — which is the real prize. The aff shipped alongside keeps only the suggestion-shaping directives (TRY/KEY/REP/MAP), so corrections still know that "cao" wants "ção" and that a missing acute accent is a near miss. What Phase 24 had to add ------------------------ The build plan recorded that the pt-PT version of this script "generalizes" to French. It did not. It handled single-character flags and plain PFX/SFX and stopped on anything else — the right call, because `fr.aff` uses four of the things it stopped on, and getting any of them wrong changes which words are accepted: * **`FLAG long`** — French flags are *two characters* (`S.`, `L'`, `Um`). The pt-PT reader took `set(flagstr)`, one flag per character, which on a French entry yields a bag of unrelated single letters: every entry would have been expanded through the wrong paradigm. This is the one that fails silently. * **Continuation flags** — pt-PT's affixes append plain text, so that script dropped anything after a `/` and asserted the drop was safe. French really does affix an affixed form: `PFX Um 0 0/S.` says the prefixed form then takes the plural suffix, and the elision prefixes arrive the same way from the other side (`SFX ... ait/n'q'l'm't's'`). * **`NEEDAFFIX`** — French marks thousands of stems "not a word on its own" (`Allemagne/S.()`), the bare form arriving instead through a zero-append rule. Ignoring the flag accepts stems the real dictionary rejects. * **`FULLSTRIP`** — a rule may strip the whole stem. `CIRCUMFIX` and `FORBIDDENWORD` are *declared* in `fr.aff` and used by nothing, which this script asserts rather than assumes: an upstream release that started using either would otherwise change what is accepted without changing this file. `KEEPCASE` and `NOSUGGEST` are honoured by being ignored on purpose — they shape casing and suggestions, not membership, and a NOSUGGEST word is still a word. Elision is handled at lookup, not here — and that is the size decision ---------------------------------------------------------------------- Most of French's affix machinery by volume is elision: `l'`, `d'`, `qu'`, `j'`, `n'`, `s'`, `jusqu'`, `puisqu'`. Hunspell treats `l'arbre` as one word, so a faithful expansion carries much of the language thirty-four times over — and Petal's tokenizer keeps internal apostrophes, so `l'arbre` really does arrive at the dictionary as one token and really would be underlined if it were absent. Both halves were built and measured. Keeping the elided forms: **3,159,832 forms, 8.25 MB gzipped**, ~45 MB of text for nspell to hash on a tablet. Dropping them: **473,326 forms, 1.19 MB gzipped**. The elided seven-eighths are not new words — they are thirteen little words glued to words already in the list — so the third option is the one taken: rules whose append carries an apostrophe are skipped here (the count is printed), and `withElision` in `useSpellChecker.ts` splits a token at a *known clitic* and checks the remainder. `l'arbre` costs one extra lookup instead of seven megabytes, and `zzz'arbre` is still flagged because `zzz` is not one of the thirteen. Stems that carry an apostrophe of their own — `aujourd'hui`, `quelqu'un`, `presqu'île`, `prud'homme` — are dictionary entries rather than affixed forms, so they are kept verbatim and matched directly. `entr'aide` and `grand'mère` are absent for the same reason they are absent from Dicollecte: modern French spells them `entraide` and `grand-mère`. Choosing the source ------------------- Both languages have a trap here, and they are different traps. **pt-PT: the wrong country.** npm's `dictionary-pt` is not European Portuguese. Both it and `dictionary-pt-br` package VERO ("Verificador Ortográfico Livre", Brasil), so vendoring the obvious npm name would have shipped Brazilian spellings under a pt-PT label — the pt-BR drift SUGGESTIONS.md §3 warns about, arriving through the packaging rather than through the model. The authentic dictionary is the Projecto Natura one (Universidade do Minho) that LibreOffice ships and Debian packages as `hunspell-pt-pt`; its aff declares `LANG pt_PT`. **es: the wrong country again, hidden one layer further down.** Spanish looked like it would repeat the pt trap — `hunspell-es` installs twenty country codes, `es_AR` through `es_VE` — and then looked like it did not, because every one of them is a symlink to a single `es_ES.aff`/`es_ES.dic`. Both readings were wrong. Debian collapses the twenty because it ships **one** of upstream's builds, and the one it ships is the **peninsular** `es_ES`. RLA (Santiago Bosio's project, `sbosio/rla-es`) publishes twenty-four dictionaries per release: one per country, plus a **generic `es`** that is the union of all of them. Debian packages neither the generic one nor a choice — it packages Spain, under a name that reads like "Spanish". Measured against the v2.9 release: Debian's file is 659,085 expanded forms and upstream `es_ES` is 659,018; the generic `es` is **717,640**. The 58,622-form difference is almost entirely **voseo** — `vení`, `tenés`, `querés`, `sabés`, `andá` — the present tense of most of Latin America, which Debian's package rejects as misspellings. Petal ships the **generic** build. **Vocabulary cannot detect this and morphology can.** The first version of the es profile asserted the pan-Hispanic lexicon — *computadora* and *ordenador*, *papa* and *patata* — and passed happily on the peninsular file, because **every** RLA variant carries the full pan-Hispanic vocabulary; only the verb paradigms are localised. The `REP` table is no help either: its `ll`↔`y` and `ás`↔`az` entries look like evidence of yeísmo and seseo, but they are shared by all twenty-four builds. What separates them is exactly two things, and the profile now demands both at once: **voseo** (absent from `es_ES`) and **vosotros** (largely absent from `es_MX`). Only the generic build has both, so only the generic build passes. This is the same decision fr made between `-classical` and `-revised`, arriving by a different road. The only thing this dictionary can do is underline something, and *tienes* and *tenés* are both correct Spanish taught in different countries — so Petal takes the build that accepts every variety rather than one that makes a writer wrong for where she is from. Nothing is generated to get there: the forms come from a real upstream package, which is what lets the MUST_ACCEPT list prove which package it was. Licensing note: RLA is tri-licensed GPL-3+ / LGPL-3+ / MPL-1.1+; Petal redistributes under the MPL. The upstream README and LICENSE are vendored beside the output. **fr: the wrong side of an argument the French have not settled.** The regional question turns out to be a non-question — Debian's `fr_FR`, `fr_CA`, `fr_BE`, `fr_CH`, `fr_LU` and `fr_MC` are all symlinks to one `fr.dic`, so unlike pt there is no country here to get wrong. What there is instead is the 1990 spelling reform, packaged three ways: `hunspell-fr-classical` (traditional), `-revised` (reform only) and `-comprehensive` (both). Petal ships **comprehensive**, because Petal never corrects her French — the only thing this dictionary can do is underline something. *coût* and *cout* are both correct French, taught in different decades to different people, and a writing companion has no business underlining one of them to take a side. The `fr` MUST_ACCEPT list is written to *prove* which package was used: classical rejects `cout`, revised rejects `coût`, and only comprehensive accepts both. Licensing: pt-PT is GPL-2 or LGPL-2.1 or MPL-1.1, (c) José João de Almeida, Rui Vilela, Alberto Simões. fr is MPL-2.0, (c) 2007-2018 the Dicollecte contributors (grammalecte.net). The upstream copyright file is vendored beside each output. Usage ----- apt-get download hunspell-fr-comprehensive # or hunspell-pt-pt dpkg-deb -x hunspell-fr-comprehensive_*.deb src python3 scripts/build_hunspell_dictionary.py fr \\ src/usr/share/hunspell/fr.aff \\ src/usr/share/hunspell/fr.dic \\ web/public/dictionaries/fr Spanish does not come from Debian — see below; `hunspell-es` is the peninsular build. Take the generic dictionary from an upstream release instead: curl -LO https://github.com/sbosio/rla-es/releases/download/v2.9/es.oxt unzip -d src es.oxt # an .oxt is a zip python3 scripts/build_hunspell_dictionary.py es \\ src/es.aff src/es.dic web/public/dictionaries/es """ import gzip import os import re import sys import unicodedata # Directives worth keeping in the shipped aff. These shape *suggestions*, not # membership: TRY orders the alphabet the corrector tries, KEY knows which keys # are adjacent, REP holds the language's own confusions (cao/ção, ss/ç), and MAP # says an accented vowel and its bare form are the same letter for scoring — # which is most of what an ESL writer gets wrong in either language. KEEP_DIRECTIVES = ("SET", "TRY", "KEY", "REP", "MAP", "WORDCHARS") # Directives that would change which words are *accepted* and that this expander # does not implement. If a future upstream release starts using one, the output # would silently disagree with the real dictionary, so the build stops instead. UNSUPPORTED = ( "COMPOUNDFLAG", "COMPOUNDMIN", "COMPOUNDRULE", "COMPOUNDBEGIN", "ONLYINCOMPOUND", "PSEUDOROOT", "AF", "AM", ) APOSTROPHES = "'’" class Aff: """The parts of an .aff file that decide which words exist.""" def __init__(self): self.pfx = {} # flag -> [(strip, append, condition, continuation)] self.sfx = {} # Cross-product, per flag — and kept per table, because PFX and SFX are # separate flag namespaces in hunspell: the same flag may name a prefix # table and a suffix table, with different cross-product settings. One # shared dict let the second block silently overwrite the first. self.cross_pfx = {} # flag -> bool self.cross_sfx = {} self.flag_kind = "char" self.needaffix = None self.circumfix = None self.forbidden = None self.dropped_apostrophe_rules = 0 def parse_flags(raw, kind): """Split a flag string into flags, per the aff's FLAG declaration.""" raw = raw.strip() if not raw: return set() if kind == "long": # Two characters per flag, exactly. An odd length is a malformed flag # string, and silently dropping the trailing character would quietly # expand an entry through the wrong paradigm — the failure mode this # whole FLAG-aware rewrite exists to avoid. if len(raw) % 2: raise SystemExit(f"odd-length long flag string {raw!r}") return {raw[i:i + 2] for i in range(0, len(raw), 2)} if kind == "num": return {f for f in raw.split(",") if f} return set(raw) def parse_aff(path): with open(path, encoding="utf-8") as fh: lines = fh.read().splitlines() aff = Aff() # FLAG has to be known before anything containing a flag is read, and it can # sit anywhere in the file. So: header pass first, rules second. for line in lines: parts = line.split() if not parts: continue head = parts[0] if head in UNSUPPORTED: raise SystemExit( f"{path}: unsupported directive {head!r} — this expander handles " "PFX/SFX affixation with continuation flags, and honouring " f"{head} would change which words are accepted. Extend the " "script before shipping." ) if len(parts) < 2: continue if head == "FLAG": aff.flag_kind = parts[1] if aff.flag_kind not in ("long", "num", "UTF-8"): raise SystemExit(f"{path}: unknown FLAG type {aff.flag_kind!r}") elif head == "NEEDAFFIX": aff.needaffix = parts[1] elif head == "CIRCUMFIX": aff.circumfix = parts[1] elif head == "FORBIDDENWORD": aff.forbidden = parts[1] i = 0 while i < len(lines): parts = lines[i].split() if parts and parts[0] in ("PFX", "SFX"): kind, flag, cross_flag, count = parts[0], parts[1], parts[2], int(parts[3]) table = aff.pfx if kind == "PFX" else aff.sfx cross = aff.cross_pfx if kind == "PFX" else aff.cross_sfx cross[flag] = cross_flag == "Y" rules = table.setdefault(flag, []) for j in range(1, count + 1): p = lines[i + j].split() strip = "" if p[2] == "0" else p[2] append, _, cont_raw = p[3].partition("/") if append == "0": append = "" # Elision. See the header: these forms are `l'` and its twelve # siblings glued to words already in the list, they multiply the # download by seven, and `withElision` reconstructs them at # lookup for the cost of one extra hash probe. if any(a in append for a in APOSTROPHES): aff.dropped_apostrophe_rules += 1 continue cont = parse_flags(cont_raw, aff.flag_kind) if aff.circumfix and aff.circumfix in cont: raise SystemExit( f"{path}: CIRCUMFIX is used by a {kind} {flag} rule. It " "was declared-but-unused when this expander was written " "and is not implemented; honouring it would change which " "words are accepted." ) cond = p[4] if len(p) > 4 else "." anchored = ("^" + cond) if kind == "PFX" else (cond + "$") rules.append((strip, append, re.compile(anchored), cont)) i += count + 1 continue i += 1 return aff def apply_suffix(word, rules): """Every (form, continuation flags) a suffix table yields for `word`.""" out = [] for strip, append, cond, cont in rules: if strip and not word.endswith(strip): continue if not cond.search(word): continue stem = word[: len(word) - len(strip)] if strip else word out.append((stem + append, cont)) return out def apply_prefix(word, rules): out = [] for strip, append, cond, cont in rules: if strip and not word.startswith(strip): continue if not cond.search(word): continue stem = word[len(strip):] if strip else word out.append((append + stem, cont)) return out def expand_entry(word, flags, aff, out): """Add every surface form of one dictionary entry to `out`. Hunspell's model without compounding: a form is the stem plus at most one prefix and at most one suffix. A flag reaches an affix either from the stem's own flags or from the continuation flags of the affix applied on the other side; when both sides apply, both rules must be declared cross-product. NEEDAFFIX is why the bare form is not simply added: the flag says *this* form is not a word, only whatever can be built from it — and it arrives both on stems and on continuations. """ def is_word(carried): return not (aff.needaffix and aff.needaffix in carried) if is_word(flags): out.add(word) suffixed = [] # (form, flag, continuation flags) for f in flags: if f in aff.sfx: for form, cont in apply_suffix(word, aff.sfx[f]): suffixed.append((form, f, cont)) if is_word(cont): out.add(form) prefixed = [] for f in flags: if f in aff.pfx: for form, cont in apply_prefix(word, aff.pfx[f]): prefixed.append((form, f, cont)) if is_word(cont): out.add(form) # Prefix then suffix. The suffix flag may come from the stem or from the # prefix's own continuation (`PFX Um 0 0/S.`), and the suffix condition is # matched against the whole prefixed word, which is what hunspell does. # NEEDAFFIX is checked here too, exactly as on the single-affix paths above: # a doubly-affixed form whose last continuation still carries the flag is # "not a word on its own", and without compounding there is no third affix # left to make it one. for form, pf, pcont in prefixed: if not aff.cross_pfx.get(pf): continue for f in flags | pcont: if f in aff.sfx and aff.cross_sfx.get(f): for full, fcont in apply_suffix(form, aff.sfx[f]): if is_word(fcont): out.add(full) # Suffix then prefix — the same pair reached from the other side, which is # how the elision prefixes arrive in French. Only the flags the suffix hands # forward are new here; the stem's own were covered above. for form, sf, scont in suffixed: if not aff.cross_sfx.get(sf): continue for f in scont: if f in aff.pfx and aff.cross_pfx.get(f): for full, fcont in apply_prefix(form, aff.pfx[f]): if is_word(fcont): out.add(full) def expand(aff_path, dic_path): aff = parse_aff(aff_path) forms = set() needaffix_stems = 0 with open(dic_path, encoding="utf-8") as fh: fh.readline() # leading entry count, not a word for raw in fh: # Morphological fields (po:nom is:fem) follow the entry, separated by # a tab in pt-PT and by a space in fr. entry = raw.strip().split("\t")[0].split(" ")[0] if not entry: continue word, _, flagstr = entry.partition("/") word = word.strip() if not word: continue flags = parse_flags(flagstr, aff.flag_kind) if aff.forbidden and aff.forbidden in flags: raise SystemExit( f"{dic_path}: FORBIDDENWORD is in use ({word!r}). It was " "declared-but-unused when this expander was written; the " "forms it removes would be wrongly accepted." ) if aff.needaffix and aff.needaffix in flags: needaffix_stems += 1 expand_entry(word, flags, aff, forms) # NFC, because the aff's own ICONV table normalises decomposed accents on the # way in and the browser hands nspell whatever the keyboard produced. forms = {unicodedata.normalize("NFC", f) for f in forms} return forms, aff, needaffix_stems def shipped_aff(aff_path): keep = [] for line in open(aff_path, encoding="utf-8").read().splitlines(): head = line.split()[0] if line.split() else "" if head in KEEP_DIRECTIVES: keep.append(line) return "\n".join(keep) + "\n" # Words the built list must accept, and must reject, before it is written. Each # set is chosen to fail loudly on the *specific* wrong source that language has a # packaged, plausible way of reaching — not to spot-check spelling in general. PROFILES = { # The pt-PT/pt-BR fault lines: post-Acordo spellings, the European lexicon, # and the first-person-plural preterite accent that only pt-PT writes. "pt-PT": { "accept": ("receção", "húmido", "telemóvel", "autocarro", "comboio", "ótimo", "pensámos", "escrevêssemos", "jardim"), "reject": ("recepção", "úmido", "ônibus", "óptimo"), "wrong": "this does not look like European Portuguese", }, # Which of the three 1990-reform packagings this is. `coût`/`cout` and # `paraître`/`paraitre` are each accepted by exactly one of classical and # revised, so a build accepting all four is comprehensive and one that drops # any of them is not. `Allemagne` is a NEEDAFFIX stem reachable only through # a zero-append rule and `km` only through a prefix continuation, so between # them they also check that this expander honoured the two features the # pt-PT one refused. "fr": { "accept": ("coût", "cout", "paraître", "paraitre", "nénuphar", "nénufar", "oignon", "ognon", "événement", "évènement", "jardin", "Allemagne", "écrivissions", "km"), "reject": ("jardinn", "écrivaitz", "xyzzyque"), "wrong": "this does not look like the comprehensive French dictionary", }, # The generic RLA build, and the accept list is written to reject the four # neighbouring builds rather than to describe this one. # # The first version of this profile demanded *computadora* and *ordenador*, # *papa* and *patata*, and passed — on the peninsular file, because **every** # RLA variant carries the whole pan-Hispanic vocabulary. Vocabulary does not # discriminate here at all; only morphology does, and it discriminates # completely: # # * **voseo** (`vení`, `tenés`, `querés`) is in `es` and `es_AR` and not in # `es_ES` or Debian's package. Demanding it rejects the peninsular build. # * **vosotros** (`tenéis`, `escribid`) is in `es`, `es_AR` and `es_ES`, and # largely absent from `es_MX`. Demanding it rejects the Mexican build. # # Requiring both at once leaves exactly one package standing: the generic # `es`, which is the only one that accepts every variety of Spanish. That is # the same reason fr ships `-comprehensive` — the only thing this dictionary # can do is underline something, and *tienes* and *tenés* are both correct # Spanish taught in different countries. # # The rest are shape checks: `escribiésemos` is the -se imperfect subjunctive, # `dámelo` proves the enclitic pronoun rules ran, and `jardín`/`niño` prove # FLAG UTF-8 was read as characters rather than bytes. "es": { "accept": ( # Rejects es_ES and Debian's hunspell-es. "vení", "tenés", "querés", "sabés", "andá", # Rejects es_MX. "tenéis", "escribid", # Rejects es_AR, which has both voseo and vosotros and would # otherwise pass. Caribbean and Andean everyday words: the generic # build is the union of all twenty-four, so it is the only one that # holds another region's vocabulary as well as its own. "arepa", "chévere", "bacán", # Pan-Hispanic vocabulary. These pass on every RLA build, so they # prove nothing on their own — kept because a source that stopped # being RLA at all would fail them. "computadora", "ordenador", "papa", "patata", "jugo", "zumo", # Morphology and encoding. "escribiéramos", "escribiésemos", "escríbeme", "dámelo", "jardín", "niño", "corazón", ), "reject": ("jardinn", "escribiz", "xyzzyque", "haiga"), "wrong": "this is not the generic RLA build (a per-country one accepts " "only some of these)", }, } def main(lang, aff_path, dic_path, out_dir): profile = PROFILES.get(lang) if profile is None: raise SystemExit(f"no profile for {lang!r}; known: {', '.join(PROFILES)}") forms, aff, needaffix_stems = expand(aff_path, dic_path) missing = [w for w in profile["accept"] if w not in forms] present = [w for w in profile["reject"] if w in forms] if missing or present: raise SystemExit( f"{profile['wrong']}: missing {missing}, unexpectedly present {present}" ) os.makedirs(out_dir, exist_ok=True) ordered = sorted(forms) body = f"{len(ordered)}\n" + "\n".join(ordered) + "\n" dic_out = os.path.join(out_dir, f"{lang}.dic.gz") # mtime=0 so rebuilding identical input produces an identical file — a # vendored asset that changes on every build is noise in the diff. with gzip.GzipFile(dic_out, "wb", compresslevel=9, mtime=0) as fh: fh.write(body.encode("utf-8")) aff_out = os.path.join(out_dir, f"{lang}.aff") with open(aff_out, "w", encoding="utf-8") as fh: fh.write(shipped_aff(aff_path)) print(f"{len(ordered)} forms -> {dic_out} " f"({os.path.getsize(dic_out) / 1e6:.2f} MB gzipped); " f"{needaffix_stems} NEEDAFFIX stems, " f"{aff.dropped_apostrophe_rules} elision rules skipped", file=sys.stderr) if __name__ == "__main__": if len(sys.argv) != 5: raise SystemExit( "usage: build_hunspell_dictionary.py \n" f" lang is one of: {', '.join(PROFILES)}" ) main(*sys.argv[1:5])