#!/usr/bin/env python3 """Build one of Petal's browser spelling dictionaries from a Hunspell one. Why this script exists at all ---------------------------- English is vendored the obvious way: `dictionary-en`'s `en.aff` + `en.dic` go into web/public/dictionaries/en and nspell reads them in the browser. The plan for Phase 21 said "Hunspell pt-PT vendored like en-US", and that turns out not to work, for a measured reason. nspell expands affixes **eagerly at load time** — it materialises every surface form into a hash the moment you construct it. English gets away with this: ~50k stems and a small rule set. European Portuguese does not. `pt_PT.aff` carries 1,340 affix rules (the full verb paradigm: six persons x a dozen tenses, plus diminutives, plus productive prefixes) over 44,257 stems. Measured on this machine, nspell needed ~340 MB of heap for the first 12,000 entries alone and had not returned after three minutes on the whole file; extrapolated, it wants well over a gigabyte. That is not something to hand a browser, still less a tablet. French is larger again: 5,600 affix rules over 84,140 stems. So the expansion happens **here**, once, at build time, and the browser gets a flat word list it can load with no affix machinery at all. The runtime code path is then *identical* to English — same nspell, same interface — which is the real prize. The aff shipped alongside keeps only the suggestion-shaping directives (TRY/KEY/REP/MAP), so corrections still know that "cao" wants "ção" and that a missing acute accent is a near miss. What Phase 24 had to add ------------------------ The build plan recorded that the pt-PT version of this script "generalizes" to French. It did not. It handled single-character flags and plain PFX/SFX and stopped on anything else — the right call, because `fr.aff` uses four of the things it stopped on, and getting any of them wrong changes which words are accepted: * **`FLAG long`** — French flags are *two characters* (`S.`, `L'`, `Um`). The pt-PT reader took `set(flagstr)`, one flag per character, which on a French entry yields a bag of unrelated single letters: every entry would have been expanded through the wrong paradigm. This is the one that fails silently. * **Continuation flags** — pt-PT's affixes append plain text, so that script dropped anything after a `/` and asserted the drop was safe. French really does affix an affixed form: `PFX Um 0 0/S.` says the prefixed form then takes the plural suffix, and the elision prefixes arrive the same way from the other side (`SFX ... ait/n'q'l'm't's'`). * **`NEEDAFFIX`** — French marks thousands of stems "not a word on its own" (`Allemagne/S.()`), the bare form arriving instead through a zero-append rule. Ignoring the flag accepts stems the real dictionary rejects. * **`FULLSTRIP`** — a rule may strip the whole stem. `CIRCUMFIX` and `FORBIDDENWORD` are *declared* in `fr.aff` and used by nothing, which this script asserts rather than assumes: an upstream release that started using either would otherwise change what is accepted without changing this file. `KEEPCASE` and `NOSUGGEST` are honoured by being ignored on purpose — they shape casing and suggestions, not membership, and a NOSUGGEST word is still a word. Elision is handled at lookup, not here — and that is the size decision ---------------------------------------------------------------------- Most of French's affix machinery by volume is elision: `l'`, `d'`, `qu'`, `j'`, `n'`, `s'`, `jusqu'`, `puisqu'`. Hunspell treats `l'arbre` as one word, so a faithful expansion carries much of the language thirty-four times over — and Petal's tokenizer keeps internal apostrophes, so `l'arbre` really does arrive at the dictionary as one token and really would be underlined if it were absent. Both halves were built and measured. Keeping the elided forms: **3,159,832 forms, 8.25 MB gzipped**, ~45 MB of text for nspell to hash on a tablet. Dropping them: **473,326 forms, 1.19 MB gzipped**. The elided seven-eighths are not new words — they are thirteen little words glued to words already in the list — so the third option is the one taken: rules whose append carries an apostrophe are skipped here (the count is printed), and `withElision` in `useSpellChecker.ts` splits a token at a *known clitic* and checks the remainder. `l'arbre` costs one extra lookup instead of seven megabytes, and `zzz'arbre` is still flagged because `zzz` is not one of the thirteen. Stems that carry an apostrophe of their own — `aujourd'hui`, `quelqu'un`, `presqu'île`, `prud'homme` — are dictionary entries rather than affixed forms, so they are kept verbatim and matched directly. `entr'aide` and `grand'mère` are absent for the same reason they are absent from Dicollecte: modern French spells them `entraide` and `grand-mère`. Choosing the source ------------------- Both languages have a trap here, and they are different traps. **pt-PT: the wrong country.** npm's `dictionary-pt` is not European Portuguese. Both it and `dictionary-pt-br` package VERO ("Verificador Ortográfico Livre", Brasil), so vendoring the obvious npm name would have shipped Brazilian spellings under a pt-PT label — the pt-BR drift SUGGESTIONS.md §3 warns about, arriving through the packaging rather than through the model. The authentic dictionary is the Projecto Natura one (Universidade do Minho) that LibreOffice ships and Debian packages as `hunspell-pt-pt`; its aff declares `LANG pt_PT`. **fr: the wrong side of an argument the French have not settled.** The regional question turns out to be a non-question — Debian's `fr_FR`, `fr_CA`, `fr_BE`, `fr_CH`, `fr_LU` and `fr_MC` are all symlinks to one `fr.dic`, so unlike pt there is no country here to get wrong. What there is instead is the 1990 spelling reform, packaged three ways: `hunspell-fr-classical` (traditional), `-revised` (reform only) and `-comprehensive` (both). Petal ships **comprehensive**, because Petal never corrects her French — the only thing this dictionary can do is underline something. *coût* and *cout* are both correct French, taught in different decades to different people, and a writing companion has no business underlining one of them to take a side. The `fr` MUST_ACCEPT list is written to *prove* which package was used: classical rejects `cout`, revised rejects `coût`, and only comprehensive accepts both. Licensing: pt-PT is GPL-2 or LGPL-2.1 or MPL-1.1, (c) José João de Almeida, Rui Vilela, Alberto Simões. fr is MPL-2.0, (c) 2007-2018 the Dicollecte contributors (grammalecte.net). The upstream copyright file is vendored beside each output. Usage ----- apt-get download hunspell-fr-comprehensive # or hunspell-pt-pt dpkg-deb -x hunspell-fr-comprehensive_*.deb src python3 scripts/build_hunspell_dictionary.py fr \\ src/usr/share/hunspell/fr.aff \\ src/usr/share/hunspell/fr.dic \\ web/public/dictionaries/fr """ import gzip import os import re import sys import unicodedata # Directives worth keeping in the shipped aff. These shape *suggestions*, not # membership: TRY orders the alphabet the corrector tries, KEY knows which keys # are adjacent, REP holds the language's own confusions (cao/ção, ss/ç), and MAP # says an accented vowel and its bare form are the same letter for scoring — # which is most of what an ESL writer gets wrong in either language. KEEP_DIRECTIVES = ("SET", "TRY", "KEY", "REP", "MAP", "WORDCHARS") # Directives that would change which words are *accepted* and that this expander # does not implement. If a future upstream release starts using one, the output # would silently disagree with the real dictionary, so the build stops instead. UNSUPPORTED = ( "COMPOUNDFLAG", "COMPOUNDMIN", "COMPOUNDRULE", "COMPOUNDBEGIN", "ONLYINCOMPOUND", "PSEUDOROOT", "AF", "AM", ) APOSTROPHES = "'’" class Aff: """The parts of an .aff file that decide which words exist.""" def __init__(self): self.pfx = {} # flag -> [(strip, append, condition, continuation)] self.sfx = {} self.cross = {} # flag -> bool self.flag_kind = "char" self.needaffix = None self.circumfix = None self.forbidden = None self.dropped_apostrophe_rules = 0 def parse_flags(raw, kind): """Split a flag string into flags, per the aff's FLAG declaration.""" raw = raw.strip() if not raw: return set() if kind == "long": return {raw[i:i + 2] for i in range(0, len(raw) - 1, 2)} if kind == "num": return {f for f in raw.split(",") if f} return set(raw) def parse_aff(path): with open(path, encoding="utf-8") as fh: lines = fh.read().splitlines() aff = Aff() # FLAG has to be known before anything containing a flag is read, and it can # sit anywhere in the file. So: header pass first, rules second. for line in lines: parts = line.split() if not parts: continue head = parts[0] if head in UNSUPPORTED: raise SystemExit( f"{path}: unsupported directive {head!r} — this expander handles " "PFX/SFX affixation with continuation flags, and honouring " f"{head} would change which words are accepted. Extend the " "script before shipping." ) if len(parts) < 2: continue if head == "FLAG": aff.flag_kind = parts[1] if aff.flag_kind not in ("long", "num", "UTF-8"): raise SystemExit(f"{path}: unknown FLAG type {aff.flag_kind!r}") elif head == "NEEDAFFIX": aff.needaffix = parts[1] elif head == "CIRCUMFIX": aff.circumfix = parts[1] elif head == "FORBIDDENWORD": aff.forbidden = parts[1] i = 0 while i < len(lines): parts = lines[i].split() if parts and parts[0] in ("PFX", "SFX"): kind, flag, cross_flag, count = parts[0], parts[1], parts[2], int(parts[3]) table = aff.pfx if kind == "PFX" else aff.sfx aff.cross[flag] = cross_flag == "Y" rules = table.setdefault(flag, []) for j in range(1, count + 1): p = lines[i + j].split() strip = "" if p[2] == "0" else p[2] append, _, cont_raw = p[3].partition("/") if append == "0": append = "" # Elision. See the header: these forms are `l'` and its twelve # siblings glued to words already in the list, they multiply the # download by seven, and `withElision` reconstructs them at # lookup for the cost of one extra hash probe. if any(a in append for a in APOSTROPHES): aff.dropped_apostrophe_rules += 1 continue cont = parse_flags(cont_raw, aff.flag_kind) if aff.circumfix and aff.circumfix in cont: raise SystemExit( f"{path}: CIRCUMFIX is used by a {kind} {flag} rule. It " "was declared-but-unused when this expander was written " "and is not implemented; honouring it would change which " "words are accepted." ) cond = p[4] if len(p) > 4 else "." anchored = ("^" + cond) if kind == "PFX" else (cond + "$") rules.append((strip, append, re.compile(anchored), cont)) i += count + 1 continue i += 1 return aff def apply_suffix(word, rules): """Every (form, continuation flags) a suffix table yields for `word`.""" out = [] for strip, append, cond, cont in rules: if strip and not word.endswith(strip): continue if not cond.search(word): continue stem = word[: len(word) - len(strip)] if strip else word out.append((stem + append, cont)) return out def apply_prefix(word, rules): out = [] for strip, append, cond, cont in rules: if strip and not word.startswith(strip): continue if not cond.search(word): continue stem = word[len(strip):] if strip else word out.append((append + stem, cont)) return out def expand_entry(word, flags, aff, out): """Add every surface form of one dictionary entry to `out`. Hunspell's model without compounding: a form is the stem plus at most one prefix and at most one suffix. A flag reaches an affix either from the stem's own flags or from the continuation flags of the affix applied on the other side; when both sides apply, both rules must be declared cross-product. NEEDAFFIX is why the bare form is not simply added: the flag says *this* form is not a word, only whatever can be built from it — and it arrives both on stems and on continuations. """ def is_word(carried): return not (aff.needaffix and aff.needaffix in carried) if is_word(flags): out.add(word) suffixed = [] # (form, flag, continuation flags) for f in flags: if f in aff.sfx: for form, cont in apply_suffix(word, aff.sfx[f]): suffixed.append((form, f, cont)) if is_word(cont): out.add(form) prefixed = [] for f in flags: if f in aff.pfx: for form, cont in apply_prefix(word, aff.pfx[f]): prefixed.append((form, f, cont)) if is_word(cont): out.add(form) # Prefix then suffix. The suffix flag may come from the stem or from the # prefix's own continuation (`PFX Um 0 0/S.`), and the suffix condition is # matched against the whole prefixed word, which is what hunspell does. for form, pf, pcont in prefixed: if not aff.cross.get(pf): continue for f in flags | pcont: if f in aff.sfx and aff.cross.get(f): for full, _ in apply_suffix(form, aff.sfx[f]): out.add(full) # Suffix then prefix — the same pair reached from the other side, which is # how the elision prefixes arrive in French. Only the flags the suffix hands # forward are new here; the stem's own were covered above. for form, sf, scont in suffixed: if not aff.cross.get(sf): continue for f in scont: if f in aff.pfx and aff.cross.get(f): for full, _ in apply_prefix(form, aff.pfx[f]): out.add(full) def expand(aff_path, dic_path): aff = parse_aff(aff_path) forms = set() needaffix_stems = 0 with open(dic_path, encoding="utf-8") as fh: fh.readline() # leading entry count, not a word for raw in fh: # Morphological fields (po:nom is:fem) follow the entry, separated by # a tab in pt-PT and by a space in fr. entry = raw.strip().split("\t")[0].split(" ")[0] if not entry: continue word, _, flagstr = entry.partition("/") word = word.strip() if not word: continue flags = parse_flags(flagstr, aff.flag_kind) if aff.forbidden and aff.forbidden in flags: raise SystemExit( f"{dic_path}: FORBIDDENWORD is in use ({word!r}). It was " "declared-but-unused when this expander was written; the " "forms it removes would be wrongly accepted." ) if aff.needaffix and aff.needaffix in flags: needaffix_stems += 1 expand_entry(word, flags, aff, forms) # NFC, because the aff's own ICONV table normalises decomposed accents on the # way in and the browser hands nspell whatever the keyboard produced. forms = {unicodedata.normalize("NFC", f) for f in forms} return forms, aff, needaffix_stems def shipped_aff(aff_path): keep = [] for line in open(aff_path, encoding="utf-8").read().splitlines(): head = line.split()[0] if line.split() else "" if head in KEEP_DIRECTIVES: keep.append(line) return "\n".join(keep) + "\n" # Words the built list must accept, and must reject, before it is written. Each # set is chosen to fail loudly on the *specific* wrong source that language has a # packaged, plausible way of reaching — not to spot-check spelling in general. PROFILES = { # The pt-PT/pt-BR fault lines: post-Acordo spellings, the European lexicon, # and the first-person-plural preterite accent that only pt-PT writes. "pt-PT": { "accept": ("receção", "húmido", "telemóvel", "autocarro", "comboio", "ótimo", "pensámos", "escrevêssemos", "jardim"), "reject": ("recepção", "úmido", "ônibus", "óptimo"), "wrong": "this does not look like European Portuguese", }, # Which of the three 1990-reform packagings this is. `coût`/`cout` and # `paraître`/`paraitre` are each accepted by exactly one of classical and # revised, so a build accepting all four is comprehensive and one that drops # any of them is not. `Allemagne` is a NEEDAFFIX stem reachable only through # a zero-append rule and `km` only through a prefix continuation, so between # them they also check that this expander honoured the two features the # pt-PT one refused. "fr": { "accept": ("coût", "cout", "paraître", "paraitre", "nénuphar", "nénufar", "oignon", "ognon", "événement", "évènement", "jardin", "Allemagne", "écrivissions", "km"), "reject": ("jardinn", "écrivaitz", "xyzzyque"), "wrong": "this does not look like the comprehensive French dictionary", }, } def main(lang, aff_path, dic_path, out_dir): profile = PROFILES.get(lang) if profile is None: raise SystemExit(f"no profile for {lang!r}; known: {', '.join(PROFILES)}") forms, aff, needaffix_stems = expand(aff_path, dic_path) missing = [w for w in profile["accept"] if w not in forms] present = [w for w in profile["reject"] if w in forms] if missing or present: raise SystemExit( f"{profile['wrong']}: missing {missing}, unexpectedly present {present}" ) os.makedirs(out_dir, exist_ok=True) ordered = sorted(forms) body = f"{len(ordered)}\n" + "\n".join(ordered) + "\n" dic_out = os.path.join(out_dir, f"{lang}.dic.gz") # mtime=0 so rebuilding identical input produces an identical file — a # vendored asset that changes on every build is noise in the diff. with gzip.GzipFile(dic_out, "wb", compresslevel=9, mtime=0) as fh: fh.write(body.encode("utf-8")) aff_out = os.path.join(out_dir, f"{lang}.aff") with open(aff_out, "w", encoding="utf-8") as fh: fh.write(shipped_aff(aff_path)) print(f"{len(ordered)} forms -> {dic_out} " f"({os.path.getsize(dic_out) / 1e6:.2f} MB gzipped); " f"{needaffix_stems} NEEDAFFIX stems, " f"{aff.dropped_apostrophe_rules} elision rules skipped", file=sys.stderr) if __name__ == "__main__": if len(sys.argv) != 5: raise SystemExit( "usage: build_hunspell_dictionary.py \n" f" lang is one of: {', '.join(PROFILES)}" ) main(*sys.argv[1:5])