petal/scripts/build_ptpt_dictionary.py

#!/usr/bin/env python3
"""Build Petal's browser pt-PT spelling dictionary from the LibreOffice Hunspell one.

Why this script exists at all
----------------------------
English is vendored the obvious way: `dictionary-en`'s `en.aff` + `en.dic` go
into web/public/dictionaries/en and nspell reads them in the browser. The plan
for Phase 21 said "Hunspell pt-PT vendored like en-US", and that turns out not to
work, for a measured reason.

nspell expands affixes **eagerly at load time** — it materialises every surface
form into a hash the moment you construct it. English gets away with this: ~50k
stems and a small rule set. European Portuguese does not. `pt_PT.aff` carries
1,340 affix rules (the full verb paradigm: six persons x a dozen tenses, plus
diminutives, plus productive prefixes) over 44,257 stems. Measured on this
machine, nspell needed ~340 MB of heap for the first 12,000 entries alone and had
not returned after three minutes on the whole file; extrapolated, it wants well
over a gigabyte. That is not something to hand a browser, still less a tablet.

So the expansion happens **here**, once, at build time, and the browser gets a
flat word list it can load with no affix machinery at all: 1,039,058 forms, 15 MB
of text, 2.7 MB gzipped, which nspell reads in ~0.8s using ~120 MB. The runtime
code path is then *identical* to English — same nspell, same interface — which is
the real prize. The aff we ship alongside keeps only the suggestion-shaping
directives (TRY/KEY/REP/MAP), so corrections still know that "cao" wants "ção"
and that a missing acute accent is a near miss.

Choosing the source
-------------------
npm's `dictionary-pt` is **not** European Portuguese. Both it and
`dictionary-pt-br` package VERO ("Verificador Ortográfico Livre", Brasil), so
vendoring the obvious npm name would have shipped Brazilian spellings under a
pt-PT label — exactly the pt-BR drift SUGGESTIONS.md §3 warns about, arriving
through the packaging rather than through the model.

The authentic dictionary is the Projecto Natura one (Universidade do Minho) that
LibreOffice ships and Debian packages as `hunspell-pt-pt`. Its aff declares
`LANG pt_PT`. Spot-checked against the built list, it accepts `receção`,
`húmido`, `telemóvel`, `autocarro`, `comboio`, `ótimo` and `pensámos`, and
rejects `recepção`, `úmido`, `ônibus` and `óptimo` — post-Acordo European
Portuguese, which is what a pt-PT writer should be held to.

Licensing: GPL-2 or LGPL-2.1 or MPL-1.1, (c) José João de Almeida, Rui Vilela,
Alberto Simões. The upstream copyright file is vendored beside the output.

Usage
-----
    apt-get download hunspell-pt-pt        # or take pt_PT.aff/.dic from LibreOffice
    dpkg-deb -x hunspell-pt-pt_*.deb ptpt
    python3 scripts/build_ptpt_dictionary.py \
        ptpt/usr/share/hunspell/pt_PT.aff \
        ptpt/usr/share/hunspell/pt_PT.dic \
        web/public/dictionaries/pt-PT

Only the directives pt_PT.aff actually uses are implemented: PFX/SFX with
single-character flags, strip/append/condition, and cross-product. It carries no
compounding, no flag aliases and no NEEDAFFIX, so there is nothing else to
honour — the script asserts that rather than assuming it.
"""
import gzip
import os
import re
import sys

# Directives worth keeping in the shipped aff. These shape *suggestions*, not
# membership: TRY orders the alphabet the corrector tries, KEY knows which keys
# are adjacent, REP holds Portuguese-specific confusions (cao/ção, ss/ç), and MAP
# says an accented vowel and its bare form are the same letter for scoring —
# which is most of what an ESL writer gets wrong in Portuguese.
KEEP_DIRECTIVES = ("SET", "TRY", "KEY", "REP", "MAP", "WORDCHARS")

# Directives that would change which words are *accepted*. If a future upstream
# release starts using one, this script's output would silently disagree with
# the real dictionary, so it stops instead.
UNSUPPORTED = (
    "COMPOUNDFLAG", "COMPOUNDMIN", "COMPOUNDRULE", "COMPOUNDBEGIN",
    "ONLYINCOMPOUND", "NEEDAFFIX", "PSEUDOROOT", "CIRCUMFIX",
    "FORBIDDENWORD", "AF", "AM", "FLAG",
)


def parse_aff(path):
    """Return (prefix rules, suffix rules, cross-product flags) keyed by flag."""
    with open(path, encoding="utf-8") as fh:
        lines = fh.read().splitlines()

    for line in lines:
        head = line.split()[0] if line.split() else ""
        if head in UNSUPPORTED:
            raise SystemExit(
                f"{path}: unsupported directive {head!r} — this expander only "
                "handles plain PFX/SFX affixation, and honouring it would change "
                "which words are accepted. Extend the script before shipping."
            )

    pfx, sfx, cross = {}, {}, {}
    i = 0
    while i < len(lines):
        parts = lines[i].split()
        if parts and parts[0] in ("PFX", "SFX"):
            kind, flag, cross_flag, count = parts[0], parts[1], parts[2], int(parts[3])
            table = pfx if kind == "PFX" else sfx
            cross[flag] = cross_flag == "Y"
            rules = table.setdefault(flag, [])
            for j in range(1, count + 1):
                p = lines[i + j].split()
                strip = "" if p[2] == "0" else p[2]
                # The appended text may carry its own continuation flags after a
                # slash (append/FLAGS). We drop them: honouring them would mean
                # affixing an affixed form, which pt_PT.aff does not do.
                append = "" if p[3] == "0" else p[3].split("/")[0]
                cond = p[4] if len(p) > 4 else "."
                anchored = ("^" + cond) if kind == "PFX" else (cond + "$")
                rules.append((strip, append, re.compile(anchored)))
            i += count + 1
            continue
        i += 1
    return pfx, sfx, cross


def apply_suffix(word, rules):
    out = []
    for strip, append, cond in rules:
        if strip and not word.endswith(strip):
            continue
        if not cond.search(word):
            continue
        stem = word[: len(word) - len(strip)] if strip else word
        out.append(stem + append)
    return out


def apply_prefix(word, rules):
    out = []
    for strip, append, cond in rules:
        if strip and not word.startswith(strip):
            continue
        if not cond.search(word):
            continue
        stem = word[len(strip):] if strip else word
        out.append(append + stem)
    return out


def expand(aff_path, dic_path):
    pfx, sfx, cross = parse_aff(aff_path)
    forms = set()
    with open(dic_path, encoding="utf-8") as fh:
        fh.readline()  # leading entry count, not a word
        for raw in fh:
            entry = raw.strip().split("\t")[0]  # drop morphological fields
            if not entry:
                continue
            word, _, flagstr = entry.partition("/")
            word = word.strip()
            if not word:
                continue
            flags = set(flagstr.strip())
            forms.add(word)

            for f in flags:
                if f in sfx:
                    forms.update(apply_suffix(word, sfx[f]))

            prefixed = []
            for f in flags:
                if f in pfx:
                    prefixed.extend(apply_prefix(word, pfx[f]))
            forms.update(prefixed)

            # Cross-product: a prefixed form may also take a suffix, but only
            # when both rules are declared cross-product ("Y").
            for f in flags:
                if f in pfx and cross.get(f):
                    for base in apply_prefix(word, pfx[f]):
                        for g in flags:
                            if g in sfx and cross.get(g):
                                forms.update(apply_suffix(base, sfx[g]))
    return forms


def shipped_aff(aff_path):
    keep = []
    for line in open(aff_path, encoding="utf-8").read().splitlines():
        head = line.split()[0] if line.split() else ""
        if head in KEEP_DIRECTIVES:
            keep.append(line)
    return "\n".join(keep) + "\n"


# Words the built list must accept, and must reject, before it is written. These
# are the pt-PT/pt-BR fault lines: post-Acordo spellings, the European lexicon,
# and the first-person-plural preterite accent that only pt-PT writes. A source
# dictionary that fails these is not the one this script is for.
MUST_ACCEPT = ("receção", "húmido", "telemóvel", "autocarro", "comboio",
               "ótimo", "pensámos", "escrevêssemos", "jardim")
MUST_REJECT = ("recepção", "úmido", "ônibus", "óptimo")


def main(aff_path, dic_path, out_dir):
    forms = expand(aff_path, dic_path)

    missing = [w for w in MUST_ACCEPT if w not in forms]
    present = [w for w in MUST_REJECT if w in forms]
    if missing or present:
        raise SystemExit(
            "this does not look like European Portuguese: "
            f"missing {missing}, unexpectedly present {present}"
        )

    os.makedirs(out_dir, exist_ok=True)
    ordered = sorted(forms)
    body = f"{len(ordered)}\n" + "\n".join(ordered) + "\n"

    dic_out = os.path.join(out_dir, "pt-PT.dic.gz")
    # mtime=0 so rebuilding identical input produces an identical file — a
    # vendored asset that changes on every build is noise in the diff.
    with gzip.GzipFile(dic_out, "wb", compresslevel=9, mtime=0) as fh:
        fh.write(body.encode("utf-8"))

    aff_out = os.path.join(out_dir, "pt-PT.aff")
    with open(aff_out, "w", encoding="utf-8") as fh:
        fh.write(shipped_aff(aff_path))

    print(f"{len(ordered)} forms -> {dic_out} "
          f"({os.path.getsize(dic_out) / 1e6:.2f} MB gzipped)", file=sys.stderr)


if __name__ == "__main__":
    if len(sys.argv) != 4:
        raise SystemExit(__doc__.strip().splitlines()[-1])
    main(*sys.argv[1:4])