drwily/petal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Petal learns French, and the pack that shipped was misspelling itself

Browse Source 
Phase 24, the fr half: langpack, Hunspell dictionary, Piper voice, and the
lexicon coverage that turned out to have been measured already (63.1%, better
than pt-PT's 62.1%). No migration; not deployed.

The plan recorded that build_ptpt_dictionary.py "generalizes" to French. It
did not. It handled single-character flags and plain PFX/SFX and stopped on
everything else, and fr.aff uses four of the things it stopped on. FLAG long
is the dangerous one: French flags are two characters, so the old reader's
set(flagstr) yields a bag of unrelated letters and expands every entry through
the wrong paradigm without ever erroring. Plus continuation flags (French
really does affix an affixed form), NEEDAFFIX on 68,075 of 84,140 stems, and
FULLSTRIP. Renamed build_hunspell_dictionary.py with a per-language profile,
asserting that CIRCUMFIX and FORBIDDENWORD are still unused rather than
assuming it — and it rebuilds pt-PT byte-identical to the shipped asset, which
is the only thing that makes "generalized" a claim rather than a hope.

Elision was decided by building both halves and measuring. Keeping l'arbre and
its thirty-three siblings: 3,159,832 forms, 8.25 MB gzipped. Dropping them:
473,326 and 1.19 MB. They are not new words, but the tokenizer keeps internal
apostrophes, so they genuinely would have been underlined — so they moved out
of the dictionary into withElision, which splits at a known clitic and still
requires the remainder to be a word (l'zzzz stays flagged). Real nspell: 369 ms
and 74 MB, against pt-PT's 842 ms and 139 MB, on the larger language.

Where the regional trap lives is the mirror image of Portuguese's: every fr_*
Piper voice is fr_FR and Debian's fr_FR/fr_CA/fr_BE dictionaries are one shared
word list, so nothing can be quietly wrong about the country and the whole
decision sits in the copy. What French has instead is the 1990 reform, packaged
three ways; comprehensive ships, because Petal never corrects her French and
coût and cout are both correct.

Then the interim review pass, at the user's suggestion and explicitly "for
now": four models read each Latin pack independently, and only findings at
least two of them reached on their own were applied — five per pack. It earned
its keep on the pack that was already live. pt-PT was carrying pre-Acordo
spellings (adjectivos, actualmente) in a file whose own header commits to
post-Acordo, plus Brazilian decepção, because the Phase 21 greps checked for
Brazilian vocabulary and never checked the pack against its own spelling
policy. That grep now exists and was confirmed to fail on the old text before
being kept. Where reviewers agreed a line was wrong but split on the fix, the
wording is mine and the reasoning is in BUILD_PLAN rather than averaged away.

Still owed, and both packs now say so precisely: a quorum of models agreeing is
agreement, not authority. No native speaker has read either pack, and none of
this has been seen in a browser.

go build/vet/test clean, tsc, vite build, vitest 190/190.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
This commit is contained in:
prosolis
2026-07-27 16:19:26 -07:00
parent 9a2e909b85
commit 071ea7b835
20 changed files with 1453 additions and 286 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/build_hunspell_dictionary.py
+432
View File
@@ -0,0 +1,432 @@
#!/usr/bin/env python3
"""Build one of Petal's browser spelling dictionaries from a Hunspell one.
Why this script exists at all
----------------------------
English is vendored the obvious way: `dictionary-en`'s `en.aff` + `en.dic` go
into web/public/dictionaries/en and nspell reads them in the browser. The plan
for Phase 21 said "Hunspell pt-PT vendored like en-US", and that turns out not to
work, for a measured reason.
nspell expands affixes **eagerly at load time** — it materialises every surface
form into a hash the moment you construct it. English gets away with this: ~50k
stems and a small rule set. European Portuguese does not. `pt_PT.aff` carries
1,340 affix rules (the full verb paradigm: six persons x a dozen tenses, plus
diminutives, plus productive prefixes) over 44,257 stems. Measured on this
machine, nspell needed ~340 MB of heap for the first 12,000 entries alone and had
not returned after three minutes on the whole file; extrapolated, it wants well
over a gigabyte. That is not something to hand a browser, still less a tablet.
French is larger again: 5,600 affix rules over 84,140 stems.
So the expansion happens **here**, once, at build time, and the browser gets a
flat word list it can load with no affix machinery at all. The runtime code path
is then *identical* to English — same nspell, same interface — which is the real
prize. The aff shipped alongside keeps only the suggestion-shaping directives
(TRY/KEY/REP/MAP), so corrections still know that "cao" wants "ção" and that a
missing acute accent is a near miss.
What Phase 24 had to add
------------------------
The build plan recorded that the pt-PT version of this script "generalizes" to
French. It did not. It handled single-character flags and plain PFX/SFX and
stopped on anything else — the right call, because `fr.aff` uses four of the
things it stopped on, and getting any of them wrong changes which words are
accepted:
* **`FLAG long`** — French flags are *two characters* (`S.`, `L'`, `Um`). The
pt-PT reader took `set(flagstr)`, one flag per character, which on a French
entry yields a bag of unrelated single letters: every entry would have been
expanded through the wrong paradigm. This is the one that fails silently.
* **Continuation flags** — pt-PT's affixes append plain text, so that script
dropped anything after a `/` and asserted the drop was safe. French really
does affix an affixed form: `PFX Um 0 0/S.` says the prefixed form then takes
the plural suffix, and the elision prefixes arrive the same way from the
other side (`SFX ... ait/n'q'l'm't's'`).
* **`NEEDAFFIX`** — French marks thousands of stems "not a word on its own"
(`Allemagne/S.()`), the bare form arriving instead through a zero-append
rule. Ignoring the flag accepts stems the real dictionary rejects.
* **`FULLSTRIP`** — a rule may strip the whole stem.
`CIRCUMFIX` and `FORBIDDENWORD` are *declared* in `fr.aff` and used by nothing,
which this script asserts rather than assumes: an upstream release that started
using either would otherwise change what is accepted without changing this file.
`KEEPCASE` and `NOSUGGEST` are honoured by being ignored on purpose — they shape
casing and suggestions, not membership, and a NOSUGGEST word is still a word.
Elision is handled at lookup, not here — and that is the size decision
----------------------------------------------------------------------
Most of French's affix machinery by volume is elision: `l'`, `d'`, `qu'`, `j'`,
`n'`, `s'`, `jusqu'`, `puisqu'`. Hunspell treats `l'arbre` as one word, so a
faithful expansion carries much of the language thirty-four times over — and
Petal's tokenizer keeps internal apostrophes, so `l'arbre` really does arrive at
the dictionary as one token and really would be underlined if it were absent.
Both halves were built and measured. Keeping the elided forms: **3,159,832 forms,
8.25 MB gzipped**, ~45 MB of text for nspell to hash on a tablet. Dropping them:
**473,326 forms, 1.19 MB gzipped**. The elided seven-eighths are not new words —
they are thirteen little words glued to words already in the list — so the third
option is the one taken: rules whose append carries an apostrophe are skipped
here (the count is printed), and `withElision` in `useSpellChecker.ts` splits a
token at a *known clitic* and checks the remainder. `l'arbre` costs one extra
lookup instead of seven megabytes, and `zzz'arbre` is still flagged because
`zzz` is not one of the thirteen.
Stems that carry an apostrophe of their own — `aujourd'hui`, `quelqu'un`,
`presqu'île`, `prud'homme` — are dictionary entries rather than affixed forms,
so they are kept verbatim and matched directly. `entr'aide` and `grand'mère` are
absent for the same reason they are absent from Dicollecte: modern French spells
them `entraide` and `grand-mère`.
Choosing the source
-------------------
Both languages have a trap here, and they are different traps.
**pt-PT: the wrong country.** npm's `dictionary-pt` is not European Portuguese.
Both it and `dictionary-pt-br` package VERO ("Verificador Ortográfico Livre",
Brasil), so vendoring the obvious npm name would have shipped Brazilian spellings
under a pt-PT label — the pt-BR drift SUGGESTIONS.md §3 warns about, arriving
through the packaging rather than through the model. The authentic dictionary is
the Projecto Natura one (Universidade do Minho) that LibreOffice ships and Debian
packages as `hunspell-pt-pt`; its aff declares `LANG pt_PT`.
**fr: the wrong side of an argument the French have not settled.** The regional
question turns out to be a non-question — Debian's `fr_FR`, `fr_CA`, `fr_BE`,
`fr_CH`, `fr_LU` and `fr_MC` are all symlinks to one `fr.dic`, so unlike pt there
is no country here to get wrong. What there is instead is the 1990 spelling
reform, packaged three ways: `hunspell-fr-classical` (traditional), `-revised`
(reform only) and `-comprehensive` (both). Petal ships **comprehensive**, because
Petal never corrects her French — the only thing this dictionary can do is
underline something. *coût* and *cout* are both correct French, taught in
different decades to different people, and a writing companion has no business
underlining one of them to take a side. The `fr` MUST_ACCEPT list is written to
*prove* which package was used: classical rejects `cout`, revised rejects `coût`,
and only comprehensive accepts both.
Licensing: pt-PT is GPL-2 or LGPL-2.1 or MPL-1.1, (c) José João de Almeida, Rui
Vilela, Alberto Simões. fr is MPL-2.0, (c) 2007-2018 the Dicollecte contributors
(grammalecte.net). The upstream copyright file is vendored beside each output.
Usage
-----
apt-get download hunspell-fr-comprehensive # or hunspell-pt-pt
dpkg-deb -x hunspell-fr-comprehensive_*.deb src
python3 scripts/build_hunspell_dictionary.py fr \\
src/usr/share/hunspell/fr.aff \\
src/usr/share/hunspell/fr.dic \\
web/public/dictionaries/fr
"""
import gzip
import os
import re
import sys
import unicodedata
# Directives worth keeping in the shipped aff. These shape *suggestions*, not
# membership: TRY orders the alphabet the corrector tries, KEY knows which keys
# are adjacent, REP holds the language's own confusions (cao/ção, ss/ç), and MAP
# says an accented vowel and its bare form are the same letter for scoring —
# which is most of what an ESL writer gets wrong in either language.
KEEP_DIRECTIVES = ("SET", "TRY", "KEY", "REP", "MAP", "WORDCHARS")
# Directives that would change which words are *accepted* and that this expander
# does not implement. If a future upstream release starts using one, the output
# would silently disagree with the real dictionary, so the build stops instead.
UNSUPPORTED = (
"COMPOUNDFLAG", "COMPOUNDMIN", "COMPOUNDRULE", "COMPOUNDBEGIN",
"ONLYINCOMPOUND", "PSEUDOROOT", "AF", "AM",
)
APOSTROPHES = "'’"
class Aff:
"""The parts of an .aff file that decide which words exist."""
def __init__(self):
self.pfx = {} # flag -> [(strip, append, condition, continuation)]
self.sfx = {}
self.cross = {} # flag -> bool
self.flag_kind = "char"
self.needaffix = None
self.circumfix = None
self.forbidden = None
self.dropped_apostrophe_rules = 0
def parse_flags(raw, kind):
"""Split a flag string into flags, per the aff's FLAG declaration."""
raw = raw.strip()
if not raw:
return set()
if kind == "long":
return {raw[i:i + 2] for i in range(0, len(raw) - 1, 2)}
if kind == "num":
return {f for f in raw.split(",") if f}
return set(raw)
def parse_aff(path):
with open(path, encoding="utf-8") as fh:
lines = fh.read().splitlines()
aff = Aff()
# FLAG has to be known before anything containing a flag is read, and it can
# sit anywhere in the file. So: header pass first, rules second.
for line in lines:
parts = line.split()
if not parts:
continue
head = parts[0]
if head in UNSUPPORTED:
raise SystemExit(
f"{path}: unsupported directive {head!r} — this expander handles "
"PFX/SFX affixation with continuation flags, and honouring "
f"{head} would change which words are accepted. Extend the "
"script before shipping."
)
if len(parts) < 2:
continue
if head == "FLAG":
aff.flag_kind = parts[1]
if aff.flag_kind not in ("long", "num", "UTF-8"):
raise SystemExit(f"{path}: unknown FLAG type {aff.flag_kind!r}")
elif head == "NEEDAFFIX":
aff.needaffix = parts[1]
elif head == "CIRCUMFIX":
aff.circumfix = parts[1]
elif head == "FORBIDDENWORD":
aff.forbidden = parts[1]
i = 0
while i < len(lines):
parts = lines[i].split()
if parts and parts[0] in ("PFX", "SFX"):
kind, flag, cross_flag, count = parts[0], parts[1], parts[2], int(parts[3])
table = aff.pfx if kind == "PFX" else aff.sfx
aff.cross[flag] = cross_flag == "Y"
rules = table.setdefault(flag, [])
for j in range(1, count + 1):
p = lines[i + j].split()
strip = "" if p[2] == "0" else p[2]
append, _, cont_raw = p[3].partition("/")
if append == "0":
append = ""
# Elision. See the header: these forms are `l'` and its twelve
# siblings glued to words already in the list, they multiply the
# download by seven, and `withElision` reconstructs them at
# lookup for the cost of one extra hash probe.
if any(a in append for a in APOSTROPHES):
aff.dropped_apostrophe_rules += 1
continue
cont = parse_flags(cont_raw, aff.flag_kind)
if aff.circumfix and aff.circumfix in cont:
raise SystemExit(
f"{path}: CIRCUMFIX is used by a {kind} {flag} rule. It "
"was declared-but-unused when this expander was written "
"and is not implemented; honouring it would change which "
"words are accepted."
)
cond = p[4] if len(p) > 4 else "."
anchored = ("^" + cond) if kind == "PFX" else (cond + "$")
rules.append((strip, append, re.compile(anchored), cont))
i += count + 1
continue
i += 1
return aff
def apply_suffix(word, rules):
"""Every (form, continuation flags) a suffix table yields for `word`."""
out = []
for strip, append, cond, cont in rules:
if strip and not word.endswith(strip):
continue
if not cond.search(word):
continue
stem = word[: len(word) - len(strip)] if strip else word
out.append((stem + append, cont))
return out
def apply_prefix(word, rules):
out = []
for strip, append, cond, cont in rules:
if strip and not word.startswith(strip):
continue
if not cond.search(word):
continue
stem = word[len(strip):] if strip else word
out.append((append + stem, cont))
return out
def expand_entry(word, flags, aff, out):
"""Add every surface form of one dictionary entry to `out`.
Hunspell's model without compounding: a form is the stem plus at most one
prefix and at most one suffix. A flag reaches an affix either from the stem's
own flags or from the continuation flags of the affix applied on the other
side; when both sides apply, both rules must be declared cross-product.
NEEDAFFIX is why the bare form is not simply added: the flag says *this* form
is not a word, only whatever can be built from it — and it arrives both on
stems and on continuations.
"""
def is_word(carried):
return not (aff.needaffix and aff.needaffix in carried)
if is_word(flags):
out.add(word)
suffixed = [] # (form, flag, continuation flags)
for f in flags:
if f in aff.sfx:
for form, cont in apply_suffix(word, aff.sfx[f]):
suffixed.append((form, f, cont))
if is_word(cont):
out.add(form)
prefixed = []
for f in flags:
if f in aff.pfx:
for form, cont in apply_prefix(word, aff.pfx[f]):
prefixed.append((form, f, cont))
if is_word(cont):
out.add(form)
# Prefix then suffix. The suffix flag may come from the stem or from the
# prefix's own continuation (`PFX Um 0 0/S.`), and the suffix condition is
# matched against the whole prefixed word, which is what hunspell does.
for form, pf, pcont in prefixed:
if not aff.cross.get(pf):
continue
for f in flags | pcont:
if f in aff.sfx and aff.cross.get(f):
for full, _ in apply_suffix(form, aff.sfx[f]):
out.add(full)
# Suffix then prefix — the same pair reached from the other side, which is
# how the elision prefixes arrive in French. Only the flags the suffix hands
# forward are new here; the stem's own were covered above.
for form, sf, scont in suffixed:
if not aff.cross.get(sf):
continue
for f in scont:
if f in aff.pfx and aff.cross.get(f):
for full, _ in apply_prefix(form, aff.pfx[f]):
out.add(full)
def expand(aff_path, dic_path):
aff = parse_aff(aff_path)
forms = set()
needaffix_stems = 0
with open(dic_path, encoding="utf-8") as fh:
fh.readline() # leading entry count, not a word
for raw in fh:
# Morphological fields (po:nom is:fem) follow the entry, separated by
# a tab in pt-PT and by a space in fr.
entry = raw.strip().split("\t")[0].split(" ")[0]
if not entry:
continue
word, _, flagstr = entry.partition("/")
word = word.strip()
if not word:
continue
flags = parse_flags(flagstr, aff.flag_kind)
if aff.forbidden and aff.forbidden in flags:
raise SystemExit(
f"{dic_path}: FORBIDDENWORD is in use ({word!r}). It was "
"declared-but-unused when this expander was written; the "
"forms it removes would be wrongly accepted."
)
if aff.needaffix and aff.needaffix in flags:
needaffix_stems += 1
expand_entry(word, flags, aff, forms)
# NFC, because the aff's own ICONV table normalises decomposed accents on the
# way in and the browser hands nspell whatever the keyboard produced.
forms = {unicodedata.normalize("NFC", f) for f in forms}
return forms, aff, needaffix_stems
def shipped_aff(aff_path):
keep = []
for line in open(aff_path, encoding="utf-8").read().splitlines():
head = line.split()[0] if line.split() else ""
if head in KEEP_DIRECTIVES:
keep.append(line)
return "\n".join(keep) + "\n"
# Words the built list must accept, and must reject, before it is written. Each
# set is chosen to fail loudly on the *specific* wrong source that language has a
# packaged, plausible way of reaching — not to spot-check spelling in general.
PROFILES = {
# The pt-PT/pt-BR fault lines: post-Acordo spellings, the European lexicon,
# and the first-person-plural preterite accent that only pt-PT writes.
"pt-PT": {
"accept": ("receção", "húmido", "telemóvel", "autocarro", "comboio",
"ótimo", "pensámos", "escrevêssemos", "jardim"),
"reject": ("recepção", "úmido", "ônibus", "óptimo"),
"wrong": "this does not look like European Portuguese",
},
# Which of the three 1990-reform packagings this is. `coût`/`cout` and
# `paraître`/`paraitre` are each accepted by exactly one of classical and
# revised, so a build accepting all four is comprehensive and one that drops
# any of them is not. `Allemagne` is a NEEDAFFIX stem reachable only through
# a zero-append rule and `km` only through a prefix continuation, so between
# them they also check that this expander honoured the two features the
# pt-PT one refused.
"fr": {
"accept": ("coût", "cout", "paraître", "paraitre", "nénuphar", "nénufar",
"oignon", "ognon", "événement", "évènement", "jardin",
"Allemagne", "écrivissions", "km"),
"reject": ("jardinn", "écrivaitz", "xyzzyque"),
"wrong": "this does not look like the comprehensive French dictionary",
},
}
def main(lang, aff_path, dic_path, out_dir):
profile = PROFILES.get(lang)
if profile is None:
raise SystemExit(f"no profile for {lang!r}; known: {', '.join(PROFILES)}")
forms, aff, needaffix_stems = expand(aff_path, dic_path)
missing = [w for w in profile["accept"] if w not in forms]
present = [w for w in profile["reject"] if w in forms]
if missing or present:
raise SystemExit(
f"{profile['wrong']}: missing {missing}, unexpectedly present {present}"
)
os.makedirs(out_dir, exist_ok=True)
ordered = sorted(forms)
body = f"{len(ordered)}\n" + "\n".join(ordered) + "\n"
dic_out = os.path.join(out_dir, f"{lang}.dic.gz")
# mtime=0 so rebuilding identical input produces an identical file — a
# vendored asset that changes on every build is noise in the diff.
with gzip.GzipFile(dic_out, "wb", compresslevel=9, mtime=0) as fh:
fh.write(body.encode("utf-8"))
aff_out = os.path.join(out_dir, f"{lang}.aff")
with open(aff_out, "w", encoding="utf-8") as fh:
fh.write(shipped_aff(aff_path))
print(f"{len(ordered)} forms -> {dic_out} "
f"({os.path.getsize(dic_out) / 1e6:.2f} MB gzipped); "
f"{needaffix_stems} NEEDAFFIX stems, "
f"{aff.dropped_apostrophe_rules} elision rules skipped", file=sys.stderr)
if __name__ == "__main__":
if len(sys.argv) != 5:
raise SystemExit(
"usage: build_hunspell_dictionary.py <lang> <aff> <dic> <out-dir>\n"
f" lang is one of: {', '.join(PROFILES)}"
)
main(*sys.argv[1:5])
scripts/build_ptpt_dictionary.py
-232
View File
@@ -1,232 +0,0 @@
#!/usr/bin/env python3
"""Build Petal's browser pt-PT spelling dictionary from the LibreOffice Hunspell one.
Why this script exists at all
----------------------------
English is vendored the obvious way: `dictionary-en`'s `en.aff` + `en.dic` go
into web/public/dictionaries/en and nspell reads them in the browser. The plan
for Phase 21 said "Hunspell pt-PT vendored like en-US", and that turns out not to
work, for a measured reason.
nspell expands affixes **eagerly at load time** — it materialises every surface
form into a hash the moment you construct it. English gets away with this: ~50k
stems and a small rule set. European Portuguese does not. `pt_PT.aff` carries
1,340 affix rules (the full verb paradigm: six persons x a dozen tenses, plus
diminutives, plus productive prefixes) over 44,257 stems. Measured on this
machine, nspell needed ~340 MB of heap for the first 12,000 entries alone and had
not returned after three minutes on the whole file; extrapolated, it wants well
over a gigabyte. That is not something to hand a browser, still less a tablet.
So the expansion happens **here**, once, at build time, and the browser gets a
flat word list it can load with no affix machinery at all: 1,039,058 forms, 15 MB
of text, 2.7 MB gzipped, which nspell reads in ~0.8s using ~120 MB. The runtime
code path is then *identical* to English — same nspell, same interface — which is
the real prize. The aff we ship alongside keeps only the suggestion-shaping
directives (TRY/KEY/REP/MAP), so corrections still know that "cao" wants "ção"
and that a missing acute accent is a near miss.
Choosing the source
-------------------
npm's `dictionary-pt` is **not** European Portuguese. Both it and
`dictionary-pt-br` package VERO ("Verificador Ortográfico Livre", Brasil), so
vendoring the obvious npm name would have shipped Brazilian spellings under a
pt-PT label — exactly the pt-BR drift SUGGESTIONS.md §3 warns about, arriving
through the packaging rather than through the model.
The authentic dictionary is the Projecto Natura one (Universidade do Minho) that
LibreOffice ships and Debian packages as `hunspell-pt-pt`. Its aff declares
`LANG pt_PT`. Spot-checked against the built list, it accepts `receção`,
`húmido`, `telemóvel`, `autocarro`, `comboio`, `ótimo` and `pensámos`, and
rejects `recepção`, `úmido`, `ônibus` and `óptimo` — post-Acordo European
Portuguese, which is what a pt-PT writer should be held to.
Licensing: GPL-2 or LGPL-2.1 or MPL-1.1, (c) José João de Almeida, Rui Vilela,
Alberto Simões. The upstream copyright file is vendored beside the output.
Usage
-----
apt-get download hunspell-pt-pt # or take pt_PT.aff/.dic from LibreOffice
dpkg-deb -x hunspell-pt-pt_*.deb ptpt
python3 scripts/build_ptpt_dictionary.py \
ptpt/usr/share/hunspell/pt_PT.aff \
ptpt/usr/share/hunspell/pt_PT.dic \
web/public/dictionaries/pt-PT
Only the directives pt_PT.aff actually uses are implemented: PFX/SFX with
single-character flags, strip/append/condition, and cross-product. It carries no
compounding, no flag aliases and no NEEDAFFIX, so there is nothing else to
honour — the script asserts that rather than assuming it.
"""
import gzip
import os
import re
import sys
# Directives worth keeping in the shipped aff. These shape *suggestions*, not
# membership: TRY orders the alphabet the corrector tries, KEY knows which keys
# are adjacent, REP holds Portuguese-specific confusions (cao/ção, ss/ç), and MAP
# says an accented vowel and its bare form are the same letter for scoring —
# which is most of what an ESL writer gets wrong in Portuguese.
KEEP_DIRECTIVES = ("SET", "TRY", "KEY", "REP", "MAP", "WORDCHARS")
# Directives that would change which words are *accepted*. If a future upstream
# release starts using one, this script's output would silently disagree with
# the real dictionary, so it stops instead.
UNSUPPORTED = (
"COMPOUNDFLAG", "COMPOUNDMIN", "COMPOUNDRULE", "COMPOUNDBEGIN",
"ONLYINCOMPOUND", "NEEDAFFIX", "PSEUDOROOT", "CIRCUMFIX",
"FORBIDDENWORD", "AF", "AM", "FLAG",
)
def parse_aff(path):
"""Return (prefix rules, suffix rules, cross-product flags) keyed by flag."""
with open(path, encoding="utf-8") as fh:
lines = fh.read().splitlines()
for line in lines:
head = line.split()[0] if line.split() else ""
if head in UNSUPPORTED:
raise SystemExit(
f"{path}: unsupported directive {head!r} — this expander only "
"handles plain PFX/SFX affixation, and honouring it would change "
"which words are accepted. Extend the script before shipping."
)
pfx, sfx, cross = {}, {}, {}
i = 0
while i < len(lines):
parts = lines[i].split()
if parts and parts[0] in ("PFX", "SFX"):
kind, flag, cross_flag, count = parts[0], parts[1], parts[2], int(parts[3])
table = pfx if kind == "PFX" else sfx
cross[flag] = cross_flag == "Y"
rules = table.setdefault(flag, [])
for j in range(1, count + 1):
p = lines[i + j].split()
strip = "" if p[2] == "0" else p[2]
# The appended text may carry its own continuation flags after a
# slash (append/FLAGS). We drop them: honouring them would mean
# affixing an affixed form, which pt_PT.aff does not do.
append = "" if p[3] == "0" else p[3].split("/")[0]
cond = p[4] if len(p) > 4 else "."
anchored = ("^" + cond) if kind == "PFX" else (cond + "$")
rules.append((strip, append, re.compile(anchored)))
i += count + 1
continue
i += 1
return pfx, sfx, cross
def apply_suffix(word, rules):
out = []
for strip, append, cond in rules:
if strip and not word.endswith(strip):
continue
if not cond.search(word):
continue
stem = word[: len(word) - len(strip)] if strip else word
out.append(stem + append)
return out
def apply_prefix(word, rules):
out = []
for strip, append, cond in rules:
if strip and not word.startswith(strip):
continue
if not cond.search(word):
continue
stem = word[len(strip):] if strip else word
out.append(append + stem)
return out
def expand(aff_path, dic_path):
pfx, sfx, cross = parse_aff(aff_path)
forms = set()
with open(dic_path, encoding="utf-8") as fh:
fh.readline() # leading entry count, not a word
for raw in fh:
entry = raw.strip().split("\t")[0] # drop morphological fields
if not entry:
continue
word, _, flagstr = entry.partition("/")
word = word.strip()
if not word:
continue
flags = set(flagstr.strip())
forms.add(word)
for f in flags:
if f in sfx:
forms.update(apply_suffix(word, sfx[f]))
prefixed = []
for f in flags:
if f in pfx:
prefixed.extend(apply_prefix(word, pfx[f]))
forms.update(prefixed)
# Cross-product: a prefixed form may also take a suffix, but only
# when both rules are declared cross-product ("Y").
for f in flags:
if f in pfx and cross.get(f):
for base in apply_prefix(word, pfx[f]):
for g in flags:
if g in sfx and cross.get(g):
forms.update(apply_suffix(base, sfx[g]))
return forms
def shipped_aff(aff_path):
keep = []
for line in open(aff_path, encoding="utf-8").read().splitlines():
head = line.split()[0] if line.split() else ""
if head in KEEP_DIRECTIVES:
keep.append(line)
return "\n".join(keep) + "\n"
# Words the built list must accept, and must reject, before it is written. These
# are the pt-PT/pt-BR fault lines: post-Acordo spellings, the European lexicon,
# and the first-person-plural preterite accent that only pt-PT writes. A source
# dictionary that fails these is not the one this script is for.
MUST_ACCEPT = ("receção", "húmido", "telemóvel", "autocarro", "comboio",
"ótimo", "pensámos", "escrevêssemos", "jardim")
MUST_REJECT = ("recepção", "úmido", "ônibus", "óptimo")
def main(aff_path, dic_path, out_dir):
forms = expand(aff_path, dic_path)
missing = [w for w in MUST_ACCEPT if w not in forms]
present = [w for w in MUST_REJECT if w in forms]
if missing or present:
raise SystemExit(
"this does not look like European Portuguese: "
f"missing {missing}, unexpectedly present {present}"
)
os.makedirs(out_dir, exist_ok=True)
ordered = sorted(forms)
body = f"{len(ordered)}\n" + "\n".join(ordered) + "\n"
dic_out = os.path.join(out_dir, "pt-PT.dic.gz")
# mtime=0 so rebuilding identical input produces an identical file — a
# vendored asset that changes on every build is noise in the diff.
with gzip.GzipFile(dic_out, "wb", compresslevel=9, mtime=0) as fh:
fh.write(body.encode("utf-8"))
aff_out = os.path.join(out_dir, "pt-PT.aff")
with open(aff_out, "w", encoding="utf-8") as fh:
fh.write(shipped_aff(aff_path))
print(f"{len(ordered)} forms -> {dic_out} "
f"({os.path.getsize(dic_out) / 1e6:.2f} MB gzipped)", file=sys.stderr)
if __name__ == "__main__":
if len(sys.argv) != 4:
raise SystemExit(__doc__.strip().splitlines()[-1])
main(*sys.argv[1:4])