Petal learns French, and the pack that shipped was misspelling itself

Phase 24, the fr half: langpack, Hunspell dictionary, Piper voice, and the
lexicon coverage that turned out to have been measured already (63.1%, better
than pt-PT's 62.1%). No migration; not deployed.

The plan recorded that build_ptpt_dictionary.py "generalizes" to French. It
did not. It handled single-character flags and plain PFX/SFX and stopped on
everything else, and fr.aff uses four of the things it stopped on. FLAG long
is the dangerous one: French flags are two characters, so the old reader's
set(flagstr) yields a bag of unrelated letters and expands every entry through
the wrong paradigm without ever erroring. Plus continuation flags (French
really does affix an affixed form), NEEDAFFIX on 68,075 of 84,140 stems, and
FULLSTRIP. Renamed build_hunspell_dictionary.py with a per-language profile,
asserting that CIRCUMFIX and FORBIDDENWORD are still unused rather than
assuming it — and it rebuilds pt-PT byte-identical to the shipped asset, which
is the only thing that makes "generalized" a claim rather than a hope.

Elision was decided by building both halves and measuring. Keeping l'arbre and
its thirty-three siblings: 3,159,832 forms, 8.25 MB gzipped. Dropping them:
473,326 and 1.19 MB. They are not new words, but the tokenizer keeps internal
apostrophes, so they genuinely would have been underlined — so they moved out
of the dictionary into withElision, which splits at a known clitic and still
requires the remainder to be a word (l'zzzz stays flagged). Real nspell: 369 ms
and 74 MB, against pt-PT's 842 ms and 139 MB, on the larger language.

Where the regional trap lives is the mirror image of Portuguese's: every fr_*
Piper voice is fr_FR and Debian's fr_FR/fr_CA/fr_BE dictionaries are one shared
word list, so nothing can be quietly wrong about the country and the whole
decision sits in the copy. What French has instead is the 1990 reform, packaged
three ways; comprehensive ships, because Petal never corrects her French and
coût and cout are both correct.

Then the interim review pass, at the user's suggestion and explicitly "for
now": four models read each Latin pack independently, and only findings at
least two of them reached on their own were applied — five per pack. It earned
its keep on the pack that was already live. pt-PT was carrying pre-Acordo
spellings (adjectivos, actualmente) in a file whose own header commits to
post-Acordo, plus Brazilian decepção, because the Phase 21 greps checked for
Brazilian vocabulary and never checked the pack against its own spelling
policy. That grep now exists and was confirmed to fail on the old text before
being kept. Where reviewers agreed a line was wrong but split on the fix, the
wording is mine and the reasoning is in BUILD_PLAN rather than averaged away.

Still owed, and both packs now say so precisely: a quorum of models agreeing is
agreement, not authority. No native speaker has read either pack, and none of
this has been seen in a browser.

go build/vet/test clean, tsc, vite build, vitest 190/190.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua

web/src/hooks/spellchecker.test.ts

@@ -1,5 +1,5 @@
 import { describe, it, expect } from 'vitest'
-import { combine, interleave, type Loaded } from './useSpellChecker'
+import { combine, interleave, withElision, type Dictionary, type Loaded } from './useSpellChecker'
 
 // The both-dictionaries rule (SUGGESTIONS.md §3a), separated from the fetching
 // so it can be checked without a 15 MB word list. What matters here is not
@@ -78,3 +78,61 @@ describe('correction pills', () => {
     expect(interleave([])).toEqual([])
   })
 })
+
+// French elision (Phase 24). The word list is built without the elided forms
+// because carrying them costs 8.25 MB gzipped instead of 1.19 MB, so the split
+// happens here instead. What is being checked is not "does French work" but the
+// same question as everywhere else in this file: which way is it allowed to be
+// wrong.
+describe('elision', () => {
+  const FR = ['l', 'd', 'j', 'qu', 'jusqu']
+  const base: Dictionary = {
+    correct: (w) => ['arbre', 'accord', 'elle', 'ici', "aujourd'hui", 'Étang'].includes(w),
+    suggest: (w) => (w === 'arbrz' ? ['arbre', 'arbres'] : []),
+    add: () => undefined,
+  }
+  const fr = withElision(base, FR)
+
+  it('looks past a clitic the language actually elides', () => {
+    // The whole reason this exists: every one of these is absent from the built
+    // word list, and every one of them is ordinary French.
+    for (const w of ["l'arbre", "d'accord", "qu'elle", "jusqu'ici"]) {
+      expect(base.correct(w), `${w} should not be in the list`).toBe(false)
+      expect(fr.correct(w), w).toBe(true)
+    }
+  })
+
+  it('leaves a word that carries its own apostrophe alone', () => {
+    // "aujourd" is not a clitic, so this is never split — it matches directly,
+    // which is why the build script keeps such stems verbatim.
+    expect(fr.correct("aujourd'hui")).toBe(true)
+  })
+
+  it('still flags an apostrophe that is not an elision', () => {
+    // Both halves have to hold up: an unknown clitic on a real word, and a real
+    // clitic on an unknown word. Splitting is a second lookup, not an amnesty.
+    expect(fr.correct("zzz'arbre")).toBe(false)
+    expect(fr.correct("l'zzzz")).toBe(false)
+  })
+
+  it('is case-insensitive about the clitic, because a sentence can start with one', () => {
+    expect(fr.correct("L'Étang")).toBe(true)
+  })
+
+  it('puts the clitic back on its corrections', () => {
+    // The pill replaces the whole token. Offering "arbre" for "l'arbrz" would
+    // silently delete the article she wrote.
+    expect(fr.suggest("l'arbrz")).toEqual(["l'arbre", "l'arbres"])
+  })
+
+  it('does not split what has no apostrophe to split on', () => {
+    expect(fr.correct("'arbre")).toBe(false) // nothing before the mark
+    expect(fr.correct("l'")).toBe(false) // nothing after it
+  })
+
+  it('hands back the same dictionary for a language that elides nothing', () => {
+    // pt-PT and English pay nothing for this.
+    expect(withElision(base, [])).toBe(base)
+    expect(withElision(base, undefined)).toBe(base)
+  })
+})

web/src/hooks/useSpellChecker.ts

@@ -49,6 +49,9 @@ interface DictSpec {
   // compressed and is inflated here; en's 550 KB does not need it.
   gzipped?: boolean
   extendedAlphabet?: boolean
+  // The little words this language glues onto the front of the next one. See
+  // withElision — a language with none simply omits it.
+  elision?: string[]
 }
 
 const EN: DictSpec = {
@@ -57,14 +60,28 @@ const EN: DictSpec = {
   dic: '/dictionaries/en/en.dic',
 }
 
+// The thirteen words French elides onto whatever follows: le/la, de, je, me, te,
+// se, ce, ne, que, jusque, lorsque, puisque, quoique. Written as the *head* they
+// leave behind, because that is the half a token can be split on.
+//
+// Hunspell carries the elided forms as thirty-four prefix rules, which is seven
+// megabytes of gzip once expanded (measured: 8.25 MB against 1.19 MB without).
+// They are not new words — they are these thirteen glued to words already in the
+// list — so the build script skips them and withElision puts them back at
+// lookup. What is *not* here is deliberate: "aujourd'hui", "quelqu'un" and
+// "presqu'île" are entries in their own right and match directly, and "entr'" is
+// missing for the same reason Dicollecte omits it — modern French writes
+// "entraide".
+const FR_ELISION = ['l', 'd', 'j', 'm', 't', 's', 'c', 'n', 'qu', 'jusqu', 'lorsqu', 'puisqu', 'quoiqu']
+
 // The writer's-language dictionary, by pair. zh has no Hunspell dictionary and
 // needs none: Chinese is not tokenized, so it is never flagged.
 //
-// pt-PT's word list is pre-expanded (see scripts/build_ptpt_dictionary.py) —
-// nspell expands affixes eagerly on load, and doing that to European
-// Portuguese's 1,340 rules in a browser wants over a gigabyte of heap. The
-// forms are computed at build time instead, so this is the same nspell reading
-// a bigger, simpler file.
+// Both Latin word lists are pre-expanded (see
+// scripts/build_hunspell_dictionary.py) — nspell expands affixes eagerly on
+// load, and doing that to European Portuguese's 1,340 rules or French's 5,600 in
+// a browser wants over a gigabyte of heap. The forms are computed at build time
+// instead, so this is the same nspell reading a bigger, simpler file.
 const PAIR_DICTS: Partial<Record<PairLang, DictSpec>> = {
   'pt-PT': {
     lang: 'pt-PT',
@@ -73,6 +90,14 @@ const PAIR_DICTS: Partial<Record<PairLang, DictSpec>> = {
     gzipped: true,
     extendedAlphabet: true,
   },
+  fr: {
+    lang: 'fr',
+    aff: '/dictionaries/fr/fr.aff',
+    dic: '/dictionaries/fr/fr.dic.gz',
+    gzipped: true,
+    extendedAlphabet: true,
+    elision: FR_ELISION,
+  },
 }
 
 // Where the list lived before it had an owner (Phase 7). Read once, handed to
@@ -132,6 +157,54 @@ export interface Loaded {
   extendedAlphabet: boolean
 }
 
+// withElision wraps a dictionary so that a clitic glued to the front of a word
+// is looked past rather than looked up.
+//
+// The tokenizer keeps internal apostrophes on purpose (don't, O'Brien), so
+// "l'arbre" reaches the dictionary whole. French writes that constantly and the
+// forms are not in the list — see FR_ELISION for why they are not — so without
+// this every elided article in a French document would be underlined, which is
+// the one failure mode this whole subsystem is built to avoid.
+//
+// It splits at the *first* apostrophe and only when the head is one of the
+// language's own clitics, and the remainder still has to be a word: "zzz'arbre"
+// is flagged because zzz is not a word French elides, and "l'zzzz" is flagged
+// because zzzz is not a word. Splitting buys a second lookup, not an amnesty.
+//
+// A language with no clitics gets the dictionary back untouched, so pt-PT and
+// English pay nothing for this.
+export function withElision(spell: Dictionary, clitics: string[] | undefined): Dictionary {
+  if (!clitics || clitics.length === 0) return spell
+  const heads = new Set(clitics)
+
+  // [clitic-with-apostrophe, remainder], or null if this isn't an elision.
+  const split = (word: string): [string, string] | null => {
+    const i = word.indexOf("'")
+    if (i <= 0 || i === word.length - 1) return null
+    if (!heads.has(word.slice(0, i).toLowerCase())) return null
+    return [word.slice(0, i + 1), word.slice(i + 1)]
+  }
+
+  return {
+    correct: (word) => {
+      if (spell.correct(word)) return true
+      const parts = split(word)
+      return parts ? spell.correct(parts[1]) : false
+    },
+    // Corrections come back with the clitic put back on, because the pill
+    // replaces the whole token: offering "arbre" for "l'arbrz" would silently
+    // delete the article she wrote.
+    suggest: (word) => {
+      const direct = spell.suggest(word)
+      if (direct.length > 0) return direct
+      const parts = split(word)
+      if (!parts) return []
+      return spell.suggest(parts[1]).map((s) => parts[0] + s)
+    },
+    add: (word) => spell.add(word),
+  }
+}
+
 // load builds one nspell instance and replays this writer's personal words into
 // it. `adoptLegacy` is passed only for English, and only on first load.
 async function load(spec: DictSpec, adoptLegacy: boolean): Promise<Loaded> {
@@ -153,7 +226,11 @@ async function load(spec: DictSpec, adoptLegacy: boolean): Promise<Loaded> {
     : await api.listPersonalWords(spec.lang)
   for (const w of stored.words) spell.add(w)
 
-  return { lang: spec.lang, spell, extendedAlphabet: spec.extendedAlphabet ?? false }
+  return {
+    lang: spec.lang,
+    spell: withElision(spell, spec.elision),
+    extendedAlphabet: spec.extendedAlphabet ?? false,
+  }
 }
 
 // interleave merges each dictionary's corrections round-robin. Concatenating