drwily/petal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 21: Petal learns to be an English+Portuguese pair

Browse Source 
The plan said "Hunspell pt-PT vendored like en-US". Measuring that first is
what saved it: nspell expands affixes eagerly on construction, and European
Portuguese's 1,340 rules over 44,257 stems want over a gigabyte of browser
heap — ~340 MB for the first 12,000 entries, and no return at all after three
minutes on the whole file. So the expansion runs once at build time instead:
1,039,058 forms, 2.66 MB gzipped, read by the same nspell in 842 ms.

The obvious npm package would also have shipped the wrong language. Both
dictionary-pt and dictionary-pt-br carry VERO, the Brazilian word list, so
vendoring by name puts pt-BR spellings behind a pt-PT label — the drift
SUGGESTIONS §3 warns about, arriving through the packaging where no reviewer
can see it. The source is Projecto Natura's, and the build script now asserts
the fault lines (receção in, recepção out) before writing anything.

Spellcheck consults both dictionaries and flags only what both reject, which
is the no-detector answer to a pair with no script boundary. The word card
does the same in the other direction: "data" is a word in both languages, so
Petal shows both readings rather than guessing which she meant.

Writing the tests caught the one real bug — extendedAlphabet was a snapshot
while correct/suggest read live, and her dictionary arrives after English, so
every lookup would have resolved "cora" while the underlines were already
right.

Not done, and not claimed: the pack has not been read by a pt-PT speaker, and
the Piper voice is deferred with the deploy.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
This commit is contained in:
prosolis
2026-07-27 12:43:02 -07:00
parent 4de83d0da5
commit ccb43e5a4d
22 changed files with 1458 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files
web/src/hooks/spellchecker.test.ts
+80
View File
@@ -0,0 +1,80 @@
import { describe, it, expect } from 'vitest'
import { combine, interleave, type Loaded } from './useSpellChecker'
// The both-dictionaries rule (SUGGESTIONS.md §3a), separated from the fetching
// so it can be checked without a 15 MB word list. What matters here is not
// "does nspell work" but which way the combination is allowed to be wrong.
// A dictionary that accepts exactly the words it was given.
function dict(lang: string, words: string[], corrections: string[] = [], extendedAlphabet = false): Loaded {
const set = new Set(words)
return {
lang,
extendedAlphabet,
spell: {
correct: (w: string) => set.has(w),
suggest: () => corrections,
add: (w: string) => set.add(w),
},
}
}
const en = dict('en', ['sale', 'the', 'river'], ['sailed', 'salt'])
const pt = dict('pt-PT', ['sale', 'coração', 'jardim'], ['salte', 'sala'], true)
describe('the both-dictionaries rule', () => {
it('accepts a word either dictionary knows', () => {
const c = combine(() => [en, pt])
expect(c.correct('river')).toBe(true) // English only
expect(c.correct('jardim')).toBe(true) // Portuguese only
expect(c.correct('sale')).toBe(true) // both — the collision case
})
it('flags only what every dictionary rejects', () => {
expect(combine(() => [en, pt]).correct('qqzzx')).toBe(false)
})
it('never flags a Portuguese word just because English has not heard of it', () => {
// The property the whole design exists for. With English alone, "coração"
// is a misspelling; with her own dictionary loaded it is a word she wrote.
expect(combine(() => [en]).correct('coração')).toBe(false)
expect(combine(() => [en, pt]).correct('coração')).toBe(true)
})
it('accepts everything when no dictionary loaded', () => {
// A failed fetch must not underline every word in the document. Silence is
// the safe failure; a page of red is not.
expect(combine(() => []).correct('qqzzx')).toBe(true)
})
it('sees a dictionary that arrives after the checker was built', () => {
// English loads immediately; hers lands a moment later, once /api/me has
// named her pair. The checker reads through a getter for exactly this.
let loaded: Loaded[] = [en]
const c = combine(() => loaded)
expect(c.correct('jardim')).toBe(false)
expect(c.extendedAlphabet).toBe(false)
loaded = [en, pt]
expect(c.correct('jardim')).toBe(true)
expect(c.extendedAlphabet).toBe(true)
})
})
describe('correction pills', () => {
it('interleaves the two dictionaries rather than letting one fill the list', () => {
// Five pills fit. Concatenating would spend all of them on English and
// leave a misspelt Portuguese word with no Portuguese correction — the one
// case the second dictionary was loaded for.
expect(combine(() => [en, pt]).suggest('salle')).toEqual(['sailed', 'salte', 'salt', 'sala'])
})
it('drops duplicates, keeping the first dictionary to offer one', () => {
expect(interleave([['a', 'b'], ['a', 'c']])).toEqual(['a', 'b', 'c'])
})
it('keeps going when one dictionary runs out of ideas', () => {
expect(interleave([['a'], ['x', 'y', 'z']])).toEqual(['a', 'x', 'y', 'z'])
expect(interleave([[], []])).toEqual([])
expect(interleave([])).toEqual([])
})
})
web/src/hooks/useSpellChecker.ts
+235 -62
View File
@@ -1,37 +1,85 @@
import { useCallback, useEffect, useMemo, useRef, useState } from 'react'
import nspell, { type NSpell } from 'nspell'
import nspell from 'nspell'
import { api } from '../api/client'
import { usePack } from '../i18n'
import type { PairLang } from '../i18n'
// useSpellChecker loads the vendored en-US Hunspell dictionary (served from
// /dictionaries/en, embedded in the Go binary via web/dist) and builds an
// in-browser nspell instance — zero backend round-trips, per spec. The
// dictionary is ~550KB, so it's fetched as a static asset (kept out of the JS
// bundle) once per app session, not per document.
// useSpellChecker builds the in-browser spell checker — zero backend round-trips
// per keystroke, per spec.
//
// English is always loaded, because English is always the target half of the
// pair. The writer's own language is loaded too when Petal ships a dictionary
// for it, and then the two are consulted together: **a token is flagged only if
// both dictionaries reject it.**
//
// That rule is the answer to the one genuinely new problem a Latin-script pair
// creates (SUGGESTIONS.md §3a). The zh pair never had to decide which language a
// word was in — the script answers it, and CJK is simply never tokenized. In an
// English+Portuguese document both halves are Latin letters, and there is no
// honest way to look at "sale" and know which language it is. Asking both
// dictionaries needs no detector and no guess. It can miss a misspelling that
// happens to be a real word in the other language; it can never squiggle
// correct writing. That is the gentle direction to be wrong in.
//
// The personal word list — the words she's told Petal to stop flagging — lives
// on the server, keyed by her account and by the dictionary's language. It used
// to be one localStorage key, which meant two people sharing a device shared a
// word list built from one person's private writing, and one person on a laptop
// and a tablet had two lists that never met. It is replayed into nspell on load;
// adding a word bumps a `version` so consumers re-run their decorations and the
// word stops flagging.
// on the server, keyed by her account and by dictionary language. It is replayed
// into each nspell instance on load; adding a word bumps a `version` so consumers
// re-run their decorations and the word stops flagging.
// SpellChecker is the minimal surface the editor decoration layer consumes.
export interface SpellChecker {
correct(word: string): boolean
suggest(word: string): string[]
// Whether the loaded dictionaries need letters beyond A-Z. Portuguese words
// carry ç and five accented vowels; tokenizing without them would cut "ação"
// into fragments. It is a property of the checker rather than a constant
// because widening the alphabet for an English-only writer would only earn
// her new squiggles under the French and Portuguese words she borrows.
extendedAlphabet: boolean
}
// The dictionary this hook loads. Only en-US ships today; pt-PT arrives with the
// first Latin pair, and its personal words are a separate list by design — an
// English exception must not silence a Portuguese flag.
const DICT_LANG = 'en'
// A dictionary Petal can load into the browser.
interface DictSpec {
// The key the personal word list is stored under. It is the dictionary's
// language, not the writer's.
lang: string
aff: string
dic: string
// Whether `dic` is gzipped. pt-PT's word list is 15 MB of text, so it ships
// compressed and is inflated here; en's 550 KB does not need it.
gzipped?: boolean
extendedAlphabet?: boolean
}
const EN: DictSpec = {
lang: 'en',
aff: '/dictionaries/en/en.aff',
dic: '/dictionaries/en/en.dic',
}
// The writer's-language dictionary, by pair. zh has no Hunspell dictionary and
// needs none: Chinese is not tokenized, so it is never flagged.
//
// pt-PT's word list is pre-expanded (see scripts/build_ptpt_dictionary.py) —
// nspell expands affixes eagerly on load, and doing that to European
// Portuguese's 1,340 rules in a browser wants over a gigabyte of heap. The
// forms are computed at build time instead, so this is the same nspell reading
// a bigger, simpler file.
const PAIR_DICTS: Partial<Record<PairLang, DictSpec>> = {
'pt-PT': {
lang: 'pt-PT',
aff: '/dictionaries/pt-PT/pt-PT.aff',
dic: '/dictionaries/pt-PT/pt-PT.dic.gz',
gzipped: true,
extendedAlphabet: true,
},
}
// Where the list lived before it had an owner (Phase 7). Read once, handed to
// the account, and then removed — see takeLegacyWords.
// the account, and then removed — see readLegacyWords.
const LEGACY_KEY = 'petal.spell.personal'
// takeLegacyWords reads the pre-account list without deleting it: the words are
// readLegacyWords reads the pre-account list without deleting it: the words are
// only dropped from the browser once the server has actually accepted them, so
// a failed request costs nothing.
function readLegacyWords(): string[] {
@@ -52,72 +100,197 @@ function clearLegacyWords() {
}
}
export function useSpellChecker() {
const spellRef = useRef<NSpell | null>(null)
const [ready, setReady] = useState(false)
// Bumped whenever the personal dictionary changes, to force re-decoration.
const [version, setVersion] = useState(0)
// fetchText retrieves a dictionary file, inflating it when it ships gzipped.
//
// DecompressionStream is used rather than a bundled inflate because it costs no
// bytes and has been in every browser since 2023. A browser without it gets an
// exception, which the caller treats as "this dictionary didn't load" — the same
// outcome as a failed fetch.
async function fetchText(url: string, gzipped: boolean | undefined): Promise<string> {
const res = await fetch(url)
if (!res.ok) throw new Error(`${url}: ${res.status}`)
if (!gzipped) return res.text()
if (!res.body) throw new Error(`${url}: no body to inflate`)
const stream = res.body.pipeThrough(new DecompressionStream('gzip'))
return new Response(stream).text()
}
// Dictionary is the three methods Petal asks of nspell. Naming them rather than
// referring to NSpell is what lets the decision logic below be exercised without
// a 15 MB word list behind it; a real nspell instance satisfies this as-is.
export interface Dictionary {
correct(word: string): boolean
suggest(word: string): string[]
add(word: string): unknown
}
// A dictionary that finished loading, with the language its personal words
// belong to.
export interface Loaded {
lang: string
spell: Dictionary
extendedAlphabet: boolean
}
// load builds one nspell instance and replays this writer's personal words into
// it. `adoptLegacy` is passed only for English, and only on first load.
async function load(spec: DictSpec, adoptLegacy: boolean): Promise<Loaded> {
const [aff, dic] = await Promise.all([
fetchText(spec.aff, false),
fetchText(spec.dic, spec.gzipped),
])
const spell = nspell(aff, dic)
// Her own words come from her account. A browser holding a list from before
// accounts existed hands it over on the way — but only lets go of it once the
// server has taken it.
const legacy = adoptLegacy ? readLegacyWords() : []
const stored = legacy.length
? await api.addPersonalWords(spec.lang, legacy).then((res) => {
clearLegacyWords()
return res
})
: await api.listPersonalWords(spec.lang)
for (const w of stored.words) spell.add(w)
return { lang: spec.lang, spell, extendedAlphabet: spec.extendedAlphabet ?? false }
}
// interleave merges each dictionary's corrections round-robin. Concatenating
// would let English fill all five pills and leave a misspelt Portuguese word
// with no Portuguese suggestion, which is the case the second dictionary exists
// for.
export function interleave(lists: string[][]): string[] {
const out: string[] = []
const seen = new Set<string>()
for (let i = 0; i < Math.max(...lists.map((l) => l.length), 0); i++) {
for (const list of lists) {
const word = list[i]
if (word && !seen.has(word)) {
seen.add(word)
out.push(word)
}
}
}
return out
}
// combine is the both-dictionaries rule itself, separated from the loading so it
// can be reasoned about (and tested) on its own. `dicts` is a getter because the
// writer's dictionary lands after English and the checker must see it when it
// does, without being rebuilt around a stale array.
export function combine(dicts: () => Loaded[]): SpellChecker {
return {
// Flag only what *every* loaded dictionary rejects. With one dictionary this
// is exactly the pre-Phase-21 behaviour; with two it is the rule from
// SUGGESTIONS.md §3a. No dictionary at all accepts everything — an editor
// that underlines every word because a fetch failed is worse than one that
// underlines nothing.
correct: (w) => {
const loaded = dicts()
if (loaded.length === 0) return true
return loaded.some((d) => d.spell.correct(w))
},
suggest: (w) => interleave(dicts().map((d) => d.spell.suggest(w))),
// A getter, not a snapshot. The alphabet is read by every surface that
// resolves a word, and if it were captured when the checker was built it
// would still say A-Z after her dictionary arrived — so a pt-PT writer's
// first lookups would silently be of "cora" while the underlines were
// already right.
get extendedAlphabet() {
return dicts().some((d) => d.extendedAlphabet)
},
}
}
export function useSpellChecker() {
const pack = usePack()
const pairSpec = PAIR_DICTS[pack.code]
const loadedRef = useRef<Loaded[]>([])
// Bumped whenever a dictionary arrives or the personal list changes, to force
// the editor to re-decorate.
const [version, setVersion] = useState(0)
const [ready, setReady] = useState(false)
// English first, once per session — it is never reloaded, whatever the pair.
useEffect(() => {
let cancelled = false
;(async () => {
try {
// Served from web/dist root (and embedded in the Go binary), same as /api.
const [aff, dic] = await Promise.all([
fetch('/dictionaries/en/en.aff').then((r) => r.text()),
fetch('/dictionaries/en/en.dic').then((r) => r.text()),
])
load(EN, true)
.then((loaded) => {
if (cancelled) return
const sp = nspell(aff, dic)
spellRef.current = sp
loadedRef.current = [...loadedRef.current.filter((l) => l.lang !== EN.lang), loaded]
setReady(true)
// Her own words come from her account. A browser holding a list from
// before accounts existed hands it over on the way — but only lets go of
// it once the server has taken it.
const legacy = readLegacyWords()
const stored = legacy.length
? await api.addPersonalWords(DICT_LANG, legacy).then((res) => {
clearLegacyWords()
return res
})
: await api.listPersonalWords(DICT_LANG)
if (cancelled) return
for (const w of stored.words) sp.add(w)
setVersion((v) => v + 1)
} catch (err) {
// A failure here costs correct words being flagged, not writing. The
// checker itself stays usable if only the word list failed to arrive.
})
.catch((err) => {
// A failure here costs correct words being flagged, not writing.
console.error('spell checker failed to load', err)
}
})()
})
return () => {
cancelled = true
}
}, [])
// Recreate the checker's identity on load and on every personal-dict change so
// the editor's effect re-pushes it and rebuilds decorations.
// The writer's own dictionary, once her pair is known. `pack.code` is 'zh'
// until /api/me answers, so a pt-PT writer loads this a moment after the
// editor is already usable — which is the right order: the English half works
// immediately and hers fills in.
useEffect(() => {
if (!pairSpec) {
// Switching away from a pair (only tests do this today) must not leave the
// old language's words still being accepted.
loadedRef.current = loadedRef.current.filter((l) => l.lang === EN.lang)
setVersion((v) => v + 1)
return
}
let cancelled = false
load(pairSpec, false)
.then((loaded) => {
if (cancelled) return
loadedRef.current = [...loadedRef.current.filter((l) => l.lang !== loaded.lang), loaded]
setVersion((v) => v + 1)
})
.catch((err) => {
// Only her half failed. English still checks, and the consequence is
// that correct Portuguese gets underlined — worth a console line, not
// worth blocking the editor.
console.error(`spell checker failed to load ${pairSpec.lang}`, err)
})
return () => {
cancelled = true
}
}, [pairSpec])
// The checker's identity changes on every load and every added word, so the
// editor's effect re-pushes it and rebuilds decorations.
const checker = useMemo<SpellChecker | null>(() => {
if (!ready) return null
return {
correct: (w) => spellRef.current?.correct(w) ?? true,
suggest: (w) => spellRef.current?.suggest(w) ?? [],
}
// Read through the ref rather than closing over a snapshot: the pair
// dictionary arrives after English, and `version` is what re-runs this.
return combine(() => loadedRef.current)
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [ready, version])
// Adding a word takes effect in the editor immediately and is persisted in the
// background: the word stops being underlined the instant she asks, whatever
// the network is doing.
//
// It is written to every loaded dictionary's list. Under the both-dictionaries
// rule a word is only ever flagged when *all* of them rejected it, so
// accepting it is a statement about her pair rather than about one language —
// and recording it against only one would silently unaccept it if the other
// dictionary is the one still loaded next time.
const addWord = useCallback((word: string) => {
const sp = spellRef.current
if (!sp) return
sp.add(word)
const dicts = loadedRef.current
if (dicts.length === 0) return
for (const d of dicts) {
d.spell.add(word)
api.addPersonalWords(d.lang, [word]).catch((err) => {
console.error('could not save personal word', err)
})
}
setVersion((v) => v + 1)
api.addPersonalWords(DICT_LANG, [word]).catch((err) => {
console.error('could not save personal word', err)
})
}, [])
return { checker, ready, addWord }