Phase 20: the dictionary stops being English and Chinese only

Word lookups now come from DreamDict's dict.db for every pair but Chinese —
opened read-only beside petal.db, no service, nothing over the VPN, because a
hover gloss has to answer in milliseconds.

`Provider` is the two questions the popover and the tooltip already asked, so
the embedded *Lexicon satisfies it with no changes at all; Set.For(lang) is the
single place the choice between them is made. The prerequisite in the dreamdict
repo turned out to be two things, not one: the module path was unfetchable
*and* the query layer sat in internal/, which no other module may import
whatever the module is called. Both fixed upstream.

The plan's central assumption did not survive the data. It mapped
Gloss ← Translate(word, "en", L1) one-to-one; against the real 452 MB database
that table answers for 17% of the 2,000 commonest English words into pt-PT.
Wiktionary's translation sections are thin in that direction — "ephemeral",
"think" and "quickly" have no en→pt-PT row at all. Shared WordNet synsets
answer for 61%, so DreamDict gained Equivalents() and Petal glosses through it.
Ordering those was wrong in an instructive way too: sorting by frequency
glosses "think" as lembrar, "remember", because lembrar is the commoner
Portuguese word even though pensar shares six of think's synsets to lembrar's
one. Counting sense agreement first asks the right question.

The same measurement is why zh stays on ECDICT: DreamDict reaches a Chinese
gloss for 53% of those words, ECDICT for nearly all of them. The plan said
converge only if quality holds. It didn't, so nothing converged.

Two decisions about failure worth keeping. A missing dict.db is not an error —
a laptop checkout has never had one — but a present-and-never-imported one is,
because that is a half-finished deploy. And a pt-PT writer with no dictionary
falls back to the embedded datasets with the gloss suppressed, keeping
definitions, synonyms and phonetics rather than blanking the popover: an empty
field reads as "not found", the wrong language reads as broken.

The new fields surface as an etymology line and a three-band chip. Three, not
five: the difficulty score separates "everyday" from "you'll have to explain
this" but cannot rank obfuscate against serendipity, and a finer scale would be
a confident-looking lie. An unscored word gets no chip.

Writing the tests found two bugs first — trimEtymology sliced by byte, which
would have emitted invalid UTF-8 for exactly the Greek and Latin etymologies
the feature exists for, and its ellipsis path overran its own cap.

go build/vet/test, tsc, vite, vitest 96/96 clean; live smoke against the real
dict.db with one instance flipped from zh to pt-PT mid-run.

Not deployed: go.mod still replaces github.com/prosolis/dreamdict with
../dreamdict, so the Docker build needs the two upstream commits pushed and the
replace dropped. The deployed dict.db also predates DreamDict's Spanish data.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua

BUILD_PLAN.md

@@ -211,13 +211,22 @@ Pure refactor, zero visible change; prerequisite for every new pair (SUGGESTIONS
 - Verified: go build/vet/test, tsc, vite build, vitest 90/90 clean; live smoke on a throwaway DB (:8074) — doc create/save, search, md export, spell add, warm 502 from the collocation pass with the LLM down; the built bundle still carries the zh strings.
 - ⚠️ **Not yet deployed.** Pure refactor with no migration, so the deploy is a rebuild; still unshipped at the end of the session.
 
-### Phase 20 — DreamDict as a lexicon provider
-Option 3 ratified (import package, read-only `dict.db`). **Prerequisite in the dreamdict repo:** rename its module path (or add a `replace` for dev).
-- [ ] Provider seam behind the existing `lexicon` interface; DreamDict provider opens `dict.db` read-only (modernc driver, second handle beside `petal.db`); graceful "no data" when the file is absent
-- [ ] pt-PT + fr + **es** wired to DreamDict (nothing to regress); **zh stays on ECDICT** until compared on real lookups from her documents — converge only if quality holds
-  - **es is no longer gated** (2026-07-26): DreamDict grew Spanish support, so the "maybe es" in the pair model is now a real option and the provider seam should cover it from day one — it costs nothing here and saves re-opening the package later.
-- [ ] Surface the new fields where cheap: frequency/difficulty chip in WordCard; etymology line (cognate hook for en-natives)
-- [ ] Deploy: `dict.db` ships in the data dir alongside `petal.db`
+### Phase 20 — DreamDict as a lexicon provider ✅ (2026-07-27)
+Option 3 ratified (import package, read-only `dict.db`).
+- [x] **The prerequisite was bigger than the plan thought.** Renaming the module was necessary but not sufficient: DreamDict's query layer lived in `internal/dictionary`, which no other module may import whatever the module is called. Both fixed upstream in one commit — `module github.com/prosolis/dreamdict`, `internal/dictionary` → `dictionary`, with a package comment saying why reading a built database is public API while building one stays internal. `internal/loader` is untouched, and DreamDict's own tests pass unchanged.
+- [x] **Provider seam** (`internal/lexicon/provider.go`): a `Provider` is the two questions the popover and the tooltip have always asked (`Lookup`, `Gloss`), which the embedded `*Lexicon` already satisfied unmodified. `Set.For(lang)` is the single place the choice is made. `OpenDreamDict` opens `dict.db` read-only beside `petal.db`; **a missing file returns `(nil, nil)`, not an error** — a laptop checkout has never had one — while a file that is *present but unimported* does error, because that one is somebody's half-finished deploy.
+- [x] **The absent-dictionary case degrades better than "no data".** A pt-PT writer with no `dict.db` falls back to the embedded datasets **with the gloss suppressed** (`glossless`), so she keeps English definitions, synonyms and phonetics — all compiled into the binary and all correct for her — and loses only the translation. Handing her the Chinese gloss would be worse than handing her nothing: empty reads as "not found", wrong-language reads as Petal being broken.
+- [x] pt-PT + fr + **es** wired to DreamDict; **zh stays on ECDICT**, and the routing test is the guard on that decision. The comparison the plan asked for was run against the real 452 MB database: DreamDict reaches a Chinese gloss for **53%** of the 2,000 commonest English words, against ECDICT's essentially total coverage of them. Quality did not hold, so nothing converged. es routes to DreamDict from day one and simply finds no rows in the April build — which is the same code path as any unglossed word.
+- [x] **The plan's central assumption was wrong, and measuring it is what found that.** `Gloss ← Translate(word, "en", L1)` was mapped 1:1 in `MULTIUSER_PLAN.md`; against real data that table answers for **17%** of common English words into pt-PT (16% into fr). Wiktionary's translation sections are thin in the en→X direction. Going through shared Princeton WordNet synset ids instead answers for **61%**, and it is where the words a learner wants live — "ephemeral", "think" and "quickly" have no en→pt-PT translation row at all. New `dictionary.Equivalents(word, from, to)` upstream does that, falling back to the translations table, for **62%** combined. A gloss absent five times in six is not a gloss.
+- [x] **Ranking, argued from a wrong answer.** Ordering equivalents by target-word frequency glosses "think" as *lembrar* — "remember" — because lembrar is the commoner Portuguese word even though pensar shares six of think's synsets to lembrar's one. Counting shared senses first, frequency second, asks which candidate means the same thing *most often*: think → pensar; achar; lembrar, write → escrever, garden → jardim, house → casa before firma.
+- [x] The de-inflection walk (`candidates`) is shared with the embedded path, because `dict.db` stores headwords — "running" has no row. The first candidate that *has definitions* becomes the headword every other field is read from, so one popover never mixes "running"'s frequency with "run"'s senses. The gloss walks separately, since a word can have an equivalent and no definition.
+- [x] **Surfaced where cheap**: a band chip beside the phonetic (`wordband.ts`) and an etymology line at the foot of the card. Following Phase 19, `wordBand` returns a band *name* and the langpack owns the wording. **Three bands, not five** — the difficulty score is a heuristic over length and corpus counts, good enough to separate "everyday" from "you will need to explain this" and not good enough to rank *obfuscate* against *serendipity*; a finer scale would be a confident-looking lie. Thresholds come from the real distribution (136k headwords bunch between 0.45 and 0.60; the words a writer reaches for sit under 0.40). An unscored word renders **no chip at all**.
+- [x] The pair language is read **per request** in `providerFor` — a word lookup has no row-scoped query to piggyback on, unlike Phase 19's handlers — and a failed read falls back to today's embedded behaviour rather than failing the lookup. `Cache-Control` dropped from `public` to `private`: the same URL now answers in a different language per writer.
+- Tests: `internal/lexicon/dreamdict_test.go` — fixture is a **real dict.db on disk**, so open/stat/seeded is the production path; missing vs. unseeded vs. unreadable, every field filled, gloss follows the writer not the word (pt-PT/fr/es/de), the synset path, de-inflection carrying *all* fields to one headword, miss-is-not-an-error, the NULL-difficulty sentinel, IPA chosen over CMU, and the handler tests (two writers/one URL/two languages, unknown caller, private caching). Upstream: `Equivalents` ordering, synset-over-translation, fallback. Frontend: `wordband.test.ts` (bands pinned to real scores; difficulty 0.0 is a score, not a missing value) and an i18n assertion that no band can be unlabelled.
+- **Two bugs the tests found before the browser did**: `trimEtymology` sliced by byte, which would put invalid UTF-8 in the JSON for exactly the etymologies that matter (ἐφήμερος, ephemerus), and its ellipsis path overran its own cap.
+- Verified: go build/vet/test, tsc, vite build, vitest 96/96 clean, both repos. Live smoke on a throwaway DB (:8075) against the real 452 MB `dict.db` — startup logs the languages it actually got, zh unchanged, then the same instance flipped to pt-PT and re-queried.
+- [x] Deploy documented (`deploy/README.md` §4b, `DICT_PATH` through Dockerfile/compose/.env.example): `dict.db` ships into the data dir, stays out of the backups because they name `petal.db` explicitly, and is rebuildable from public data.
+- ⚠️ **Not deployed, and two things are outstanding.** (1) `go.mod` carries a `replace` to `../dreamdict`; the Docker build needs the two dreamdict commits **pushed** and the replace dropped. (2) The deployed `dict.db` is from **2026-04-04** and has **no Spanish data** and none of the intervening loader fixes — rebuild it (needs the es sources downloaded on millenia) before the Spanish pair ships. Neither blocks pt-PT or French.
 
 ### Phase 21 — The pt-PT pair (first Latin pair, proves the model)
 SUGGESTIONS §1/§3/§3a. French and **Spanish** follow the same groove afterwards — es is no longer gated now that DreamDict has Spanish data (2026-07-26). pt-PT still goes first: it's the pair with a real user behind it, and it's the one that proves the langpack + both-dictionaries model.
@@ -251,6 +260,7 @@ Each item independent and small; order within is free (SUGGESTIONS §5–§6).
 - [x] **Phase 14 — companion warmth + bedtime nag + night mode**: more encouraging phrases, a gentle "go to bed" nudge after 11pm, and a calm dark theme + falling stars at night. ✅ (see Phase 14 above)
 
 ## Session log
+- 2026-07-27: **Phase 20 — DreamDict becomes the dictionary for every pair but Chinese** (user: "let's continue the build plan"; scope confirmed: build the seam against the existing April `dict.db`, rebuild it later, code + local verification only). The prerequisite was bigger than the plan recorded: renaming DreamDict's module path was necessary but useless on its own, because the query layer lived in `internal/dictionary` and no module may import another's `internal`. Both fixed upstream — the package is now `dictionary`, with a comment saying why *reading* a built database is public API while the loaders that build one stay internal. In Petal, `Provider` is the two questions the popover already asked, so the embedded `*Lexicon` satisfied it with no changes at all, and `Set.For(lang)` is the one place the choice is made. **The measurement is the story of the phase.** `MULTIUSER_PLAN.md` mapped `Gloss ← Translate(word, "en", L1)` 1:1; against the real 452 MB database that table answers for **17%** of the 2,000 commonest English words into pt-PT. Wiktionary's translation sections are thin in that direction — "ephemeral", "think" and "quickly" have no en→pt-PT row at all. The shared-synset path answers for **61%**, so a new upstream `Equivalents` queries that and falls back to translations for 62% combined. Then the *ordering* was wrong in an instructive way: sorting by target frequency glosses "think" as *lembrar* — "remember" — because lembrar is commoner in Portuguese, even though pensar shares six of think's synsets to lembrar's one. Counting sense agreement first fixes it (think → pensar; write → escrever; garden → jardim). The same measurement is what kept **zh on ECDICT**: DreamDict reaches a Chinese gloss for 53% of those words where ECDICT reaches nearly all — the plan said converge only if quality holds, and it didn't. Two other decisions worth keeping: a missing `dict.db` is **not an error** (a laptop has never had one) but a present-and-unimported one is; and a pt-PT writer without a dictionary falls back to the embedded datasets **with the gloss suppressed**, keeping the English half rather than blanking the popover — an empty field reads as "not found", the wrong language reads as broken. The new fields surface as **three** bands, not five, because the difficulty score can separate "everyday" from "you'll have to explain this" but cannot rank *obfuscate* against *serendipity*, and a finer scale would be a confident-looking lie. Writing the tests found two bugs first: `trimEtymology` sliced by byte, which would have emitted invalid UTF-8 for precisely the Greek and Latin etymologies the feature exists for, and its ellipsis path overran its own cap. go build/vet/test, tsc, vite, vitest 96/96 clean in both repos; live smoke on a throwaway DB against the real dictionary, one instance flipped from zh to pt-PT mid-run. **Not deployed**: `go.mod` still carries a `replace` to `../dreamdict` (the two upstream commits need pushing first), and the deployed `dict.db` predates DreamDict's Spanish support and needs rebuilding before the es pair ships.
 - 2026-07-27: **Phase 18 deployed, and Phase 19 — the copy stops being hardcoded Mandarin** (user: "let's continue the build plan"; sequencing confirmed: rehearse + deploy 18, then start 19). The rehearsal the previous session was blocked from running went first: a `VACUUM INTO` snapshot of the live VPS database, migrated locally by the Phase-18 binary, every count unchanged and FTS/integrity/foreign keys clean, `personal_words` created empty — then the deploy itself (off-box encrypted backup, rebuild, all three containers healthy, `0011` applied to the live DB with her writing untouched, `/api/spell/words` 401 without a session over public HTTPS). **One check was refused and not worked around**: minting a probe session row to see the endpoint answer 200 for a real cookie reads as credential fabrication to this session's classifier; the endpoint's lifecycle is covered by tests and the shared middleware governs that last step for every other route. **Phase 19** then lifted every `中文 · English` literal out of ~29 files into `web/src/i18n` — one `Pack` type, a verbatim `zh` pack, and two access paths chosen by *when* copy is built: `usePack()` for components, `pack()` for the companion and prose checker, which compose a line when something happens rather than when something renders. The interesting decisions were about what a pack must be allowed to control: **every string with a value in it is a function** (`reviewDue(n)`, `daysAgo(n)`, even English pluralisation) because word order isn't universal; the roster constants keep only value + emoji so a label can never drift from its key; and `gradeBand` returns a band *name* rather than a label. On the server, `internal/llm/lang.go` replaces "Simplified Chinese" in the three prompts that name her language — with pt-PT spelled **"European Portuguese (pt-PT, never Brazilian Portuguese)"** in the prompt itself, and her word for "why" carried alongside so the tutor still recognises the question. `pair_lang` is read **in the row-scoped query each handler already ran**, not a second lookup that could disagree with it — and the test for that was checked by breaking the join and watching it fail. go build/vet/test, tsc, vite, vitest 90/90 clean; live smoke on a throwaway DB. **Phase 19 is not deployed** — no migration, so it's a rebuild whenever the user wants it.
 - 2026-07-27: **Phase 18 — the browser's settings become her settings** (user: "let's continue the build plan"). Two scope calls taken with the user: the personal spell dictionary goes **server-side** rather than being namespaced in place, and the pre-account `localStorage` keys are **adopted then rescoped** by the first writer to sign in. New `web/src/lib/prefs.ts` namespaces `petal.sound`/`petal.petals`/`petal.companion` by user id; the interesting part is timing — those modules read their value at *import* time, before `/api/me` can possibly have answered, so a pre-scope read deliberately sees the legacy key (the right value on a single-writer browser) and `setPrefsScope`, called from `useSession`, adopts it and notifies every reader. Adoption **moves** rather than copies, so account two starts from Petal's defaults instead of inheriting a stranger's mascot. New `internal/spell` package + migration `0011`: `personal_words` keyed `(user_id, lang, word)`, where `lang` is the **dictionary's** language, not the writer's — an English exception must not silence a pt-PT flag when the second pair ships. `useSpellChecker` now replays her list from her account, hands over any browser-held Phase-7 list on first load (releasing it only once the server has taken it), and persists an added word in the background so the underline vanishes the instant she asks. **The reason for the server table over cheaper namespacing**: keying the existing list by user in `localStorage` would have *fragmented* the words she already has across her laptop and tablet — the "cheap" fix was the one that made things worse. Tests: full lifecycle + languages-don't-merge + junk + the standing-rule two-user isolation suite in Go, and legacy-adoption/move-not-copy/two-accounts/storage-throws in vitest. go build/vet/test, tsc, vite, vitest 82/82 clean; live smoke on a throwaway DB. **Not deployed** — and the customary rehearsal of `0011` against a copy of the live VPS database was blocked by the session's permission classifier, so that check is outstanding (it is a plain `CREATE TABLE`, so lower-risk than `0005`/`0010`, but the convention exists for a reason).
 - 2026-07-27: **Phase 17 — Claire's writing moved onto her real account** (user: "claire is local user today in Petal. let's make sure to migrate existing data to her account"). `scripts/migrate_local_user.py`: dry-run by default, own `VACUUM INTO` backup, one transaction with foreign keys off, re-points `documents`/`tags`/`vocab_words`/`images`, verifies every expected row moved before committing. **The plan's stated prerequisite — "she logs in once so her sub exists" — turned out to be false**: authentik's `hashed_user_id` sub is `User.uid`, derived from her id and the instance secret, so it is readable in advance and the data could move *first*; she signs in to find her writing already there instead of to an empty Petal. Her 8 documents, 33 snapshots, 103 suggestions, 3 vocabulary words and 1 image now belong to `5f47d955…`, verified end to end over public HTTPS. Per the user's call the VPS is now canonical and millenia was left running and untouched as a frozen fallback (it diverges the moment either is written to — retire it rather than sync it). **Three bugs, each found by a different kind of contact with reality**: (1) the image backfill claims files for `local`, which stops existing after a migration — a foreign-key error inside `images.New`, which `main.go` treats as fatal, so Petal would have crash-looped on first start against a migrated database; (2) `BEGIN EXCLUSIVE` was the wrong liveness check, since in WAL mode it only conflicts with another *writer* and sails past a running-but-idle Petal — exactly the case the guard exists for; (3) the replacement, `PRAGMA locking_mode = EXCLUSIVE`, holds its lock past being reset to `NORMAL`, so on a real WAL database the script locked itself out of its own backup — invisible locally because the test file had come from `VACUUM INTO` and wasn't in WAL mode. Same shape as Phase 16's trailing-slash issuer: the fixture didn't look like production.