Record the Spanish rebuild and how to do the next one

dict.db now holds all five languages on the VPS. The deploy notes carry the
recipe, the two traps (SUBTLEX-US fails harmlessly; check the other languages'
counts are unchanged before shipping) and the measured coverage, so the next
rebuild is not archaeology.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua

BUILD_PLAN.md

@@ -227,11 +227,14 @@ Option 3 ratified (import package, read-only `dict.db`).
 - Verified: go build/vet/test, tsc, vite build, vitest 96/96 clean, both repos. Live smoke on a throwaway DB (:8075) against the real 452 MB `dict.db` — startup logs the languages it actually got, zh unchanged, then the same instance flipped to pt-PT and re-queried.
 - [x] Deploy documented (`deploy/README.md` §4b, `DICT_PATH` through Dockerfile/compose/.env.example): `dict.db` ships into the data dir, stays out of the backups because they name `petal.db` explicitly, and is rebuildable from public data.
 - [x] **Deployed 2026-07-27**, with Phase 19. The two dreamdict commits went to GitHub (`prosolis/dreamdict` main), the `replace` came out for a real pseudo-version, and Petal shipped as a rebuild — no migration in either phase, so `schema_migrations` is still 11. Order: off-box encrypted backup first (`✓ petal.db.age`), then `dict.db` copied into the LUKS volume and **SHA-256 verified end to end**, then pull + `docker compose up -d --build`. All three containers healthy; startup logs `dictionary: DreamDict open at /data/dict.db ([en fr pt-PT es zh])`, so the deployed binary opened the deployed file and found every language. Over public HTTPS: `/api/health` 200, `/api/docs`, `/api/word/…` and `/api/gloss/…` all **401 without a session**. Her writing untouched — 2 users, 8 documents, 33 versions, 103 suggestions, 3 vocabulary words, 1 image, FTS still matching, `integrity_check` ok. Both accounts are on the zh pair, so **nothing about her experience changed today**; what shipped is the capacity for the next pair.
-- ⚠️ **One item outstanding.** The deployed `dict.db` is from **2026-04-04**: no Spanish data, and none of the intervening loader fixes. Rebuild it (needs the es sources downloaded on millenia, ~3–4 GB, then `go run ./cmd/dictimport`) before the Spanish pair ships. It does not block pt-PT or French.
+- [x] **`dict.db` rebuilt with Spanish and deployed** (2026-07-27, user: "if we need to redeploy DreamDict to add Spanish support, then do so"). Millenia's dreamdict checkout turned out to carry ~490 lines of **uncommitted** changes; checking rather than pulling over them showed an *earlier draft* of the regional-variant work since committed upstream (main has the reviewed `wordListQuery` refactor, millenia the pre-refactor `Words`) — nothing unique at risk, but not mine to discard, so that checkout was left untouched and the build ran from a clean clone. Import: 6m15s, `es` 102,971 words / 71,680 definitions, **every other language byte-identical to April** — which is what says a language was added rather than the rest quietly shifted. Coverage of the 2,000 commonest English words: **es 68.6%** (best of the four), fr 63.1%, pt-PT 62.1% (both unchanged), **zh 53.2% — re-measured, still under ECDICT, so Chinese stays put**. Shipped millenia→parodia direct over headscale, SHA-256 verified both ends, April file kept as `dict.db.april-backup`.
+- **The startup log was lying, and the rebuild is what exposed it.** It printed `dictionary.Langs()` — a compile-time constant of the languages DreamDict *supports* — so it had been reporting a confident `[en fr pt-PT es zh]` over the April file that contained no Spanish at all: exactly the failure the line existed to catch, rendered as success. It now counts rows (`en=136615 es=102971 fr=56096 pt-PT=136300 zh=120883`), with a test asserting a fixture holding only two languages cannot name five. For a file somebody copies onto the box by hand, "what is in it" is the only question worth asking.
+- A second thing worth recording from the rebuild: the SUBTLEX-US download now fails (source moved behind a manual export), which looked like it would silently cost English frequency data. It doesn't — the loader falls back to `SUBTLEX-US.txt`. Chasing it down showed English "frequency" is mostly **SCOWL's commonness bucket** (1000/800/600/…/50), refined by SUBTLEX for ~1,600 words — which is the quantised distribution measured earlier, and independent confirmation that the band chip was right to read `difficulty` rather than `frequency`.
 - ⚠️ As in Phase 18, the authenticated live probe — seeing `/api/word` answer 200 for a real cookie — was **not performed**: minting a session row reads as credential fabrication to this session's classifier. The lookup path was smoke-tested end to end against this exact `dict.db` (same SHA-256) on a throwaway instance, including the zh→pt-PT flip, and the shared middleware governs that last step for every other route.
 
 ### Phase 21 — The pt-PT pair (first Latin pair, proves the model)
 SUGGESTIONS §1/§3/§3a. French and **Spanish** follow the same groove afterwards — es is no longer gated now that DreamDict has Spanish data (2026-07-26). pt-PT still goes first: it's the pair with a real user behind it, and it's the one that proves the langpack + both-dictionaries model.
+Phase 20 left this ready: `dict.db` on the VPS now holds all five languages, and pt-PT gloss coverage of common English words is 62%.
 - [ ] Hunspell pt-PT vendored like en-US; **both-dictionaries spellcheck** (flag only if wrong in both; pills from both) — the no-detector stance, Q1 settled
 - [ ] Gloss/WordCard both directions; on en/pt collisions show both compactly, never hide either
 - [ ] Prompts pinned to **European Portuguese, never pt-BR** (explicit in every prompt); pt-PT langpack copy written and **reviewed by a pt-PT speaker before trusted**
@@ -262,6 +265,7 @@ Each item independent and small; order within is free (SUGGESTIONS §5–§6).
 - [x] **Phase 14 — companion warmth + bedtime nag + night mode**: more encouraging phrases, a gentle "go to bed" nudge after 11pm, and a calm dark theme + falling stars at night. ✅ (see Phase 14 above)
 
 ## Session log
+- 2026-07-27: **dict.db rebuilt with Spanish, and a log line caught lying** (user: "if we need to redeploy DreamDict to add Spanish support, then do so"). Millenia's dreamdict checkout held ~490 lines of uncommitted work; rather than pull over it, comparing file contents showed an earlier draft of the regional-variant work already committed upstream — nothing unique, but not mine to discard, so it was left alone and the rebuild ran from a clean clone pushed over from the laptop (millenia has no GitHub SSH). Import took 6m15s and added **es: 102,971 words**, leaving en/fr/pt-PT/zh byte-identical — the check that distinguishes "added a language" from "quietly changed everything". Coverage measured before shipping: **es 68.6%**, the best of the four; **zh re-measured at 53.2%**, so the ECDICT decision stands on fresh evidence rather than on the earlier number. Shipped direct millenia→parodia over headscale, hashed both ends, kept the April file for rollback. **The rebuild's real find was in Petal, not DreamDict**: the startup line reported `dictionary.Langs()`, a compile-time constant of *supported* languages, so it had been printing a cheerful `[en fr pt-PT es zh]` over a database with no Spanish in it — the exact failure it existed to catch, reported as success, and something I had already claimed as proof the deploy was good. It now counts rows. Chasing a failed SUBTLEX-US download (benign — the loader falls back to `.txt`) also confirmed English "frequency" is mostly SCOWL's commonness bucket, which independently vindicates the band chip reading `difficulty` instead.
 - 2026-07-27: **Phase 20 — DreamDict becomes the dictionary for every pair but Chinese** (user: "let's continue the build plan"; scope confirmed: build the seam against the existing April `dict.db`, rebuild it later, code + local verification only). The prerequisite was bigger than the plan recorded: renaming DreamDict's module path was necessary but useless on its own, because the query layer lived in `internal/dictionary` and no module may import another's `internal`. Both fixed upstream — the package is now `dictionary`, with a comment saying why *reading* a built database is public API while the loaders that build one stay internal. In Petal, `Provider` is the two questions the popover already asked, so the embedded `*Lexicon` satisfied it with no changes at all, and `Set.For(lang)` is the one place the choice is made. **The measurement is the story of the phase.** `MULTIUSER_PLAN.md` mapped `Gloss ← Translate(word, "en", L1)` 1:1; against the real 452 MB database that table answers for **17%** of the 2,000 commonest English words into pt-PT. Wiktionary's translation sections are thin in that direction — "ephemeral", "think" and "quickly" have no en→pt-PT row at all. The shared-synset path answers for **61%**, so a new upstream `Equivalents` queries that and falls back to translations for 62% combined. Then the *ordering* was wrong in an instructive way: sorting by target frequency glosses "think" as *lembrar* — "remember" — because lembrar is commoner in Portuguese, even though pensar shares six of think's synsets to lembrar's one. Counting sense agreement first fixes it (think → pensar; write → escrever; garden → jardim). The same measurement is what kept **zh on ECDICT**: DreamDict reaches a Chinese gloss for 53% of those words where ECDICT reaches nearly all — the plan said converge only if quality holds, and it didn't. Two other decisions worth keeping: a missing `dict.db` is **not an error** (a laptop has never had one) but a present-and-unimported one is; and a pt-PT writer without a dictionary falls back to the embedded datasets **with the gloss suppressed**, keeping the English half rather than blanking the popover — an empty field reads as "not found", the wrong language reads as broken. The new fields surface as **three** bands, not five, because the difficulty score can separate "everyday" from "you'll have to explain this" but cannot rank *obfuscate* against *serendipity*, and a finer scale would be a confident-looking lie. Writing the tests found two bugs first: `trimEtymology` sliced by byte, which would have emitted invalid UTF-8 for precisely the Greek and Latin etymologies the feature exists for, and its ellipsis path overran its own cap. go build/vet/test, tsc, vite, vitest 96/96 clean in both repos; live smoke on a throwaway DB against the real dictionary, one instance flipped from zh to pt-PT mid-run. **Then deployed, with Phase 19** (user: "do it"): dreamdict pushed to GitHub, the `replace` swapped for a real pseudo-version, encrypted off-box backup first, `dict.db` copied into the LUKS volume and SHA-256-verified, then a rebuild — no migration in either phase, so `schema_migrations` stayed at 11 and her writing came through untouched (8 documents, 33 versions, 103 suggestions, FTS matching, integrity ok). Both accounts are on the zh pair, so **nothing she sees changed today**; what shipped is the capacity for the next pair. Outstanding: the deployed `dict.db` predates DreamDict's Spanish data and needs rebuilding before the es pair ships.
 - 2026-07-27: **Phase 18 deployed, and Phase 19 — the copy stops being hardcoded Mandarin** (user: "let's continue the build plan"; sequencing confirmed: rehearse + deploy 18, then start 19). The rehearsal the previous session was blocked from running went first: a `VACUUM INTO` snapshot of the live VPS database, migrated locally by the Phase-18 binary, every count unchanged and FTS/integrity/foreign keys clean, `personal_words` created empty — then the deploy itself (off-box encrypted backup, rebuild, all three containers healthy, `0011` applied to the live DB with her writing untouched, `/api/spell/words` 401 without a session over public HTTPS). **One check was refused and not worked around**: minting a probe session row to see the endpoint answer 200 for a real cookie reads as credential fabrication to this session's classifier; the endpoint's lifecycle is covered by tests and the shared middleware governs that last step for every other route. **Phase 19** then lifted every `中文 · English` literal out of ~29 files into `web/src/i18n` — one `Pack` type, a verbatim `zh` pack, and two access paths chosen by *when* copy is built: `usePack()` for components, `pack()` for the companion and prose checker, which compose a line when something happens rather than when something renders. The interesting decisions were about what a pack must be allowed to control: **every string with a value in it is a function** (`reviewDue(n)`, `daysAgo(n)`, even English pluralisation) because word order isn't universal; the roster constants keep only value + emoji so a label can never drift from its key; and `gradeBand` returns a band *name* rather than a label. On the server, `internal/llm/lang.go` replaces "Simplified Chinese" in the three prompts that name her language — with pt-PT spelled **"European Portuguese (pt-PT, never Brazilian Portuguese)"** in the prompt itself, and her word for "why" carried alongside so the tutor still recognises the question. `pair_lang` is read **in the row-scoped query each handler already ran**, not a second lookup that could disagree with it — and the test for that was checked by breaking the join and watching it fail. go build/vet/test, tsc, vite, vitest 90/90 clean; live smoke on a throwaway DB. **Phase 19 is not deployed** — no migration, so it's a rebuild whenever the user wants it.
 - 2026-07-27: **Phase 18 — the browser's settings become her settings** (user: "let's continue the build plan"). Two scope calls taken with the user: the personal spell dictionary goes **server-side** rather than being namespaced in place, and the pre-account `localStorage` keys are **adopted then rescoped** by the first writer to sign in. New `web/src/lib/prefs.ts` namespaces `petal.sound`/`petal.petals`/`petal.companion` by user id; the interesting part is timing — those modules read their value at *import* time, before `/api/me` can possibly have answered, so a pre-scope read deliberately sees the legacy key (the right value on a single-writer browser) and `setPrefsScope`, called from `useSession`, adopts it and notifies every reader. Adoption **moves** rather than copies, so account two starts from Petal's defaults instead of inheriting a stranger's mascot. New `internal/spell` package + migration `0011`: `personal_words` keyed `(user_id, lang, word)`, where `lang` is the **dictionary's** language, not the writer's — an English exception must not silence a pt-PT flag when the second pair ships. `useSpellChecker` now replays her list from her account, hands over any browser-held Phase-7 list on first load (releasing it only once the server has taken it), and persists an added word in the background so the underline vanishes the instant she asks. **The reason for the server table over cheaper namespacing**: keying the existing list by user in `localStorage` would have *fragmented* the words she already has across her laptop and tablet — the "cheap" fix was the one that made things worse. Tests: full lifecycle + languages-don't-merge + junk + the standing-rule two-user isolation suite in Go, and legacy-adoption/move-not-copy/two-accounts/storage-throws in vitest. go build/vet/test, tsc, vite, vitest 82/82 clean; live smoke on a throwaway DB. **Not deployed** — and the customary rehearsal of `0011` against a copy of the live VPS database was blocked by the session's permission classifier, so that check is outstanding (it is a plain `CREATE TABLE`, so lower-risk than `0005`/`0010`, but the convention exists for a reason).