diff --git a/BUILD_PLAN.md b/BUILD_PLAN.md index 0c54528..5df485a 100644 --- a/BUILD_PLAN.md +++ b/BUILD_PLAN.md @@ -209,7 +209,7 @@ Pure refactor, zero visible change; prerequisite for every new pair (SUGGESTIONS - [x] Wired to `users.pair_lang` on both sides: `useSession` calls `setPackLang` the moment `/api/me` answers, and each LLM handler reads the column **in the row-scoped query it already ran** (the one that proves she owns the document) rather than in a second lookup that could disagree with it. - Tests: `internal/llm/lang_test.go` (fallback matrix; each prompt names the writer's language and *not* Chinese; the zh pair reads exactly as before), `internal/suggestions/pairlang_test.go` (the column reaches the model for collocation + translate, zh unchanged — **verified to fail when the join is removed**), `web/src/i18n/i18n.test.ts` (default before `/api/me`, fallback for an unshipped pair, no spurious notifications, verbatim sentinels, interpolation incl. plurals, no empty string anywhere in the pack, every companion/tone/style labelled). - Verified: go build/vet/test, tsc, vite build, vitest 90/90 clean; live smoke on a throwaway DB (:8074) — doc create/save, search, md export, spell add, warm 502 from the collocation pass with the LLM down; the built bundle still carries the zh strings. -- ⚠️ **Not yet deployed.** Pure refactor with no migration, so the deploy is a rebuild; still unshipped at the end of the session. +- [x] **Deployed 2026-07-27**, together with Phase 20. Pure refactor with no migration, so it was a rebuild; the langpack is live and reads exactly as before, which is the whole point of a verbatim `zh` pack. ### Phase 20 — DreamDict as a lexicon provider ✅ (2026-07-27) Option 3 ratified (import package, read-only `dict.db`). @@ -226,7 +226,9 @@ Option 3 ratified (import package, read-only `dict.db`). - **Two bugs the tests found before the browser did**: `trimEtymology` sliced by byte, which would put invalid UTF-8 in the JSON for exactly the etymologies that matter (ἐφήμερος, ephemerus), and its ellipsis path overran its own cap. - Verified: go build/vet/test, tsc, vite build, vitest 96/96 clean, both repos. Live smoke on a throwaway DB (:8075) against the real 452 MB `dict.db` — startup logs the languages it actually got, zh unchanged, then the same instance flipped to pt-PT and re-queried. - [x] Deploy documented (`deploy/README.md` §4b, `DICT_PATH` through Dockerfile/compose/.env.example): `dict.db` ships into the data dir, stays out of the backups because they name `petal.db` explicitly, and is rebuildable from public data. -- ⚠️ **Not deployed, and two things are outstanding.** (1) `go.mod` carries a `replace` to `../dreamdict`; the Docker build needs the two dreamdict commits **pushed** and the replace dropped. (2) The deployed `dict.db` is from **2026-04-04** and has **no Spanish data** and none of the intervening loader fixes — rebuild it (needs the es sources downloaded on millenia) before the Spanish pair ships. Neither blocks pt-PT or French. +- [x] **Deployed 2026-07-27**, with Phase 19. The two dreamdict commits went to GitHub (`prosolis/dreamdict` main), the `replace` came out for a real pseudo-version, and Petal shipped as a rebuild — no migration in either phase, so `schema_migrations` is still 11. Order: off-box encrypted backup first (`✓ petal.db.age`), then `dict.db` copied into the LUKS volume and **SHA-256 verified end to end**, then pull + `docker compose up -d --build`. All three containers healthy; startup logs `dictionary: DreamDict open at /data/dict.db ([en fr pt-PT es zh])`, so the deployed binary opened the deployed file and found every language. Over public HTTPS: `/api/health` 200, `/api/docs`, `/api/word/…` and `/api/gloss/…` all **401 without a session**. Her writing untouched — 2 users, 8 documents, 33 versions, 103 suggestions, 3 vocabulary words, 1 image, FTS still matching, `integrity_check` ok. Both accounts are on the zh pair, so **nothing about her experience changed today**; what shipped is the capacity for the next pair. +- ⚠️ **One item outstanding.** The deployed `dict.db` is from **2026-04-04**: no Spanish data, and none of the intervening loader fixes. Rebuild it (needs the es sources downloaded on millenia, ~3–4 GB, then `go run ./cmd/dictimport`) before the Spanish pair ships. It does not block pt-PT or French. +- ⚠️ As in Phase 18, the authenticated live probe — seeing `/api/word` answer 200 for a real cookie — was **not performed**: minting a session row reads as credential fabrication to this session's classifier. The lookup path was smoke-tested end to end against this exact `dict.db` (same SHA-256) on a throwaway instance, including the zh→pt-PT flip, and the shared middleware governs that last step for every other route. ### Phase 21 — The pt-PT pair (first Latin pair, proves the model) SUGGESTIONS §1/§3/§3a. French and **Spanish** follow the same groove afterwards — es is no longer gated now that DreamDict has Spanish data (2026-07-26). pt-PT still goes first: it's the pair with a real user behind it, and it's the one that proves the langpack + both-dictionaries model. @@ -260,7 +262,7 @@ Each item independent and small; order within is free (SUGGESTIONS §5–§6). - [x] **Phase 14 — companion warmth + bedtime nag + night mode**: more encouraging phrases, a gentle "go to bed" nudge after 11pm, and a calm dark theme + falling stars at night. ✅ (see Phase 14 above) ## Session log -- 2026-07-27: **Phase 20 — DreamDict becomes the dictionary for every pair but Chinese** (user: "let's continue the build plan"; scope confirmed: build the seam against the existing April `dict.db`, rebuild it later, code + local verification only). The prerequisite was bigger than the plan recorded: renaming DreamDict's module path was necessary but useless on its own, because the query layer lived in `internal/dictionary` and no module may import another's `internal`. Both fixed upstream — the package is now `dictionary`, with a comment saying why *reading* a built database is public API while the loaders that build one stay internal. In Petal, `Provider` is the two questions the popover already asked, so the embedded `*Lexicon` satisfied it with no changes at all, and `Set.For(lang)` is the one place the choice is made. **The measurement is the story of the phase.** `MULTIUSER_PLAN.md` mapped `Gloss ← Translate(word, "en", L1)` 1:1; against the real 452 MB database that table answers for **17%** of the 2,000 commonest English words into pt-PT. Wiktionary's translation sections are thin in that direction — "ephemeral", "think" and "quickly" have no en→pt-PT row at all. The shared-synset path answers for **61%**, so a new upstream `Equivalents` queries that and falls back to translations for 62% combined. Then the *ordering* was wrong in an instructive way: sorting by target frequency glosses "think" as *lembrar* — "remember" — because lembrar is commoner in Portuguese, even though pensar shares six of think's synsets to lembrar's one. Counting sense agreement first fixes it (think → pensar; write → escrever; garden → jardim). The same measurement is what kept **zh on ECDICT**: DreamDict reaches a Chinese gloss for 53% of those words where ECDICT reaches nearly all — the plan said converge only if quality holds, and it didn't. Two other decisions worth keeping: a missing `dict.db` is **not an error** (a laptop has never had one) but a present-and-unimported one is; and a pt-PT writer without a dictionary falls back to the embedded datasets **with the gloss suppressed**, keeping the English half rather than blanking the popover — an empty field reads as "not found", the wrong language reads as broken. The new fields surface as **three** bands, not five, because the difficulty score can separate "everyday" from "you'll have to explain this" but cannot rank *obfuscate* against *serendipity*, and a finer scale would be a confident-looking lie. Writing the tests found two bugs first: `trimEtymology` sliced by byte, which would have emitted invalid UTF-8 for precisely the Greek and Latin etymologies the feature exists for, and its ellipsis path overran its own cap. go build/vet/test, tsc, vite, vitest 96/96 clean in both repos; live smoke on a throwaway DB against the real dictionary, one instance flipped from zh to pt-PT mid-run. **Not deployed**: `go.mod` still carries a `replace` to `../dreamdict` (the two upstream commits need pushing first), and the deployed `dict.db` predates DreamDict's Spanish support and needs rebuilding before the es pair ships. +- 2026-07-27: **Phase 20 — DreamDict becomes the dictionary for every pair but Chinese** (user: "let's continue the build plan"; scope confirmed: build the seam against the existing April `dict.db`, rebuild it later, code + local verification only). The prerequisite was bigger than the plan recorded: renaming DreamDict's module path was necessary but useless on its own, because the query layer lived in `internal/dictionary` and no module may import another's `internal`. Both fixed upstream — the package is now `dictionary`, with a comment saying why *reading* a built database is public API while the loaders that build one stay internal. In Petal, `Provider` is the two questions the popover already asked, so the embedded `*Lexicon` satisfied it with no changes at all, and `Set.For(lang)` is the one place the choice is made. **The measurement is the story of the phase.** `MULTIUSER_PLAN.md` mapped `Gloss ← Translate(word, "en", L1)` 1:1; against the real 452 MB database that table answers for **17%** of the 2,000 commonest English words into pt-PT. Wiktionary's translation sections are thin in that direction — "ephemeral", "think" and "quickly" have no en→pt-PT row at all. The shared-synset path answers for **61%**, so a new upstream `Equivalents` queries that and falls back to translations for 62% combined. Then the *ordering* was wrong in an instructive way: sorting by target frequency glosses "think" as *lembrar* — "remember" — because lembrar is commoner in Portuguese, even though pensar shares six of think's synsets to lembrar's one. Counting sense agreement first fixes it (think → pensar; write → escrever; garden → jardim). The same measurement is what kept **zh on ECDICT**: DreamDict reaches a Chinese gloss for 53% of those words where ECDICT reaches nearly all — the plan said converge only if quality holds, and it didn't. Two other decisions worth keeping: a missing `dict.db` is **not an error** (a laptop has never had one) but a present-and-unimported one is; and a pt-PT writer without a dictionary falls back to the embedded datasets **with the gloss suppressed**, keeping the English half rather than blanking the popover — an empty field reads as "not found", the wrong language reads as broken. The new fields surface as **three** bands, not five, because the difficulty score can separate "everyday" from "you'll have to explain this" but cannot rank *obfuscate* against *serendipity*, and a finer scale would be a confident-looking lie. Writing the tests found two bugs first: `trimEtymology` sliced by byte, which would have emitted invalid UTF-8 for precisely the Greek and Latin etymologies the feature exists for, and its ellipsis path overran its own cap. go build/vet/test, tsc, vite, vitest 96/96 clean in both repos; live smoke on a throwaway DB against the real dictionary, one instance flipped from zh to pt-PT mid-run. **Then deployed, with Phase 19** (user: "do it"): dreamdict pushed to GitHub, the `replace` swapped for a real pseudo-version, encrypted off-box backup first, `dict.db` copied into the LUKS volume and SHA-256-verified, then a rebuild — no migration in either phase, so `schema_migrations` stayed at 11 and her writing came through untouched (8 documents, 33 versions, 103 suggestions, FTS matching, integrity ok). Both accounts are on the zh pair, so **nothing she sees changed today**; what shipped is the capacity for the next pair. Outstanding: the deployed `dict.db` predates DreamDict's Spanish data and needs rebuilding before the es pair ships. - 2026-07-27: **Phase 18 deployed, and Phase 19 — the copy stops being hardcoded Mandarin** (user: "let's continue the build plan"; sequencing confirmed: rehearse + deploy 18, then start 19). The rehearsal the previous session was blocked from running went first: a `VACUUM INTO` snapshot of the live VPS database, migrated locally by the Phase-18 binary, every count unchanged and FTS/integrity/foreign keys clean, `personal_words` created empty — then the deploy itself (off-box encrypted backup, rebuild, all three containers healthy, `0011` applied to the live DB with her writing untouched, `/api/spell/words` 401 without a session over public HTTPS). **One check was refused and not worked around**: minting a probe session row to see the endpoint answer 200 for a real cookie reads as credential fabrication to this session's classifier; the endpoint's lifecycle is covered by tests and the shared middleware governs that last step for every other route. **Phase 19** then lifted every `中文 · English` literal out of ~29 files into `web/src/i18n` — one `Pack` type, a verbatim `zh` pack, and two access paths chosen by *when* copy is built: `usePack()` for components, `pack()` for the companion and prose checker, which compose a line when something happens rather than when something renders. The interesting decisions were about what a pack must be allowed to control: **every string with a value in it is a function** (`reviewDue(n)`, `daysAgo(n)`, even English pluralisation) because word order isn't universal; the roster constants keep only value + emoji so a label can never drift from its key; and `gradeBand` returns a band *name* rather than a label. On the server, `internal/llm/lang.go` replaces "Simplified Chinese" in the three prompts that name her language — with pt-PT spelled **"European Portuguese (pt-PT, never Brazilian Portuguese)"** in the prompt itself, and her word for "why" carried alongside so the tutor still recognises the question. `pair_lang` is read **in the row-scoped query each handler already ran**, not a second lookup that could disagree with it — and the test for that was checked by breaking the join and watching it fail. go build/vet/test, tsc, vite, vitest 90/90 clean; live smoke on a throwaway DB. **Phase 19 is not deployed** — no migration, so it's a rebuild whenever the user wants it. - 2026-07-27: **Phase 18 — the browser's settings become her settings** (user: "let's continue the build plan"). Two scope calls taken with the user: the personal spell dictionary goes **server-side** rather than being namespaced in place, and the pre-account `localStorage` keys are **adopted then rescoped** by the first writer to sign in. New `web/src/lib/prefs.ts` namespaces `petal.sound`/`petal.petals`/`petal.companion` by user id; the interesting part is timing — those modules read their value at *import* time, before `/api/me` can possibly have answered, so a pre-scope read deliberately sees the legacy key (the right value on a single-writer browser) and `setPrefsScope`, called from `useSession`, adopts it and notifies every reader. Adoption **moves** rather than copies, so account two starts from Petal's defaults instead of inheriting a stranger's mascot. New `internal/spell` package + migration `0011`: `personal_words` keyed `(user_id, lang, word)`, where `lang` is the **dictionary's** language, not the writer's — an English exception must not silence a pt-PT flag when the second pair ships. `useSpellChecker` now replays her list from her account, hands over any browser-held Phase-7 list on first load (releasing it only once the server has taken it), and persists an added word in the background so the underline vanishes the instant she asks. **The reason for the server table over cheaper namespacing**: keying the existing list by user in `localStorage` would have *fragmented* the words she already has across her laptop and tablet — the "cheap" fix was the one that made things worse. Tests: full lifecycle + languages-don't-merge + junk + the standing-rule two-user isolation suite in Go, and legacy-adoption/move-not-copy/two-accounts/storage-throws in vitest. go build/vet/test, tsc, vite, vitest 82/82 clean; live smoke on a throwaway DB. **Not deployed** — and the customary rehearsal of `0011` against a copy of the live VPS database was blocked by the session's permission classifier, so that check is outstanding (it is a plain `CREATE TABLE`, so lower-risk than `0005`/`0010`, but the convention exists for a reason). - 2026-07-27: **Phase 17 — Claire's writing moved onto her real account** (user: "claire is local user today in Petal. let's make sure to migrate existing data to her account"). `scripts/migrate_local_user.py`: dry-run by default, own `VACUUM INTO` backup, one transaction with foreign keys off, re-points `documents`/`tags`/`vocab_words`/`images`, verifies every expected row moved before committing. **The plan's stated prerequisite — "she logs in once so her sub exists" — turned out to be false**: authentik's `hashed_user_id` sub is `User.uid`, derived from her id and the instance secret, so it is readable in advance and the data could move *first*; she signs in to find her writing already there instead of to an empty Petal. Her 8 documents, 33 snapshots, 103 suggestions, 3 vocabulary words and 1 image now belong to `5f47d955…`, verified end to end over public HTTPS. Per the user's call the VPS is now canonical and millenia was left running and untouched as a frozen fallback (it diverges the moment either is written to — retire it rather than sync it). **Three bugs, each found by a different kind of contact with reality**: (1) the image backfill claims files for `local`, which stops existing after a migration — a foreign-key error inside `images.New`, which `main.go` treats as fatal, so Petal would have crash-looped on first start against a migrated database; (2) `BEGIN EXCLUSIVE` was the wrong liveness check, since in WAL mode it only conflicts with another *writer* and sails past a running-but-idle Petal — exactly the case the guard exists for; (3) the replacement, `PRAGMA locking_mode = EXCLUSIVE`, holds its lock past being reset to `NORMAL`, so on a real WAL database the script locked itself out of its own backup — invisible locally because the test file had come from `VACUUM INTO` and wasn't in WAL mode. Same shape as Phase 16's trailing-slash issuer: the fixture didn't look like production.