Finish Phase 22: the half of Petal that works with the tunnel down

Grammar lite, the false-friend list, the daily invitation and the offline
miscollocations — the four remaining §5–§6 items, all client-side and all
alive on a box that cannot reach the model.

The offline collocations forced a schema change. `type` had been doubling
as the answer to "which engine found this" — `mechanics` meant offline —
and that stops being true the moment an offline rule proposes a
collocation. Migration 0013 adds `source` (llm | local) and every pass now
scopes its DELETE by engine; without it the coach silently wiped every
offline chunk on the page. Existing rows backfill by type, so a pre-0013
collocation row is claimed as the coach's, which it was: the offline list
did not exist yet.

The rule pack is hand-curated rather than mined, and the entries left out
are the point — `married with` is wrong until "married with children",
`arrive to` wants at or in depending on the noun. A pack running on every
keystroke must not correct correct writing.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
This commit is contained in:
prosolis
2026-07-27 15:05:55 -07:00
parent e9b8595456
commit 1bbc8fc8d3
20 changed files with 1678 additions and 50 deletions
+21 -5
View File
@@ -255,9 +255,10 @@ Phase 20 left this ready: `dict.db` on the VPS now holds all five languages, and
- **A bug the test found, not the code review**: `extendedAlphabet` was a value computed when the checker was built while `correct`/`suggest` read live. Her dictionary arrives *after* English, so the underlines would have been right while every lookup was still resolving "cora". It is a getter now.
- Verified: go build/vet/test, tsc, vite build, vitest 116/116 clean. The shipped asset loaded in a real nspell (842 ms, 139 MB, pt-PT variants correct both ways). Live smoke on a throwaway DB (:8091): both dictionary files served (577 B aff, 2,661,813 B gz), the gz inflating to 1,039,058 forms with `receção` present, and the zh word lookup unchanged. **Not verified against real data**: this laptop has no `dict.db`, so the reverse-lookup path is exercised by the fixture only — the first real pt-PT collision lookup happens on the VPS.
### Phase 22 — Learning loop + code-first layers
### Phase 22 — Learning loop + code-first layers ✅ (2026-07-27) — the last phase of the plan
Each item independent and small; order within is free (SUGGESTIONS §5–§6).
**First two built 2026-07-27** (user: "continue the build plan"; code only, no VPS work — not deployed, and there is no migration to undo, so it is a rebuild whenever the user wants it).
**Remaining four built 2026-07-27** (user: "let's finish the last phase of the build plan"). With them the left-hand column of the SUGGESTIONS §6 table is complete: **spell, define, gloss, pronounce, catch the common mistakes, review vocabulary, prove authorship — every daily-writing need now works on a box with the tunnel down.** The model adds depth and conversation when it is reachable and holds nothing hostage when it isn't. Carries one migration (`0013_suggestion_source`), so unlike the earlier code-only sessions this is a deploy rather than a rebuild.
- [x] **Growth journal** (Q3 settled) ✅ (2026-07-27) — `GET /api/suggestions/growth`, a read-side view of a table Petal already keeps: no new capture, no model call, nothing leaves the box. Three signals, and the work was in deciding which ones are *honest* rather than in computing them.
- **Kept** — edits she took on board in the last 30 days, with the 30 before it offered flat beside it. That second number is the whole of the self-comparison rule: there is no target, no average and no other account anywhere in these queries.
- **Stuck** — accepted phrasing that now appears in **two or more** of her own documents. One document is not evidence: it is the edit itself, still sitting where it was applied. The second is her reaching for the phrase on her own, which is the only thing the line actually claims. Candidate phrases are filtered through `vocab.PhraseKey`, the *same* definition of "a learnable chunk" the garden plants, so the journal and the garden can never disagree about what counts.
@@ -276,10 +277,24 @@ Each item independent and small; order within is free (SUGGESTIONS §5–§6).
- Tests: `internal/vocab/plant_test.go` (PhraseKey table incl. rune-vs-byte, plant-once, unplantable is a silent no-op), `internal/suggestions/plant_test.go` (corrected-sentence example, only-collocations, idempotent-and-never-resets, sentence-rewrite skipped without failing the accept), `internal/suggestions/growth_test.go` (both windows, stuck needs a second document, the wrote-recently guard, still-happening excluded, and a per-writer isolation test seeding bob), `internal/db/db_test.go` (the backfill). Frontend: `journalCheers.test.ts` (silent before the fetch lands, once per line, one fetch however often warmed, silent on failure, pack resolved at call time) plus journal assertions in `i18n.test.ts`.
- Verified: go build/vet, `go test ./internal/...` clean, tsc, vite build, vitest 131/131.
- ⚠️ **Not deployed and not seen in a browser.** Same standing gap as Phase 21: no pt-PT account exists, and this was a code-only session. The pt-PT journal copy is part of the pack a native speaker still has not reviewed.
- [ ] **Daily writing invitation** from the companion (no streaks, declining is fine)
- [ ] **False-friend list** per pair (curated data, WordCard heads-up + gentle flag)
- [ ] **Embedded miscollocation list** (code-first under the collocation family; LLM adds the long tail when reachable)
- [ ] **Grammar lite** rule-pack as a fourth suggestion family: instant, offline, precision-over-recall (near-certain or silent); per-pair L1-interference rules; sourcing per SUGGESTIONS Q6 (hand-curate vs mine LanguageTool's corpus — decide at build time)
- [x] **Daily writing invitation** from the companion ✅ (2026-07-27) — offered to a *blank page* about a minute into a session, at most once a day. Petal always has a document open, so "a session that starts with no doc open" became "the page in front of her is still empty", which is the state the invitation was actually for.
- **The stored value is a date, and that is the entire mechanism.** No count, no run of days, nothing that degrades with absence: coming back after a month reads exactly like coming back tomorrow. That is the one property this feature could lose silently, so the rule lives in its own file (`invitation.ts`) rather than inside the heartbeat, and the test names it — *treats a month away the same as a day away*.
- **Both answers spend the day's invitation.** Being asked again after "not today" would make no into a negotiation. Declining costs a sleepy `好吧,我继续睡 😴` and nothing else; letting the bubble time out is a third way of saying no.
- **Accepting titles the blank page with the prompt**, so the question she agreed to answer is still in front of her once the bubble has gone.
- Copy is bound the way the journal's is: a test greps both packs for *streak / in a row / every day / missed / 连续 / 打卡 / todos os dias* — the framing is the feature.
- [x] **False-friend list** per pair ✅ (2026-07-27) — ~19 curated en↔pt entries in the pt-PT pack; **zh has none, and that is the honest answer**, not an unwritten one: the trap needs a shared script to spring.
- **Never a correction.** Two surfaces, both heads-up only: a lavender block at the top of the WordCard (above the definition — it is the thing she would not think to check), and at most one companion note per pass. No `fix`, so it never becomes a card. *Actually* may well be the word she meant; the flag says what the English one means and stops. A test greps the entries for *wrong / mistake / errado* — this is the mistake that makes a learner feel foolish, and the tone is the whole point.
- [x] **Embedded miscollocation list** ✅ (2026-07-27) — the do/make, say/tell, heavy-rain families as ten curated patterns, and **they file as `collocation`, not as a new family**. Same rail, same warm phrasing, and — the reason it matters — an accepted chunk plants in the vocabulary garden exactly as the coach's would. The writer never learns which engine spoke.
- **That forced a schema change**: `type` had been doubling as the answer to "which engine found this" (`mechanics` meant offline). The moment an offline rule proposes a collocation that breaks — so migration `0013_suggestion_source` adds `source` (llm | local) and every pass now scopes its DELETE by engine. Without it the coach silently wiped every offline chunk on the page, and the offline pass left the coach's rows to accumulate. Both directions are tested; existing rows backfill by type, and a pre-0013 collocation row is correctly claimed as the coach's, since the offline list did not exist yet.
- **The span tiebreak moved with it**: an exact offline card beats an overlapping LLM one by *source*, not by type — an offline miscollocation is as exact as an offline comma.
- Replacements agree with the tense she wrote in (`did a mistake``made a mistake`), and a rule never proposes a phrase identical to what she already wrote.
- [x] **Grammar lite** rule-pack ✅ (2026-07-27) — the deterministic `mechanics` family already *was* the fourth family (Phase 8), so this was the rule pack it had been waiting for rather than new plumbing: preposition pairs, doubled comparatives, `people is`, and per-pair L1 interference. All client-side, instant, no debounce, no rate limit, alive on a VPN-down box.
- **Sourcing decision (SUGGESTIONS Q6): hand-curated, not mined.** LanguageTool's corpus is broad because it aims at recall; this pack aims at the opposite. Every entry here is a pairing that is wrong in essentially *all* contexts, and the ones that are only usually wrong were left out on purpose — `married with` is a mistake until "married with children", `arrive to` wants at or in depending on the noun, `different than` is ordinary American English. Each rule is tested in both directions, and the guard cases are the correct English sitting next to the mistake.
- **L1 rules are gated by pair, and the gating is what lets them be confident**: a near-certainty for a Portuguese speaker is only a guess for anybody else. pt/fr/es get *ter 30 anos* → "I am 30 years old" (subject and tense carried into the correction), "I am agree", "since three years" → "for three years". zh gets 很喜欢 → "very like", 开灯 → "open the light", and 虽然…但是 → "although … but".
- **The zh rules the plan named and this pack does not implement**: dropped articles and he/she slips. Neither is detectable from text alone — "She said he was late" is a perfect sentence whichever pronoun was meant — and flagging them would mean correcting correct writing, which is the one thing a rule pack running on every keystroke must not do. Said in a comment where the rules are, not only here.
- Verified live on a throwaway DB (:8099, no dictionary, **no LLM configured at all**): an offline `did a mistake` → card → accept → garden card *made a mistake* with the example bounded to its own corrected sentence, and the journal reporting `kept:1`.
- Tests: `grammarLite.test.ts` (30, every rule in both directions), `invitation.test.ts` (7), `offline_test.go` (the six engine-split cases), `db_test.go` (the 0013 backfill), plus false-friend shape/tone guards in `i18n.test.ts`.
- ⚠️ **Not deployed and not seen in a browser.** Same standing gap as Phases 2122: the pt-PT copy added here is part of the pack a native speaker still has not reviewed.
### Later / explicitly not now
- Learner-facing Chinese writing (the zh pair's second direction) — own phase with its own spec (SUGGESTIONS §4); only after Phases 1921 prove the pair model
@@ -295,6 +310,7 @@ Each item independent and small; order within is free (SUGGESTIONS §5–§6).
- [x] **Phase 14 — companion warmth + bedtime nag + night mode**: more encouraging phrases, a gentle "go to bed" nudge after 11pm, and a calm dark theme + falling stars at night. ✅ (see Phase 14 above)
## Session log
- 2026-07-27: **Phase 22 finished — the build plan's last four items, and the LLM stops holding anything hostage** (user: "let's finish the last phase of the build plan"; code only, no VPS work). The four remaining items shared one theme, and it only became visible while building them: **§6's left-hand column is now complete.** Spell, define, gloss, pronounce, catch the common mistakes, review vocabulary, prove authorship — every daily-writing need works with the tunnel down. **The plan asked for "grammar lite as a fourth suggestion family", and the fourth family already existed**: Phase 8's deterministic `mechanics` pass was the plumbing, so this was the rule pack it had been waiting for rather than new machinery — preposition pairs, doubled comparatives, `people is`, plus per-pair L1 interference. **Q6 answered by hand-curating rather than mining LanguageTool**: that corpus is broad because it aims at recall, and this pack aims at the exact opposite, so every entry is a pairing wrong in essentially *all* contexts and the ones only *usually* wrong were left out on purpose — `married with` is a mistake until "married with children", `arrive to` wants at or in depending on the noun, `different than` is ordinary American English. Each rule is pinned in both directions, the guard case being the correct English next to the mistake. **The L1 rules are gated by pair, and the gating is what earns them their confidence***ter 30 anos* → "I am 30 years old" is a near-certainty for a Portuguese writer and only a guess for anyone else. The two zh rules the plan itself named are the ones this pack **refuses** to implement: dropped articles and he/she slips are not detectable from text alone ("She said he was late" is perfect whichever pronoun was meant), and flagging them would mean correcting correct writing. **The miscollocation list forced the session's one real design change.** It had to file as `collocation` rather than as its own family — same rail, same phrasing, and an accepted chunk plants in the garden exactly as the coach's would — but `type` had been quietly doubling as the answer to *which engine found this*, and that breaks the instant an offline rule proposes a collocation. Migration `0013_suggestion_source` splits the two apart: each pass now scopes its DELETE by engine, and the span tiebreak moved with it (an exact offline card beats an overlapping LLM one by source, not by type — an offline miscollocation is as exact as an offline comma). Without it the coach silently wiped every offline chunk on the page and the offline pass left the coach's rows to pile up; both directions are now tested, and a pre-0013 collocation row correctly backfills to the coach, since the offline list did not exist yet. **The daily invitation's whole substance is one stored date** — no count, no run of days, nothing that gets worse for being away, so a month away reads exactly like a day away; it lives in its own file because that is the property this feature would lose silently, and the test is named for it rather than for the query. Both answers spend the day's invitation, because being asked again after "not today" would make no a negotiation. **False friends are the one thing here that never becomes a card**: ~19 curated en↔pt entries, shown as a lavender block above the WordCard's definition and as at most one companion note per pass, with no `fix` anywhere — *actually* may well be the word she meant, and this is the mistake that makes a learner feel foolish rather than merely corrected. zh has none, which is the honest answer and not an unwritten one: the trap needs a shared script. Copy for the invitation and the false friends is greped by tests the same way the journal's is (*streak / in a row / 连续 / todos os dias*; *wrong / mistake / errado*) — the framing is the feature, and it is the part a future edit would undo while meaning well. Verified: go build/vet, `go test ./internal/...` clean, tsc, vite build, vitest 172/172 (30 new rule cases, 7 invitation, plus false-friend shape/tone guards), and a live throwaway DB on :8099 with **no LLM configured at all** — offline `did a mistake` → card → accept → garden card *made a mistake*, example bounded to its own corrected sentence, journal `kept:1`. ⚠️ **Not deployed and not seen in a browser**, and this one carries a migration, so it is a deploy rather than a rebuild. The pt-PT copy added here joins the pack a native speaker still has not reviewed.
- 2026-07-27: **Phase 21 deployed — the pt-PT pair has a voice** (user: "continue the build plan"; scope chosen: deploy Phase 21 to the VPS rather than start Phase 22). The plan's remaining line was "Piper pt-PT voice instance on parodia", and it hid two things. **A language was still a code change**: read-aloud knew exactly two, named in the Config struct as `TTSEndpointZH`/`TTSVoiceZH`, so adding Portuguese meant editing Go to add Portuguese. Petal now discovers its Piper instances from the environment — English keeps the unsuffixed pair, everything else is `TTS_ENDPOINT_<LANG>`/`TTS_VOICE_<LANG>`, base tag only because an env var name cannot hold pt-PT's hyphen — and a language configured by halves is dropped rather than routed, so it reaches the client as "no voice, use Web Speech" instead of erroring on every tap. fr and es now cost a compose service and two `.env` lines. **And the voice itself repeated Phase 21's own lesson in a new place**: `pt_PT-tugão-medium` is the *only* European Portuguese voice in Piper's catalogue — the other five are Brazilian — so, exactly as with `dictionary-pt` packaging VERO, the default anyone reaches for ships the wrong country. Then it wouldn't download at all: `piper.download_voices` pastes the voice name into the HTTP request line and `http.client` encodes that as ASCII, so it dies with `UnicodeEncodeError` on the *ã* before a byte leaves the container — a failure that lands on precisely the one voice this pair needs and on no other. The entrypoint falls back to fetching the model and its config itself with the path percent-encoded, which is all the downloader was missing. **The slow replay** (§5e) went in while there: `slow: true` raises `length_scale` to ~4/3, and the pace is part of the **cache key** — without that, asking to hear slowly a word already heard at speed serves the fast clip back, which is the one request where the difference is the entire point. **The L1 voice asks the pack, not the letters**: a new `locale` field, because "comum" is spelled the same in both halves and a detector would have to guess — the same reason the gloss shows both directions. **Deploying is what finally ran the reverse lookup against real data**, the item the previous session left open because this laptop has no `dict.db`: *data* → "date", *comum* → "common; usual", *tarde* → "evening; afternoon", *ali* → "there", with *think*, *computer* and *garden* correctly silent; and *think* glossing to **pensar** first confirms Phase 20's sense-agreement ordering on the real 550 MB database rather than on a fixture. zh flipped back is byte-for-byte ECDICT again. go build/vet/test, tsc, vitest 125/125, vite; laptop smoke against two fake Pipers, then the real thing on the box. Her data untouched: 8 documents, 33 versions, 103 suggestions, FTS matching, integrity ok, `schema_migrations` still at 11 (no migration in this phase). **Two things Phase 21 still owes, both said plainly**: the pack has not been read by a pt-PT speaker, and no pt-PT account exists — both writers are on the zh pair, so nothing she sees changed today and the browser half of the Portuguese experience has never had a human in front of it.
- 2026-07-27: **Phase 21 (code half) — the pt-PT pair, and the plan's one-line assumption about the dictionary** (user: "let's continue the build plan"; scope confirmed: code only, the Piper voice and the deploy deferred, the pack written but flagged unreviewed). The plan said "Hunspell pt-PT vendored like en-US", and that turned out to be the load-bearing sentence. **nspell expands affixes eagerly on construction** — it materialises every surface form the moment you build it. English survives that; European Portuguese's 1,340 affix rules over 44,257 stems do not. Measured before deciding anything: ~340 MB of heap for the first 12,000 entries, and no return at all after three minutes on the whole file — over a gigabyte, in a browser, on a tablet. So the expansion moved to build time: `scripts/build_ptpt_dictionary.py` writes 1,039,058 forms, 2.66 MB gzipped, which the *same* nspell then reads in 842 ms using ~120 MB, and the runtime path stays byte-for-byte the English one. The `.aff` keeps only TRY/KEY/REP/MAP, which shape corrections rather than membership, so "telemovel" still corrects to "telemóvel". **A second thing the obvious route would have got wrong quietly**: npm's `dictionary-pt` is not European Portuguese — both it and `dictionary-pt-br` package VERO (Brasil), so vendoring the obvious package name ships Brazilian spellings under a pt-PT label. That is §3's pt-BR drift arriving through the *packaging* rather than through the model, and nobody reviewing the diff would see it. The real source is Projecto Natura's, packaged as `hunspell-pt-pt`; the build script now asserts the fault lines (`receção`/`húmido`/`pensámos` in, `recepção`/`úmido`/`ônibus`/`óptimo` out) before it writes a byte, and a vitest greps the built langpack for *sinônimo*, *arquivo*, *tela*, *você*. **Both-dictionaries spellcheck** landed as §3a specifies — flag only what every loaded dictionary rejects, interleave the correction pills so English can't fill all five — and dragged a smaller thing with it: the tokenizer had to become a property of the checker rather than a constant, because `[A-Za-z]` cuts "coração" into "cora", which is both silently unchecked *and* what a right-click would have looked up. The wide alphabet stays off for a writer with no Latin second language, where it could only earn her new squiggles. **Gloss both directions**: a Latin pair has no script boundary, so *data*, *sale* and *comum* are words on both sides and there is no honest way to know which she meant — Petal asks both and shows what answers, which needs no detector and therefore cannot be wrong about her writing. The reverse direction deliberately skips the English de-inflection walk, which over Portuguese would be right by accident and wrong by rule. **Writing the tests found the bug**: `extendedAlphabet` was a snapshot taken when the checker was built while `correct`/`suggest` read live — and her dictionary arrives *after* English, so the underlines would have been right while every lookup still resolved "cora". go build/vet/test, tsc, vite, vitest 116/116 clean; the shipped asset loaded in a real nspell; live smoke on a throwaway DB served both files and left the zh lookup untouched. **Two things outstanding and both said plainly**: the pack has *not* been read by a pt-PT speaker (SUGGESTIONS §3's own bar, and not one I can meet), and this laptop has no `dict.db`, so the reverse-lookup path is covered by a fixture rather than by a real collision — the first of those happens on the VPS.
- 2026-07-27: **dict.db rebuilt with Spanish, and a log line caught lying** (user: "if we need to redeploy DreamDict to add Spanish support, then do so"). Millenia's dreamdict checkout held ~490 lines of uncommitted work; rather than pull over it, comparing file contents showed an earlier draft of the regional-variant work already committed upstream — nothing unique, but not mine to discard, so it was left alone and the rebuild ran from a clean clone pushed over from the laptop (millenia has no GitHub SSH). Import took 6m15s and added **es: 102,971 words**, leaving en/fr/pt-PT/zh byte-identical — the check that distinguishes "added a language" from "quietly changed everything". Coverage measured before shipping: **es 68.6%**, the best of the four; **zh re-measured at 53.2%**, so the ECDICT decision stands on fresh evidence rather than on the earlier number. Shipped direct millenia→parodia over headscale, hashed both ends, kept the April file for rollback. **The rebuild's real find was in Petal, not DreamDict**: the startup line reported `dictionary.Langs()`, a compile-time constant of *supported* languages, so it had been printing a cheerful `[en fr pt-PT es zh]` over a database with no Spanish in it — the exact failure it existed to catch, reported as success, and something I had already claimed as proof the deploy was good. It now counts rows. Chasing a failed SUBTLEX-US download (benign — the loader falls back to `.txt`) also confirmed English "frequency" is mostly SCOWL's commonness bucket, which independently vindicates the band chip reading `difficulty` instead.