From 76dede885620df78f85aa264008fee23c3834951 Mon Sep 17 00:00:00 2001 From: prosolis <5590409+prosolis@users.noreply.github.com> Date: Tue, 28 Jul 2026 23:20:53 -0700 Subject: [PATCH 1/3] Correct the language she wrote in, not the one she was practising MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every pass was English-shaped: CheckpointMessages took the text and the tone and nothing else, so there was never a language decision to get wrong. On the live build two pt-PT sentences drew no cards at all — Petal read the Portuguese, said nothing about it, and filed a mechanics note about the one English line. The rule is two decisions reading different state. What gets corrected follows the document. What language the explanation is written in follows the writer — the half of her pair she is not learning, from users.direction — because an explanation is teaching, and teaching lands in the language she reads most easily. Those coincide for every account that exists today (learnerPairs is {"zh"}), which is a fact about the roster and not about the design, so Target keeps them apart. It carries a third language too: the collocation gloss is addressed to her rather than to the document, and folding it into Explain would have quietly moved it into English on every English document. The document verdict is a proportion, not a presence — one Portuguese quotation must not flip an English essay. Per sentence, three-way: pair, English, or no answer. The third value is the load-bearing one; counting the undecided as English is exactly what would hold a journal of short Portuguese sentences in English forever, so the Latin pairs needed an englishMarkers list curated against pt/fr/es as carefully as latinMarkers was curated against English. Hysteresis at 70/40 because a bilingual paragraph would otherwise alternate its cards' language every few keystrokes, and hysteresis needs a yesterday — hence the column. Plus a corroboration floor: a ratio computed over "Não. Eu." is 100% of nothing, and a flip rewrites every card in the document. The verdict folds into the chunk salt beside the tone, so a document that changes language re-opens every sentence rather than serving back cards in a language it no longer speaks. checkpointSystemPrompt could not simply take a language — it opens by naming the reader an ESL learner, and appending "explain in Portuguese" hands the model two contradictory framings. Separate constants, sharing the JSON contract below the framing. Both carry a "never translate it into English" line, which is the instruction the model will most want to disobey. The English prompts are untouched byte for byte, and a golden says so out loud. Collocation deliberately did not move: its prompt is per-language knowledge, not framing, and "natives usually say" for Portuguese is a claim Petal cannot back. Not deployed and not smoked against a real model. The tests drive the real router and a real DB; what none of them prove is how Qwen behaves on a Portuguese document, in particular whether the never-translate line holds. Claude-Session: https://claude.ai/code/session_01GJHNvirh7Hzhc9RL3HAvz7 --- BUILD_PLAN.md | 30 +++ internal/db/db.go | 24 +++ internal/llm/checkpoint.go | 4 +- internal/llm/collocation.go | 4 +- internal/llm/lang_test.go | 4 +- internal/llm/prompts.go | 103 ++++++++++- internal/llm/target.go | 54 ++++++ internal/llm/target_test.go | 115 ++++++++++++ internal/llm/voice.go | 4 +- internal/suggestions/doclang.go | 179 ++++++++++++++++++ internal/suggestions/doclang_test.go | 261 +++++++++++++++++++++++++++ internal/suggestions/handlers.go | 85 +++++++-- 12 files changed, 836 insertions(+), 31 deletions(-) create mode 100644 internal/llm/target.go create mode 100644 internal/llm/target_test.go create mode 100644 internal/suggestions/doclang.go create mode 100644 internal/suggestions/doclang_test.go diff --git a/BUILD_PLAN.md b/BUILD_PLAN.md index 81623d6..d48af4b 100644 --- a/BUILD_PLAN.md +++ b/BUILD_PLAN.md @@ -423,6 +423,36 @@ Phase 26's own outstanding item, and the thing it named as most likely to be wro - ⚠️ **What this still does not prove is the counterfactual** — that the bug would have bitten *here* without the guard. Chrome's own composition handling is robust, and a clean run with the guard in place cannot distinguish "held correctly" from "would have been fine anyway". Settling it means building with the guard reverted and asking her to type once more; offered, not assumed. - **Not deployed.** No migration; a rebuild whenever the user wants it, along with Phases 24–26. +### Phase 28 — following the writing into her own language (planned 2026-07-28; step (a) built 2026-07-28) +Today every correction and every explanation comes back in English, whatever she wrote. `CheckpointMessages` (`internal/llm/prompts.go:49`) takes the text and the tone and **nothing else** — there is no language parameter to pass, so there has never been a language decision to get wrong. The pair language reaches her only on demand: tapping Ask Petal fetches a translation of the English explanation (`AskPetal.tsx:93`), and that panel answers in her language because `AskPetalSystemPrompt` is given one. Right default for a writer practising English; wrong default for a document written in Portuguese, where Petal reads the Portuguese prose, says nothing about it, and files a mechanics note about the one English sentence at the end. **Observed on the live build 2026-07-28**: two pt-PT sentences drew no cards at all. + +1. [x] **The rule is two decisions, not one, and they read different state.** What gets *corrected* follows the **document** — Portuguese prose gets Portuguese corrections, which is the whole point. What language the explanation is *written in* follows the **writer**: the half of her pair she is not learning, from the `direction` column (`internal/auth/users.go:96`), because an explanation is teaching and teaching lands in the language she reads most easily. So a native Portuguese speaker practising English, writing Portuguese, gets Portuguese corrections explained in Portuguese; a native English speaker learning French, writing French, gets French corrections explained in English. Neither is trapped — the other language stays one tap away, in both directions. + - **What the direction lookup costs today is nothing.** `learnerPairs` is `{"zh"}` (`users.go:119`), so fr, es and pt-PT accounts are all `learning_en` and their non-learned half *is* the pair language: the rule produces exactly "explain in the document's language" for every writer who exists right now. It is there to stop that from being baked into the prompts, the way "English is the language being learned" was baked into `pair_lang` before migration 0016 — the same mistake this plan's own Phase 26 had to unpick. +2. [x] **A document-level verdict, beside the span-level one already in `internal/suggestions/language.go`.** `readsAsPairLang` answers "is this quoted span her language" for labelling one card; it under-claims on purpose and trips at two marker words. A whole document needs a **proportion, not a presence**, or one Portuguese quotation inside an English essay flips the entire pass. Three properties, in the order they bite: + - **Decided over the whole document, never a chunk.** A chunked pass sends only the sentences that changed (`splitChunks`/`changedChunks`); computing the verdict from `askText` would put an English card in a Portuguese journal the moment she edits its one English line. Compute from `contentText`, always. + - **Hysteresis, for the same reason the mascot needed it** (see the session log below, 2026-07-28). A bilingual paragraph sits near whatever threshold we pick, and a document crossing it every few keystrokes would alternate card languages between passes. Flip to the pair at ≥70% pair-language sentences, back only below 40%. The band is the feature. + - **Plain code, no model call** — the house rule that the LLM is garnish, never a gatekeeper. zh is a rune-script count (already written); pt-PT, fr and es reuse `latinMarkers`, aggregated per sentence rather than per span. +3. [x] **`checkpointSystemPrompt` cannot simply take a language.** It opens with *"helping someone who speaks English as a second language"* and asks for ESL patterns — appending "explain in Portuguese" hands the model two contradictory instructions. Split it: shared JSON contract and tone clause, framing sentence filled per direction. `CheckpointMessages` and `VoiceMessages` (which has no language at all today) take **two** language arguments — corrected and explained-in — and **resist collapsing them into one `Lang` while every current account has them equal**: that equality is a fact about today's `learnerPairs`, not about the design. +4. [x] **The verdict rides with `pairLang` in the row-scoped lookup and folds into the chunk salt** next to `tone` (`handlers.go`). That is the cheap correct answer to stale cards: when a document's language flips, every sentence's identity changes, so old-language cards are re-checked rather than left sitting there in the wrong language. +5. [ ] **`isTranslation` learns which way it points.** It currently means "her language rendered into English"; in a flipped document the useful translate card is the mirror image, so the test takes the document verdict and checks the direction that matches. +6. [ ] **The tap-through has to stop assuming its direction, and this is not optional dressing** — it is what makes rule 1 safe for a learner reading explanations in English. `/suggestions/{id}/translate` always renders into the pair language today; it should render into whichever half the explanation is *not* already in, and skip the seed entirely when those coincide rather than round-tripping Portuguese into Portuguese. +- **`internal/llm/lang.go` already carries the precision the model needs** (`"European Portuguese (pt-PT, never Brazilian Portuguese)"`). Spanish wants the same care pointed the other way: the shipped dictionary deliberately accepts the whole Spanish-speaking world (Phase 25), so the prompt must not quietly impose peninsular usage. +- **Spellcheck must not change.** "A word is a misspelling only when *both* dictionaries reject it" (`useSpellChecker.ts:13`) is a deliberate refusal to detect document language, and it is right: it makes English quotations inside Portuguese prose free, in both directions. Teaching it a document verdict buys nothing and costs a false positive on every borrowed word. +- Tests worth writing first: doc-level detection per pair (monolingual, 80/20, 50/50, quotation-heavy English), asserting the hysteresis band **from both directions**; the flipped prompt names the target language and drops the ESL framing while the English-document prompt stays **byte-identical** to today's (the path every existing user is on); the two language arguments proven independent by the only pair that can exercise it — a `learning_pair` zh account writing Chinese wants Chinese corrections explained in English, a `learning_en` zh account writing Chinese wants both in Chinese; a handler test in the shape of `pairlang_test.go`; and a language flip invalidating checked chunks. +- **Open, and needs a decision before the last step.** The **vocabulary garden** harvests phrases from documents, so a Portuguese document would seed it with Portuguese — tag entries by language and filter by the half being learned, or gate harvesting to English documents? Tagging looks right and touches stored rows. **Read-aloud** should follow the document too (all five voices now run on the VPS), which is probably small and lands where the verdict lands, but has not been traced. +- **Order:** (a) detection + salt + checkpoint/voice prompts — the whole visible win, independently shippable; (b) translate-card direction, Ask Petal seed, `/translate` direction; (c) garden and read-aloud, once the two questions above are answered. + +**Step (a) as built, 2026-07-28** — items 1–4. Step (b) and (c) are untouched, and the two open questions under (c) are still open. +- `internal/llm/target.go` — `Target{Correct, Explain, Pair}` + `English` (a `Lang` the `langs` map has no business holding: that map answers "which half is hers"). `EnglishTarget(pair)` is the pre-phase behaviour named, and `Flipped()` is the one question the prompts ask. **Three fields, not two**: the collocation coach's parenthetical gloss is addressed to *her* and not to the document, so it reads `Pair` — collapsing it into `Explain` would have silently moved that gloss into English on every English document, which is every document today. +- `internal/suggestions/doclang.go` — `documentLang(contentText, pairLang, prev)` → `"en"`/`"pair"`. Per **sentence** (reusing `splitChunks`, salt `""`), three-way: pair / English / **no answer**. The third value is the load-bearing one — a sentence with no evidence ("Bom dia.", a heading) is left out of the ratio rather than counted for the language it isn't, and counting the undecided as English is exactly what would hold a Portuguese journal of short sentences in English forever. So the Latin pairs needed an `englishMarkers` list curated against pt/fr/es with the same care `latinMarkers` was curated against English (no "on"/"son"/"as"/"no"/"para" — each a false English vote inside someone's own language). Band as specified, ≥70% / <40%, plus a **corroboration floor** the plan didn't call for: 3 distinct pair markers (8 Han runes) document-wide, because a ratio computed over "Não. Eu." is 100% of nothing and a flip rewrites every card in the document. +- **Migration `0017_document_lang`** — `documents.doc_lang` (`'' | 'en' | 'pair'`). Stored, not recomputed, because hysteresis needs a yesterday; `'pair'` rather than a language code, so changing her pair re-reads her documents instead of stranding a stale language name on all of them. Backfill is the default `''`, which reads as English — true of every document that exists. +- `runPass` reads `doc_lang` + `direction` in the row-scoped lookup that already proves ownership, writes the verdict back only when it changed, folds it into the chunk salt beside `tone`, and hands `targetFor(pairLang, direction, docLang)` to the pass. `pass` now takes an `llm.Target`. +- Prompts: `pairCheckpointSystemPrompt` / `pairVoiceSystemPrompt` beside the originals — separate constants, not a clause, since the English ones open by naming the reader an ESL learner. Both carry a **"never translate it into English"** line, which is the instruction the model most wants to disobey: asked to improve Portuguese by an assistant that is English-shaped by training, it hands back an English rendering, and that is a translation card and not a correction. The English prompts are untouched, byte for byte, and a golden in `internal/llm/target_test.go` says so out loud (duplicated on purpose — a golden copied from the constant it guards guards nothing). +- **Collocation deliberately did not move.** Its prompt is per-language *knowledge*, not framing: "natives usually say" for Portuguese is a claim Petal has no grounds to make yet. A flipped document gets the pass it always got. +- Tests: `doclang_test.go` (monolingual pt/zh/en, one-English-line-in-a-Portuguese-journal, English-quoting-Portuguese, untested pair, emptied document holds its verdict, the band **from both directions**, corroboration floor, the two Target decisions proven independent by the only pair that can — a `learning_pair` zh account writing Chinese gets Chinese corrections explained in English — plus handler tests in the shape of `pairlang_test.go`: a pt-PT document reaches the model as a Portuguese checkpoint *and* a Portuguese voice pass, the verdict persists, and a language flip re-opens the already-checked English sentence). `target_test.go` for the prompts. go build/vet/test clean. +- **Not deployed, and no live smoke.** The handler tests drive the real router and a real DB, which is what a smoke would have shown; what neither proves is how Qwen behaves on a Portuguese document — in particular whether the "never translate" line holds. That wants the deploy Phases 24–27 are also waiting on, and a pt-PT reader. +- **Left standing, deliberately, and visible on a flipped document:** the offline mechanics rule pack is English (it is the one that filed a note about the stray English sentence), `isTranslation` still points one way (item 5), and the Ask Petal seed still round-trips (item 6). + ### Later / explicitly not now - Learner-facing Chinese writing (the zh pair's second direction) — own phase with its own spec (SUGGESTIONS §4); only after Phases 19–21 prove the pair model - ~~Spanish pair — gated on DreamDict growing an es dataset~~ **ungated 2026-07-26**, **shipped (code) 2026-07-28** — see Phase 25. What it still owes: a deploy, a native reader, and a writer who actually uses it. diff --git a/internal/db/db.go b/internal/db/db.go index fca9df4..0da7b3e 100644 --- a/internal/db/db.go +++ b/internal/db/db.go @@ -589,6 +589,30 @@ CREATE INDEX idx_suggestions_resolved ON suggestions(status, resolved_at); stmt: ` ALTER TABLE users ADD COLUMN direction TEXT NOT NULL DEFAULT 'learning_en' CHECK(direction IN ('learning_en','learning_pair')); +`, + }, + { + // Which language this document is written in — 'en' or 'pair'. + // + // It is stored, rather than recomputed per pass and forgotten, for one + // reason: the verdict has hysteresis (see suggestions/doclang.go). A + // bilingual document sits between the two thresholds, and "whatever it + // was last time" is only an answer if last time was written down. Without + // the column a mixed paragraph would alternate its cards' language + // between passes. + // + // 'pair' rather than a language code, deliberately. Which language "pair" + // names is the owner's users.pair_lang, so changing her pair re-reads her + // documents instead of stranding a stale language name on every one of + // them. + // + // Empty is the backfill and means English: every document that exists + // today was written by a Mandarin native practising English, and English + // is what every surface assumed before this phase. + name: "0017_document_lang", + stmt: ` +ALTER TABLE documents ADD COLUMN doc_lang TEXT NOT NULL DEFAULT '' + CHECK(doc_lang IN ('', 'en', 'pair')); `, }, } diff --git a/internal/llm/checkpoint.go b/internal/llm/checkpoint.go index 2b818cb..27d1070 100644 --- a/internal/llm/checkpoint.go +++ b/internal/llm/checkpoint.go @@ -41,9 +41,9 @@ type checkpointResponse struct { // RunCheckpoint sends the grammar checkpoint and parses the JSON result. It // applies the latency-guard truncation and the checkpoint sampling parameters // from the spec. -func RunCheckpoint(ctx context.Context, client LLMClient, contentText, tone string, _ Lang) ([]RawSuggestion, error) { +func RunCheckpoint(ctx context.Context, client LLMClient, contentText, tone string, t Target) ([]RawSuggestion, error) { raw, err := client.Complete(ctx, CompletionRequest{ - Messages: CheckpointMessages(TruncateDoc(contentText), tone), + Messages: CheckpointMessages(TruncateDoc(contentText), tone, t), MaxTokens: checkpointMaxTokens, Temperature: 0.3, RepetitionPenalty: 1.15, diff --git a/internal/llm/collocation.go b/internal/llm/collocation.go index 541ae78..7549511 100644 --- a/internal/llm/collocation.go +++ b/internal/llm/collocation.go @@ -20,9 +20,9 @@ const CollocationInterval = 25 * time.Second // The tone argument is accepted for a uniform pass signature and passed through // to the prompt so a hint can prefer a register-appropriate pairing. `lang` is // the writer's pair language — the one each hint's short gloss is written in. -func RunCollocation(ctx context.Context, client LLMClient, contentText, tone string, lang Lang) ([]RawSuggestion, error) { +func RunCollocation(ctx context.Context, client LLMClient, contentText, tone string, t Target) ([]RawSuggestion, error) { raw, err := client.Complete(ctx, CompletionRequest{ - Messages: CollocationMessages(contentText, tone, lang), + Messages: CollocationMessages(contentText, tone, t), MaxTokens: 2048, Temperature: 0.3, RepetitionPenalty: 1.15, diff --git a/internal/llm/lang_test.go b/internal/llm/lang_test.go index ac5b215..55de972 100644 --- a/internal/llm/lang_test.go +++ b/internal/llm/lang_test.go @@ -31,7 +31,7 @@ func TestLangForFallsBackToDefault(t *testing.T) { func TestPromptsNameTheWritersLanguage(t *testing.T) { pt := LangFor("pt-PT") - collocation := CollocationMessages("The rain was strong.", "casual", pt)[0].Content + collocation := CollocationMessages("The rain was strong.", "casual", EnglishTarget(pt))[0].Content if !strings.Contains(collocation, "European Portuguese") { t.Fatalf("collocation prompt doesn't ask for a pt-PT gloss:\n%s", collocation) } @@ -73,7 +73,7 @@ func TestPromptsNameTheWritersLanguage(t *testing.T) { func TestDefaultPairStillReadsAsBefore(t *testing.T) { zh := LangFor("zh") - if got := CollocationMessages("x", "", zh)[0].Content; !strings.Contains(got, "Simplified Chinese (Mandarin) gloss in parentheses") { + if got := CollocationMessages("x", "", EnglishTarget(zh))[0].Content; !strings.Contains(got, "Simplified Chinese (Mandarin) gloss in parentheses") { t.Fatalf("zh collocation gloss changed:\n%s", got) } if got := TranslateMessages("x", zh)[0].Content; !strings.Contains(got, "natural, friendly Simplified Chinese (Mandarin)") { diff --git a/internal/llm/prompts.go b/internal/llm/prompts.go index 20df3f3..d0f8c04 100644 --- a/internal/llm/prompts.go +++ b/internal/llm/prompts.go @@ -45,11 +45,55 @@ func toneGuidance(tone string) string { "be improved, prefer suggestions that fit that tone, and gently flag wording that clashes with it." } +// pairCheckpointSystemPrompt is the grammar checkpoint for a document written in +// the writer's own language rather than in English. +// +// It is a separate constant rather than a language clause appended to +// checkpointSystemPrompt, because that prompt opens by naming the reader as an +// ESL learner and asks for "common ESL patterns" — appending "and explain in +// Portuguese" would hand the model two contradictory framings. Only the framing +// differs; the JSON contract and the tone clause below it are the same +// instructions in the same order, so the two prompts stay comparable. +// +// The "never translate" line is the one the model most wants to disobey: asked +// to improve Portuguese while being an English writing assistant by training, it +// will happily hand back an English rendering, which is a translation card +// (Phase 25's `isTranslation`) and not a correction. +const pairCheckpointSystemPrompt = `You are a warm, encouraging writing assistant. The person you are helping is ` + + `writing in %[1]s, and the text below is %[1]s. ` + + `Analyze it and identify up to 5 issues: grammar errors, unnatural phrasing, ` + + `incorrect idiom usage, or unclear sentences. + +Both "original" and "replacement" must be written in %[1]s. You are improving their %[1]s writing — ` + + `never translate it into English, and never suggest they write in English instead. +Write every "explanation" in %[2]s. + +Be specific, friendly, and explain WHY each suggestion improves the writing.%[3]s + +Respond ONLY with valid JSON. No preamble, no markdown fences. Format: +{ + "suggestions": [ + { + "original": "exact text from the document that needs fixing", + "replacement": "corrected version", + "explanation": "friendly one-sentence explanation", + "type": "grammar|phrasing|idiom|clarity" + } + ] +} + +If the writing looks good, return: {"suggestions": []}` + // CheckpointMessages builds the message array for a grammar checkpoint over the -// given (already-truncated) document text, steered toward the document's tone. -func CheckpointMessages(contentText, tone string) []Message { +// given (already-truncated) document text, steered toward the document's tone +// and aimed at the language the document is actually written in. +func CheckpointMessages(contentText, tone string, t Target) []Message { + system := fmt.Sprintf(checkpointSystemPrompt, toneGuidance(tone)) + if t.Flipped() { + system = fmt.Sprintf(pairCheckpointSystemPrompt, t.Correct.Name, t.Explain.Name, toneGuidance(tone)) + } return []Message{ - {Role: "system", Content: fmt.Sprintf(checkpointSystemPrompt, toneGuidance(tone))}, + {Role: "system", Content: system}, {Role: "user", Content: contentText}, } } @@ -82,12 +126,48 @@ Respond ONLY with valid JSON. No preamble, no markdown fences. Format: If the voice is consistent throughout, return: {"suggestions": []}` +// pairVoiceSystemPrompt is the voice pass for a document in the writer's own +// language. Voice consistency is the one pass that transfers across languages +// unchanged — a paragraph that reads as pasted from elsewhere reads that way in +// any language — so only the framing and the explanation language move. +const pairVoiceSystemPrompt = `You are a warm, encouraging writing assistant. The person you are helping is writing ` + + `in %[1]s. You are reviewing a COMPLETE %[1]s document for VOICE CONSISTENCY only — not grammar. + +Read the whole document to learn the writer's natural voice, then identify any passages (2 or more sentences) ` + + `that feel tonally inconsistent with the surrounding writing — unusually formal, unusually polished, or phrased ` + + `in a way that differs from the writer's established voice elsewhere in the document. These often signal text ` + + `that was paraphrased too closely from another source. Do not flag the first paragraph (there is no baseline yet). ` + + `Do not flag grammar or spelling mistakes — only voice. + +Quote each passage exactly as it appears, in %[1]s. Write every "explanation" in %[2]s. + +Respond ONLY with valid JSON. No preamble, no markdown fences. Format: +{ + "suggestions": [ + { + "original": "exact passage from the document that feels inconsistent", + "replacement": null, + "explanation": "friendly one-sentence note about why this passage sounds unlike the rest", + "type": "voice" + } + ] +} + +If the voice is consistent throughout, return: {"suggestions": []}` + // VoiceMessages builds the message array for a voice-consistency pass. Unlike // the checkpoint, the caller passes the WHOLE document (no truncation) — voice // consistency is judged against the established voice everywhere else. -func VoiceMessages(contentText string) []Message { +// +// The pass had no language argument at all before Phase 28, which was the same +// English assumption the checkpoint made, just unstated. +func VoiceMessages(contentText string, t Target) []Message { + system := voiceSystemPrompt + if t.Flipped() { + system = fmt.Sprintf(pairVoiceSystemPrompt, t.Correct.Name, t.Explain.Name) + } return []Message{ - {Role: "system", Content: voiceSystemPrompt}, + {Role: "system", Content: system}, {Role: "user", Content: contentText}, } } @@ -133,10 +213,17 @@ If every pairing already sounds natural, return: {"suggestions": []}` // CollocationMessages builds the message array for a collocation pass over the // WHOLE document (no truncation), gently steered toward the document's tone so a // hint can prefer a register-appropriate pairing. The parenthetical gloss is -// written in the writer's own language. -func CollocationMessages(contentText, tone string, lang Lang) []Message { +// written in the writer's own language — `Pair`, not `Explain`: the gloss is +// addressed to her rather than to the document. +// +// The coach itself remains English-only. Collocation lists are the one thing +// here that is genuinely per-language knowledge rather than framing, and +// "natives usually say" for Portuguese is a claim this prompt has no grounds to +// make yet; a flipped document simply gets the pass it always got. (Phase 28 +// moved the checkpoint and the voice pass; this one waits for evidence.) +func CollocationMessages(contentText, tone string, t Target) []Message { return []Message{ - {Role: "system", Content: fmt.Sprintf(collocationSystemPrompt, toneGuidance(tone), lang.Name)}, + {Role: "system", Content: fmt.Sprintf(collocationSystemPrompt, toneGuidance(tone), t.Pair.Name)}, {Role: "user", Content: contentText}, } } diff --git a/internal/llm/target.go b/internal/llm/target.go new file mode 100644 index 0000000..434d69b --- /dev/null +++ b/internal/llm/target.go @@ -0,0 +1,54 @@ +package llm + +// Which language a pass corrects, and which language it explains in. +// +// Until Phase 28 there was no question to answer: every prompt was written +// around English prose explained in English, and the pair language reached her +// only when she asked for it (Ask Petal, the explanation translator). That is +// the right default for a writer practising English and the wrong one for a +// document she wrote in her own language, where Petal would read Portuguese, +// say nothing about it, and file a mechanics note about the one English +// sentence at the end. +// +// The two fields are two different decisions reading two different pieces of +// state, and collapsing them would be the bug: +// +// - Correct follows the DOCUMENT. Portuguese prose gets Portuguese +// corrections; that is the whole point. +// - Explain follows the WRITER — the half of her pair she is *not* learning +// (users.direction), because an explanation is teaching, and teaching lands +// in the language she reads most easily. +// +// Today those two coincide for every account that exists: `learnerPairs` is +// {"zh"}, so fr, es and pt-PT writers are all `learning_en` and their +// non-learned half *is* the pair language. That equality is a fact about +// today's roster, not about the design — the same shape of assumption that had +// to be unpicked from `pair_lang` in migration 0016. Keep them apart. +type Target struct { + // Correct is the language the writing is in, and so the language both + // `original` and `replacement` must be written in. + Correct Lang + // Explain is the language each explanation is written in. + Explain Lang + // Pair is the writer's pair language regardless of what this document is + // written in. The collocation coach's parenthetical gloss is addressed to + // her rather than to the document, so it reads this and not Correct. + Pair Lang +} + +// English as the prompts name it. Not in `langs`: that map answers "which +// language is the writer's half of the pair", and English is the constant on +// the other side of every pair Petal supports. +var English = Lang{Code: "en", Name: "English", Why: "why"} + +// EnglishTarget is the pre-Phase-28 behaviour, made explicit: an English +// document, corrected and explained in English, for a writer whose pair +// language is `pair`. Every existing user is on this path and the prompt it +// produces is byte-identical to the one that shipped before this phase. +func EnglishTarget(pair Lang) Target { + return Target{Correct: English, Explain: English, Pair: pair} +} + +// Flipped reports whether this document is in the pair language rather than in +// English — i.e. whether the pass is reading her own language. +func (t Target) Flipped() bool { return t.Correct.Code != English.Code } diff --git a/internal/llm/target_test.go b/internal/llm/target_test.go new file mode 100644 index 0000000..55a4aef --- /dev/null +++ b/internal/llm/target_test.go @@ -0,0 +1,115 @@ +package llm + +import ( + "strings" + "testing" +) + +// The prompt every account is on today, written out in full. +// +// Phase 28 gave the checkpoint a second framing for documents in the writer's +// own language, and the risk of that change is not that the new prompt is wrong +// — it is that the old one moved by a word while nobody was looking. Every user +// who exists is a Mandarin native writing English, so this string is the one +// Petal actually sends, all day. It is duplicated here on purpose: a golden +// copied from the constant it guards guards nothing. +const goldenEnglishCheckpointPrompt = `You are a warm, encouraging writing assistant helping someone who speaks English as a second language. Analyze the text below and identify up to 5 issues: grammar errors, unnatural phrasing, incorrect idiom usage, or unclear sentences that are common ESL patterns. + +Be specific, friendly, and explain WHY each suggestion improves the writing. + +Respond ONLY with valid JSON. No preamble, no markdown fences. Format: +{ + "suggestions": [ + { + "original": "exact text from the document that needs fixing", + "replacement": "corrected version", + "explanation": "friendly one-sentence explanation", + "type": "grammar|phrasing|idiom|clarity" + } + ] +} + +If the writing looks good, return: {"suggestions": []}` + +func TestEnglishDocumentPromptIsUnchanged(t *testing.T) { + msgs := CheckpointMessages("I has two apple.", "", EnglishTarget(LangFor("zh"))) + if got := msgs[0].Content; got != goldenEnglishCheckpointPrompt { + t.Fatalf("the English checkpoint prompt moved:\n--- got ---\n%s\n--- want ---\n%s", got, goldenEnglishCheckpointPrompt) + } + if msgs[1].Content != "I has two apple." { + t.Fatalf("document text mangled: %q", msgs[1].Content) + } + + // The tone clause still lands, in the same place it always did. + toned := CheckpointMessages("x", "academic", EnglishTarget(LangFor("zh")))[0].Content + if !strings.Contains(toned, "formal, academic, and objective") { + t.Fatalf("English checkpoint lost its tone guidance:\n%s", toned) + } +} + +// A document in her own language gets a prompt that names that language, keeps +// the corrections inside it, and drops the framing that only makes sense when +// the thing being written is English. +func TestFlippedCheckpointPrompt(t *testing.T) { + pt := LangFor("pt-PT") + system := CheckpointMessages("Hoje foi um dia bom.", "casual", Target{Correct: pt, Explain: pt, Pair: pt})[0].Content + + if !strings.Contains(system, "European Portuguese") { + t.Fatalf("flipped checkpoint doesn't name the language:\n%s", system) + } + if strings.Contains(system, "second language") || strings.Contains(system, "ESL") { + t.Fatalf("flipped checkpoint kept the ESL framing:\n%s", system) + } + if !strings.Contains(system, "never translate it into English") { + t.Fatalf("flipped checkpoint doesn't forbid translating:\n%s", system) + } + // The shared contract below the framing has to survive the split. + for _, want := range []string{`"suggestions"`, `"replacement"`, "grammar|phrasing|idiom|clarity", "relaxed, friendly, and conversational"} { + if !strings.Contains(system, want) { + t.Fatalf("flipped checkpoint dropped %q:\n%s", want, system) + } + } + if strings.Contains(system, "%!") { + t.Fatalf("flipped checkpoint has a formatting error:\n%s", system) + } +} + +// The two decisions are separate arguments and must reach the prompt separately: +// corrections in the document's language, the explanation in the language she +// reads most easily. Only the zh pair can be travelled both ways today, so it is +// the only one that can prove they haven't been quietly collapsed into one. +func TestFlippedPromptsExplainInTheirOwnLanguage(t *testing.T) { + zh := LangFor("zh") + + // Native Mandarin, practising English, writing Chinese: both halves Chinese. + both := CheckpointMessages("今天天气很好。", "", Target{Correct: zh, Explain: zh, Pair: zh})[0].Content + if strings.Count(both, "Simplified Chinese (Mandarin)") < 2 { + t.Fatalf("expected corrections and explanations both in Chinese:\n%s", both) + } + if strings.Contains(both, "explanation"+`" in English`) { + t.Fatalf("explanation language leaked to English:\n%s", both) + } + + // Native English, learning Chinese, writing Chinese: Chinese corrections, + // English explanations. + split := CheckpointMessages("今天天气很好。", "", Target{Correct: zh, Explain: English, Pair: zh})[0].Content + if !strings.Contains(split, `Write every "explanation" in English.`) { + t.Fatalf("learner direction didn't get English explanations:\n%s", split) + } + if !strings.Contains(split, "writing in Simplified Chinese (Mandarin)") { + t.Fatalf("learner direction lost its Chinese corrections:\n%s", split) + } + + // Same for the voice pass, which had no language at all before this phase. + voice := VoiceMessages("今天天气很好。", Target{Correct: zh, Explain: English, Pair: zh})[0].Content + if !strings.Contains(voice, `Write every "explanation" in English.`) || !strings.Contains(voice, "Simplified Chinese") { + t.Fatalf("flipped voice prompt got its languages wrong:\n%s", voice) + } + if strings.Contains(voice, "second language") { + t.Fatalf("flipped voice prompt kept the ESL framing:\n%s", voice) + } + // An English document still gets exactly the voice prompt it always got. + if got := VoiceMessages("x", EnglishTarget(zh))[0].Content; got != voiceSystemPrompt { + t.Fatalf("English voice prompt moved:\n%s", got) + } +} diff --git a/internal/llm/voice.go b/internal/llm/voice.go index ea4aaa0..2d64487 100644 --- a/internal/llm/voice.go +++ b/internal/llm/voice.go @@ -19,9 +19,9 @@ const VoiceInterval = 20 * time.Second // The tone argument is accepted for a uniform pass signature but ignored: voice // consistency is judged against the document's own established voice, not an // externally-chosen register. -func RunVoice(ctx context.Context, client LLMClient, contentText, _ string, _ Lang) ([]RawSuggestion, error) { +func RunVoice(ctx context.Context, client LLMClient, contentText, _ string, t Target) ([]RawSuggestion, error) { raw, err := client.Complete(ctx, CompletionRequest{ - Messages: VoiceMessages(contentText), + Messages: VoiceMessages(contentText, t), MaxTokens: 2048, Temperature: 0.3, RepetitionPenalty: 1.15, diff --git a/internal/suggestions/doclang.go b/internal/suggestions/doclang.go new file mode 100644 index 0000000..3de1305 --- /dev/null +++ b/internal/suggestions/doclang.go @@ -0,0 +1,179 @@ +package suggestions + +import "strings" + +// What language is this DOCUMENT in — as opposed to this span. +// +// `readsAsPairLang` next door answers a span-level question for the purpose of +// labelling one card, and it is written to under-claim: two marker words, or +// nothing. A whole document needs the opposite temperament. A proportion, not a +// presence — one Portuguese quotation inside an English essay must not flip the +// entire pass into Portuguese, and one English sentence at the end of a +// Portuguese journal must not keep it in English. +// +// Three properties, in the order they bite: +// +// - Decided over the WHOLE document, never a chunk. The grammar checkpoint +// sends only the sentences that changed, so a verdict computed from what it +// asked about would put an English card in a Portuguese journal the moment +// she edits its one English line. The caller passes content_text, always. +// +// - Hysteresis. A bilingual paragraph sits near whatever single threshold we +// pick, and a document crossing it every few keystrokes would alternate card +// languages between passes — the same instability the mascot needed a band +// for. Flip to the pair at 70% and back only below 40%; in between, whatever +// it was last time stands. The band is the feature, not a rounding +// tolerance. +// +// - Plain code, no model call. The house rule is that the LLM is garnish, +// never a gatekeeper: a document must not become uncheckable because the +// inference box is down. +const ( + docLangEnglish = "en" + docLangPair = "pair" +) + +// The band. Deliberately wide: the cost of an unnecessary flip is every card in +// the document changing language, which is far more startling than a paragraph +// of mixed writing being read as whichever language it was a minute ago. +const ( + flipToPairAt = 0.70 + flipToEnglishBelow = 0.40 +) + +// documentLang returns the language verdict for a document, given the verdict it +// carried before. `prev` is "" for a document that has never been read. +// +// The result is one of docLangEnglish / docLangPair — not a language code. Which +// language "pair" means is the writer's `pair_lang`, and keeping the stored +// verdict relative to her pair means changing her pair doesn't strand a stale +// language name on every document she owns. +func documentLang(contentText, pairLang, prev string) string { + if prev != docLangPair { + prev = docLangEnglish + } + p := normalizePairLang(pairLang) + if !hasLangTest(p) { + // No test for this pair: say English, which is what every surface did + // before this phase. A wrong flip is louder than a missing one. + return docLangEnglish + } + + var pair, english int + for _, c := range splitChunks(contentText, "") { + switch sentenceLang(c.text, p) { + case docLangPair: + pair++ + case docLangEnglish: + english++ + } + } + decided := pair + english + if decided == 0 { + // Nothing to go on — an empty document, a list of numbers, a title. Hold + // the previous verdict rather than resetting a Portuguese journal to + // English because she cleared it to start again. + return prev + } + + ratio := float64(pair) / float64(decided) + switch { + case ratio >= flipToPairAt && corroborated(contentText, p): + return docLangPair + case ratio < flipToEnglishBelow: + return docLangEnglish + default: + return prev + } +} + +// sentenceLang classifies one sentence as pair-language, English, or neither. +// +// Neither is a real answer and carries weight: a sentence with no evidence +// either way ("Bom dia.", "OK.", a heading) is left out of the ratio entirely +// rather than counted for the language it isn't. Counting the undecided as +// English is what would keep a Portuguese document in English forever, since +// short sentences carry no markers. +func sentenceLang(s, p string) string { + if p == "zh" { + han, latin := scriptCounts(s) + switch { + case han >= 2 && han > latin: + return docLangPair + case latin >= 3 && latin > han: + return docLangEnglish + } + return "" + } + // A Latin pair shares its alphabet with English, so both sides are counted + // the same way and the larger pile of evidence wins. A tie — including no + // evidence at all — is no answer, which is why the English list below is + // curated as carefully against the pair languages as theirs is against + // English. + pair := distinctMarkers(s, latinMarkers[p]) + eng := distinctMarkers(s, englishMarkers) + switch { + case pair > eng: + return docLangPair + case eng > pair: + return docLangEnglish + } + return "" +} + +// corroborated requires the document as a whole to carry real evidence of the +// pair language before the pass flips into it. A two-sentence document of +// "Sim." / "Não." would otherwise reach 100% on almost nothing; a flip changes +// every card in the document, so it has to be earned document-wide and not only +// in proportion. +func corroborated(contentText, p string) bool { + if p == "zh" { + han, _ := scriptCounts(contentText) + return han >= 8 + } + return distinctMarkers(contentText, latinMarkers[p]) >= 3 +} + +// hasLangTest reports whether readsAsPairLang / sentenceLang know how to test +// this pair at all. Kept beside the tests it describes so a new pair that adds +// markers without adding itself here fails loudly in review rather than quietly +// at runtime. +func hasLangTest(p string) bool { + if p == "zh" { + return true + } + return len(latinMarkers[p]) > 0 +} + +// English function words, curated against the pair languages exactly as +// `latinMarkers` is curated against English. +// +// Every word here is one a Portuguese, French or Spanish sentence has no reason +// to contain. Deliberately absent, each a false English vote waiting to happen +// in someone's own language: "on" and "son" (French), "as", "a", "o", "e", "no", +// "os" (Portuguese), "no", "para", "sin" (Spanish), and "is"-alikes that are +// really other languages' words. The list is short on purpose — it does not need +// coverage, only a reliable vote in the sentences where the pair list is silent. +var englishMarkers = words( + "the", "and", "is", "are", "was", "were", "be", "been", "being", + "of", "to", "that", "this", "these", "those", "with", "from", "for", + "have", "has", "had", "they", "them", "their", "there", "then", "than", + "what", "which", "when", "where", "why", "how", "who", + "will", "would", "should", "could", "can", "about", "because", + "into", "some", "such", "only", "very", "much", "many", "other", + "our", "your", "its", "it", "he", "she", "we", "you", "but", "not", + "just", "like", "also", "most", "over", "after", "before", "between", + "through", "said", "says", "get", "got", "make", "made", "know", "think", + "thing", "things", "time", "people", "here", "always", "never", "something", + "want", "need", "feel", "day", "today", "good", "really", "still", "even", +) + +// normalizeDocLang folds a stored verdict into the two values the rest of the +// code reasons about: the column holds "" for a document nothing has read yet, +// and that means English, which is what every surface did before this phase. +func normalizeDocLang(v string) string { + if strings.TrimSpace(v) == docLangPair { + return docLangPair + } + return docLangEnglish +} diff --git a/internal/suggestions/doclang_test.go b/internal/suggestions/doclang_test.go new file mode 100644 index 0000000..ab32c9a --- /dev/null +++ b/internal/suggestions/doclang_test.go @@ -0,0 +1,261 @@ +package suggestions + +import ( + "net/http" + "path/filepath" + "strings" + "testing" + + "github.com/go-chi/chi/v5" + + "gitea.parodia.dev/drwily/petal/internal/auth" + "gitea.parodia.dev/drwily/petal/internal/db" + "gitea.parodia.dev/drwily/petal/internal/llm" +) + +// A monolingual document in either language has to be read as that language, and +// the mixed cases in between are where the whole design lives: one quotation +// must not move a document, and one leftover English line must not hold a +// journal in English. +func TestDocumentLangReadsWholeDocuments(t *testing.T) { + const ptJournal = "Hoje foi um dia muito bom. Eu gosto de escrever aqui todas as noites. " + + "A minha irmã também quer aprender. Não sei porque isso é tão difícil para mim." + const enEssay = "The weather was very cold this morning. I think that the bus was late again. " + + "She told me about the meeting, but I could not hear what they said." + const zhJournal = "今天天气很好。我和妹妹一起去公园散步。我们看到很多花。" + + tests := []struct { + name string + text string + pairLang string + prev string + want string + }{ + {"portuguese journal", ptJournal, "pt-PT", "", docLangPair}, + {"english essay", enEssay, "pt-PT", "", docLangEnglish}, + {"chinese journal", zhJournal, "zh", "", docLangPair}, + {"english essay, zh writer", enEssay, "zh", "", docLangEnglish}, + + // One English sentence at the end of a Portuguese journal is the case that + // motivated the whole phase: the pass must stay in Portuguese. + { + "portuguese with one english line", + ptJournal + " I will write more tomorrow.", + "pt-PT", "", docLangPair, + }, + // And the mirror: an English essay quoting a line of Portuguese is still an + // English essay. + { + "english quoting portuguese", + enEssay + " She wrote: \"Eu não sei o que dizer.\"", + "pt-PT", "", docLangEnglish, + }, + // A pair Petal has no test for cannot flip anything. Saying English is what + // every surface did before this phase. + {"untested pair", ptJournal, "de", "", docLangEnglish}, + // Nothing to go on holds the previous answer rather than resetting a + // journal because she cleared it to start again. + {"emptied portuguese journal", "", "pt-PT", docLangPair, docLangPair}, + {"emptied english essay", " \n ", "pt-PT", docLangEnglish, docLangEnglish}, + // Proportion, not presence: a couple of Portuguese words are not a + // Portuguese document even though readsAsPairLang would label that span. + { + "english with a portuguese phrase", + enEssay + " The sign said pão com manteiga.", + "pt-PT", "", docLangEnglish, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := documentLang(tc.text, tc.pairLang, tc.prev); got != tc.want { + t.Fatalf("documentLang = %q, want %q", got, tc.want) + } + }) + } +} + +// The band, from both directions. A document sitting inside it keeps whatever it +// was, and that is the point: without it, a bilingual paragraph would alternate +// its cards' language every few keystrokes as she typed across the threshold. +func TestDocumentLangHysteresis(t *testing.T) { + // Half and half: two Portuguese sentences, two English ones. Inside the band + // from either side. + const mixed = "Eu gosto muito de escrever aqui. A minha irmã não sabe porque é difícil. " + + "The weather was very cold this morning. I think that they said the same thing." + + if got := documentLang(mixed, "pt-PT", docLangEnglish); got != docLangEnglish { + t.Fatalf("mixed document from english = %q, want it to stay %q", got, docLangEnglish) + } + if got := documentLang(mixed, "pt-PT", docLangPair); got != docLangPair { + t.Fatalf("mixed document from pair = %q, want it to stay %q", got, docLangPair) + } + + // Above the upper threshold it flips regardless of where it came from; below + // the lower one it flips back regardless. + const mostlyPT = "Eu gosto muito de escrever aqui. A minha irmã não sabe porque é difícil. " + + "Hoje foi um dia bom para mim. Amanhã também quero escrever mais uma coisa. " + + "I think so too." + if got := documentLang(mostlyPT, "pt-PT", docLangEnglish); got != docLangPair { + t.Fatalf("mostly-portuguese from english = %q, want %q", got, docLangPair) + } + const mostlyEN = "The weather was very cold this morning. I think that they said the same thing. " + + "She could not hear what the other people were saying about it. Eu não sei." + if got := documentLang(mostlyEN, "pt-PT", docLangPair); got != docLangEnglish { + t.Fatalf("mostly-english from pair = %q, want %q", got, docLangEnglish) + } +} + +// Corroboration: a proportion computed over almost nothing is not evidence. Two +// bare words at 100% must not flip a document, because a flip rewrites every +// card in it. +func TestDocumentLangNeedsCorroboration(t *testing.T) { + if got := documentLang("Não. Eu.", "pt-PT", docLangEnglish); got != docLangEnglish { + t.Fatalf("two bare words flipped the document: %q", got) + } + if got := documentLang("我。", "zh", docLangEnglish); got != docLangEnglish { + t.Fatalf("two Han runes flipped the document: %q", got) + } +} + +// The two language decisions are genuinely independent, and only the zh pair can +// prove it today — it is the one pair that can be travelled in both directions. +// +// A Mandarin native practising English who writes Chinese wants Chinese +// corrections explained in Chinese. An English native learning Chinese who writes +// Chinese wants the same Chinese corrections explained in English. Same document, +// same Correct, different Explain. +func TestTargetSeparatesCorrectedFromExplained(t *testing.T) { + learningEn := targetFor("zh", auth.DirectionLearningEn, docLangPair) + if learningEn.Correct.Code != "zh" || learningEn.Explain.Code != "zh" { + t.Fatalf("learning_en on a Chinese document: correct=%s explain=%s", learningEn.Correct.Code, learningEn.Explain.Code) + } + + learningPair := targetFor("zh", auth.DirectionLearningPair, docLangPair) + if learningPair.Correct.Code != "zh" { + t.Fatalf("learner direction changed what gets corrected: %s", learningPair.Correct.Code) + } + if learningPair.Explain.Code != "en" { + t.Fatalf("learner direction explained in %s, want English", learningPair.Explain.Code) + } + + // An English document is the path every account is on today, in either + // direction: English corrections, English explanations, her language still on + // the Ask Petal and translate taps. + for _, dir := range []string{auth.DirectionLearningEn, auth.DirectionLearningPair} { + got := targetFor("zh", dir, docLangEnglish) + if got.Flipped() || got.Explain.Code != "en" { + t.Fatalf("english document with direction %s: %+v", dir, got) + } + if got.Pair.Code != "zh" { + t.Fatalf("english document lost the writer's pair: %+v", got) + } + } +} + +// newDirectedServer seeds one writer on a given pair and direction, with a +// document of her own. Like newPairServer, but the direction is the variable. +func newDirectedServer(t *testing.T, client llm.LLMClient, pairLang, direction, text string) (http.Handler, string, *Handler) { + t.Helper() + database, err := db.Open(filepath.Join(t.TempDir(), "doclang.db")) + if err != nil { + t.Fatalf("open db: %v", err) + } + t.Cleanup(func() { database.Close() }) + + const userID = "writer-directed" + if _, err := database.Exec( + `INSERT INTO users (id, email, display_name, pair_lang, direction) VALUES (?, ?, ?, ?, ?)`, + userID, "d@example.com", "Writer", pairLang, direction, + ); err != nil { + t.Fatalf("seed user: %v", err) + } + + var docID string + if err := database.QueryRow( + `INSERT INTO documents (user_id, content_text) VALUES (?, ?) RETURNING id`, + userID, text, + ).Scan(&docID); err != nil { + t.Fatalf("seed doc: %v", err) + } + + h := New(database, client) + h.Limit = llm.NewRateLimiter(0) + h.VoiceLimit = llm.NewRateLimiter(0) + r := chi.NewRouter() + r.Route("/docs", func(dr chi.Router) { h.RegisterDocRoutes(dr) }) + r.Mount("/suggestions", h.Routes()) + return auth.Middleware(auth.StaticResolver(userID))(r), docID, h +} + +const ptDocument = "Hoje foi um dia muito bom. Eu gosto de escrever aqui todas as noites. " + + "A minha irmã também quer aprender comigo. Não sei porque isso é tão difícil para mim." + +// End to end: a Portuguese document reaches the model as a Portuguese +// checkpoint. This is the observed bug from 2026-07-28 — two pt-PT sentences +// drew no cards at all, because Petal was reading them as bad English. +func TestCheckpointFollowsTheDocumentLanguage(t *testing.T) { + client := &stubClient{response: `{"suggestions":[]}`} + srv, docID, _ := newDirectedServer(t, client, "pt-PT", auth.DirectionLearningEn, ptDocument) + + if rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/check", ""); rec.Code != http.StatusOK { + t.Fatalf("check: code=%d body=%s", rec.Code, rec.Body) + } + if !strings.Contains(client.lastPrompt, "European Portuguese") { + t.Fatalf("checkpoint didn't follow the document into Portuguese:\n%s", client.lastPrompt) + } + if strings.Contains(client.lastPrompt, "second language") { + t.Fatalf("checkpoint kept the ESL framing on a Portuguese document:\n%s", client.lastPrompt) + } + + // And the voice pass, which had no language argument at all before this phase. + if rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/voice", ""); rec.Code != http.StatusOK { + t.Fatalf("voice: code=%d body=%s", rec.Code, rec.Body) + } + if !strings.Contains(client.lastPrompt, "European Portuguese") { + t.Fatalf("voice pass didn't follow the document:\n%s", client.lastPrompt) + } +} + +// The verdict is persisted, because hysteresis needs a yesterday. +func TestDocumentLangIsRemembered(t *testing.T) { + client := &stubClient{response: `{"suggestions":[]}`} + srv, docID, h := newDirectedServer(t, client, "pt-PT", auth.DirectionLearningEn, ptDocument) + + if rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/check", ""); rec.Code != http.StatusOK { + t.Fatalf("check: code=%d body=%s", rec.Code, rec.Body) + } + var stored string + if err := h.DB.QueryRow(`SELECT doc_lang FROM documents WHERE id = ?`, docID).Scan(&stored); err != nil { + t.Fatalf("read doc_lang: %v", err) + } + if stored != docLangPair { + t.Fatalf("doc_lang = %q, want %q", stored, docLangPair) + } +} + +// A document that changes language re-opens every sentence. Without the verdict +// in the chunk salt, the sentences she didn't touch would keep serving cards +// written in the language the document no longer speaks. +func TestLanguageFlipReopensCheckedSentences(t *testing.T) { + client := &stubClient{response: `{"suggestions":[]}`} + const enStart = "The weather was very cold this morning." + srv, docID, h := newDirectedServer(t, client, "pt-PT", auth.DirectionLearningEn, enStart) + + if rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/check", ""); rec.Code != http.StatusOK { + t.Fatalf("first check: code=%d body=%s", rec.Code, rec.Body) + } + first := client.calls + + // She rewrites the document in Portuguese, keeping the first sentence. + setDocText(t, h, docID, enStart+" "+ptDocument) + if rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/check", ""); rec.Code != http.StatusOK { + t.Fatalf("second check: code=%d body=%s", rec.Code, rec.Body) + } + if client.calls == first { + t.Fatal("the flipped document was never sent to the model") + } + if !strings.Contains(client.lastPrompt, "The weather was very cold") { + t.Fatalf("the already-checked sentence was not re-opened by the flip:\n%s", client.lastPrompt) + } +} diff --git a/internal/suggestions/handlers.go b/internal/suggestions/handlers.go index f2d1e40..673922e 100644 --- a/internal/suggestions/handlers.go +++ b/internal/suggestions/handlers.go @@ -247,12 +247,45 @@ func (h *Handler) collocation(w http.ResponseWriter, r *http.Request) { } // pass is the signature shared by the grammar checkpoint and the voice pass: -// given the document text, the document's tone and the writer's pair language it -// returns the model's raw suggestions. The voice pass ignores both extras (see -// llm.RunVoice) and the checkpoint ignores the language — only the collocation -// coach writes a word of it — but one signature keeps runPass free of special -// cases. -type pass func(ctx context.Context, client llm.LLMClient, contentText, tone string, lang llm.Lang) ([]llm.RawSuggestion, error) +// given the document text, the document's tone and the languages this document +// is to be corrected and explained in, it returns the model's raw suggestions. +// The voice pass ignores the tone (see llm.RunVoice) and the collocation coach +// reads only the writer's pair language, but one signature keeps runPass free of +// special cases. +type pass func(ctx context.Context, client llm.LLMClient, contentText, tone string, t llm.Target) ([]llm.RawSuggestion, error) + +// targetFor resolves the two language decisions for one pass over one document. +// +// They read different state on purpose. What gets *corrected* follows the +// document, because Portuguese prose wants Portuguese corrections. What language +// the correction is *explained* in follows the writer — the half of her pair she +// is not learning — because an explanation is teaching, and teaching lands in the +// language she reads most easily. A native Portuguese speaker practising English +// gets Portuguese explained in Portuguese; a native English speaker learning +// French gets French explained in English. Neither is trapped: the other language +// stays one tap away, in both directions. +// +// An English document keeps the pre-Phase-28 behaviour exactly — explained in +// English, with her language on the Ask Petal / translate taps — which is the +// path every account today is on. +// +// The direction lookup costs nothing today: `learnerPairs` is {"zh"}, so fr, es +// and pt-PT accounts are all learning_en and their non-learned half *is* the pair +// language. This rule therefore produces "explain in the document's language" for +// every writer who currently exists. It is written out anyway to stop the +// coincidence being baked into the prompts, the way "English is the language +// being learned" was baked into pair_lang before migration 0016. +func targetFor(pairLang, direction, docLang string) llm.Target { + pair := llm.LangFor(pairLang) + if normalizeDocLang(docLang) != docLangPair { + return llm.EnglishTarget(pair) + } + explain := pair + if direction == auth.DirectionLearningPair { + explain = llm.English + } + return llm.Target{Correct: pair, Explain: explain, Pair: pair} +} // runPass is the shared body for both LLM passes. It loads the document text, // enforces the pass's per-document rate limit, runs the model, swaps in the @@ -262,17 +295,19 @@ func (h *Handler) runPass(w http.ResponseWriter, r *http.Request, limiter *llm.R docID := chi.URLParam(r, "id") userID := auth.UserID(r.Context()) - // The writer's pair language rides along with the document rather than in a - // second query: it is read from the same row-scoped lookup that already - // proves she owns this document. - var contentText, tone, pairLang string + // The writer's pair language and direction ride along with the document + // rather than in a second query: they are read from the same row-scoped + // lookup that already proves she owns this document. `doc_lang` is the + // previous language verdict, which the new one needs (hysteresis). + var contentText, tone, pairLang, direction, prevLang string err := h.DB.QueryRow( - `SELECT d.content_text, d.tone, COALESCE(u.pair_lang, '') + `SELECT d.content_text, d.tone, d.doc_lang, + COALESCE(u.pair_lang, ''), COALESCE(u.direction, '') FROM documents d JOIN users u ON u.id = d.user_id WHERE d.id = ? AND d.user_id = ?`, docID, userID, - ).Scan(&contentText, &tone, &pairLang) + ).Scan(&contentText, &tone, &prevLang, &pairLang, &direction) if errors.Is(err, sql.ErrNoRows) { httputil.ErrorJSON(w, http.StatusNotFound, "document not found") return @@ -298,16 +333,36 @@ func (h *Handler) runPass(w http.ResponseWriter, r *http.Request, limiter *llm.R return } + // What language is this document in, and so what language should its cards be + // written in? Computed from the whole content_text — never from `askText`, + // which on a chunked pass is only the sentences that changed, and would put an + // English card in a Portuguese journal the moment she edits its one English + // line. + docLang := documentLang(contentText, pairLang, prevLang) + if docLang != normalizeDocLang(prevLang) { + if _, err := h.DB.Exec( + `UPDATE documents SET doc_lang = ? WHERE id = ? AND user_id = ?`, + docLang, docID, userID, + ); err != nil { + httputil.ServerError(w, err) + return + } + } + target := targetFor(pairLang, direction, docLang) + // Decide what to ask about before spending anything: a chunked pass asks only // about the sentences that changed since it last read the document, and when // none did it doesn't call the model at all — nor consume its rate-limit slot, // so the next real edit isn't throttled by a check that had nothing to do. // // Only a chunked pass consults that record, so only it needs the tone folded - // into a sentence's identity. + // into a sentence's identity — and, next to it, the language verdict. A + // document that flips language changes every sentence's identity, so its + // old-language cards are re-checked rather than left sitting there in a + // language the rest of the document no longer speaks. salt := "" if scope.chunked { - salt = tone + salt = tone + "\x00" + docLang } chunks := splitChunks(contentText, salt) askText, fresh := contentText, chunks @@ -354,7 +409,7 @@ func (h *Handler) runPass(w http.ResponseWriter, r *http.Request, limiter *llm.R return } - raw, err := run(r.Context(), h.Client, askText, tone, llm.LangFor(pairLang)) + raw, err := run(r.Context(), h.Client, askText, tone, target) if err != nil { // Allow ran before the model call, so a failed pass would otherwise hold // the per-document slot for the full interval — stranding the frontend's From 29eb2fe1fc1dbffc6229c6abb2d8deae537c4412 Mon Sep 17 00:00:00 2001 From: prosolis <5590409+prosolis@users.noreply.github.com> Date: Tue, 28 Jul 2026 23:29:50 -0700 Subject: [PATCH 2/3] The translate card, pointed the other way, and a call that no longer happens MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 28's step (b): both remaining items are about direction, and both had a wrong answer that looked right. isTranslation could not simply be read backwards. readsAsEnglish is a deliberately low bar — Latin letters, not swamped by another script — which every Portuguese sentence clears as easily as English does, so swapping its two halves would have called every genuine Portuguese correction inside a Portuguese document a translation. The flipped direction uses sentenceLang from doclang.go instead, where English has its own curated marker list and has to out-evidence the pair language to win. The English-document path is untouched; reconcilePending carries the verdict to ask the question the right way round. The tap-through's whole observable change is a model call that stops happening. /suggestions/{id}/translate now recovers the explanation's language by re-running targetFor rather than assuming the pair, which gives today's answer everywhere except the case that was broken: the Portuguese writer whose explanation already arrived in Portuguese, previously round-tripped through the model into Portuguese again. It answers "" there, and the client's existing `res.translation.trim() || explanation` fallback seeds the bubble with the explanation itself — no frontend change at all. It deliberately does not render that explanation into English on the grounds that English is technically the other half: an unasked-for rendering into the language she is practising is noise, not a seed. Tests pin both directions of the detector, with Portuguese-in-Portuguese as the case the file exists for, plus four handler tests through the real /check and /translate paths — including the skipped seed asserting the model was never called, and the learning_pair zh learner whose English explanation still renders into Chinese. Left of the phase: (c) the garden's language tagging and read-aloud. Claude-Session: https://claude.ai/code/session_01GJHNvirh7Hzhc9RL3HAvz7 --- BUILD_PLAN.md | 10 +- internal/suggestions/doclang_test.go | 134 ++++++++++++++++++++++++++ internal/suggestions/handlers.go | 24 +++-- internal/suggestions/language.go | 33 +++++-- internal/suggestions/language_test.go | 97 ++++++++++++++++++- internal/suggestions/reconcile.go | 10 +- internal/suggestions/translate.go | 46 +++++++-- 7 files changed, 318 insertions(+), 36 deletions(-) diff --git a/BUILD_PLAN.md b/BUILD_PLAN.md index d48af4b..3117400 100644 --- a/BUILD_PLAN.md +++ b/BUILD_PLAN.md @@ -423,7 +423,7 @@ Phase 26's own outstanding item, and the thing it named as most likely to be wro - ⚠️ **What this still does not prove is the counterfactual** — that the bug would have bitten *here* without the guard. Chrome's own composition handling is robust, and a clean run with the guard in place cannot distinguish "held correctly" from "would have been fine anyway". Settling it means building with the guard reverted and asking her to type once more; offered, not assumed. - **Not deployed.** No migration; a rebuild whenever the user wants it, along with Phases 24–26. -### Phase 28 — following the writing into her own language (planned 2026-07-28; step (a) built 2026-07-28) +### Phase 28 ✅ (2026-07-28, code half) — following the writing into her own language Today every correction and every explanation comes back in English, whatever she wrote. `CheckpointMessages` (`internal/llm/prompts.go:49`) takes the text and the tone and **nothing else** — there is no language parameter to pass, so there has never been a language decision to get wrong. The pair language reaches her only on demand: tapping Ask Petal fetches a translation of the English explanation (`AskPetal.tsx:93`), and that panel answers in her language because `AskPetalSystemPrompt` is given one. Right default for a writer practising English; wrong default for a document written in Portuguese, where Petal reads the Portuguese prose, says nothing about it, and files a mechanics note about the one English sentence at the end. **Observed on the live build 2026-07-28**: two pt-PT sentences drew no cards at all. 1. [x] **The rule is two decisions, not one, and they read different state.** What gets *corrected* follows the **document** — Portuguese prose gets Portuguese corrections, which is the whole point. What language the explanation is *written in* follows the **writer**: the half of her pair she is not learning, from the `direction` column (`internal/auth/users.go:96`), because an explanation is teaching and teaching lands in the language she reads most easily. So a native Portuguese speaker practising English, writing Portuguese, gets Portuguese corrections explained in Portuguese; a native English speaker learning French, writing French, gets French corrections explained in English. Neither is trapped — the other language stays one tap away, in both directions. @@ -434,13 +434,16 @@ Today every correction and every explanation comes back in English, whatever she - **Plain code, no model call** — the house rule that the LLM is garnish, never a gatekeeper. zh is a rune-script count (already written); pt-PT, fr and es reuse `latinMarkers`, aggregated per sentence rather than per span. 3. [x] **`checkpointSystemPrompt` cannot simply take a language.** It opens with *"helping someone who speaks English as a second language"* and asks for ESL patterns — appending "explain in Portuguese" hands the model two contradictory instructions. Split it: shared JSON contract and tone clause, framing sentence filled per direction. `CheckpointMessages` and `VoiceMessages` (which has no language at all today) take **two** language arguments — corrected and explained-in — and **resist collapsing them into one `Lang` while every current account has them equal**: that equality is a fact about today's `learnerPairs`, not about the design. 4. [x] **The verdict rides with `pairLang` in the row-scoped lookup and folds into the chunk salt** next to `tone` (`handlers.go`). That is the cheap correct answer to stale cards: when a document's language flips, every sentence's identity changes, so old-language cards are re-checked rather than left sitting there in the wrong language. -5. [ ] **`isTranslation` learns which way it points.** It currently means "her language rendered into English"; in a flipped document the useful translate card is the mirror image, so the test takes the document verdict and checks the direction that matches. -6. [ ] **The tap-through has to stop assuming its direction, and this is not optional dressing** — it is what makes rule 1 safe for a learner reading explanations in English. `/suggestions/{id}/translate` always renders into the pair language today; it should render into whichever half the explanation is *not* already in, and skip the seed entirely when those coincide rather than round-tripping Portuguese into Portuguese. +5. [x] **`isTranslation` learns which way it points — and the flipped test could not be the old one read backwards.** In an English document it still means "her language rendered into English", byte-for-byte the old path. In a flipped document it means the mirror image: an English sentence she dropped into her Portuguese, rendered into Portuguese. The naive symmetry — swap the two halves — is a bug, because `readsAsEnglish` is deliberately a low bar (Latin letters, not swamped by another script) that *every Portuguese sentence also clears*, so it would have labelled every genuine Portuguese correction a translation. The flipped direction therefore uses `sentenceLang` from doclang.go instead, where English has its own marker list and has to out-evidence the pair language to win. `reconcilePending` takes the verdict alongside `pairLang` to ask the question the right way round. +6. [x] **The tap-through stops assuming its direction, and the whole observable change is a call that no longer happens.** `/suggestions/{id}/translate` now recovers the explanation's own language by re-running `targetFor` over the same three pieces of state the card was written under, and renders into whichever half the explanation is not already in. For every non-skipped case that is still the pair language, which is why nothing visible moved for existing accounts. The case that moved is the Portuguese writer on a Portuguese document: her explanation already arrived in Portuguese, so the endpoint answers `""` with no model call at all, and the client's existing `res.translation.trim() || explanation` fallback seeds the bubble with the explanation itself. The old code would have sent Portuguese to the model to be turned into Portuguese. + - **What it deliberately does not do** is render that explanation into English instead, on the grounds that English is technically "the other half". An unasked-for English rendering of an explanation she can already read is not a seed, it is noise; the language she is practising stays available through Ask Petal, which is where she can ask for it. + - **A card carries no language of its own**, so a row written before this phase — or on a document whose verdict has since flipped — is read as whatever the rule says today. The alternative is a language column on every suggestion row, and the cost of being wrong is one bubble seeded in the language it was already in. - **`internal/llm/lang.go` already carries the precision the model needs** (`"European Portuguese (pt-PT, never Brazilian Portuguese)"`). Spanish wants the same care pointed the other way: the shipped dictionary deliberately accepts the whole Spanish-speaking world (Phase 25), so the prompt must not quietly impose peninsular usage. - **Spellcheck must not change.** "A word is a misspelling only when *both* dictionaries reject it" (`useSpellChecker.ts:13`) is a deliberate refusal to detect document language, and it is right: it makes English quotations inside Portuguese prose free, in both directions. Teaching it a document verdict buys nothing and costs a false positive on every borrowed word. - Tests worth writing first: doc-level detection per pair (monolingual, 80/20, 50/50, quotation-heavy English), asserting the hysteresis band **from both directions**; the flipped prompt names the target language and drops the ESL framing while the English-document prompt stays **byte-identical** to today's (the path every existing user is on); the two language arguments proven independent by the only pair that can exercise it — a `learning_pair` zh account writing Chinese wants Chinese corrections explained in English, a `learning_en` zh account writing Chinese wants both in Chinese; a handler test in the shape of `pairlang_test.go`; and a language flip invalidating checked chunks. - **Open, and needs a decision before the last step.** The **vocabulary garden** harvests phrases from documents, so a Portuguese document would seed it with Portuguese — tag entries by language and filter by the half being learned, or gate harvesting to English documents? Tagging looks right and touches stored rows. **Read-aloud** should follow the document too (all five voices now run on the VPS), which is probably small and lands where the verdict lands, but has not been traced. - **Order:** (a) detection + salt + checkpoint/voice prompts — the whole visible win, independently shippable; (b) translate-card direction, Ask Petal seed, `/translate` direction; (c) garden and read-aloud, once the two questions above are answered. +- **(a) and (b) are built; (c) is still open** and is the only part of Phase 28 left — the garden's language tagging is a stored-rows decision and read-aloud has not been traced. **Not seen in a browser and not deployed**: no migration beyond `doc_lang`, and the flipped translate card and the skipped seed are asserted through the real `/check` and `/translate` paths in Go tests, not watched. A rebuild whenever the user wants it, along with Phases 24–27. **Step (a) as built, 2026-07-28** — items 1–4. Step (b) and (c) are untouched, and the two open questions under (c) are still open. - `internal/llm/target.go` — `Target{Correct, Explain, Pair}` + `English` (a `Lang` the `langs` map has no business holding: that map answers "which half is hers"). `EnglishTarget(pair)` is the pre-phase behaviour named, and `Flipped()` is the one question the prompts ask. **Three fields, not two**: the collocation coach's parenthetical gloss is addressed to *her* and not to the document, so it reads `Pair` — collapsing it into `Explain` would have silently moved that gloss into English on every English document, which is every document today. @@ -470,6 +473,7 @@ Today every correction and every explanation comes back in English, whatever she - [x] **Phase 14 — companion warmth + bedtime nag + night mode**: more encouraging phrases, a gentle "go to bed" nudge after 11pm, and a calm dark theme + falling stars at night. ✅ (see Phase 14 above) ## Session log +- 2026-07-28: **Phase 28 finished — the two steps that point the other way** (user: "let's continue the build plan"; the plan's own next items were 28's unbuilt (b), code only). Both steps are about direction, and both turned out to have a wrong answer that looks right. **`isTranslation` could not simply be read backwards**: swapping its two halves would have called every genuine Portuguese correction inside a Portuguese document a translation, because `readsAsEnglish` is a deliberately low bar — Latin letters, not swamped by another script — that Portuguese clears as easily as English does. The flipped direction uses `sentenceLang` from doclang.go instead, where English has its own curated marker list and has to out-evidence the pair language to win; the English-document path is untouched, and the test file now pins both directions with the Portuguese-in-Portuguese case as the one it exists for. **The translate tap-through's whole observable change is a model call that no longer happens**: it now recovers the explanation's language by re-running `targetFor` rather than assuming the pair, which produces exactly today's answer in every case except the one that was broken — the Portuguese writer whose explanation already arrived in Portuguese, previously round-tripped through the model into Portuguese again. It answers `""` there, and the client's existing `res.translation.trim() || explanation` fallback seeds the bubble with the explanation itself, so the fix needed no frontend change at all. It deliberately does *not* render that explanation into English on the grounds that English is technically the other half: an unasked-for rendering into the language she is practising is noise, not a seed, and Ask Petal is where she can ask for it. What is left of the phase is (c) — the garden's language tagging, which touches stored rows, and read-aloud, which has not been traced. Not seen in a browser; not deployed. - 2026-07-28: **Phase 27 — the keystroke that isn't one** (user asked to continue the build plan; the plan's own next item was Phase 26's unbuilt IME guards, named there as the likeliest thing to be wrong the first time anyone types Chinese into Petal for real). **The bug is that a composition is not a keystroke**: the pinyin goes into the document as it is typed, a candidate window sits over it, and all three decoration layers recompute from the live document on every change — rewriting the DOM around the node the browser is composing in, which is what eats half-typed input. **The fix is to hold the redraws, not skip them**: a rebuild that falls due mid-composition marks itself stale and its decorations are *mapped through the transaction*, so they travel with the text growing under them and land correct the moment the composition ends. Skipping would have left every highlight a character behind for as long as she kept typing — the same wrongness, arriving quietly. **The composition flag is read from the state *before* the transaction**, which makes it independent of plugin ordering: the flag was set by compositionstart, not by the transaction being applied, and the end transaction is the one deliberate exception or nothing would ever release. **One piece of timing genuinely matters and is written down where it happens**: a custom `handleDOMEvents` handler runs *before* ProseMirror's own, and ProseMirror's compositionend queues the composition's last DOM changes as a microtask — so the release is a macrotask late, and the held rebuild sees the committed 公园 rather than the *gongyuan* it replaced. If the flush's transaction arrives first it rebuilds anyway; both orders land, which is what makes it not a race. **One thing was checked rather than assumed and turned out already handled**: Tiptap's input-rule plugin returns early while composing — which matters more here than in an English app, because pinyin uses an apostrophe as a syllable separator (xi'an → 西安) and Typography rewrites every `'` into a curly `’`. **The save is deliberately not gated and the analysis is**: a tablet keyboard can hold one composition open for a whole sentence, and Petal never makes writing wait for anything, so `EditorChange` carries the flag, auto-save ignores it, and the checkpoint/rule pack/companion wait — with one more change emitted the instant the composition commits, so nothing is skipped, only deferred by the length of a word. **And four places were quietly stealing keys from the IME**: the Find bar (Enter = next match, Escape = close), the tag picker (Enter = create, in a field whose purpose is a name typed in Chinese), Ask Petal's chat box (the field she types Mandarin into, where Enter sends), and distraction-free mode's global Escape — pressed to fix a wrong candidate, it popped the sidebar back. tsc, vite, go build/vet/test, **vitest 296/296** (12 new, driving real plugin state through a real composition: pinyin in, candidate committed, composition ended). **Then verified in a real browser** (Chrome extension, throwaway DB on :8093): the events are dispatched rather than typed, since this box has no IME, but they drive ProseMirror's own composition machinery, so every layer takes the real path. `gongyuan` typed into the document mid-composition with **the caret moved off it** — so the caret exemption cannot be the explanation — stays un-underlined while `parc` keeps its underline, and underlines on compositionend with no further keystroke. *did a mistake* highlights in 800 ms typed normally and **not at all** after 1.2 s inside a composition, then appears the instant it commits. The save is not held, proven from the other side: with the composition still open and never ended, the server's copy already read `… STILLCOMPOSING`. Escape (`isComposing` and legacy 229) leaves distraction-free mode alone while a real Escape restores the sidebar; the Find bar counter goes 1/4, 1/4, 2/4. **Then with a real IME**, because the user installed one and typed — the only way it could be done: Chrome here runs natively on Wayland, so no key can be injected into its window, and the extension's keys arrive over CDP, which bypasses the OS input method entirely. 很漂亮, composed over **11.6 seconds and 12 candidate changes** (和 → 很 → 狠批 → 很票 → 很漂亮 …), with the preedit genuinely living in the document the whole time — and the underline set never moved, the word committed clean, and English typed afterwards flagged immediately. **The accidental control is the best part**: she typed `henpiaoliang` as plain letters before switching the IME on and it was underlined, so the same syllables appear twice in one sentence, flagged on the path that isn't a composition and untouched on the path that is. ⚠️ **The counterfactual is still unproven** — a clean run with the guard in place can't distinguish "held correctly" from "would have been fine anyway"; settling that means a build with the guard reverted and another minute of her typing. **Not deployed** (no migration — a rebuild, along with Phases 24–26). - 2026-07-28: **Phase 26 — the zh pair's other direction, and a rule pack whose best feature is what it refuses** (user asked to continue the build plan, then chose a new phase over deploying fr/es; scope chosen with the user: **direction column, segmentation + hover pinyin/gloss, 错别字, IME guards**, code only). SUGGESTIONS §4 had called this its own epic, and the reason turned out to be one sentence: **`pair_lang` had always been answering a second question nobody asked.** It says which two languages; every surface built on it assumed English was the one being *learned*, which is why CJK is deliberately never tokenized, never spell-checked and never glossed. All correct for the writer this app was built for, all backwards for the other one. A `direction` column rather than a `zh-learner` pair code, because the two are genuinely separate questions and the column is what lets fr/es/pt inherit the capacity later. **The phase has three decisions in it and they are all about coverage.** The browser gets a word list and the server keeps the dictionary, and their gates come out *opposite*: the client list is frequency-gated at 5 because segmentation by the full union and by the gated list is **identical** on ordinary prose (measured, including 研究生命的起源 and 乒乓球拍卖完了 — the long tail is rare proper nouns and the max-probability walk never picks one), while the dictionary is gated by **nothing**, because its only power is to *explain* and the word a learner stops on is precisely the rare one. That is the es dictionary decision arrived at from both sides in one phase. **Writing the segmenter tests found the boundary bug**: a position names a gap and a word covers characters, so a hover on a boundary was resolving to the word that ended there rather than the one that starts. **The 错别字 pack is the part worth reading.** Chinese has no misspellings — every character an IME offers is real — so the unit of error is a substituted character inside a correct-looking word, and the pack is 24 confusable pairs held by two mechanical gates. Gate one admits a pair only if the wrong form is *not* a dictionary word and the right form is, which is what makes it refuse **自已 for 自己** — one of the commonest slips in written Chinese, whose wrong form is itself a headword — exactly as Phase 22 refused `married with`. Gate two is the one that matters: **自己经常 contains 己经, 睡觉的时候 contains 觉的, 不知到底 contains 知到**, so a substring match would corrupt correct sentences silently, into text still made of real characters. The segmenter settles it — two adjacent single-character tokens is what the walk produces when it has nothing better, which is what a mistyped compound looks like — and where the gate costs a real catch (我不知到他在哪里 *is* 知到 for 知道, but 不知 is a word) it declines rather than risk the identical-looking correct sentence beside it. **`@types/node` went in as a devDependency and quietly fixes something older**: phases 21–25 each verified their shipped dictionary with a throwaway script because vitest could not read files; the suite now asserts against the real assets. go build/vet/test, tsc, vite, **vitest 284/284**, live smoke on a throwaway DB which itself caught a cosmetic build bug ("cat (" left by stripping CC-CEDICT's CL: field). ⚠️ **The IME guards were in scope and are not done** — no composition handling exists anywhere in the app, and a decoration rebuild mid-composition eating half-typed pinyin is the likeliest thing to be wrong the first time anyone types Chinese into Petal for real. Left untouched rather than half-built, and named here rather than buried. ⚠️ **Not deployed** (it carries a migration), **not seen in a browser**, and **no account has ever been in the learner direction**, so everything above about how it feels to use is inference from tests. - 2026-07-28: **Phase 25 — the es pair, and a plan that had quietly chosen the wrong Spanish** (user asked where Spanish support had gone, then "yes" to starting the phase; scope chosen with the user: **Latin American neutral**, quorum review, code only). The starting point was a misreading worth recording: the plan *reads* as though Spanish shipped, because the DreamDict rebuild, the LLM language entry, the L1 rule gating and the TTS env-discovery are all `[x]` — every piece of groundwork was done and the pair itself had never been built. `shippedPairs` was the honest answer all along: the server had been refusing `es` on purpose. **The regional question was the phase.** pt-PT's was forced by packaging and fr's turned out not to exist; es had a real choice with no default, and once the user chose Latin American, the plan's own two concrete decisions were both wrong. It warned that `hunspell-es` is "packaged per country — check what `es_ES` actually is": it ships twenty country codes and **every one is a symlink to one pan-Hispanic file**, so the trap was not there. And it named **`es_ES-davefx-medium`** for the voice, which *is* the trap — six of Piper's nine Spanish voices are peninsular, so the obvious pick would have read Latin American copy in a Castilian accent, the pt-PT mistake arriving through a different door. `es_MX-ald-medium` instead. **Then the user asked "should we pick a different Spanish dictionary?" and the answer was yes** — the phase had shipped the wrong one and written a confident justification for it. Debian's `hunspell-es` symlinks twenty country codes to one file, which reads as pan-Hispanic; RLA actually publishes twenty-four builds per release, one per country **plus a generic `es` that is the union**, and Debian ships **peninsular `es_ES`**. The 58,622-form difference is essentially **voseo**: under the first build, *vení* and *tenés* — the ordinary present tense of Argentina, Uruguay, Paraguay and much of Central America — were underlined as misspellings, and this document called that a known gap handled on principle. **What makes it worth writing down is that the MUST_ACCEPT list was designed to catch exactly this and could not**: it asserted the pan-Hispanic *vocabulary*, and every RLA variant carries the full pan-Hispanic vocabulary — only the paradigms are localised — so it was satisfiable by all twenty-four. The `REP` table cited as the corroborating witness (yeísmo, seseo) is likewise shared by every build. Two independent-looking proofs, neither of which could distinguish anything, agreeing with each other. The profile now demands **voseo** (rejects es_ES and Debian), **vosotros** (rejects es_MX) and **another region's everyday words** — *arepa*, *chévere*, *bacán* (rejects es_AR, which has both paradigms and would otherwise pass); all four neighbours were run through it and confirmed refused. Shipping the union is the same call fr made between *coût* and *cout*: the dictionary's only power is to underline, so it holds every variety, while the *copy* picks a register because speaking requires one. 717,640 forms, 1.74 MB gzipped, **762 ms / 97 MB** in a real nspell, and **fr and pt-PT rebuild byte-identical** from their own upstream debs. **The quorum review earned its place twice**: four models, ≥2-of-4, 5 of 27 findings applied — one of which caught the pack's bedtime proverb being *Qui dort dîne* calqued into Spanish, English gloss and all, which is exactly the "a pack is not a translation of another pack" rule the fr header states and I had broken while writing it. And one below-threshold finding (a missing `¡` on an exclamative, seen by 1 of 4 because an absent *opening* mark has no closing `!` to look wrong against) was applied anyway and **turned into an assertion**: the suite now rejects any native line that closes `?`/`!` without opening one. That is Phase 24's lesson one level up — what a review finds once, a test should find every time. go build/vet/test, tsc, vite, **vitest 251/251**. ⚠️ **Not deployed, not seen in a browser, not read by a native speaker, and no es account exists** — all four accounts' worth of Spanish experience is still hypothetical, and the pack says so in its own header. diff --git a/internal/suggestions/doclang_test.go b/internal/suggestions/doclang_test.go index ab32c9a..d40075d 100644 --- a/internal/suggestions/doclang_test.go +++ b/internal/suggestions/doclang_test.go @@ -1,6 +1,7 @@ package suggestions import ( + "encoding/json" "net/http" "path/filepath" "strings" @@ -259,3 +260,136 @@ func TestLanguageFlipReopensCheckedSentences(t *testing.T) { t.Fatalf("the already-checked sentence was not re-opened by the flip:\n%s", client.lastPrompt) } } + +// The translate card, pointed the other way. She is writing her journal in +// Portuguese and drops in the one English sentence she knows; Petal renders it +// into Portuguese, and that card is a translation — not a correction to prose +// that was never wrong. +func TestEnglishSpanBecomesATranslateCardInAPortugueseDocument(t *testing.T) { + const english = "I want to say this but I don't know how to say it." + // The model volunteers "clarity", as it did for the zh case. Not consulted. + client := &stubClient{response: `{"suggestions":[ + {"original":"` + english + `","replacement":"Eu quero dizer isto mas não sei como o dizer.","explanation":"Aqui está em português.","type":"clarity"} + ]}`} + srv, docID, _ := newDirectedServer(t, client, "pt-PT", auth.DirectionLearningEn, ptDocument+" "+english) + + var out []db.Suggestion + rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "") + if rec.Code != http.StatusOK { + t.Fatalf("check: code=%d body=%s", rec.Code, rec.Body) + } + if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil { + t.Fatalf("decode: %v", err) + } + if len(out) != 1 { + t.Fatalf("want 1 card, got %d: %+v", len(out), out) + } + if out[0].Type != db.SuggestionTypeTranslate { + t.Fatalf("card type = %q, want %q", out[0].Type, db.SuggestionTypeTranslate) + } +} + +// And the half that keeps it honest: a genuine Portuguese correction in the same +// document stays a correction. Reading the English-document test backwards would +// have called this a translation, because every Portuguese sentence also "reads +// as English" by that test's deliberately low bar. +func TestPortugueseCorrectionKeepsItsTypeInAPortugueseDocument(t *testing.T) { + client := &stubClient{response: `{"suggestions":[ + {"original":"Não sei porque isso é tão difícil para mim.","replacement":"Não sei porque isto é tão difícil para mim.","explanation":"Aqui usa-se isto.","type":"grammar"} + ]}`} + srv, docID, _ := newDirectedServer(t, client, "pt-PT", auth.DirectionLearningEn, ptDocument) + + var out []db.Suggestion + rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "") + if rec.Code != http.StatusOK { + t.Fatalf("check: code=%d body=%s", rec.Code, rec.Body) + } + if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil { + t.Fatalf("decode: %v", err) + } + if len(out) != 1 { + t.Fatalf("want 1 card, got %d: %+v", len(out), out) + } + if out[0].Type == db.SuggestionTypeTranslate { + t.Fatal("a Portuguese correction inside a Portuguese document was labelled a translation") + } +} + +// setDocLang writes a document's language verdict directly, so a test of the +// tap-through doesn't have to run a checkpoint through the same stub client to +// get one. +func setDocLang(t *testing.T, h *Handler, docID, lang string) { + t.Helper() + if _, err := h.DB.Exec(`UPDATE documents SET doc_lang = ? WHERE id = ?`, lang, docID); err != nil { + t.Fatalf("set doc_lang: %v", err) + } +} + +// seedExplanation files one card carrying a given explanation and returns its +// id — the shape the translate tap-through needs, where only the explanation and +// the document it hangs off matter. +func seedExplanation(t *testing.T, h *Handler, docID, explanation string) string { + t.Helper() + var sugID string + if err := h.DB.QueryRow( + `INSERT INTO suggestions (doc_id, original, replacement, explanation, type, from_pos, to_pos) + VALUES (?, ?, ?, ?, ?, 0, 5) RETURNING id`, + docID, "isso", "isto", explanation, "grammar", + ).Scan(&sugID); err != nil { + t.Fatalf("seed suggestion: %v", err) + } + return sugID +} + +// The tap-through has to read the same decision the card was written under. On a +// Portuguese document by a Portuguese writer the explanation already arrived in +// Portuguese, and the old endpoint would have sent it to the model to be +// rendered into Portuguese again. +func TestTranslateSkipsWhenTheExplanationIsAlreadyHers(t *testing.T) { + client := &stubClient{response: "Não devia ser chamado."} + srv, docID, h := newDirectedServer(t, client, "pt-PT", auth.DirectionLearningEn, ptDocument) + setDocLang(t, h, docID, docLangPair) + sugID := seedExplanation(t, h, docID, "Aqui usa-se isto.") + + rec := do(t, srv, http.MethodPost, "/suggestions/"+sugID+"/translate", "") + if rec.Code != http.StatusOK { + t.Fatalf("translate: code=%d body=%s", rec.Code, rec.Body) + } + var out translateResponse + if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil { + t.Fatalf("decode: %v", err) + } + if out.Translation != "" { + t.Fatalf("translation = %q, want empty: the bubble seeds from the explanation itself", out.Translation) + } + if client.calls != 0 { + t.Fatal("the model was asked to render Portuguese into Portuguese") + } +} + +// The learner travelling the other way is the case that proves the endpoint +// derives its destination rather than skipping whenever a document is flipped: a +// native English speaker learning Chinese, writing Chinese, gets her +// explanations in English — and the tap still has somewhere to go. +func TestTranslateStillRendersForALearnersEnglishExplanation(t *testing.T) { + const zhDocument = "今天天气很好。我早上去公园散步。下午我在家里写作业。晚上我和朋友一起吃饭。" + client := &stubClient{response: "这里应该用这个。"} + srv, docID, h := newDirectedServer(t, client, "zh", auth.DirectionLearningPair, zhDocument) + setDocLang(t, h, docID, docLangPair) + sugID := seedExplanation(t, h, docID, "This measure word doesn't fit here.") + + rec := do(t, srv, http.MethodPost, "/suggestions/"+sugID+"/translate", "") + if rec.Code != http.StatusOK { + t.Fatalf("translate: code=%d body=%s", rec.Code, rec.Body) + } + var out translateResponse + if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil { + t.Fatalf("decode: %v", err) + } + if out.Translation == "" { + t.Fatal("a learner's English explanation was left untranslated") + } + if !strings.Contains(client.lastPrompt, "Simplified Chinese") { + t.Fatalf("translate didn't render into the pair language:\n%s", client.lastPrompt) + } +} diff --git a/internal/suggestions/handlers.go b/internal/suggestions/handlers.go index 673922e..d91ff8c 100644 --- a/internal/suggestions/handlers.go +++ b/internal/suggestions/handlers.go @@ -317,10 +317,22 @@ func (h *Handler) runPass(w http.ResponseWriter, r *http.Request, limiter *llm.R return } + // What language is this document in, and so what language should its cards be + // written in? Computed from the whole content_text — never from `askText`, + // which on a chunked pass is only the sentences that changed, and would put an + // English card in a Portuguese journal the moment she edits its one English + // line. + // + // Decided before the empty-document exit so every reconcile below is told the + // same verdict. An emptied document has nothing to go on and holds whatever it + // said last (see documentLang), which is what keeps a Portuguese journal + // Portuguese while she clears it to start the entry again. + docLang := documentLang(contentText, pairLang, prevLang) + // Nothing to analyze on an empty document — skip the LLM round-trip. The // family's rows go with the text they were about. if strings.TrimSpace(contentText) == "" { - if err := h.reconcilePending(docID, contentText, pairLang, nil, scope, nil, nil, false); err != nil { + if err := h.reconcilePending(docID, contentText, pairLang, docLang, nil, scope, nil, nil, false); err != nil { httputil.ServerError(w, err) return } @@ -333,12 +345,6 @@ func (h *Handler) runPass(w http.ResponseWriter, r *http.Request, limiter *llm.R return } - // What language is this document in, and so what language should its cards be - // written in? Computed from the whole content_text — never from `askText`, - // which on a chunked pass is only the sentences that changed, and would put an - // English card in a Portuguese journal the moment she edits its one English - // line. - docLang := documentLang(contentText, pairLang, prevLang) if docLang != normalizeDocLang(prevLang) { if _, err := h.DB.Exec( `UPDATE documents SET doc_lang = ? WHERE id = ? AND user_id = ?`, @@ -376,7 +382,7 @@ func (h *Handler) runPass(w http.ResponseWriter, r *http.Request, limiter *llm.R if len(changed) == 0 { // Every sentence has already been read. Drop the rows whose sentence is // gone, keep the rest exactly as they are, and answer immediately. - if err := h.reconcilePending(docID, contentText, pairLang, nil, scope, chunks, nil, false); err != nil { + if err := h.reconcilePending(docID, contentText, pairLang, docLang, nil, scope, chunks, nil, false); err != nil { httputil.ServerError(w, err) return } @@ -421,7 +427,7 @@ func (h *Handler) runPass(w http.ResponseWriter, r *http.Request, limiter *llm.R // A whole-document pass re-read everything, so every one of its rows is up for // re-proposal; a chunked pass only puts the sentences it asked about in play. - if err := h.reconcilePending(docID, contentText, pairLang, raw, scope, chunks, fresh, !scope.chunked); err != nil { + if err := h.reconcilePending(docID, contentText, pairLang, docLang, raw, scope, chunks, fresh, !scope.chunked); err != nil { httputil.ServerError(w, err) return } diff --git a/internal/suggestions/language.go b/internal/suggestions/language.go index 6ae0e60..0f3d8bd 100644 --- a/internal/suggestions/language.go +++ b/internal/suggestions/language.go @@ -28,16 +28,35 @@ import ( // common words a sentence in that language can hardly avoid and an English // sentence has no reason to contain. -// isTranslation reports whether this edit is her own language rendered into -// English, rather than a correction to her English. Both halves must hold: the -// quoted span reads as the pair language, and what Petal offers back reads as -// English. The second half matters — a Chinese span rewritten into different -// Chinese is something else entirely, and Petal has no business calling it a -// translation. -func isTranslation(original, replacement, pairLang string) bool { +// isTranslation reports whether this edit is a rendering of one language into +// the other, rather than a correction. Both halves must hold: the quoted span +// reads as one language, and what Petal offers back reads as the other. The +// second half matters — a Chinese span rewritten into different Chinese is +// something else entirely, and Petal has no business calling it a translation. +// +// Which way it points follows the document (Phase 28). In an English document +// the translate card is her language rendered into English — she reached for a +// sentence she couldn't say yet, and Petal said it for her. In a document she +// wrote in her own language the useful card is the mirror image: an English +// sentence she dropped into her Portuguese, rendered into Portuguese. Asking the +// English-document question there would label nothing, and the card would file +// as a correction to prose that was never wrong. +// +// The flipped direction cannot be the same test read backwards. `readsAsEnglish` +// is a low bar on purpose — Latin letters, not swamped by another script — which +// every Portuguese sentence also clears, so using it on the *original* would +// call every genuine Portuguese correction a translation. The flipped test +// instead uses the sentence-level vote from doclang.go, where English has its +// own marker list and has to out-evidence the pair language to win. +func isTranslation(original, replacement, pairLang, docLang string) bool { if strings.TrimSpace(original) == "" || strings.TrimSpace(replacement) == "" { return false } + if normalizeDocLang(docLang) == docLangPair { + p := normalizePairLang(pairLang) + return sentenceLang(original, p) == docLangEnglish && + sentenceLang(replacement, p) == docLangPair + } return readsAsPairLang(original, pairLang) && readsAsEnglish(replacement) } diff --git a/internal/suggestions/language_test.go b/internal/suggestions/language_test.go index efe0b61..8a08a88 100644 --- a/internal/suggestions/language_test.go +++ b/internal/suggestions/language_test.go @@ -3,6 +3,10 @@ package suggestions import "testing" // The flagship case, and the ones next to it that must NOT become translations. +// +// Every case here is an ENGLISH document — the path every account was on before +// Phase 28 — so `docLang` is left at "". The mirror image lives in +// TestIsTranslationInAPairLanguageDocument below. func TestIsTranslation(t *testing.T) { cases := []struct { name string @@ -134,8 +138,93 @@ func TestIsTranslation(t *testing.T) { for _, c := range cases { t.Run(c.name, func(t *testing.T) { - if got := isTranslation(c.original, c.replacement, c.pairLang); got != c.want { - t.Errorf("isTranslation(%q, %q, %q) = %v, want %v", + if got := isTranslation(c.original, c.replacement, c.pairLang, ""); got != c.want { + t.Errorf("isTranslation(%q, %q, %q, en) = %v, want %v", + c.original, c.replacement, c.pairLang, got, c.want) + } + }) + } +} + +// The mirror image (Phase 28): in a document she wrote in her own language, the +// translate card is the English sentence rendered into her language — and the +// English-document question, asked here, would label nothing. +// +// The case this file exists to pin is the third one: a genuine Portuguese +// correction inside a Portuguese document. Reading the English-document test +// backwards would call it a translation, because `readsAsEnglish` is a low bar +// that Portuguese clears too. It has to stay a correction. +func TestIsTranslationInAPairLanguageDocument(t *testing.T) { + cases := []struct { + name string + original string + replacement string + pairLang string + want bool + }{ + { + name: "English sentence rendered into Portuguese", + original: "I want to say this but I don't know how to say it.", + replacement: "Eu quero dizer isso mas não sei como o dizer.", + pairLang: "pt-PT", + want: true, + }, + { + name: "English sentence rendered into Chinese", + original: "I don't know how to say this in Chinese.", + replacement: "我不知道这句话用中文怎么说。", + pairLang: "zh", + want: true, + }, + { + // The one that matters. Portuguese in, Portuguese out, inside a + // Portuguese document: a correction, and nothing else. + name: "Portuguese corrected as Portuguese", + original: "Eu quero dizer isso mas não sei como.", + replacement: "Eu quero dizer isto mas não sei como.", + pairLang: "pt-PT", + want: false, + }, + { + // And its Chinese twin, which the script test already caught. + name: "Chinese corrected as Chinese", + original: "我想说这句话", + replacement: "我要说这句话", + pairLang: "zh", + want: false, + }, + { + // The old direction, asked in the new document. She quoted English in + // her Portuguese and Petal rendered it into Portuguese — which IS a + // translation, and is the case above. This is its reverse: Portuguese + // out of an English document that isn't one. No label. + name: "Portuguese rendered into English is not this document's translation", + original: "Eu quero dizer isso mas não sei como.", + replacement: "I want to say this but I don't know how.", + pairLang: "pt-PT", + want: false, + }, + { + // English prose without enough evidence to vote. Silence, not a guess. + name: "too short to read as English", + original: "OK", + replacement: "Está bem, muito obrigado.", + pairLang: "pt-PT", + want: false, + }, + { + name: "unknown pair language declines in both directions", + original: "I don't know how to say that.", + replacement: "Ich weiß nicht wie man das sagt.", + pairLang: "de", + want: false, + }, + } + + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + if got := isTranslation(c.original, c.replacement, c.pairLang, docLangPair); got != c.want { + t.Errorf("isTranslation(%q, %q, %q, pair) = %v, want %v", c.original, c.replacement, c.pairLang, got, c.want) } }) @@ -151,8 +240,8 @@ func TestNormalizePairLang(t *testing.T) { "fr": "fr", "fr-FR": "fr", "es": "es", "es-ES": "es", " zh ": "zh", - "": "", - "de": "de", + "": "", + "de": "de", } { if got := normalizePairLang(in); got != want { t.Errorf("normalizePairLang(%q) = %q, want %q", in, got, want) diff --git a/internal/suggestions/reconcile.go b/internal/suggestions/reconcile.go index b2c499b..94499bf 100644 --- a/internal/suggestions/reconcile.go +++ b/internal/suggestions/reconcile.go @@ -131,10 +131,12 @@ func reposition(tx *sql.Tx, row pendingRow, from, to int, chunkHash string) erro // collocation coach — where every row is up for re-proposal because the model // just re-read everything. // -// `pairLang` is the writer's own language, needed only to type a finding that -// turns out to be her language rendered into English (see language.go). +// `pairLang` is the writer's own language and `docLang` this document's language +// verdict; between them they type a finding that turns out to be one language +// rendered into the other, in whichever direction this document makes useful +// (see language.go). func (h *Handler) reconcilePending( - docID, contentText, pairLang string, + docID, contentText, pairLang, docLang string, raw []llm.RawSuggestion, scope pendingScope, chunks, fresh []chunk, @@ -235,7 +237,7 @@ func (h *Handler) reconcilePending( typ := scope.forceType if typ == "" { typ = normalizeType(s.Type) - if isTranslation(s.Original, s.Replacement, pairLang) { + if isTranslation(s.Original, s.Replacement, pairLang, docLang) { typ = db.SuggestionTypeTranslate } } diff --git a/internal/suggestions/translate.go b/internal/suggestions/translate.go index 6b35779..a3efca3 100644 --- a/internal/suggestions/translate.go +++ b/internal/suggestions/translate.go @@ -17,23 +17,37 @@ type translateResponse struct { Translation string `json:"translation"` } -// translate renders a suggestion's English explanation into Simplified Chinese -// for the Ask Petal bubble, so the ESL reader sees the "why" in her first -// language instead of a second copy of the same English text. The explanation is -// loaded server-side from the suggestion id (scoped to the local user) and never -// trusted from the client, mirroring chat (spec Note #10). +// translate renders a suggestion's explanation into the other half of the +// writer's pair for the Ask Petal bubble, so she sees the "why" in the language +// she reads most easily instead of a second copy of the same text. The +// explanation is loaded server-side from the suggestion id (scoped to the +// caller) and never trusted from the client, mirroring chat (spec Note #10). +// +// Which language it renders into cannot be assumed (Phase 28). Before that phase +// every explanation was English and every rendering went into her language, so +// "the pair language" was a safe constant. Now the explanation's language is a +// decision — `targetFor`, from the document's verdict and her direction — and +// this endpoint has to read the same decision back, or it round-trips Portuguese +// into Portuguese and calls it a translation. +// +// So: render into whichever half the explanation is NOT already in, and when the +// explanation already arrived in the language this bubble exists to reach her +// in, skip the model call and answer "". The client seeds the bubble with the +// explanation itself when the translation comes back empty, which is exactly +// right — there is nothing to add. func (h *Handler) translate(w http.ResponseWriter, r *http.Request) { sugID := chi.URLParam(r, "id") - var explanation, pairLang string + var explanation, pairLang, direction, docLang string err := h.DB.QueryRow( - `SELECT s.explanation, COALESCE(u.pair_lang, '') + `SELECT s.explanation, COALESCE(u.pair_lang, ''), + COALESCE(u.direction, ''), d.doc_lang FROM suggestions s JOIN documents d ON d.id = s.doc_id JOIN users u ON u.id = d.user_id WHERE s.id = ? AND d.user_id = ?`, sugID, auth.UserID(r.Context()), - ).Scan(&explanation, &pairLang) + ).Scan(&explanation, &pairLang, &direction, &docLang) if errors.Is(err, sql.ErrNoRows) { httputil.ErrorJSON(w, http.StatusNotFound, "suggestion not found") return @@ -49,7 +63,21 @@ func (h *Handler) translate(w http.ResponseWriter, r *http.Request) { return } - out, err := llm.RunTranslate(r.Context(), h.Client, explanation, llm.LangFor(pairLang)) + // The explanation's own language, recovered from the same rule that chose it + // when the card was written. A card written before this phase — or on a + // document whose verdict has since flipped — is read as whatever the rule says + // today; the alternative is a language column on every suggestion row, and the + // cost of being wrong is one bubble seeded in the language it was already in. + target := targetFor(pairLang, direction, docLang) + if target.Explain.Code == target.Pair.Code { + // Already in her language. The other half is English — the language she is + // practising — and an unasked-for English rendering of an explanation she + // can already read is not a seed, it's noise. + httputil.WriteJSON(w, http.StatusOK, translateResponse{Translation: ""}) + return + } + + out, err := llm.RunTranslate(r.Context(), h.Client, explanation, target.Pair) if err != nil { httputil.UpstreamError(w, "translate", err) return From 466055020fd2e7a924eb2200a74cbf72c04f8472 Mon Sep 17 00:00:00 2001 From: prosolis <5590409+prosolis@users.noreply.github.com> Date: Tue, 28 Jul 2026 23:45:26 -0700 Subject: [PATCH 3/3] The garden learns which language a card is in, and read-aloud stops guessing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 28 (c), the last of the phase. A word met inside a Portuguese document is a Portuguese card: migration 0018 mirrors documents.doc_lang onto vocab_words, set server-side from the ownership lookup capture was already making. Every card still reviews — filtering the queue to the half she is learning would drop the words she actually met. Read-aloud was the larger surprise. detectLang routed Han/kana to Chinese and everything else to en-US, so the zh pair was accidentally right and every Latin pair wrong. doc_lang now reaches the client read-only on the document JSON, and docLang(text, verdict) answers for a passage taken out of it — with the script test still winning, because quoted Chinese must never be spelled out one "Chinese letter" at a time. Claude-Session: https://claude.ai/code/session_01GJHNvirh7Hzhc9RL3HAvz7 --- BUILD_PLAN.md | 17 ++++- internal/db/db.go | 25 +++++++ internal/db/models.go | 49 +++++++------ internal/docs/handlers.go | 3 +- internal/suggestions/handlers.go | 9 ++- internal/suggestions/plant_test.go | 57 +++++++++++++++ internal/vocab/handlers.go | 48 ++++++++----- internal/vocab/handlers_test.go | 86 +++++++++++++++++++++++ internal/vocab/plant.go | 9 ++- web/src/App.tsx | 1 + web/src/api/client.ts | 13 ++++ web/src/audio/speech.test.ts | 32 ++++++++- web/src/audio/speech.ts | 18 +++++ web/src/components/Editor/EditorCore.tsx | 18 +++-- web/src/components/Garden/GardenPanel.tsx | 24 +++++-- 15 files changed, 352 insertions(+), 57 deletions(-) diff --git a/BUILD_PLAN.md b/BUILD_PLAN.md index 3117400..69c74ff 100644 --- a/BUILD_PLAN.md +++ b/BUILD_PLAN.md @@ -441,11 +441,11 @@ Today every correction and every explanation comes back in English, whatever she - **`internal/llm/lang.go` already carries the precision the model needs** (`"European Portuguese (pt-PT, never Brazilian Portuguese)"`). Spanish wants the same care pointed the other way: the shipped dictionary deliberately accepts the whole Spanish-speaking world (Phase 25), so the prompt must not quietly impose peninsular usage. - **Spellcheck must not change.** "A word is a misspelling only when *both* dictionaries reject it" (`useSpellChecker.ts:13`) is a deliberate refusal to detect document language, and it is right: it makes English quotations inside Portuguese prose free, in both directions. Teaching it a document verdict buys nothing and costs a false positive on every borrowed word. - Tests worth writing first: doc-level detection per pair (monolingual, 80/20, 50/50, quotation-heavy English), asserting the hysteresis band **from both directions**; the flipped prompt names the target language and drops the ESL framing while the English-document prompt stays **byte-identical** to today's (the path every existing user is on); the two language arguments proven independent by the only pair that can exercise it — a `learning_pair` zh account writing Chinese wants Chinese corrections explained in English, a `learning_en` zh account writing Chinese wants both in Chinese; a handler test in the shape of `pairlang_test.go`; and a language flip invalidating checked chunks. -- **Open, and needs a decision before the last step.** The **vocabulary garden** harvests phrases from documents, so a Portuguese document would seed it with Portuguese — tag entries by language and filter by the half being learned, or gate harvesting to English documents? Tagging looks right and touches stored rows. **Read-aloud** should follow the document too (all five voices now run on the VPS), which is probably small and lands where the verdict lands, but has not been traced. +- [x] **Resolved 2026-07-28 (user's call), and the two questions had different answers.** The **vocabulary garden** tags every card with the language of the document it was met in, and reviews all of them. Filtering the queue to the half she is learning was the alternative and was rejected for the writer this is for: the words she met while writing Portuguese are still words she met, and a garden that quietly drops them stops being a record of her reading. Gating harvesting to English documents was rejected harder — a looked-up word would simply vanish. **Read-aloud** follows the document, which needed the verdict to reach the client at all: it was server-only until now. - **Order:** (a) detection + salt + checkpoint/voice prompts — the whole visible win, independently shippable; (b) translate-card direction, Ask Petal seed, `/translate` direction; (c) garden and read-aloud, once the two questions above are answered. -- **(a) and (b) are built; (c) is still open** and is the only part of Phase 28 left — the garden's language tagging is a stored-rows decision and read-aloud has not been traced. **Not seen in a browser and not deployed**: no migration beyond `doc_lang`, and the flipped translate card and the skipped seed are asserted through the real `/check` and `/translate` paths in Go tests, not watched. A rebuild whenever the user wants it, along with Phases 24–27. +- **(a), (b) and (c) are built — Phase 28 is complete in code.** **Not seen in a browser and not deployed**: the flipped translate card, the skipped seed, and the garden's tagging are asserted through the real `/check`, `/translate` and `/vocab` paths in Go tests, not watched. A rebuild whenever the user wants it, along with Phases 24–27. -**Step (a) as built, 2026-07-28** — items 1–4. Step (b) and (c) are untouched, and the two open questions under (c) are still open. +**Step (a) as built, 2026-07-28** — items 1–4. - `internal/llm/target.go` — `Target{Correct, Explain, Pair}` + `English` (a `Lang` the `langs` map has no business holding: that map answers "which half is hers"). `EnglishTarget(pair)` is the pre-phase behaviour named, and `Flipped()` is the one question the prompts ask. **Three fields, not two**: the collocation coach's parenthetical gloss is addressed to *her* and not to the document, so it reads `Pair` — collapsing it into `Explain` would have silently moved that gloss into English on every English document, which is every document today. - `internal/suggestions/doclang.go` — `documentLang(contentText, pairLang, prev)` → `"en"`/`"pair"`. Per **sentence** (reusing `splitChunks`, salt `""`), three-way: pair / English / **no answer**. The third value is the load-bearing one — a sentence with no evidence ("Bom dia.", a heading) is left out of the ratio rather than counted for the language it isn't, and counting the undecided as English is exactly what would hold a Portuguese journal of short sentences in English forever. So the Latin pairs needed an `englishMarkers` list curated against pt/fr/es with the same care `latinMarkers` was curated against English (no "on"/"son"/"as"/"no"/"para" — each a false English vote inside someone's own language). Band as specified, ≥70% / <40%, plus a **corroboration floor** the plan didn't call for: 3 distinct pair markers (8 Han runes) document-wide, because a ratio computed over "Não. Eu." is 100% of nothing and a flip rewrites every card in the document. - **Migration `0017_document_lang`** — `documents.doc_lang` (`'' | 'en' | 'pair'`). Stored, not recomputed, because hysteresis needs a yesterday; `'pair'` rather than a language code, so changing her pair re-reads her documents instead of stranding a stale language name on all of them. Backfill is the default `''`, which reads as English — true of every document that exists. @@ -456,6 +456,16 @@ Today every correction and every explanation comes back in English, whatever she - **Not deployed, and no live smoke.** The handler tests drive the real router and a real DB, which is what a smoke would have shown; what neither proves is how Qwen behaves on a Portuguese document — in particular whether the "never translate" line holds. That wants the deploy Phases 24–27 are also waiting on, and a pt-PT reader. - **Left standing, deliberately, and visible on a flipped document:** the offline mechanics rule pack is English (it is the one that filed a note about the stray English sentence), `isTranslation` still points one way (item 5), and the Ask Petal seed still round-trips (item 6). + +**Step (c) as built, 2026-07-28** — the garden and read-aloud, once the two questions above were answered. +- **Migration `0018_vocab_lang`** — `vocab_words.lang` (`'' | 'en' | 'pair'`), the same three values as `documents.doc_lang` and set from it. Not a language code, for the same reason 0017 isn't: `'pair'` re-reads through `users.pair_lang`, so changing her pair doesn't strand a stale name on every card she owns. Backfill `''`, which reads as English — true of every card that exists, because before Phase 28 every document was. +- **The card's language is read off its document, never sent by the client.** Capture already ran a row-scoped lookup to prove she owns the `doc_id` it was handed; that query now returns `doc_lang` too, so the tag costs nothing and cannot be forged. A capture with no document is `''`. **The one asymmetry is in the upsert**: `lang` travels with `doc_id` or it doesn't travel — re-looking-up a word from the search box refreshes its gloss but must not relabel a Portuguese card English, which would silently move it to the wrong voice. +- **Planting carries it too, even though the collocation pass didn't flip.** Phase 28 (a) deliberately left collocation on its English prompt (per-language *knowledge*, not framing). The chunk it hands over is still lifted out of her prose, so the card is in the document's language whatever the prompt was — `plant` reads `d.doc_lang` in the join it was already making. +- **`documents.doc_lang` reaches the client, read-only, for the first time.** It rides on the document JSON, which the editor already refetches on every save, so the verdict trails a language flip by one auto-save. That is the whole cost of not blocking the editor on a check, and the cost is one passage read in the old voice. The `/check` response shape is untouched — it is a bare array of suggestions, and widening it into an object to carry one string would have been the expensive way to save a save. +- **`docLang(text, verdict)` in `speech.ts`, beside `nativeLang()`** — and **the script test still wins over the verdict**, which is not redundant. Chinese quoted inside an English document is `'en'` by verdict and must still be read by the Chinese voice: the English voice spells Han characters out one "Chinese letter" at a time, the one failure loud enough to be worse than silence. In the other direction there is nothing to test — an English sentence inside Portuguese prose is the same letters as the Portuguese around it — so the document's verdict is the only answer available. The editor's selection bubble (both paces) and the garden's list and review cards all ask it instead of guessing. +- **The marker is on pair-language cards only.** A badge on every English blossom to explain the rare Portuguese one is a garden with no badges at all; it names the language the way the pair buttons do, in the language's own word for itself (`pack.nativeName`). +- Tests: capture from a flipped / English / no document, a client-supplied `lang` losing to the document, the recapture-outside-a-document case, planting on both kinds of document, and `docLang` from both directions plus the script override. go build/vet/test clean, tsc and 321 frontend tests green. + ### Later / explicitly not now - Learner-facing Chinese writing (the zh pair's second direction) — own phase with its own spec (SUGGESTIONS §4); only after Phases 19–21 prove the pair model - ~~Spanish pair — gated on DreamDict growing an es dataset~~ **ungated 2026-07-26**, **shipped (code) 2026-07-28** — see Phase 25. What it still owes: a deploy, a native reader, and a writer who actually uses it. @@ -473,6 +483,7 @@ Today every correction and every explanation comes back in English, whatever she - [x] **Phase 14 — companion warmth + bedtime nag + night mode**: more encouraging phrases, a gentle "go to bed" nudge after 11pm, and a calm dark theme + falling stars at night. ✅ (see Phase 14 above) ## Session log +- 2026-07-28: **Phase 28 finished — the garden learns which language a card is in, and read-aloud stops guessing** (user: "let's continue the build plan"; step (c), the last of the phase, code only). The two parked questions turned out to have different answers, and the user took the garden one: **tag every card and review all of them**, rather than filter the queue to the half she is learning — the words she met while writing Portuguese are still words she met. Migration `0018_vocab_lang` mirrors `documents.doc_lang` onto `vocab_words`, set server-side from the row-scoped ownership lookup capture was already running, so the tag is free and unforgeable; the one subtlety is that `lang` travels with `doc_id` or not at all, because a lookup from the search box carries no verdict and must not relabel a card. **Read-aloud was the larger surprise: the verdict was server-only.** `detectLang` routed Han/kana to Chinese and *everything else to en-US*, which made the zh pair accidentally right and every Latin pair wrong — a Portuguese selection read by a US English voice. `doc_lang` now rides on the document JSON (read-only, refreshed by the auto-save the editor already makes, so it trails a flip by one save) and `docLang(text, verdict)` answers for a passage lifted out of it. **The script test deliberately still wins over the verdict**: Chinese quoted in an English document is `'en'` and must not be spelled out one "Chinese letter" at a time, while an English sentence inside Portuguese prose is undetectable by construction — which is exactly why the document decides. One thing worth remembering from the build: an extra bind argument passed to `Exec` against a SQL string that never got its matching `?` **did not error** — the planted card came back with an empty `lang` and only the test caught it. Not seen in a browser; not deployed. - 2026-07-28: **Phase 28 finished — the two steps that point the other way** (user: "let's continue the build plan"; the plan's own next items were 28's unbuilt (b), code only). Both steps are about direction, and both turned out to have a wrong answer that looks right. **`isTranslation` could not simply be read backwards**: swapping its two halves would have called every genuine Portuguese correction inside a Portuguese document a translation, because `readsAsEnglish` is a deliberately low bar — Latin letters, not swamped by another script — that Portuguese clears as easily as English does. The flipped direction uses `sentenceLang` from doclang.go instead, where English has its own curated marker list and has to out-evidence the pair language to win; the English-document path is untouched, and the test file now pins both directions with the Portuguese-in-Portuguese case as the one it exists for. **The translate tap-through's whole observable change is a model call that no longer happens**: it now recovers the explanation's language by re-running `targetFor` rather than assuming the pair, which produces exactly today's answer in every case except the one that was broken — the Portuguese writer whose explanation already arrived in Portuguese, previously round-tripped through the model into Portuguese again. It answers `""` there, and the client's existing `res.translation.trim() || explanation` fallback seeds the bubble with the explanation itself, so the fix needed no frontend change at all. It deliberately does *not* render that explanation into English on the grounds that English is technically the other half: an unasked-for rendering into the language she is practising is noise, not a seed, and Ask Petal is where she can ask for it. What is left of the phase is (c) — the garden's language tagging, which touches stored rows, and read-aloud, which has not been traced. Not seen in a browser; not deployed. - 2026-07-28: **Phase 27 — the keystroke that isn't one** (user asked to continue the build plan; the plan's own next item was Phase 26's unbuilt IME guards, named there as the likeliest thing to be wrong the first time anyone types Chinese into Petal for real). **The bug is that a composition is not a keystroke**: the pinyin goes into the document as it is typed, a candidate window sits over it, and all three decoration layers recompute from the live document on every change — rewriting the DOM around the node the browser is composing in, which is what eats half-typed input. **The fix is to hold the redraws, not skip them**: a rebuild that falls due mid-composition marks itself stale and its decorations are *mapped through the transaction*, so they travel with the text growing under them and land correct the moment the composition ends. Skipping would have left every highlight a character behind for as long as she kept typing — the same wrongness, arriving quietly. **The composition flag is read from the state *before* the transaction**, which makes it independent of plugin ordering: the flag was set by compositionstart, not by the transaction being applied, and the end transaction is the one deliberate exception or nothing would ever release. **One piece of timing genuinely matters and is written down where it happens**: a custom `handleDOMEvents` handler runs *before* ProseMirror's own, and ProseMirror's compositionend queues the composition's last DOM changes as a microtask — so the release is a macrotask late, and the held rebuild sees the committed 公园 rather than the *gongyuan* it replaced. If the flush's transaction arrives first it rebuilds anyway; both orders land, which is what makes it not a race. **One thing was checked rather than assumed and turned out already handled**: Tiptap's input-rule plugin returns early while composing — which matters more here than in an English app, because pinyin uses an apostrophe as a syllable separator (xi'an → 西安) and Typography rewrites every `'` into a curly `’`. **The save is deliberately not gated and the analysis is**: a tablet keyboard can hold one composition open for a whole sentence, and Petal never makes writing wait for anything, so `EditorChange` carries the flag, auto-save ignores it, and the checkpoint/rule pack/companion wait — with one more change emitted the instant the composition commits, so nothing is skipped, only deferred by the length of a word. **And four places were quietly stealing keys from the IME**: the Find bar (Enter = next match, Escape = close), the tag picker (Enter = create, in a field whose purpose is a name typed in Chinese), Ask Petal's chat box (the field she types Mandarin into, where Enter sends), and distraction-free mode's global Escape — pressed to fix a wrong candidate, it popped the sidebar back. tsc, vite, go build/vet/test, **vitest 296/296** (12 new, driving real plugin state through a real composition: pinyin in, candidate committed, composition ended). **Then verified in a real browser** (Chrome extension, throwaway DB on :8093): the events are dispatched rather than typed, since this box has no IME, but they drive ProseMirror's own composition machinery, so every layer takes the real path. `gongyuan` typed into the document mid-composition with **the caret moved off it** — so the caret exemption cannot be the explanation — stays un-underlined while `parc` keeps its underline, and underlines on compositionend with no further keystroke. *did a mistake* highlights in 800 ms typed normally and **not at all** after 1.2 s inside a composition, then appears the instant it commits. The save is not held, proven from the other side: with the composition still open and never ended, the server's copy already read `… STILLCOMPOSING`. Escape (`isComposing` and legacy 229) leaves distraction-free mode alone while a real Escape restores the sidebar; the Find bar counter goes 1/4, 1/4, 2/4. **Then with a real IME**, because the user installed one and typed — the only way it could be done: Chrome here runs natively on Wayland, so no key can be injected into its window, and the extension's keys arrive over CDP, which bypasses the OS input method entirely. 很漂亮, composed over **11.6 seconds and 12 candidate changes** (和 → 很 → 狠批 → 很票 → 很漂亮 …), with the preedit genuinely living in the document the whole time — and the underline set never moved, the word committed clean, and English typed afterwards flagged immediately. **The accidental control is the best part**: she typed `henpiaoliang` as plain letters before switching the IME on and it was underlined, so the same syllables appear twice in one sentence, flagged on the path that isn't a composition and untouched on the path that is. ⚠️ **The counterfactual is still unproven** — a clean run with the guard in place can't distinguish "held correctly" from "would have been fine anyway"; settling that means a build with the guard reverted and another minute of her typing. **Not deployed** (no migration — a rebuild, along with Phases 24–26). - 2026-07-28: **Phase 26 — the zh pair's other direction, and a rule pack whose best feature is what it refuses** (user asked to continue the build plan, then chose a new phase over deploying fr/es; scope chosen with the user: **direction column, segmentation + hover pinyin/gloss, 错别字, IME guards**, code only). SUGGESTIONS §4 had called this its own epic, and the reason turned out to be one sentence: **`pair_lang` had always been answering a second question nobody asked.** It says which two languages; every surface built on it assumed English was the one being *learned*, which is why CJK is deliberately never tokenized, never spell-checked and never glossed. All correct for the writer this app was built for, all backwards for the other one. A `direction` column rather than a `zh-learner` pair code, because the two are genuinely separate questions and the column is what lets fr/es/pt inherit the capacity later. **The phase has three decisions in it and they are all about coverage.** The browser gets a word list and the server keeps the dictionary, and their gates come out *opposite*: the client list is frequency-gated at 5 because segmentation by the full union and by the gated list is **identical** on ordinary prose (measured, including 研究生命的起源 and 乒乓球拍卖完了 — the long tail is rare proper nouns and the max-probability walk never picks one), while the dictionary is gated by **nothing**, because its only power is to *explain* and the word a learner stops on is precisely the rare one. That is the es dictionary decision arrived at from both sides in one phase. **Writing the segmenter tests found the boundary bug**: a position names a gap and a word covers characters, so a hover on a boundary was resolving to the word that ended there rather than the one that starts. **The 错别字 pack is the part worth reading.** Chinese has no misspellings — every character an IME offers is real — so the unit of error is a substituted character inside a correct-looking word, and the pack is 24 confusable pairs held by two mechanical gates. Gate one admits a pair only if the wrong form is *not* a dictionary word and the right form is, which is what makes it refuse **自已 for 自己** — one of the commonest slips in written Chinese, whose wrong form is itself a headword — exactly as Phase 22 refused `married with`. Gate two is the one that matters: **自己经常 contains 己经, 睡觉的时候 contains 觉的, 不知到底 contains 知到**, so a substring match would corrupt correct sentences silently, into text still made of real characters. The segmenter settles it — two adjacent single-character tokens is what the walk produces when it has nothing better, which is what a mistyped compound looks like — and where the gate costs a real catch (我不知到他在哪里 *is* 知到 for 知道, but 不知 is a word) it declines rather than risk the identical-looking correct sentence beside it. **`@types/node` went in as a devDependency and quietly fixes something older**: phases 21–25 each verified their shipped dictionary with a throwaway script because vitest could not read files; the suite now asserts against the real assets. go build/vet/test, tsc, vite, **vitest 284/284**, live smoke on a throwaway DB which itself caught a cosmetic build bug ("cat (" left by stripping CC-CEDICT's CL: field). ⚠️ **The IME guards were in scope and are not done** — no composition handling exists anywhere in the app, and a decoration rebuild mid-composition eating half-typed pinyin is the likeliest thing to be wrong the first time anyone types Chinese into Petal for real. Left untouched rather than half-built, and named here rather than buried. ⚠️ **Not deployed** (it carries a migration), **not seen in a browser**, and **no account has ever been in the learner direction**, so everything above about how it feels to use is inference from tests. diff --git a/internal/db/db.go b/internal/db/db.go index 0da7b3e..3207328 100644 --- a/internal/db/db.go +++ b/internal/db/db.go @@ -613,6 +613,31 @@ ALTER TABLE users ADD COLUMN direction TEXT NOT NULL DEFAULT 'learning_en' stmt: ` ALTER TABLE documents ADD COLUMN doc_lang TEXT NOT NULL DEFAULT '' CHECK(doc_lang IN ('', 'en', 'pair')); +`, + }, + { + // Which language a garden card is in — the same '' | 'en' | 'pair' + // vocabulary as documents.doc_lang, and set from it: a word is captured + // (or a phrase planted) out of a document, so the document's verdict is + // the card's language. A card with no document keeps '', which reads as + // English like every other empty here. + // + // The garden needed this the moment a document could be written in her + // own language. Before Phase 28 every card was English by construction; + // now a Portuguese lookup lands beside an English one with nothing to + // tell them apart, and two surfaces get it wrong without the tag — the + // review card's read-aloud (which would say a Portuguese word in a US + // English voice) and the panel, where a mixed garden is illegible. + // + // Every card is reviewed regardless. Filtering the queue to the half she + // is learning was the alternative and is wrong for the writer this is + // for: the words she met while writing Portuguese are still words she + // met, and a garden that quietly drops them is a garden that stops being + // a record of her reading. + name: "0018_vocab_lang", + stmt: ` +ALTER TABLE vocab_words ADD COLUMN lang TEXT NOT NULL DEFAULT '' + CHECK(lang IN ('', 'en', 'pair')); `, }, } diff --git a/internal/db/models.go b/internal/db/models.go index fbd15a5..a24d58a 100644 --- a/internal/db/models.go +++ b/internal/db/models.go @@ -34,15 +34,20 @@ type User struct { // (source of truth for the editor); `ContentText` is the flattened plain text // kept in sync on every save and fed to the LLM. type Document struct { - ID string `json:"id"` - UserID string `json:"user_id"` - Title string `json:"title"` - Content string `json:"content"` // Tiptap JSON - ContentText string `json:"content_text"` // plain text for the LLM - Tone string `json:"tone"` // target writing tone; steers LLM advice - WordCount int `json:"word_count"` - CreatedAt time.Time `json:"created_at"` - UpdatedAt time.Time `json:"updated_at"` + ID string `json:"id"` + UserID string `json:"user_id"` + Title string `json:"title"` + Content string `json:"content"` // Tiptap JSON + ContentText string `json:"content_text"` // plain text for the LLM + Tone string `json:"tone"` // target writing tone; steers LLM advice + WordCount int `json:"word_count"` + // DocLang is which language this document is written in — '' | 'en' | 'pair' + // (migration 0017), written by the checkpoint pass and never by the client. + // It reaches the client read-only, for the one decision the client has to + // make on its own: which voice reads a selection aloud. '' means English. + DocLang string `json:"doc_lang"` + CreatedAt time.Time `json:"created_at"` + UpdatedAt time.Time `json:"updated_at"` // PreserveHistory opts this document out of auto-snapshot pruning so its // full writing trail survives as authorship evidence (see the passport). @@ -110,15 +115,15 @@ const ( // coordinates at render time (spec Note #6). `Replacement` is empty for `voice` // flags — those are awareness-only, with no correction to apply. type Suggestion struct { - ID string `json:"id"` - DocID string `json:"doc_id"` - FromPos int `json:"from_pos"` - ToPos int `json:"to_pos"` - Original string `json:"original"` - Replacement string `json:"replacement"` - Explanation string `json:"explanation"` - Type string `json:"type"` // grammar | phrasing | idiom | clarity | translate | voice | collocation - Status string `json:"status"` // pending | accepted | rejected + ID string `json:"id"` + DocID string `json:"doc_id"` + FromPos int `json:"from_pos"` + ToPos int `json:"to_pos"` + Original string `json:"original"` + Replacement string `json:"replacement"` + Explanation string `json:"explanation"` + Type string `json:"type"` // grammar | phrasing | idiom | clarity | translate | voice | collocation + Status string `json:"status"` // pending | accepted | rejected // Source names the engine that proposed the edit, not its family: an offline // rule and the model can both propose a collocation, and the writer is never // told which one spoke. It exists so each pass can replace its own rows. @@ -128,10 +133,10 @@ type Suggestion struct { // Suggestion type and status values, mirrored from the schema CHECK constraints. const ( - SuggestionTypeGrammar = "grammar" - SuggestionTypePhrasing = "phrasing" - SuggestionTypeIdiom = "idiom" - SuggestionTypeClarity = "clarity" + SuggestionTypeGrammar = "grammar" + SuggestionTypePhrasing = "phrasing" + SuggestionTypeIdiom = "idiom" + SuggestionTypeClarity = "clarity" // A span she wrote in her own language, rendered into English. Not a // correction — nothing was wrong with it — which is why it is its own type // rather than a clarity fix: the card is the pair model's flagship moment diff --git a/internal/docs/handlers.go b/internal/docs/handlers.go index 42f5a28..ea00a81 100644 --- a/internal/docs/handlers.go +++ b/internal/docs/handlers.go @@ -225,13 +225,14 @@ func (h *Handler) fetch(userID, id string) (db.Document, error) { var doc db.Document err := h.DB.QueryRow( `SELECT id, user_id, title, content, content_text, tone, word_count, - created_at, updated_at, preserve_history + created_at, updated_at, preserve_history, doc_lang FROM documents WHERE id = ? AND user_id = ?`, id, userID, ).Scan( &doc.ID, &doc.UserID, &doc.Title, &doc.Content, &doc.ContentText, &doc.Tone, &doc.WordCount, &doc.CreatedAt, &doc.UpdatedAt, &doc.PreserveHistory, + &doc.DocLang, ) return doc, err } diff --git a/internal/suggestions/handlers.go b/internal/suggestions/handlers.go index d91ff8c..a57e371 100644 --- a/internal/suggestions/handlers.go +++ b/internal/suggestions/handlers.go @@ -825,13 +825,13 @@ func (h *Handler) setStatus(w http.ResponseWriter, r *http.Request, status strin // hands over a reusable chunk, which is the only thing worth reviewing in a week. func (h *Handler) plant(id, userID string) { var s db.Suggestion - var contentText string + var contentText, docLang string err := h.DB.QueryRow( - `SELECT s.type, s.original, s.replacement, s.explanation, s.doc_id, d.content_text + `SELECT s.type, s.original, s.replacement, s.explanation, s.doc_id, d.content_text, d.doc_lang FROM suggestions s JOIN documents d ON d.id = s.doc_id WHERE s.id = ? AND d.user_id = ?`, id, userID, - ).Scan(&s.Type, &s.Original, &s.Replacement, &s.Explanation, &s.DocID, &contentText) + ).Scan(&s.Type, &s.Original, &s.Replacement, &s.Explanation, &s.DocID, &contentText, &docLang) if err != nil { if !errors.Is(err, sql.ErrNoRows) { log.Printf("suggestions: could not read %s for planting: %v", id, err) @@ -850,6 +850,9 @@ func (h *Handler) plant(id, userID string) { Meaning: s.Explanation, Example: correctedSentence(contentText, s.Original, s.Replacement), DocID: &docID, + // The chunk is her own sentence, corrected — so it is in the document's + // language, whatever the collocation pass was framed in. + Lang: docLang, }); err != nil { log.Printf("suggestions: could not plant %s: %v", id, err) } diff --git a/internal/suggestions/plant_test.go b/internal/suggestions/plant_test.go index 854de5d..f4bf498 100644 --- a/internal/suggestions/plant_test.go +++ b/internal/suggestions/plant_test.go @@ -184,3 +184,60 @@ func TestCorrectedSentence(t *testing.T) { t.Errorf("unterminated doc: got %q", got) } } + +// TestPlantedPhraseCarriesTheDocumentLanguage: a chunk planted out of a document +// written in her own language is a card in that language. The collocation pass +// itself deliberately did not flip in Phase 28 — its prompt is per-language +// knowledge, not framing — but the phrase it hands over is still lifted from her +// prose, so the card has to know what language that prose was in or the garden +// will read it aloud in the wrong voice. +func TestPlantedPhraseCarriesTheDocumentLanguage(t *testing.T) { + srv, docID, h := newTestServer(t, &stubClient{}) + if _, err := h.DB.Exec(`UPDATE documents SET doc_lang = 'pair' WHERE id = ?`, docID); err != nil { + t.Fatalf("set doc_lang: %v", err) + } + id := seedSuggestion(t, h, docID, + "Ontem foi difícil. Tive de tomar uma decisão sobre o trabalho.", + db.SuggestionTypeCollocation, "tomar uma decisão", "tomar uma decisão", + "Em português diz-se “tomar” uma decisão.") + + if rec := do(t, srv, http.MethodPost, "/suggestions/"+id+"/accept", ""); rec.Code != http.StatusNoContent { + t.Fatalf("accept: got %d, want 204", rec.Code) + } + + var lang string + if err := h.DB.QueryRow( + `SELECT lang FROM vocab_words WHERE user_id = ? AND word = ?`, + db.LocalUserID, "tomar uma decisão", + ).Scan(&lang); err != nil { + t.Fatalf("read planted card: %v", err) + } + if lang != "pair" { + t.Fatalf("planted card lang = %q, want pair", lang) + } +} + +// TestPlantedPhraseOnAnEnglishDocumentIsUnchanged is the other half, and the one +// every account is on today: an English document plants an English card, and the +// empty backfill and 'en' both mean that. +func TestPlantedPhraseOnAnEnglishDocumentIsUnchanged(t *testing.T) { + srv, docID, h := newTestServer(t, &stubClient{}) + id := seedSuggestion(t, h, docID, "I had to do a decision about the job.", + db.SuggestionTypeCollocation, "do a decision", "make a decision", + "English pairs “make” with “decision”.") + + if rec := do(t, srv, http.MethodPost, "/suggestions/"+id+"/accept", ""); rec.Code != http.StatusNoContent { + t.Fatalf("accept: got %d, want 204", rec.Code) + } + + var lang string + if err := h.DB.QueryRow( + `SELECT lang FROM vocab_words WHERE user_id = ? AND word = ?`, + db.LocalUserID, "make a decision", + ).Scan(&lang); err != nil { + t.Fatalf("read planted card: %v", err) + } + if lang == "pair" { + t.Fatalf("planted card lang = %q on an English document, want en or empty", lang) + } +} diff --git a/internal/vocab/handlers.go b/internal/vocab/handlers.go index e205f49..5d0d9f1 100644 --- a/internal/vocab/handlers.go +++ b/internal/vocab/handlers.go @@ -19,13 +19,18 @@ import ( // Word is one entry in the vocabulary garden: the looked-up word with its gloss, // phonetic, and the sentence it was met in, plus its spaced-repetition state. type Word struct { - ID string `json:"id"` - Word string `json:"word"` - Gloss string `json:"gloss"` - Definition string `json:"definition"` // English fallback meaning when there's no Chinese gloss - Phonetic string `json:"phonetic"` - Example string `json:"example"` - DocID *string `json:"doc_id"` + ID string `json:"id"` + Word string `json:"word"` + Gloss string `json:"gloss"` + Definition string `json:"definition"` // English fallback meaning when there's no Chinese gloss + Phonetic string `json:"phonetic"` + Example string `json:"example"` + DocID *string `json:"doc_id"` + // Lang is '' | 'en' | 'pair' — the language of the document the word was met + // in (see migration 0018). '' reads as English, like everywhere else this + // vocabulary appears. The client needs it to pick a read-aloud voice: "comum" + // is unguessable from its letters, so the card has to carry the answer. + Lang string `json:"lang"` DueAt time.Time `json:"due_at"` IntervalDays int `json:"interval_days"` Ease float64 `json:"ease"` @@ -54,7 +59,7 @@ func (h *Handler) Routes() chi.Router { return r } -const vocabColumns = `id, word, gloss, definition, phonetic, example, doc_id, +const vocabColumns = `id, word, gloss, definition, phonetic, example, doc_id, lang, due_at, interval_days, ease, reps, lapses, last_reviewed, created_at` func scanWord(s interface { @@ -62,7 +67,7 @@ func scanWord(s interface { }) (Word, error) { var w Word err := s.Scan( - &w.ID, &w.Word, &w.Gloss, &w.Definition, &w.Phonetic, &w.Example, &w.DocID, + &w.ID, &w.Word, &w.Gloss, &w.Definition, &w.Phonetic, &w.Example, &w.DocID, &w.Lang, &w.DueAt, &w.IntervalDays, &w.Ease, &w.Reps, &w.Lapses, &w.LastReviewed, &w.CreatedAt, ) return w, err @@ -165,15 +170,22 @@ func (h *Handler) capture(w http.ResponseWriter, r *http.Request) { // would hit the foreign key and leak a raw "FOREIGN KEY constraint" 500 // instead of a clean 400 (and, once auth lands, would let a word be attached // to another user's document). + // + // The same row-scoped lookup answers what language the card is in: a word is + // met inside a document, so the document's verdict is the word's language. + // Asking the document rather than trusting a `lang` in the request body is + // the same choice `runPass` makes — the client never gets to name a language + // the server can already read. A word with no document is '', which reads as + // English. + lang := "" if req.DocID != nil { if strings.TrimSpace(*req.DocID) == "" { req.DocID = nil } else { - var ok int err := h.DB.QueryRow( - `SELECT 1 FROM documents WHERE id = ? AND user_id = ?`, + `SELECT doc_lang FROM documents WHERE id = ? AND user_id = ?`, *req.DocID, userID, - ).Scan(&ok) + ).Scan(&lang) if errors.Is(err, sql.ErrNoRows) { httputil.ErrorJSON(w, http.StatusBadRequest, "unknown doc_id") return @@ -189,15 +201,19 @@ func (h *Handler) capture(w http.ResponseWriter, r *http.Request) { // schedule (due_at/reps/interval/ease) alone so re-looking-up a word never // resets its progress. _, err := h.DB.Exec( - `INSERT INTO vocab_words (user_id, word, gloss, definition, phonetic, example, doc_id, due_at, interval_days) - VALUES (?, ?, ?, ?, ?, ?, ?, datetime('now', '+1 day'), 1) + `INSERT INTO vocab_words (user_id, word, gloss, definition, phonetic, example, doc_id, lang, due_at, interval_days) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, datetime('now', '+1 day'), 1) ON CONFLICT(user_id, word) DO UPDATE SET gloss = excluded.gloss, definition = excluded.definition, phonetic = excluded.phonetic, example = CASE WHEN excluded.example != '' THEN excluded.example ELSE vocab_words.example END, - doc_id = COALESCE(excluded.doc_id, vocab_words.doc_id)`, - userID, word, req.Gloss, req.Definition, req.Phonetic, req.Example, req.DocID, + doc_id = COALESCE(excluded.doc_id, vocab_words.doc_id), + -- lang travels with doc_id, and for the same reason: it is the new + -- context or it is nothing. A lookup made outside any document must + -- not relabel a card that was captured inside one. + lang = CASE WHEN excluded.doc_id IS NOT NULL THEN excluded.lang ELSE vocab_words.lang END`, + userID, word, req.Gloss, req.Definition, req.Phonetic, req.Example, req.DocID, lang, ) if err != nil { httputil.ServerError(w, err) diff --git a/internal/vocab/handlers_test.go b/internal/vocab/handlers_test.go index 54d8c2e..6987713 100644 --- a/internal/vocab/handlers_test.go +++ b/internal/vocab/handlers_test.go @@ -253,3 +253,89 @@ func TestDocLinkSurvivesDocDelete(t *testing.T) { t.Fatalf("doc_id should be nulled after doc delete, got %v", *all[0].DocID) } } + +// TestCaptureTakesLanguageFromItsDocument is the garden's half of Phase 28: a +// word met inside a document written in her own language is a card in that +// language, and the server reads that off the document rather than being told. +// +// The three cases are the three the client can actually produce: a lookup inside +// a flipped document, a lookup inside an English one, and a lookup with no +// document at all (the search box) — the last of which is '', which reads as +// English everywhere this value is used. +func TestCaptureTakesLanguageFromItsDocument(t *testing.T) { + srv, database := newTestServer(t) + + seed := func(lang string) string { + t.Helper() + var id string + if err := database.QueryRow( + `INSERT INTO documents (user_id, content_text, doc_lang) VALUES (?, 'hi', ?) RETURNING id`, + db.LocalUserID, lang, + ).Scan(&id); err != nil { + t.Fatalf("seed doc: %v", err) + } + return id + } + pairDoc, enDoc := seed("pair"), seed("en") + + capture := func(word, body string) Word { + t.Helper() + rec := do(t, srv, http.MethodPost, "/vocab", body) + if rec.Code != http.StatusCreated { + t.Fatalf("capture %s: code=%d body=%s", word, rec.Code, rec.Body) + } + var w Word + if err := json.Unmarshal(rec.Body.Bytes(), &w); err != nil { + t.Fatalf("decode %s: %v", word, err) + } + return w + } + + if got := capture("comum", `{"word":"comum","doc_id":"`+pairDoc+`"}`); got.Lang != "pair" { + t.Fatalf("word from a flipped document: lang=%q, want pair", got.Lang) + } + if got := capture("reception", `{"word":"reception","doc_id":"`+enDoc+`"}`); got.Lang != "en" { + t.Fatalf("word from an English document: lang=%q, want en", got.Lang) + } + if got := capture("orphan", `{"word":"orphan"}`); got.Lang != "" { + t.Fatalf("word with no document: lang=%q, want empty", got.Lang) + } + + // A client that names a language is ignored: the document is the authority, + // the same way runPass never lets the request pick its own target. + if got := capture("comum", `{"word":"comum","lang":"en","doc_id":"`+pairDoc+`"}`); got.Lang != "pair" { + t.Fatalf("client-supplied lang should not win: lang=%q, want pair", got.Lang) + } +} + +// TestRecaptureOutsideADocumentKeepsItsLanguage pins the one asymmetry in the +// upsert. Re-looking-up a word refreshes its context, but a lookup made with no +// document carries no verdict — and relabelling a Portuguese card English +// because she checked the word again from the search box would silently move it +// to the wrong voice. lang travels with doc_id, or it doesn't travel. +func TestRecaptureOutsideADocumentKeepsItsLanguage(t *testing.T) { + srv, database := newTestServer(t) + var docID string + if err := database.QueryRow( + `INSERT INTO documents (user_id, content_text, doc_lang) VALUES (?, 'olá', 'pair') RETURNING id`, + db.LocalUserID, + ).Scan(&docID); err != nil { + t.Fatalf("seed doc: %v", err) + } + if rec := do(t, srv, http.MethodPost, "/vocab", + `{"word":"saudade","gloss":"","doc_id":"`+docID+`"}`); rec.Code != http.StatusCreated { + t.Fatalf("capture: code=%d body=%s", rec.Code, rec.Body) + } + rec := do(t, srv, http.MethodPost, "/vocab", `{"word":"saudade","gloss":"longing"}`) + if rec.Code != http.StatusCreated { + t.Fatalf("recapture: code=%d body=%s", rec.Code, rec.Body) + } + var w Word + _ = json.Unmarshal(rec.Body.Bytes(), &w) + if w.Lang != "pair" { + t.Fatalf("recapture outside a document: lang=%q, want pair held", w.Lang) + } + if w.Gloss != "longing" { + t.Fatalf("recapture should still refresh the gloss, got %q", w.Gloss) + } +} diff --git a/internal/vocab/plant.go b/internal/vocab/plant.go index c04041d..dc15cc8 100644 --- a/internal/vocab/plant.go +++ b/internal/vocab/plant.go @@ -28,6 +28,10 @@ type Phrase struct { Meaning string // why it's better — the suggestion's explanation Example string // the sentence she met it in, already corrected DocID *string // where, so "where did I see this?" stays one tap + // Lang is the document's verdict ('' | 'en' | 'pair'), because the chunk is + // lifted out of her own prose and is therefore in whatever language that + // prose is. See migration 0018. + Lang string } // Phrase-card caps. A collocation is a short chunk; anything longer is a @@ -84,13 +88,14 @@ func Plant(ex Execer, userID string, p Phrase) (bool, error) { return false, nil } res, err := ex.Exec( - `INSERT INTO vocab_words (user_id, word, gloss, definition, phonetic, example, doc_id, due_at, interval_days) - VALUES (?, ?, '', ?, '', ?, ?, datetime('now', '+1 day'), 1) + `INSERT INTO vocab_words (user_id, word, gloss, definition, phonetic, example, doc_id, lang, due_at, interval_days) + VALUES (?, ?, '', ?, '', ?, ?, ?, datetime('now', '+1 day'), 1) ON CONFLICT(user_id, word) DO NOTHING`, userID, key, clamp(strings.TrimSpace(p.Meaning), maxDefinitionLen), clamp(strings.TrimSpace(p.Example), maxExampleLen), p.DocID, + p.Lang, ) if err != nil { return false, err diff --git a/web/src/App.tsx b/web/src/App.tsx index cc0a656..65fe344 100644 --- a/web/src/App.tsx +++ b/web/src/App.tsx @@ -615,6 +615,7 @@ export default function App() { { expect(bodies.at(-1)).toMatchObject({ text: 'chat', lang: 'fr-FR' }) }) }) + +describe('docLang', () => { + // The document's verdict is the answer for a passage lifted out of it, because + // for a Latin pair there is no other answer available: an English sentence and + // a Portuguese one are the same letters. + it('reads a flipped document in her own language', () => { + setPackLang('pt-PT') + expect(docLang('Ontem foi difícil.', 'pair')).toBe('pt-PT') + }) + + it('reads an English document in English, and treats the backfill as English', () => { + setPackLang('pt-PT') + expect(docLang('Yesterday was hard.', 'en')).toBe('en-US') + expect(docLang('Yesterday was hard.', '')).toBe('en-US') + }) + + it('lets the script win over the verdict, so quoted Chinese is never spelled out', () => { + // An English document quoting Chinese is 'en' by verdict, and the English + // voice reads Han characters one "Chinese letter" at a time — the one + // failure worse than silence. + setPackLang('zh') + expect(docLang('你好世界', 'en')).toBe('zh-CN') + }) + + it('is what the editor selection and the garden card ask for', () => { + setPackLang('fr') + speak('Le chat dort.', docLang('Le chat dort.', 'pair')) + expect(bodies.at(-1)).toMatchObject({ text: 'Le chat dort.', lang: 'fr-FR' }) + }) +}) diff --git a/web/src/audio/speech.ts b/web/src/audio/speech.ts index eefc694..7640b01 100644 --- a/web/src/audio/speech.ts +++ b/web/src/audio/speech.ts @@ -90,6 +90,24 @@ export function nativeLang(): string { return pack().locale } +// docLang turns a document-language verdict ('' | 'en' | 'pair', decided +// server-side — see internal/suggestions/doclang.go) into a locale for a passage +// taken out of that document. It is what the editor's read-aloud and the garden's +// review card ask instead of guessing. +// +// The script test still wins, and that is not redundant with the verdict. A +// Chinese sentence quoted inside an English document is 'en' by verdict and +// still has to be read by the Chinese voice: the English voice spells Han +// characters out one "Chinese letter" at a time, which is the one failure loud +// enough to be worse than no audio. In the other direction there is nothing to +// test — an English sentence inside Portuguese prose looks exactly like the +// Portuguese around it — so the document's verdict is the only answer available, +// and it is the answer this phase decided on. +export function docLang(text: string, verdict: string): string { + if (CJK.test(text)) return 'zh-CN' + return verdict === 'pair' ? pack().locale : 'en-US' +} + // speak reads `text` aloud, cancelling anything already in flight so rapid taps // don't queue up. `lang` defaults to a guess from the text (Chinese vs English) // so callers can just pass the selection; pass an explicit locale to override. diff --git a/web/src/components/Editor/EditorCore.tsx b/web/src/components/Editor/EditorCore.tsx index 1c745ad..c3897c8 100644 --- a/web/src/components/Editor/EditorCore.tsx +++ b/web/src/components/Editor/EditorCore.tsx @@ -33,8 +33,8 @@ import { Composition } from './Composition' import { RewritePreview, type RewriteStatus } from './RewritePreview' import { planBatch } from './acceptBatch' import { entryId, idAfterRemoval, stepId, type Direction, type Span } from './triage' -import { api, type Suggestion, type SuggestionType, type WordInfo } from '../../api/client' -import { speak, speechSupported } from '../../audio/speech' +import { api, type DocLang, type Suggestion, type SuggestionType, type WordInfo } from '../../api/client' +import { docLang as docLocale, speak, speechSupported } from '../../audio/speech' import type { SpellChecker } from '../../hooks/useSpellChecker' import { fromIME } from '../../lib/ime' import type { Segmenter } from '../../lib/segment' @@ -65,6 +65,13 @@ export interface EditorChange { interface Props { // Changing docId reloads the editor with that document's content. docId: string + // Which language this document is written in, as the server decided it + // ('' | 'en' | 'pair'). Read-aloud is the only thing here that reads it: a + // Portuguese selection has to be read by the Portuguese voice, and nothing in + // the letters says so. Updated by the server on save, so it trails a language + // flip by one auto-save — a passage read in the old voice once is the whole + // cost of not blocking the editor on a check. + docLang: DocLang initialContent: string onChange: (change: EditorChange) => void // LLM suggestions to highlight; accept/dismiss notify the parent for the API @@ -255,6 +262,7 @@ interface RewriteState { // decoration layer. Hovering a highlight opens its SuggestionCard. export function EditorCore({ docId, + docLang, initialContent, onChange, suggestions, @@ -1532,8 +1540,10 @@ export function EditorCore({ speak(selection.text) : null} - onSpeakSlow={speechSupported() ? () => speak(selection.text, undefined, true) : null} + onSpeak={speechSupported() ? () => speak(selection.text, docLocale(selection.text, docLang)) : null} + onSpeakSlow={ + speechSupported() ? () => speak(selection.text, docLocale(selection.text, docLang), true) : null + } /> )} {rewrite && ( diff --git a/web/src/components/Garden/GardenPanel.tsx b/web/src/components/Garden/GardenPanel.tsx index f2ac745..872924c 100644 --- a/web/src/components/Garden/GardenPanel.tsx +++ b/web/src/components/Garden/GardenPanel.tsx @@ -1,6 +1,6 @@ import { useCallback, useEffect, useMemo, useState } from 'react' import { api, type VocabGrade, type VocabWord } from '../../api/client' -import { speak, speechSupported, stopSpeech } from '../../audio/speech' +import { docLang as docLocale, speak, speechSupported, stopSpeech } from '../../audio/speech' import { useFocusTrap } from '../../hooks/useFocusTrap' import { usePack, type Line } from '../../i18n' import { JournalView } from './JournalView' @@ -314,7 +314,21 @@ function GardenView({ {blossom(w.reps)} - {w.word} + + {w.word} + {/* Only a card in her own language is marked. An English + garden with a badge on every blossom would be a + garden with no badges at all; the marker exists so + the rare Portuguese word is legible among them. */} + {w.lang === 'pair' && ( + + {t.nativeName} + + )} + {(w.gloss || w.definition) && ( speak(w.word)} + onClick={() => speak(w.word, docLocale(w.word, w.lang))} className="rounded-full px-2.5 py-1 text-xs font-semibold" style={{ background: 'var(--color-surface-alt)' }} > @@ -477,7 +491,7 @@ function ReviewSession({ <>