Correct the language she wrote in, not the one she was practising
Every pass was English-shaped: CheckpointMessages took the text and the tone and
nothing else, so there was never a language decision to get wrong. On the live
build two pt-PT sentences drew no cards at all — Petal read the Portuguese, said
nothing about it, and filed a mechanics note about the one English line.
The rule is two decisions reading different state. What gets corrected follows
the document. What language the explanation is written in follows the writer —
the half of her pair she is not learning, from users.direction — because an
explanation is teaching, and teaching lands in the language she reads most
easily. Those coincide for every account that exists today (learnerPairs is
{"zh"}), which is a fact about the roster and not about the design, so Target
keeps them apart. It carries a third language too: the collocation gloss is
addressed to her rather than to the document, and folding it into Explain would
have quietly moved it into English on every English document.
The document verdict is a proportion, not a presence — one Portuguese quotation
must not flip an English essay. Per sentence, three-way: pair, English, or no
answer. The third value is the load-bearing one; counting the undecided as
English is exactly what would hold a journal of short Portuguese sentences in
English forever, so the Latin pairs needed an englishMarkers list curated against
pt/fr/es as carefully as latinMarkers was curated against English. Hysteresis at
70/40 because a bilingual paragraph would otherwise alternate its cards' language
every few keystrokes, and hysteresis needs a yesterday — hence the column. Plus a
corroboration floor: a ratio computed over "Não. Eu." is 100% of nothing, and a
flip rewrites every card in the document.
The verdict folds into the chunk salt beside the tone, so a document that changes
language re-opens every sentence rather than serving back cards in a language it
no longer speaks.
checkpointSystemPrompt could not simply take a language — it opens by naming the
reader an ESL learner, and appending "explain in Portuguese" hands the model two
contradictory framings. Separate constants, sharing the JSON contract below the
framing. Both carry a "never translate it into English" line, which is the
instruction the model will most want to disobey. The English prompts are
untouched byte for byte, and a golden says so out loud.
Collocation deliberately did not move: its prompt is per-language knowledge, not
framing, and "natives usually say" for Portuguese is a claim Petal cannot back.
Not deployed and not smoked against a real model. The tests drive the real router
and a real DB; what none of them prove is how Qwen behaves on a Portuguese
document, in particular whether the never-translate line holds.
Claude-Session: https://claude.ai/code/session_01GJHNvirh7Hzhc9RL3HAvz7
This commit is contained in:
prosolis
@@ -41,9 +41,9 @@ type checkpointResponse struct {
|
||||
// RunCheckpoint sends the grammar checkpoint and parses the JSON result. It
|
||||
// applies the latency-guard truncation and the checkpoint sampling parameters
|
||||
// from the spec.
|
||||
func RunCheckpoint(ctx context.Context, client LLMClient, contentText, tone string, _ Lang) ([]RawSuggestion, error) {
|
||||
func RunCheckpoint(ctx context.Context, client LLMClient, contentText, tone string, t Target) ([]RawSuggestion, error) {
|
||||
raw, err := client.Complete(ctx, CompletionRequest{
|
||||
Messages: CheckpointMessages(TruncateDoc(contentText), tone),
|
||||
Messages: CheckpointMessages(TruncateDoc(contentText), tone, t),
|
||||
MaxTokens: checkpointMaxTokens,
|
||||
Temperature: 0.3,
|
||||
RepetitionPenalty: 1.15,
|
||||
|
||||
@@ -20,9 +20,9 @@ const CollocationInterval = 25 * time.Second
|
||||
// The tone argument is accepted for a uniform pass signature and passed through
|
||||
// to the prompt so a hint can prefer a register-appropriate pairing. `lang` is
|
||||
// the writer's pair language — the one each hint's short gloss is written in.
|
||||
func RunCollocation(ctx context.Context, client LLMClient, contentText, tone string, lang Lang) ([]RawSuggestion, error) {
|
||||
func RunCollocation(ctx context.Context, client LLMClient, contentText, tone string, t Target) ([]RawSuggestion, error) {
|
||||
raw, err := client.Complete(ctx, CompletionRequest{
|
||||
Messages: CollocationMessages(contentText, tone, lang),
|
||||
Messages: CollocationMessages(contentText, tone, t),
|
||||
MaxTokens: 2048,
|
||||
Temperature: 0.3,
|
||||
RepetitionPenalty: 1.15,
|
||||
|
||||
@@ -31,7 +31,7 @@ func TestLangForFallsBackToDefault(t *testing.T) {
|
||||
func TestPromptsNameTheWritersLanguage(t *testing.T) {
|
||||
pt := LangFor("pt-PT")
|
||||
|
||||
collocation := CollocationMessages("The rain was strong.", "casual", pt)[0].Content
|
||||
collocation := CollocationMessages("The rain was strong.", "casual", EnglishTarget(pt))[0].Content
|
||||
if !strings.Contains(collocation, "European Portuguese") {
|
||||
t.Fatalf("collocation prompt doesn't ask for a pt-PT gloss:\n%s", collocation)
|
||||
}
|
||||
@@ -73,7 +73,7 @@ func TestPromptsNameTheWritersLanguage(t *testing.T) {
|
||||
func TestDefaultPairStillReadsAsBefore(t *testing.T) {
|
||||
zh := LangFor("zh")
|
||||
|
||||
if got := CollocationMessages("x", "", zh)[0].Content; !strings.Contains(got, "Simplified Chinese (Mandarin) gloss in parentheses") {
|
||||
if got := CollocationMessages("x", "", EnglishTarget(zh))[0].Content; !strings.Contains(got, "Simplified Chinese (Mandarin) gloss in parentheses") {
|
||||
t.Fatalf("zh collocation gloss changed:\n%s", got)
|
||||
}
|
||||
if got := TranslateMessages("x", zh)[0].Content; !strings.Contains(got, "natural, friendly Simplified Chinese (Mandarin)") {
|
||||
|
||||
+95
-8
@@ -45,11 +45,55 @@ func toneGuidance(tone string) string {
|
||||
"be improved, prefer suggestions that fit that tone, and gently flag wording that clashes with it."
|
||||
}
|
||||
|
||||
// pairCheckpointSystemPrompt is the grammar checkpoint for a document written in
|
||||
// the writer's own language rather than in English.
|
||||
//
|
||||
// It is a separate constant rather than a language clause appended to
|
||||
// checkpointSystemPrompt, because that prompt opens by naming the reader as an
|
||||
// ESL learner and asks for "common ESL patterns" — appending "and explain in
|
||||
// Portuguese" would hand the model two contradictory framings. Only the framing
|
||||
// differs; the JSON contract and the tone clause below it are the same
|
||||
// instructions in the same order, so the two prompts stay comparable.
|
||||
//
|
||||
// The "never translate" line is the one the model most wants to disobey: asked
|
||||
// to improve Portuguese while being an English writing assistant by training, it
|
||||
// will happily hand back an English rendering, which is a translation card
|
||||
// (Phase 25's `isTranslation`) and not a correction.
|
||||
const pairCheckpointSystemPrompt = `You are a warm, encouraging writing assistant. The person you are helping is ` +
|
||||
`writing in %[1]s, and the text below is %[1]s. ` +
|
||||
`Analyze it and identify up to 5 issues: grammar errors, unnatural phrasing, ` +
|
||||
`incorrect idiom usage, or unclear sentences.
|
||||
|
||||
Both "original" and "replacement" must be written in %[1]s. You are improving their %[1]s writing — ` +
|
||||
`never translate it into English, and never suggest they write in English instead.
|
||||
Write every "explanation" in %[2]s.
|
||||
|
||||
Be specific, friendly, and explain WHY each suggestion improves the writing.%[3]s
|
||||
|
||||
Respond ONLY with valid JSON. No preamble, no markdown fences. Format:
|
||||
{
|
||||
"suggestions": [
|
||||
{
|
||||
"original": "exact text from the document that needs fixing",
|
||||
"replacement": "corrected version",
|
||||
"explanation": "friendly one-sentence explanation",
|
||||
"type": "grammar|phrasing|idiom|clarity"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
If the writing looks good, return: {"suggestions": []}`
|
||||
|
||||
// CheckpointMessages builds the message array for a grammar checkpoint over the
|
||||
// given (already-truncated) document text, steered toward the document's tone.
|
||||
func CheckpointMessages(contentText, tone string) []Message {
|
||||
// given (already-truncated) document text, steered toward the document's tone
|
||||
// and aimed at the language the document is actually written in.
|
||||
func CheckpointMessages(contentText, tone string, t Target) []Message {
|
||||
system := fmt.Sprintf(checkpointSystemPrompt, toneGuidance(tone))
|
||||
if t.Flipped() {
|
||||
system = fmt.Sprintf(pairCheckpointSystemPrompt, t.Correct.Name, t.Explain.Name, toneGuidance(tone))
|
||||
}
|
||||
return []Message{
|
||||
{Role: "system", Content: fmt.Sprintf(checkpointSystemPrompt, toneGuidance(tone))},
|
||||
{Role: "system", Content: system},
|
||||
{Role: "user", Content: contentText},
|
||||
}
|
||||
}
|
||||
@@ -82,12 +126,48 @@ Respond ONLY with valid JSON. No preamble, no markdown fences. Format:
|
||||
|
||||
If the voice is consistent throughout, return: {"suggestions": []}`
|
||||
|
||||
// pairVoiceSystemPrompt is the voice pass for a document in the writer's own
|
||||
// language. Voice consistency is the one pass that transfers across languages
|
||||
// unchanged — a paragraph that reads as pasted from elsewhere reads that way in
|
||||
// any language — so only the framing and the explanation language move.
|
||||
const pairVoiceSystemPrompt = `You are a warm, encouraging writing assistant. The person you are helping is writing ` +
|
||||
`in %[1]s. You are reviewing a COMPLETE %[1]s document for VOICE CONSISTENCY only — not grammar.
|
||||
|
||||
Read the whole document to learn the writer's natural voice, then identify any passages (2 or more sentences) ` +
|
||||
`that feel tonally inconsistent with the surrounding writing — unusually formal, unusually polished, or phrased ` +
|
||||
`in a way that differs from the writer's established voice elsewhere in the document. These often signal text ` +
|
||||
`that was paraphrased too closely from another source. Do not flag the first paragraph (there is no baseline yet). ` +
|
||||
`Do not flag grammar or spelling mistakes — only voice.
|
||||
|
||||
Quote each passage exactly as it appears, in %[1]s. Write every "explanation" in %[2]s.
|
||||
|
||||
Respond ONLY with valid JSON. No preamble, no markdown fences. Format:
|
||||
{
|
||||
"suggestions": [
|
||||
{
|
||||
"original": "exact passage from the document that feels inconsistent",
|
||||
"replacement": null,
|
||||
"explanation": "friendly one-sentence note about why this passage sounds unlike the rest",
|
||||
"type": "voice"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
If the voice is consistent throughout, return: {"suggestions": []}`
|
||||
|
||||
// VoiceMessages builds the message array for a voice-consistency pass. Unlike
|
||||
// the checkpoint, the caller passes the WHOLE document (no truncation) — voice
|
||||
// consistency is judged against the established voice everywhere else.
|
||||
func VoiceMessages(contentText string) []Message {
|
||||
//
|
||||
// The pass had no language argument at all before Phase 28, which was the same
|
||||
// English assumption the checkpoint made, just unstated.
|
||||
func VoiceMessages(contentText string, t Target) []Message {
|
||||
system := voiceSystemPrompt
|
||||
if t.Flipped() {
|
||||
system = fmt.Sprintf(pairVoiceSystemPrompt, t.Correct.Name, t.Explain.Name)
|
||||
}
|
||||
return []Message{
|
||||
{Role: "system", Content: voiceSystemPrompt},
|
||||
{Role: "system", Content: system},
|
||||
{Role: "user", Content: contentText},
|
||||
}
|
||||
}
|
||||
@@ -133,10 +213,17 @@ If every pairing already sounds natural, return: {"suggestions": []}`
|
||||
// CollocationMessages builds the message array for a collocation pass over the
|
||||
// WHOLE document (no truncation), gently steered toward the document's tone so a
|
||||
// hint can prefer a register-appropriate pairing. The parenthetical gloss is
|
||||
// written in the writer's own language.
|
||||
func CollocationMessages(contentText, tone string, lang Lang) []Message {
|
||||
// written in the writer's own language — `Pair`, not `Explain`: the gloss is
|
||||
// addressed to her rather than to the document.
|
||||
//
|
||||
// The coach itself remains English-only. Collocation lists are the one thing
|
||||
// here that is genuinely per-language knowledge rather than framing, and
|
||||
// "natives usually say" for Portuguese is a claim this prompt has no grounds to
|
||||
// make yet; a flipped document simply gets the pass it always got. (Phase 28
|
||||
// moved the checkpoint and the voice pass; this one waits for evidence.)
|
||||
func CollocationMessages(contentText, tone string, t Target) []Message {
|
||||
return []Message{
|
||||
{Role: "system", Content: fmt.Sprintf(collocationSystemPrompt, toneGuidance(tone), lang.Name)},
|
||||
{Role: "system", Content: fmt.Sprintf(collocationSystemPrompt, toneGuidance(tone), t.Pair.Name)},
|
||||
{Role: "user", Content: contentText},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
package llm
|
||||
|
||||
// Which language a pass corrects, and which language it explains in.
|
||||
//
|
||||
// Until Phase 28 there was no question to answer: every prompt was written
|
||||
// around English prose explained in English, and the pair language reached her
|
||||
// only when she asked for it (Ask Petal, the explanation translator). That is
|
||||
// the right default for a writer practising English and the wrong one for a
|
||||
// document she wrote in her own language, where Petal would read Portuguese,
|
||||
// say nothing about it, and file a mechanics note about the one English
|
||||
// sentence at the end.
|
||||
//
|
||||
// The two fields are two different decisions reading two different pieces of
|
||||
// state, and collapsing them would be the bug:
|
||||
//
|
||||
// - Correct follows the DOCUMENT. Portuguese prose gets Portuguese
|
||||
// corrections; that is the whole point.
|
||||
// - Explain follows the WRITER — the half of her pair she is *not* learning
|
||||
// (users.direction), because an explanation is teaching, and teaching lands
|
||||
// in the language she reads most easily.
|
||||
//
|
||||
// Today those two coincide for every account that exists: `learnerPairs` is
|
||||
// {"zh"}, so fr, es and pt-PT writers are all `learning_en` and their
|
||||
// non-learned half *is* the pair language. That equality is a fact about
|
||||
// today's roster, not about the design — the same shape of assumption that had
|
||||
// to be unpicked from `pair_lang` in migration 0016. Keep them apart.
|
||||
type Target struct {
|
||||
// Correct is the language the writing is in, and so the language both
|
||||
// `original` and `replacement` must be written in.
|
||||
Correct Lang
|
||||
// Explain is the language each explanation is written in.
|
||||
Explain Lang
|
||||
// Pair is the writer's pair language regardless of what this document is
|
||||
// written in. The collocation coach's parenthetical gloss is addressed to
|
||||
// her rather than to the document, so it reads this and not Correct.
|
||||
Pair Lang
|
||||
}
|
||||
|
||||
// English as the prompts name it. Not in `langs`: that map answers "which
|
||||
// language is the writer's half of the pair", and English is the constant on
|
||||
// the other side of every pair Petal supports.
|
||||
var English = Lang{Code: "en", Name: "English", Why: "why"}
|
||||
|
||||
// EnglishTarget is the pre-Phase-28 behaviour, made explicit: an English
|
||||
// document, corrected and explained in English, for a writer whose pair
|
||||
// language is `pair`. Every existing user is on this path and the prompt it
|
||||
// produces is byte-identical to the one that shipped before this phase.
|
||||
func EnglishTarget(pair Lang) Target {
|
||||
return Target{Correct: English, Explain: English, Pair: pair}
|
||||
}
|
||||
|
||||
// Flipped reports whether this document is in the pair language rather than in
|
||||
// English — i.e. whether the pass is reading her own language.
|
||||
func (t Target) Flipped() bool { return t.Correct.Code != English.Code }
|
||||
@@ -0,0 +1,115 @@
|
||||
package llm
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The prompt every account is on today, written out in full.
|
||||
//
|
||||
// Phase 28 gave the checkpoint a second framing for documents in the writer's
|
||||
// own language, and the risk of that change is not that the new prompt is wrong
|
||||
// — it is that the old one moved by a word while nobody was looking. Every user
|
||||
// who exists is a Mandarin native writing English, so this string is the one
|
||||
// Petal actually sends, all day. It is duplicated here on purpose: a golden
|
||||
// copied from the constant it guards guards nothing.
|
||||
const goldenEnglishCheckpointPrompt = `You are a warm, encouraging writing assistant helping someone who speaks English as a second language. Analyze the text below and identify up to 5 issues: grammar errors, unnatural phrasing, incorrect idiom usage, or unclear sentences that are common ESL patterns.
|
||||
|
||||
Be specific, friendly, and explain WHY each suggestion improves the writing.
|
||||
|
||||
Respond ONLY with valid JSON. No preamble, no markdown fences. Format:
|
||||
{
|
||||
"suggestions": [
|
||||
{
|
||||
"original": "exact text from the document that needs fixing",
|
||||
"replacement": "corrected version",
|
||||
"explanation": "friendly one-sentence explanation",
|
||||
"type": "grammar|phrasing|idiom|clarity"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
If the writing looks good, return: {"suggestions": []}`
|
||||
|
||||
func TestEnglishDocumentPromptIsUnchanged(t *testing.T) {
|
||||
msgs := CheckpointMessages("I has two apple.", "", EnglishTarget(LangFor("zh")))
|
||||
if got := msgs[0].Content; got != goldenEnglishCheckpointPrompt {
|
||||
t.Fatalf("the English checkpoint prompt moved:\n--- got ---\n%s\n--- want ---\n%s", got, goldenEnglishCheckpointPrompt)
|
||||
}
|
||||
if msgs[1].Content != "I has two apple." {
|
||||
t.Fatalf("document text mangled: %q", msgs[1].Content)
|
||||
}
|
||||
|
||||
// The tone clause still lands, in the same place it always did.
|
||||
toned := CheckpointMessages("x", "academic", EnglishTarget(LangFor("zh")))[0].Content
|
||||
if !strings.Contains(toned, "formal, academic, and objective") {
|
||||
t.Fatalf("English checkpoint lost its tone guidance:\n%s", toned)
|
||||
}
|
||||
}
|
||||
|
||||
// A document in her own language gets a prompt that names that language, keeps
|
||||
// the corrections inside it, and drops the framing that only makes sense when
|
||||
// the thing being written is English.
|
||||
func TestFlippedCheckpointPrompt(t *testing.T) {
|
||||
pt := LangFor("pt-PT")
|
||||
system := CheckpointMessages("Hoje foi um dia bom.", "casual", Target{Correct: pt, Explain: pt, Pair: pt})[0].Content
|
||||
|
||||
if !strings.Contains(system, "European Portuguese") {
|
||||
t.Fatalf("flipped checkpoint doesn't name the language:\n%s", system)
|
||||
}
|
||||
if strings.Contains(system, "second language") || strings.Contains(system, "ESL") {
|
||||
t.Fatalf("flipped checkpoint kept the ESL framing:\n%s", system)
|
||||
}
|
||||
if !strings.Contains(system, "never translate it into English") {
|
||||
t.Fatalf("flipped checkpoint doesn't forbid translating:\n%s", system)
|
||||
}
|
||||
// The shared contract below the framing has to survive the split.
|
||||
for _, want := range []string{`"suggestions"`, `"replacement"`, "grammar|phrasing|idiom|clarity", "relaxed, friendly, and conversational"} {
|
||||
if !strings.Contains(system, want) {
|
||||
t.Fatalf("flipped checkpoint dropped %q:\n%s", want, system)
|
||||
}
|
||||
}
|
||||
if strings.Contains(system, "%!") {
|
||||
t.Fatalf("flipped checkpoint has a formatting error:\n%s", system)
|
||||
}
|
||||
}
|
||||
|
||||
// The two decisions are separate arguments and must reach the prompt separately:
|
||||
// corrections in the document's language, the explanation in the language she
|
||||
// reads most easily. Only the zh pair can be travelled both ways today, so it is
|
||||
// the only one that can prove they haven't been quietly collapsed into one.
|
||||
func TestFlippedPromptsExplainInTheirOwnLanguage(t *testing.T) {
|
||||
zh := LangFor("zh")
|
||||
|
||||
// Native Mandarin, practising English, writing Chinese: both halves Chinese.
|
||||
both := CheckpointMessages("今天天气很好。", "", Target{Correct: zh, Explain: zh, Pair: zh})[0].Content
|
||||
if strings.Count(both, "Simplified Chinese (Mandarin)") < 2 {
|
||||
t.Fatalf("expected corrections and explanations both in Chinese:\n%s", both)
|
||||
}
|
||||
if strings.Contains(both, "explanation"+`" in English`) {
|
||||
t.Fatalf("explanation language leaked to English:\n%s", both)
|
||||
}
|
||||
|
||||
// Native English, learning Chinese, writing Chinese: Chinese corrections,
|
||||
// English explanations.
|
||||
split := CheckpointMessages("今天天气很好。", "", Target{Correct: zh, Explain: English, Pair: zh})[0].Content
|
||||
if !strings.Contains(split, `Write every "explanation" in English.`) {
|
||||
t.Fatalf("learner direction didn't get English explanations:\n%s", split)
|
||||
}
|
||||
if !strings.Contains(split, "writing in Simplified Chinese (Mandarin)") {
|
||||
t.Fatalf("learner direction lost its Chinese corrections:\n%s", split)
|
||||
}
|
||||
|
||||
// Same for the voice pass, which had no language at all before this phase.
|
||||
voice := VoiceMessages("今天天气很好。", Target{Correct: zh, Explain: English, Pair: zh})[0].Content
|
||||
if !strings.Contains(voice, `Write every "explanation" in English.`) || !strings.Contains(voice, "Simplified Chinese") {
|
||||
t.Fatalf("flipped voice prompt got its languages wrong:\n%s", voice)
|
||||
}
|
||||
if strings.Contains(voice, "second language") {
|
||||
t.Fatalf("flipped voice prompt kept the ESL framing:\n%s", voice)
|
||||
}
|
||||
// An English document still gets exactly the voice prompt it always got.
|
||||
if got := VoiceMessages("x", EnglishTarget(zh))[0].Content; got != voiceSystemPrompt {
|
||||
t.Fatalf("English voice prompt moved:\n%s", got)
|
||||
}
|
||||
}
|
||||
@@ -19,9 +19,9 @@ const VoiceInterval = 20 * time.Second
|
||||
// The tone argument is accepted for a uniform pass signature but ignored: voice
|
||||
// consistency is judged against the document's own established voice, not an
|
||||
// externally-chosen register.
|
||||
func RunVoice(ctx context.Context, client LLMClient, contentText, _ string, _ Lang) ([]RawSuggestion, error) {
|
||||
func RunVoice(ctx context.Context, client LLMClient, contentText, _ string, t Target) ([]RawSuggestion, error) {
|
||||
raw, err := client.Complete(ctx, CompletionRequest{
|
||||
Messages: VoiceMessages(contentText),
|
||||
Messages: VoiceMessages(contentText, t),
|
||||
MaxTokens: 2048,
|
||||
Temperature: 0.3,
|
||||
RepetitionPenalty: 1.15,
|
||||
|
||||
Reference in New Issue
Block a user