From 5659312358923d528c05523cec93da9f55a39742 Mon Sep 17 00:00:00 2001
From: prosolis <5590409+prosolis@users.noreply.github.com>
Date: Tue, 28 Jul 2026 19:39:21 -0700
Subject: [PATCH] A real IME, a real candidate window, and 11.6 seconds of
 holding still

---
 BUILD_PLAN.md | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/BUILD_PLAN.md b/BUILD_PLAN.md
index a476ad6..81623d6 100644
--- a/BUILD_PLAN.md
+++ b/BUILD_PLAN.md
@@ -415,7 +415,12 @@ Phase 26's own outstanding item, and the thing it named as most likely to be wro
   - **The rule pack**: `Yesterday at work I did a mistake.` typed normally highlights *did a mistake* inside 800 ms. The same sentence typed inside a composition shows **nothing** after 1.2 s — five times the 250 ms fuse — and the card appears the moment the composition ends. That is the commit-nudge, end to end.
   - **The save is not held**, proven from the other side: with a composition still open and never ended, the server's copy already read `… STILLCOMPOSING`.
   - **The stolen keys**: Escape with `isComposing`, and with the legacy `keyCode 229`, both leave distraction-free mode alone; a real Escape restores the sidebar. In the Find bar the same three: `1 / 4`, `1 / 4`, then `2 / 4`.
-- ⚠️ **Still not a hardware IME.** What a synthetic composition cannot show is the thing at the very bottom: whether a real candidate window survives the DOM under it holding still. Everything above it — held, mapped, released, on the real ProseMirror composition path — now is shown.
+- [x] **Then with a real IME** (2026-07-28, same session). The user installed `ibus-libpinyin` and typed, because nothing else could: Chrome here runs **natively on Wayland**, so there is no way to inject key events into its window, and the keys the extension sends arrive over CDP — which bypasses the operating system's input method entirely. An IME test needs a person's hands, and the automation exists to *watch*, not to type. The page was instrumented to sample the document every 150 ms and on every composition event; she typed 很漂亮 into an English sentence.
+  - **The composition was open for 11.6 seconds across 12 `compositionupdate`s** — 和 → 很 → 很片 → 狠批 → 很皮阿 → 很票 → 很漂亮 → 很瓢里 → 很票俩 → 很票连 → 很漂亮 — and **the preedit really does live in the document**: the samples show the paragraph carrying each of those in turn. That is exactly the dangerous shape, the composing text inside the doc with ProseMirror rewriting the node under the candidate window on every keystroke.
+  - **Through all of it the underline set never moved**: `[parc, henpiaoliang]` before, during and at the end, held rather than rebuilt, and never applied to the composing text. The word committed cleanly as **很漂亮** — nothing eaten, nothing doubled — and typing English afterwards flagged a new word immediately, so the layers were released and not merely switched off.
+  - **An accidental control, and the neatest evidence in the run**: she typed `henpiaoliang` as plain letters *before* switching the IME on, and it was underlined. The same syllables, typed through the IME, produced hanzi that were not — the two paths side by side in one sentence.
+  - A brief flicker on `henpiaoliang`'s underline after the commit turned out to be **the caret exemption** (Phase 7's: the word under the cursor is not flagged), confirmed by putting the caret in and out of `parc` and watching its underline drop and return. Not composition-related.
+- ⚠️ **What this still does not prove is the counterfactual** — that the bug would have bitten *here* without the guard. Chrome's own composition handling is robust, and a clean run with the guard in place cannot distinguish "held correctly" from "would have been fine anyway". Settling it means building with the guard reverted and asking her to type once more; offered, not assumed.
 - **Not deployed.** No migration; a rebuild whenever the user wants it, along with Phases 24–26.
 
 ### Later / explicitly not now
@@ -435,7 +440,7 @@ Phase 26's own outstanding item, and the thing it named as most likely to be wro
 - [x] **Phase 14 — companion warmth + bedtime nag + night mode**: more encouraging phrases, a gentle "go to bed" nudge after 11pm, and a calm dark theme + falling stars at night. ✅ (see Phase 14 above)
 
 ## Session log
-- 2026-07-28: **Phase 27 — the keystroke that isn't one** (user asked to continue the build plan; the plan's own next item was Phase 26's unbuilt IME guards, named there as the likeliest thing to be wrong the first time anyone types Chinese into Petal for real). **The bug is that a composition is not a keystroke**: the pinyin goes into the document as it is typed, a candidate window sits over it, and all three decoration layers recompute from the live document on every change — rewriting the DOM around the node the browser is composing in, which is what eats half-typed input. **The fix is to hold the redraws, not skip them**: a rebuild that falls due mid-composition marks itself stale and its decorations are *mapped through the transaction*, so they travel with the text growing under them and land correct the moment the composition ends. Skipping would have left every highlight a character behind for as long as she kept typing — the same wrongness, arriving quietly. **The composition flag is read from the state *before* the transaction**, which makes it independent of plugin ordering: the flag was set by compositionstart, not by the transaction being applied, and the end transaction is the one deliberate exception or nothing would ever release. **One piece of timing genuinely matters and is written down where it happens**: a custom `handleDOMEvents` handler runs *before* ProseMirror's own, and ProseMirror's compositionend queues the composition's last DOM changes as a microtask — so the release is a macrotask late, and the held rebuild sees the committed 公园 rather than the *gongyuan* it replaced. If the flush's transaction arrives first it rebuilds anyway; both orders land, which is what makes it not a race. **One thing was checked rather than assumed and turned out already handled**: Tiptap's input-rule plugin returns early while composing — which matters more here than in an English app, because pinyin uses an apostrophe as a syllable separator (xi'an → 西安) and Typography rewrites every `'` into a curly `’`. **The save is deliberately not gated and the analysis is**: a tablet keyboard can hold one composition open for a whole sentence, and Petal never makes writing wait for anything, so `EditorChange` carries the flag, auto-save ignores it, and the checkpoint/rule pack/companion wait — with one more change emitted the instant the composition commits, so nothing is skipped, only deferred by the length of a word. **And four places were quietly stealing keys from the IME**: the Find bar (Enter = next match, Escape = close), the tag picker (Enter = create, in a field whose purpose is a name typed in Chinese), Ask Petal's chat box (the field she types Mandarin into, where Enter sends), and distraction-free mode's global Escape — pressed to fix a wrong candidate, it popped the sidebar back. tsc, vite, go build/vet/test, **vitest 296/296** (12 new, driving real plugin state through a real composition: pinyin in, candidate committed, composition ended). **Then verified in a real browser** (Chrome extension, throwaway DB on :8093): the events are dispatched rather than typed, since this box has no IME, but they drive ProseMirror's own composition machinery, so every layer takes the real path. `gongyuan` typed into the document mid-composition with **the caret moved off it** — so the caret exemption cannot be the explanation — stays un-underlined while `parc` keeps its underline, and underlines on compositionend with no further keystroke. *did a mistake* highlights in 800 ms typed normally and **not at all** after 1.2 s inside a composition, then appears the instant it commits. The save is not held, proven from the other side: with the composition still open and never ended, the server's copy already read `… STILLCOMPOSING`. Escape (`isComposing` and legacy 229) leaves distraction-free mode alone while a real Escape restores the sidebar; the Find bar counter goes 1/4, 1/4, 2/4. ⚠️ **Still not a hardware IME** — what a synthetic composition cannot show is whether a real candidate window survives the DOM under it holding still. Everything above that now is shown. **Not deployed** (no migration — a rebuild, along with Phases 24–26).
+- 2026-07-28: **Phase 27 — the keystroke that isn't one** (user asked to continue the build plan; the plan's own next item was Phase 26's unbuilt IME guards, named there as the likeliest thing to be wrong the first time anyone types Chinese into Petal for real). **The bug is that a composition is not a keystroke**: the pinyin goes into the document as it is typed, a candidate window sits over it, and all three decoration layers recompute from the live document on every change — rewriting the DOM around the node the browser is composing in, which is what eats half-typed input. **The fix is to hold the redraws, not skip them**: a rebuild that falls due mid-composition marks itself stale and its decorations are *mapped through the transaction*, so they travel with the text growing under them and land correct the moment the composition ends. Skipping would have left every highlight a character behind for as long as she kept typing — the same wrongness, arriving quietly. **The composition flag is read from the state *before* the transaction**, which makes it independent of plugin ordering: the flag was set by compositionstart, not by the transaction being applied, and the end transaction is the one deliberate exception or nothing would ever release. **One piece of timing genuinely matters and is written down where it happens**: a custom `handleDOMEvents` handler runs *before* ProseMirror's own, and ProseMirror's compositionend queues the composition's last DOM changes as a microtask — so the release is a macrotask late, and the held rebuild sees the committed 公园 rather than the *gongyuan* it replaced. If the flush's transaction arrives first it rebuilds anyway; both orders land, which is what makes it not a race. **One thing was checked rather than assumed and turned out already handled**: Tiptap's input-rule plugin returns early while composing — which matters more here than in an English app, because pinyin uses an apostrophe as a syllable separator (xi'an → 西安) and Typography rewrites every `'` into a curly `’`. **The save is deliberately not gated and the analysis is**: a tablet keyboard can hold one composition open for a whole sentence, and Petal never makes writing wait for anything, so `EditorChange` carries the flag, auto-save ignores it, and the checkpoint/rule pack/companion wait — with one more change emitted the instant the composition commits, so nothing is skipped, only deferred by the length of a word. **And four places were quietly stealing keys from the IME**: the Find bar (Enter = next match, Escape = close), the tag picker (Enter = create, in a field whose purpose is a name typed in Chinese), Ask Petal's chat box (the field she types Mandarin into, where Enter sends), and distraction-free mode's global Escape — pressed to fix a wrong candidate, it popped the sidebar back. tsc, vite, go build/vet/test, **vitest 296/296** (12 new, driving real plugin state through a real composition: pinyin in, candidate committed, composition ended). **Then verified in a real browser** (Chrome extension, throwaway DB on :8093): the events are dispatched rather than typed, since this box has no IME, but they drive ProseMirror's own composition machinery, so every layer takes the real path. `gongyuan` typed into the document mid-composition with **the caret moved off it** — so the caret exemption cannot be the explanation — stays un-underlined while `parc` keeps its underline, and underlines on compositionend with no further keystroke. *did a mistake* highlights in 800 ms typed normally and **not at all** after 1.2 s inside a composition, then appears the instant it commits. The save is not held, proven from the other side: with the composition still open and never ended, the server's copy already read `… STILLCOMPOSING`. Escape (`isComposing` and legacy 229) leaves distraction-free mode alone while a real Escape restores the sidebar; the Find bar counter goes 1/4, 1/4, 2/4. **Then with a real IME**, because the user installed one and typed — the only way it could be done: Chrome here runs natively on Wayland, so no key can be injected into its window, and the extension's keys arrive over CDP, which bypasses the OS input method entirely. 很漂亮, composed over **11.6 seconds and 12 candidate changes** (和 → 很 → 狠批 → 很票 → 很漂亮 …), with the preedit genuinely living in the document the whole time — and the underline set never moved, the word committed clean, and English typed afterwards flagged immediately. **The accidental control is the best part**: she typed `henpiaoliang` as plain letters before switching the IME on and it was underlined, so the same syllables appear twice in one sentence, flagged on the path that isn't a composition and untouched on the path that is. ⚠️ **The counterfactual is still unproven** — a clean run with the guard in place can't distinguish "held correctly" from "would have been fine anyway"; settling that means a build with the guard reverted and another minute of her typing. **Not deployed** (no migration — a rebuild, along with Phases 24–26).
 - 2026-07-28: **Phase 26 — the zh pair's other direction, and a rule pack whose best feature is what it refuses** (user asked to continue the build plan, then chose a new phase over deploying fr/es; scope chosen with the user: **direction column, segmentation + hover pinyin/gloss, 错别字, IME guards**, code only). SUGGESTIONS §4 had called this its own epic, and the reason turned out to be one sentence: **`pair_lang` had always been answering a second question nobody asked.** It says which two languages; every surface built on it assumed English was the one being *learned*, which is why CJK is deliberately never tokenized, never spell-checked and never glossed. All correct for the writer this app was built for, all backwards for the other one. A `direction` column rather than a `zh-learner` pair code, because the two are genuinely separate questions and the column is what lets fr/es/pt inherit the capacity later. **The phase has three decisions in it and they are all about coverage.** The browser gets a word list and the server keeps the dictionary, and their gates come out *opposite*: the client list is frequency-gated at 5 because segmentation by the full union and by the gated list is **identical** on ordinary prose (measured, including 研究生命的起源 and 乒乓球拍卖完了 — the long tail is rare proper nouns and the max-probability walk never picks one), while the dictionary is gated by **nothing**, because its only power is to *explain* and the word a learner stops on is precisely the rare one. That is the es dictionary decision arrived at from both sides in one phase. **Writing the segmenter tests found the boundary bug**: a position names a gap and a word covers characters, so a hover on a boundary was resolving to the word that ended there rather than the one that starts. **The 错别字 pack is the part worth reading.** Chinese has no misspellings — every character an IME offers is real — so the unit of error is a substituted character inside a correct-looking word, and the pack is 24 confusable pairs held by two mechanical gates. Gate one admits a pair only if the wrong form is *not* a dictionary word and the right form is, which is what makes it refuse **自已 for 自己** — one of the commonest slips in written Chinese, whose wrong form is itself a headword — exactly as Phase 22 refused `married with`. Gate two is the one that matters: **自己经常 contains 己经, 睡觉的时候 contains 觉的, 不知到底 contains 知到**, so a substring match would corrupt correct sentences silently, into text still made of real characters. The segmenter settles it — two adjacent single-character tokens is what the walk produces when it has nothing better, which is what a mistyped compound looks like — and where the gate costs a real catch (我不知到他在哪里 *is* 知到 for 知道, but 不知 is a word) it declines rather than risk the identical-looking correct sentence beside it. **`@types/node` went in as a devDependency and quietly fixes something older**: phases 21–25 each verified their shipped dictionary with a throwaway script because vitest could not read files; the suite now asserts against the real assets. go build/vet/test, tsc, vite, **vitest 284/284**, live smoke on a throwaway DB which itself caught a cosmetic build bug ("cat (" left by stripping CC-CEDICT's CL: field). ⚠️ **The IME guards were in scope and are not done** — no composition handling exists anywhere in the app, and a decoration rebuild mid-composition eating half-typed pinyin is the likeliest thing to be wrong the first time anyone types Chinese into Petal for real. Left untouched rather than half-built, and named here rather than buried. ⚠️ **Not deployed** (it carries a migration), **not seen in a browser**, and **no account has ever been in the learner direction**, so everything above about how it feels to use is inference from tests.
 - 2026-07-28: **Phase 25 — the es pair, and a plan that had quietly chosen the wrong Spanish** (user asked where Spanish support had gone, then "yes" to starting the phase; scope chosen with the user: **Latin American neutral**, quorum review, code only). The starting point was a misreading worth recording: the plan *reads* as though Spanish shipped, because the DreamDict rebuild, the LLM language entry, the L1 rule gating and the TTS env-discovery are all `[x]` — every piece of groundwork was done and the pair itself had never been built. `shippedPairs` was the honest answer all along: the server had been refusing `es` on purpose. **The regional question was the phase.** pt-PT's was forced by packaging and fr's turned out not to exist; es had a real choice with no default, and once the user chose Latin American, the plan's own two concrete decisions were both wrong. It warned that `hunspell-es` is "packaged per country — check what `es_ES` actually is": it ships twenty country codes and **every one is a symlink to one pan-Hispanic file**, so the trap was not there. And it named **`es_ES-davefx-medium`** for the voice, which *is* the trap — six of Piper's nine Spanish voices are peninsular, so the obvious pick would have read Latin American copy in a Castilian accent, the pt-PT mistake arriving through a different door. `es_MX-ald-medium` instead. **Then the user asked "should we pick a different Spanish dictionary?" and the answer was yes** — the phase had shipped the wrong one and written a confident justification for it. Debian's `hunspell-es` symlinks twenty country codes to one file, which reads as pan-Hispanic; RLA actually publishes twenty-four builds per release, one per country **plus a generic `es` that is the union**, and Debian ships **peninsular `es_ES`**. The 58,622-form difference is essentially **voseo**: under the first build, *vení* and *tenés* — the ordinary present tense of Argentina, Uruguay, Paraguay and much of Central America — were underlined as misspellings, and this document called that a known gap handled on principle. **What makes it worth writing down is that the MUST_ACCEPT list was designed to catch exactly this and could not**: it asserted the pan-Hispanic *vocabulary*, and every RLA variant carries the full pan-Hispanic vocabulary — only the paradigms are localised — so it was satisfiable by all twenty-four. The `REP` table cited as the corroborating witness (yeísmo, seseo) is likewise shared by every build. Two independent-looking proofs, neither of which could distinguish anything, agreeing with each other. The profile now demands **voseo** (rejects es_ES and Debian), **vosotros** (rejects es_MX) and **another region's everyday words** — *arepa*, *chévere*, *bacán* (rejects es_AR, which has both paradigms and would otherwise pass); all four neighbours were run through it and confirmed refused. Shipping the union is the same call fr made between *coût* and *cout*: the dictionary's only power is to underline, so it holds every variety, while the *copy* picks a register because speaking requires one. 717,640 forms, 1.74 MB gzipped, **762 ms / 97 MB** in a real nspell, and **fr and pt-PT rebuild byte-identical** from their own upstream debs. **The quorum review earned its place twice**: four models, ≥2-of-4, 5 of 27 findings applied — one of which caught the pack's bedtime proverb being *Qui dort dîne* calqued into Spanish, English gloss and all, which is exactly the "a pack is not a translation of another pack" rule the fr header states and I had broken while writing it. And one below-threshold finding (a missing `¡` on an exclamative, seen by 1 of 4 because an absent *opening* mark has no closing `!` to look wrong against) was applied anyway and **turned into an assertion**: the suite now rejects any native line that closes `?`/`!` without opening one. That is Phase 24's lesson one level up — what a review finds once, a test should find every time. go build/vet/test, tsc, vite, **vitest 251/251**. ⚠️ **Not deployed, not seen in a browser, not read by a native speaker, and no es account exists** — all four accounts' worth of Spanish experience is still hypothetical, and the pack says so in its own header.
 - 2026-07-27: **Phase 24 — the fr pair, and a "generalizes" that did not** (user: "resume the build plan"; scope chosen with the user: French end to end, code only, deploy its own step). The plan's five items were meant to be mechanical, and four of them were — the Piper voice is a compose service and two env lines because Phase 21 made a language configuration; the lexicon needed nothing at all, fr having been measured at 63.1% during Phase 20's rebuild, better than the pair that already shipped; the sidebar picker grew a third entry without a line of UI because it derives itself from the shipped packs. **Item 3 was the one that had been recorded as done and wasn't.** `build_ptpt_dictionary.py` was said to generalize; it handled single-character flags and plain PFX/SFX and stopped on everything else, and `fr.aff` uses four of the things it stopped on. `FLAG long` is the dangerous one: French flags are two characters, so the pt-PT reader's `set(flagstr)` yields a bag of unrelated letters and expands every entry through the wrong paradigm without erroring. Plus continuation flags (French really does affix an affixed form), NEEDAFFIX on 68,075 of 84,140 stems, and FULLSTRIP. The rewritten `build_hunspell_dictionary.py` carries a per-language profile and asserts that CIRCUMFIX and FORBIDDENWORD are still unused — and **rebuilds pt-PT byte-identical to the shipped asset**, which is the only thing that makes "generalized" a claim rather than a hope. **The second decision was elision, and it was made by measuring both halves**: keeping `l'arbre` and its thirty-three siblings costs 3,159,832 forms and 8.25 MB gzipped; dropping them costs 473,326 and 1.19 MB. They are not new words, but the tokenizer keeps internal apostrophes, so they really would have been underlined — so they moved out of the dictionary and into `withElision`, which splits at a known clitic and still requires the remainder to be a word (`l'zzzz` stays flagged). Real nspell: 369 ms and 74 MB for the larger language, against pt-PT's 842 ms and 139 MB. **Where the regional trap lives is the mirror image of Portuguese's**: every `fr_*` Piper voice is fr_FR and every Debian fr dictionary is one shared word list, so nothing can be quietly wrong about the country — the whole decision is in the copy, which is why the pack is greped for *courriel* and *magasiner* the way pt-PT is greped for *arquivo*. What French does have instead is the 1990 reform, packaged three ways; Petal ships comprehensive, because Petal never corrects her French and *coût* and *cout* are both correct. go build/vet/test, tsc, vite, vitest 190/190. **Two things owed and both said plainly**: no native speaker has read the pack (SUGGESTIONS §3's bar, unmet for pt-PT too), and nothing here has been seen in a browser. **Then, same session, an interim answer to the first of those** (user: "perhaps for now, we could leverage multiple LLMs to act as reviewers?"): four models reviewed each Latin pack independently, and only findings ≥2 of them reached on their own were applied — five per pack. It earned its keep on the pack that was *already shipped*: pt-PT had **pre-Acordo spellings in a file whose own header commits to post-Acordo**, because the Phase 21 greps checked for Brazilian vocabulary and never checked the pack against its own spelling policy. That grep now exists and was confirmed to fail on the old text. Where reviewers agreed a line was wrong but split on the fix, the wording is mine and the reasoning is in the phase entry rather than averaged away. Still not a native speaker, and both packs now say so precisely.