Compare commits

..
Author SHA1 Message Date
prosolis 8f2ad34a10 Four ways the two scripts weren't the same app, and a smaller cat
A review of the pair work found the seams — every one of them a place where
the Chinese half was written and the older Latin half was left standing.

The right-click menu still asked the Latin tokenizer whether there was a word
under the pointer, so right-clicking a hanzi opened the browser's own menu
instead of the card. Hover, long-press and Ctrl+D had all moved to the shared
resolver; this one hadn't, and it is the surface the segmenter's own header
names first.

isHan is a property escape precisely so the extension blocks are covered, and
then every call site handed it one UTF-16 code unit — half a surrogate pair
for anything above the BMP, which \p{Script=Han} rightly says is not Han. The
run split in two around the character and the words either side stopped being
looked up. The walk, the scan and wordAt now step by code point, the regex is
anchored, and the test that passed by accident (unanchored, so it searched a
two-unit string rather than testing one character) is joined by one that
would have failed.

The pair picker sent the pair alone. The server validates pair and direction
as one decision and refuses a learner direction for a pair it has no word
list for — so an English speaker learning Chinese could not move to French at
all: every button failed with the generic message. It now names both, keeps
her direction where the target pack has a learner side, and returns her to
learning_en where it does not. Routed through useSession rather than the
picker's own api call, so me.direction — which decides whether the word list
stays loaded — moves with it.

UpdateMe answered every error from Get with 401. A SQLite fault on a PATCH
would have tripped the client's session interceptor and thrown a writer into
the signed-out overlay while her session was fine. Only a missing row means
not signed in, which is the distinction SetPair already made below it.

emitCommittedRef was assigned during render and called later from
compositionend; a render React discards must not leave its closure behind for
a DOM event.

And the kitten is 10% smaller — one clamp, three terms, everything else
calc()s off it.

vitest 297/297, tsc, go build/vet/test clean.
2026-07-28 20:03:45 -07:00
prosolis 5659312358 A real IME, a real candidate window, and 11.6 seconds of holding still 2026-07-28 19:39:21 -07:00
prosolis a216614c81 Phase 27 seen in a browser: held, released, and the save never held with it 2026-07-28 19:32:14 -07:00
prosolis c348a9b8ae The keystroke that isn't one: IME composition guards
Phase 26 scoped these and left them unbuilt, naming them as the likeliest
thing to be wrong the first time anyone types Chinese into Petal for real.

A composition is not a keystroke: the pinyin goes into the document as it
is typed, a candidate window sits over it, and all three decoration layers
recompute from the live document on every change — rewriting the DOM around
the node the browser is composing in, which is what eats half-typed input.

The layers now hold their redraws rather than skip them: a rebuild that falls
due mid-composition marks itself stale and its decorations are mapped through
the transaction, so they travel with the text and land correct the moment the
composition ends. The flag is read from the state before the transaction, so
the answer doesn't depend on plugin ordering; the end transaction is the one
deliberate exception, or nothing would ever release. The release is a
macrotask late because a custom handleDOMEvents handler runs before
ProseMirror's own and ProseMirror flushes the composition's last changes in a
microtask — so the held rebuild sees the committed hanzi, not the pinyin it
replaced.

Input rules needed no guard (Tiptap already returns early while composing),
which was checked rather than assumed: pinyin uses an apostrophe as a
syllable separator and Typography rewrites every ' into a curly one.

The save is deliberately not gated and the analysis is. A tablet keyboard can
hold one composition open for a whole sentence, and Petal never makes writing
wait for anything — so EditorChange carries the flag, auto-save ignores it,
and the checkpoint, rule pack and companion wait for the word to commit. One
more change is emitted the instant it does, so nothing is skipped.

Four places were taking keys that belong to the IME: the Find bar, the tag
picker, Ask Petal's chat box, and distraction-free mode's global Escape.

vitest 296/296, tsc, vite, go build/vet/test clean. Not verified with a real
IME — no browser or IME here, and that is the half the tests cannot reach.
2026-07-28 19:20:11 -07:00
prosolis 77f284f65c The zh pair's other direction, and a rule pack that mostly says no
`pair_lang` had always been answering a second question nobody asked: it
says which two languages, and every surface built on it assumed English
was the one being learned. That is why hanzi is never tokenized, never
spell-checked, never glossed — correct for a Mandarin native practising
English, backwards for an English native practising Mandarin.
`users.direction` (migration 0016) separates the two questions; a
`zh-learner` pair code would have been cheaper and would have made two
directions of one pair look like two unrelated languages to every query.

Segmentation is what replaces `wordAt` where there are no spaces: a
shortest-path walk over log-probabilities, 232 ms and 14 MB for 188,522
words. The browser gets the word list because segmentation runs on hover;
the server keeps the whole dictionary. Their coverage gates come out
opposite on purpose — the client list is frequency-gated because the
segmentation is measurably identical without the tail, and the dictionary
is gated by nothing, because its only power is to explain and the word a
learner stops on is the rare one.

The 错别字 pack is 24 confusable pairs behind two mechanical gates. One
admits a pair only if the wrong form is not a dictionary word and the
right form is, which is why it refuses 自已 for 自己 — a real error whose
wrong form is a headword. The other asks the segmenter whether the two
characters already belong to two different words, without which 自己经常,
睡觉的时候 and 不知到底 would all be corrupted silently into text still
made of real characters.

Not deployed (this carries a migration), not seen in a browser, and no
account has ever been in the learner direction. The IME composition
guards were in scope and are not done — see BUILD_PLAN Phase 26.
2026-07-28 19:04:53 -07:00
prosolis 9224c44fff The es pair, and a dictionary that was quietly Spain's
Phase 25. Spanish was never built — the groundwork was all [x] (DreamDict
data, the prompt language, the L1 rule gating, TTS env-discovery), which is
why the plan read as though it had shipped. shippedPairs was the honest
answer: the server had been refusing es on purpose.

The langpack is neutral Latin American, chosen with the user: tú, ustedes,
no vosotros, and the pan-American half of every vocabulary split. A vitest
greps for the peninsular twins the way fr is greped for québécismes —
including coger, which is not merely regional but obscene through most of
Latin America.

The dictionary is the story. Debian's hunspell-es symlinks twenty country
codes to one file, which reads as pan-Hispanic; RLA publishes twenty-four
builds per release, one per country plus a generic es that is the union,
and Debian ships peninsular es_ES. The 58,622-form gap is essentially
voseo, so the first version of this commit underlined vení and tenés as
misspellings and called it a considered gap.

The MUST_ACCEPT list was written to catch exactly that and structurally
could not: it asserted the pan-Hispanic vocabulary, and every RLA variant
carries the full pan-Hispanic vocabulary — only the paradigms are
localised. The REP table cited as the second witness is shared by all
builds too. Two independent-looking proofs, neither able to distinguish
anything, agreeing with each other.

The profile now demands what discriminates, each verified against the build
it targets: voseo rejects es_ES and Debian, vosotros rejects es_MX, and
arepa/chévere/bacán reject es_AR, which has both paradigms and would
otherwise pass. 717,640 forms, 1.74 MB gzipped, 762 ms / 97 MB in a real
nspell. fr and pt-PT rebuild byte-identical from their own upstream debs,
so the shared script still means what it meant.

Shipping the union is fr's call arrived at from the other side: coût and
cout are both correct French, tienes and tenés are both correct Spanish.
The dictionary holds every variety because underlining is all it can do;
the copy picks a register because speaking requires one.

Reviewed by four models at the usual >=2-of-4 threshold, 5 of 27 findings
applied — one catching the bedtime proverb as fr's Qui dort dîne calqued
into Spanish, gloss and all, which is the rule the fr header states. One
below-threshold finding (a missing ¡, seen by 1 of 4 because an absent
opening mark has no closing ! to look wrong against) was applied and turned
into an assertion instead: the suite now rejects any native line that
closes ? or ! without opening one.

piper-es on es_MX-ald-medium, not the es_ES-davefx-medium the plan named —
six of Piper's nine Spanish voices are peninsular, so the obvious pick was
the pt-PT trap through a different door.

go build/vet/test, tsc, vite, vitest 251/251.

Not deployed, not seen in a browser, not read by a native speaker, and no
es account exists.
2026-07-28 18:25:59 -07:00
prosolis 39d4e4770a Merge feat/accept-all-category: a whole category in one click, and one undo 2026-07-28 17:30:51 -07:00
prosolis bd92cdc9b6 A whole category accepted in one click, and one undo
Five article fixes were five clicks, five confetti bursts and five undo
steps. "Accept all Tidy-up (5)" makes them one of each.

The single undo decided the implementation: every replacement goes into one
Tiptap chain, which applies as one transaction and so undoes as one history
event. That only works if the spans can't move under each other, so the
plan resolves every span against the document as it stands and applies them
last-first.

Three outcomes rather than one, because a batch that quietly dropped a card
would be reporting edits it never made: a span she already fixed herself is
settled without an edit (what a single Accept does too), and a card quoting
the same words as one already taken is left on screen, since findRange
would resolve both to the same place.

The control sits on the first card of its kind — the rail can't carry a
category header, its cards are anchored to their own sentences — and only
when the category has company. It is outlined rather than filled: it acts
on cards she can't see from where she's standing.
2026-07-28 17:30:44 -07:00
prosolis 1acc23244e UX review: item 8's two are done, and this stack is deliberately not deployed 2026-07-28 17:07:46 -07:00
prosolis 40de65b3d1 Merge feat/settled-spans: a dismissed card stays dismissed, and the bar counts the petals 2026-07-28 07:27:55 -07:00
prosolis b2d50e9136 A dismissed card stays dismissed, even offline
The server already suppressed every span she had accepted or dismissed, on
both the LLM reconcile and the mechanics pass, with tests either side. What
had no memory was the half that never asks it: item 3b's rule pack renders
250 ms after a keystroke with no network, and its record of "she already
answered this" was a set cleared on every document switch and added to only
for cards dismissed while still provisional.

So dismissing a persisted rule-pack card recorded nothing client-side and the
next keystroke put it straight back until the server's reply removed it again;
and after a reload the client knew nothing at all — permanently so with the
server unreachable, which is the case the rule pack exists for.

GET /docs/{id}/settled hands over the normalized originals of the document's
actioned rows, scoped through documents because an original quotes her
sentence. The client seeds a SettledSpans from it on open and adds to it for
every card that leaves, keyed on the original alone the way the server keys
it. The load adds rather than assigns, so a dismissal made while it is in
flight survives it.

normalizeForDedup now exists in both languages, compared across a network
boundary, so the same nine cases are asserted on both sides and each test
names the other.

Also: the status-bar count — "🌸 5片花瓣待打磨 · 5 petals to polish" beside the
word count, from the packs, hidden at zero. An empty rail already says nothing
is waiting; a badge announcing it after every check is a verdict, which the
review's non-goals rule out.

Verified in Chrome at 1517x810 with the server killed: a new violation was
detected, underlined and counted with no network, while the dismissed span
stayed gone.
2026-07-28 07:27:49 -07:00
prosolis 1aa3a14030 Merge fix/rail-follows-mode: the rail follows the mode, and Ask Petal answers in both languages
Item 7 (the rail is a mode, not a screen size; click opens the anchored
card even with the rail up) and item 6 (bilingual Ask Petal answers with
room to read) from the 2026-07-27 UX review.
2026-07-28 07:04:23 -07:00
prosolis 978cb80642 Ask Petal answers in both languages, with room to read
The tutor prompt said "never mix languages in a single response" and
mirrored the language of the question, so asking in English — which she
does, because she is practising — returned the one explanation surface
that gives nothing in her own language. It now answers in both, pair
language first, halves separated by a blank line.

Which half is the safety net and which is the lesson depends on who is
writing: the pair is (English + X) and Petal is used from both ends, so
the prompt asks for both and says it doesn't know which way round.

The split is a rendering nicety, never a parse the reply depends on: a
half-streamed reply is all one half, a model that ignores the
instruction renders as one block, and nothing is ever dropped.

For the height, the first attempt clamped the box to the room left below
the anchored card so it could never overhang — measured, that gave 176px
against a 442px answer, worse than the 220px it replaced. The card's own
chrome spends ~290px of an 810px window, so "fits below the word" and
"room to read" are not both available. The ceiling is now a flat 50vh and
the overhang is made navigable instead, per item 4: the card reports its
reach like the rail already does, the column grows, and the page can
scroll to the actions below it.
2026-07-28 07:04:16 -07:00
prosolis f082a930cb The rail is a mode, not a screen size
Item 7 said to confirm before building, and confirming is what mattered.

The rail's 348px threshold is measured against a fixed 720px column centred
in the pane. The doc-list sidebar is 280px, so at her 1517px viewport the
right margin is 258 with it open and 406 without — either side of the
threshold. What moves between them is distraction-free mode, which engages on
its own when the editor takes focus. The rail therefore appears when she
starts writing and disappears when she stops; items 4 and 5 disagreed about
whether it exists at 1517px only because they caught it in different states.

Re-centring a fixed-width column changes its position and not its size, so
the wrapper's ResizeObserver reported nothing and no window resize fired.
railEnabled kept whatever value it last had. Leaving distraction-free with
the rail up left a 300px column in a 266px margin: overhanging the viewport
by 66px, cards clipped mid-sentence, the page scrolling sideways. Entering it
with the rail down opened 406px of margin and put nothing in it. Both
persisted until something else happened to resize the window.

Observe the scrollport too — it spans the pane, so it resizes whenever the
chrome around the editor does. That covers any future chrome that moves the
editor, which threading focusMode down as a prop would not.

Clicking a highlight now opens the anchored card even when the rail is up.
That is the item's own acceptance criterion and was previously false by
design; the measured distance from the first flagged span to its rail card is
651px, not the ~400 the review guessed. Hover still defers to the rail, since
the reasoning against an unbidden second card was about hover and still
holds — but a click is her asking to deal with that word. The rail card glows
instead of expanding, so nothing is ever open twice.

Verified in Chrome at the review's own 1517x810, driving the rule pack from
item 3b so no model was involved: the rail follows the mode in both
directions with no resize event anywhere; the popover lands 6px under the
word with the full explanation, Ask Petal, Accept and Dismiss; accepting from
it applied the edit and took the rail 6 cards to 5, leaving the rest with
their ids, positions and wording intact.

railFit.test.ts pins the threshold to the margins actually measured. The
observer wiring has no unit test and can't have a useful one: jsdom has no
layout, so every rect is zero and the rail branch is unreachable there. That
half is browser-verified only, and the doc says so.
2026-07-28 06:34:27 -07:00
prosolis ce5de68b9b UX review: item 5 is live, and what the backup nearly missed
Stopping the container does not checkpoint the WAL: petal.db on disk was four
hours stale while 2 MB of her writing sat in petal.db-wal. A lone
`cp data/petal.db` would have backed up the wrong day — and 0015 is the first
migration here that rebuilds a table, so that backup was the one that mattered.
Recorded for the next one.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-28 00:35:45 -07:00
prosolis b23c5a9a13 Merge feat/translate-card: her own language gets its own card
When she reaches for Chinese mid-sentence, the card now says 翻译 · Translate
rather than Clarity. Petal already found the span and already rendered it into
English; the type was the whole gap, and it is derived from the span rather
than asked of the model so it can't drift between passes.

Carries migration 0015 (suggestions table rebuild for the extended type CHECK)
and two findings from the running page: the inline underline needs a per-type
colour rule or it renders invisibly, and at her viewport the rail is disabled —
the inline hover card is what she sees.

Green: go vet, go test ./..., tsc --noEmit, 199 vitest tests.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-28 00:32:04 -07:00
prosolis 25e415daa2 When she writes in Chinese, say Translate — not Clarity
She reaches for her own language mid-sentence when English won't come, and
Petal already handled it: it found the span and rendered it into English. It
just filed the result as a Clarity fix, so the pair model's flagship moment
read as tidying up her Chinese.

The type is now derived from the span rather than asked of the model. A type is
structural, and a model that re-reasons every pass would drift between labels
for a sentence nobody had touched — the instability the last session spent
itself removing. The label the model volunteers is still ignored.

Only the grammar checkpoint can be promoted. A pass with a forced type owns its
family: voice reads paragraphs for tone and its rows carry no replacement, so a
"translation" there would be a card offering nothing to accept.

zh is a different script and counting Han runes is close to certain. The Latin
pairs share an alphabet with English and get none of that, so they fall back to
function words and need two before Petal claims anything — with every word that
is also English left out, even the common ones. The heuristic is justified by
how cheap being wrong is: it changes a coloured pill, and nothing else.

The pill is the one bilingual type name in the rail. Every other type stays
English because those are the terms she is learning; this card's whole subject
is her own language. And it stops truncating its two lines — elsewhere the diff
is a word and the explanation is what she reads, but here the two sentences are
the card.

Two things only the running page could report. The inline underline was
invisible: the decoration carries a per-type class and the base rule is a
transparent border, so a type with no colour rule gets no mark at all. And at
1517×810 with the document list open there is no rail — the margin is 258 where
railEnabled wants 348 — so what she gets is the inline hover card. Item 7 is
written the other way round.

Migration 0015 rebuilds the suggestions table for the CHECK, which makes it the
first one here that could quietly drop her rows; there is a test that carries
every column, both timestamps and both indexes across it.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-28 00:09:42 -07:00
prosolis 3bcc967f51 UX review: the stack is deployed, and the handoff says so
Three sessions running, the handoff opened with "nothing deployed" and
closed by recommending item 3b — which had shipped two sessions earlier.
Both are now false, and a stale recommendation is worse than none: it sends
the next session to re-scope finished work.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 23:40:20 -07:00
prosolis 963fc1754d Merge the suggestion-loop stack: instant rules, stable cards, reachable rail
Three sessions of UX_REVIEW work land together, because they are one
change to how suggestions arrive and sit:

- item 3b — the deterministic rule pack renders on its own 250 ms fuse
  instead of waiting behind the LLM's 4 s checkpoint and a round-trip.
- item 2 — passes reconcile instead of replacing, so an untouched card
  keeps its id, its arrival chime and its original explanation, and an
  unchanged document doesn't call the model at all.
- item 4 — the rail's overhang becomes real scrollable page, with the
  prose pinned bottom-anchored so the sentences the lower cards flag
  stay on screen.

Green: go test ./... , tsc --noEmit, 195 vitest tests.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 23:37:56 -07:00
prosolis de251ceae2 Make the suggestion rail's overhang reachable, and keep the text in view
The margin rail hangs off an absolutely-positioned column, so its cards add
no layout height. On her live document that meant four 173px cards anchored
inside 126px of text: a 714px stack over a page whose scrollHeight equalled
its clientHeight. The lower cards weren't far from their sentence, they were
off-screen with nothing to scroll.

The rail now reports how far its resolved stack reaches and the wrapper takes
that as a minimum height, so the space those cards occupy is real, scrollable
page. minHeight never shrinks the column, so a rail that fits beside its text
is unaffected.

Scrolling into that space would have carried every sentence off the top, so
the prose is pinned while the stack overhangs it. The offset is
min(0, port - content): prose shorter than the viewport pins at the top,
taller prose pins by its bottom edge, keeping the last lines visible — those
are the ones the overhanging cards flag.

The prose box has to stay at its natural height. Keeping the old h-full made
it measure the wrapper this change had just grown, reporting the cards'
height as the text's own, so the pin could never trip.

Verified in a browser at the review's 1517x810, driven offline by the rule
pack: 8 cards over 95px of prose gained 675px of scroll where there was none,
the last card lands fully in view with the text still on screen, tall prose
pins bottom-anchored without disturbing ordinary scrolling, and hover-linking
still glows the right span.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 23:07:09 -07:00
prosolis 10e8aef86c Stop regenerating the world on every check
A card vanishing and coming back seconds later, with different words, was
never about latency: every pass deleted its whole family and re-inserted
it, so each round minted new row ids. The rail keys on suggestion.id, so a
full remount was guaranteed — new id, new created_at (hence the re-fired
chime), and a fresh explanation from a model that re-reasons every time it
is asked. One unchanged mistake carried three different explanations in a
single sitting.

Passes now reconcile instead of replace. A re-proposed edit keeps its row:
its id, its created_at, and the wording she has already read. And the
grammar checkpoint stops asking about sentences nobody touched — the
document is split into hashed sentences, checked_chunks records which ones
a family has read, and only the difference is sent. When nothing changed
it doesn't call the model at all, and doesn't spend its rate-limit slot on
having done nothing.

The tone is part of a sentence's identity: cached advice was written for
the old register, so switching doc type re-reads every line.

replaceMechanics reconciles too, which mattered more than expected — the
rule pack fires 250 ms after a keystroke, so it was re-minting every local
card's id several times a sentence.

Only the grammar checkpoint is chunked. Voice is a property of the whole
document, and the collocation coach is a button she pressed asking for a
fresh read.

No client change was needed; stable ids were the whole of it.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 22:46:12 -07:00
prosolis c33de1175b Render deterministic rule hits instantly, not on the LLM's clock
The rule pack in prose.ts already found "a apple" — articles,
pluralAfterNumber, subjectVerbAgreement, uncountables are all there, and
they already surface as real mechanics cards. But mechanicsFindings only
ran inside runCheck, behind the same 4s checkpoint debounce as the model,
and only reached the screen via the server's reply. A free, instant,
offline-capable detection was being delivered on an LLM-shaped delay.

The rule pack now runs on its own 250ms fuse and renders its findings with
no network at all, as provisional cards. The mechanics submit follows; its
reply is authoritative and clears them. If the reply never comes — offline,
server down — the cards simply stay, which is the whole point of having
rules that need no model.

Provisional cards are keyed by wording rather than position, so one can't
flicker into a duplicate of its own persisted twin while she types around
it. resolveServerId maps a card to the row the API can act on, awaiting the
in-flight submit, so accepting inside that window still records the keep and
plants its word in the garden instead of being quietly dropped; null means
there is no row and the edit has landed regardless. Findings she actions
while provisional are remembered client-side, because the detector has no
memory between runs. runCheck no longer re-submits what the fast pass
already filed — it's the catch-up path for when that submit failed.

The arrival chime keys rule-pack cards by wording too, so a finding doesn't
chime once as provisional and again as persisted.

Not done, deliberately: no distinct style for unconfirmed local hits. The
rail renders both engines identically on purpose, and a provisional card now
lives for one LAN round-trip.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 22:28:37 -07:00
prosolis ba06d904f0 UX review: correct the item 3b handoff advice
The previous handoff sent the next session off to read
feat/mechanics-deterministic-pass and feat/calm-suggestions as unmerged
branches. Both are in main and have been for a while — that came from
misreading `git branch -vv` tracking info as merge status.

It matters because it inverts the advice. The deterministic rules engine
(prose.ts) already ships, already emits exact-span fixes as suggestion
cards under a 'mechanics' family, and already suppresses re-edits of
settled sentences. So item 3b's remaining work is most likely the
latency/ordering half — render local hits before the LLM pass — not
writing a rules engine. Point at the code and the two commits instead.

Same for item 5: internal/suggestions/translate.go already exists.

Also record that every topic branch was fully merged and has now been
deleted locally and on origin; main is the only branch left.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 22:20:54 -07:00
prosolis ea14eb5e88 UX review: handoff notes for the next session
Records what shipped and is live, what was closed without code (items 1
and 5), what's untouched, and the suggested next step — item 3b, with a
warning to read grammarLite.test.ts and the unmerged
feat/mechanics-deterministic-pass and feat/calm-suggestions branches
before writing a new rules engine.

Also writes down how to instrument a production build, since item 1
looked airtight in source and was wrong: fiber-walk from .ProseMirror to
the Tiptap editor, then read the prosemirror-history state directly.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 22:17:26 -07:00
prosolis aac15b5ac5 Merge fix/companion-yields-to-cards: the kitten gets out of the way
Fades, shrinks and goes click-through when suggestion cards or the
History/Garden drawers reach its corner, so nothing it sits on top of is
ever unreadable or unclickable.

Also lands the 2026-07-27 UX review doc, with items 1 and 5 corrected
against the live build.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 22:12:47 -07:00
prosolis be9aa13287 Kitten yields to panels too; redo bug not reproducible
The overlap hook only watched .petal-rail-card, so the History and Garden
drawers still sat under the mascot — with a real control ("写作证明 ·
Writing passport") buried under the halo on the live build.

Match [role="dialog"][aria-modal="true"] as well. Both drawers already
render it, so this covers them and any future drawer without a selector
list to keep in sync.

Two things the follow-up note didn't anticipate:

- The hook now reports { cards, modal } separately. A card overlap still
  lets the kitten wake for a bubble; a modal overlap yields
  unconditionally — a cheer isn't worth covering the panel she just
  opened on purpose.
- The speech bubble is its own layer, so fading the badge didn't hide it.
  Hold it back while a panel is open; useCompanion keeps it in state, so
  it reappears when she closes the panel.

Also guard the poll's setState on value equality, so the 500 ms tick
stops re-rendering the companion for an unchanged answer.

UX_REVIEW item 1 (redo does not re-apply an accepted suggestion) is
recorded as NOT REPRODUCIBLE. Read the prosemirror-history state directly
and hooked view.dispatch: redo works pressed immediately, after an 18 s
pause that lets a full re-check land, and with the editor never focused.
The doc's hypothesis is false — every re-check transaction is
decoration-only, which prosemirror-history ignores, and canRedo stayed
true throughout. Two real findings from that dig are written into the doc
instead: keyboard undo dies when focus isn't in the editor, and an undone
suggestion stays accepted server-side so its card doesn't reliably return.

Item 5's premise is also partly wrong and now re-scoped: the Chinese
sentence does produce a card with an English rendering, just labeled
Clarity rather than a first-class Translate type.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 22:12:30 -07:00
prosolis ec9fba9252 Kitten yields to cards: fade, shrink, click-through; plus UX review doc
When a suggestion card drifts into the mascot's corner, the kitten turns
translucent (15%), steps back 10% (standalone `scale` so it composes with
the bob animation), and lets clicks pass through to the card. It wakes
while its bubble or the picker is open, or once the corner clears.

Also adds UX_REVIEW_2026-07-27.md — the hands-on review of the live deploy
turned into implementation-ready items (repro, location, fix, acceptance).

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 21:57:20 -07:00
prosolis 9c40a8ad3f Merge fix/mobile-chrome-overflow: keep the sideways scrolling in the pills
On a phone, switching off the Mandarin pair set the whole page of writing
scrolling left and right. The chrome pills say everything twice and refuse to
shrink, so the title row's width is a property of the langpack, and in French it
wanted 551px of a 338px column. It scrolls itself now, with a faded edge
pointing at whatever is off-screen, and the bilingual labels survive at every
width.

Also: the header no longer overflows the viewport below 360px, and the language
picker no longer spills out of the drawer.

Frontend only — nothing to deploy beyond the usual rebuild.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 20:13:54 -07:00
prosolis ac1c6cddb0 French dragged her whole page sideways
The bug reads as a language-picker bug — switch off Mandarin on a phone and
the writing starts scrolling left and right — but the picker only changes
how wide Petal's own chrome wants to be. Every pill says its thing twice,
历史 · History against Historique · History, and every pill is nowrap and
shrink-0. So the title row's width is a property of the langpack: 459px in
Chinese, 551px in French, 506px in Portuguese, against the 338px column a
390px phone gives it. That row lives in the editor pane, and a pane that is
overflow-y: auto has an overflow-x of auto too, whatever the stylesheet
says. The overflow had nowhere to go but the page of writing.

Chinese was already 120px over. French is simply where it stopped being
possible to ignore.

The pills now live in a strip that scrolls itself, with overscroll-behavior
so a swipe off the end doesn't turn into a page gesture. What that buys is
the thing worth keeping: every label stays bilingual at every width. The
first version of this fix dropped the English half on phones, which fixed
the geometry by taking away the half she is learning from — on the device
she writes on most.

A scrolled pill that has left the screen is indistinguishable from a pill
that isn't there, so each edge with more behind it fades, the same hint
.petal-toolbar gives with its clipped right edge, except this row can be
scrolled from either end and has to point the right way. ChromeStrip sets
data-edge from the scroll position and re-measures when the pills resize —
which is also what catches every label changing width at once when she
switches her pair.

The tone and export menus had to leave with them. A scroll container clips
its absolutely-positioned children, so both menus would have been trapped in
a 36px-tall box; they anchor against the viewport now.

Two smaller ones the same measurement turned up:

  - The header overflowed the viewport itself below 360px — a real
    page-level scroll, 40px of it at 320px, off 🌷 Jardim de palavras ·
    Garden. Narrower padding on phones, and under 360px the wordmark
    yields. Of everything in that row it is the one thing that can go: she
    is already inside the app, and the blossom stays.
  - The language picker wanted 307px inside a 280px drawer and spilled out
    of it. It wraps now. That one was broken in Chinese too, at 291px.

Verified in a real browser rather than by arithmetic: 310px to 780px in 10px
steps, in all three packs, no page overflow and no editor-pane overflow at
any width. Edge fades flip correctly, the export menu opens unclipped with
the strip scrolled to its end, tsc clean, 195 frontend tests pass. Desktop
is untouched — one 40px header row, as before.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 20:10:17 -07:00
prosolis 3e714b6f00 Merge fix/security-review: refuse to start unauthenticated, and mean it
Petal now fails closed rather than falling back to the `local` user on a
public host, serves stored images inert, and keeps its response headers
where a route can tighten them instead of where the edge can overwrite
them. Plus the smaller findings: allowlist warning, upstream errors kept
out of responses, export link schemes, per-account draft rescue, POST-only
logout, storage quotas, __Host- session cookie, npm audit.

Deploying this needs one check first: if the live .env is missing any of
AUTHENTIK_URL / AUTHENTIK_CLIENT_ID / AUTHENTIK_CLIENT_SECRET, the
container will refuse to start — which is the guard working, but better
found before the deploy than during it.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 18:25:07 -07:00
prosolis 69bf3ffde1 Close the door the edge gate used to hold
A security review of the whole repo. The queries were already scoped, the
OIDC flow already did state and nonce and PKCE, the session tokens were
already stored as hashes. What it found was mostly the seam between the
code and the deployment — and one place where the deployment quietly
undid the code.

The one that matters: with any AUTHENTIK_* variable missing, Petal fell
back to resolving every request to the single `local` user. That is right
on a laptop and a catastrophe on a public host, and Phase 16 removed the
Traefik basic-auth gate that used to stand behind the mistake. A typo in
the client secret would have served her journals to the open internet and
said so only in a log line nobody reads. It now refuses to start, guarded
by default for any BASE_URL that isn't loopback.

Then the one that would have been fixed and wasn't: stored images now
serve under `default-src 'none'; sandbox`, so an SVG pasted into a
document can't run as a page on Petal's own origin. Traefik's
customresponseheaders *overwrites*, so the CSP declared in the compose
labels would have silently replaced that per-route policy in production.
The whole header block moved into the binary, where a route can tighten
its own and a test can prove it; only HSTS stays at the edge, where TLS
actually terminates.

The rest, smaller:

  - PETAL_ALLOWED_SUBS empty means everyone authentik authenticates, and
    authentik here fronts half a dozen applications. Still legal, now
    said out loud every boot, and set in both env examples.
  - LLM failures relayed err.Error() to the browser, which carries the
    address of the inference box on the far side of the VPN. Logged
    instead; the client only ever rendered "the helper is resting".
  - Exports scheme-check their links. Escaping makes a URL safe to sit
    in an attribute and says nothing about following it, and an export
    is the one artifact here meant to leave. Writing the test found the
    markdown image src, which I'd missed reading it.
  - The draft rescue is namespaced per account and cleared on sign-out.
    Everything else in localStorage is a preference; this is her unsaved
    writing, sitting in a profile two people share.
  - /auth/logout is POST-only. With SameSite=Lax a GET route lets any
    page on the internet sign her out mid-draft.
  - Image uploads get a per-account allowance and the TTS cache a size
    cap. Both share the encrypted volume the database is on, and a full
    disk is SQLite failing to write, not a feature degrading.
  - The session cookie takes the __Host- prefix over https, so nothing
    else under parodia.dev can plant one. Old cookies still resolve;
    nobody is signed out to get there.
  - npm audit: linkify-it and postcss.

Verified: go build, go vet, the full Go suite, tsc, 195 frontend tests,
npm audit clean. The startup guard and both CSPs checked against a
running server rather than only asserted.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 18:24:47 -07:00
prosolis 9a0edd6679 Merge feat/deploy-phase15: containerized VPS deploy, auth, multi-language pairs
Brings Phases 15-23 onto main: Docker/Traefik deploy at petal.parodia.dev,
OIDC sign-in replacing the basic-auth gate, per-user settings, a dictionary
that is no longer English+Chinese only, English+Portuguese and English+French
pairs with their Piper voices, and encrypted nightly backups on millenia.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 17:16:19 -07:00
prosolis be1ab5cef7 Her apostrophe was cutting French words in half
Typography.ts rewrites every ' typed in the editor into a curly ’, but both
word regexes only counted the straight one. So "aujourd’hui" reached the
dictionary as "aujourd" + "hui", neither of them a French word, and one of the
commonest words in the language came back wearing two red underlines. Same for
quelqu’un, presqu’île, prud’homme. l’arbre only survived by accident, because
"l" happens to be a bare entry. withElision, written for exactly this, could
only ever fire on pasted text.

Both marks are word characters now, and combine() straightens on lookup — the
one place every lookup passes through — since the shipped word lists spell
theirs straight. Suggestions come back wearing whichever mark she actually
used, so accepting a pill never swaps her apostrophe.

œ was untokenizable too: U+0152/U+0153 sit outside the Latin-1 ranges, so
"cœur" split into "c" + "ur" and the orphan was long enough to underline. 586 œ
forms ship in fr.dic.gz and not one of them was reachable.

In the dictionary builder, the two cross-product paths added their forms
without the NEEDAFFIX check the single-affix paths apply, so a doubly-affixed
form that is still "not a word on its own" was accepted anyway — the exact
class of error the FLAG-aware rewrite exists to close. PFX and SFX are also
separate flag namespaces, and one shared `cross` dict let the second block
overwrite the first. Odd-length long-flag strings now stop the build instead of
dropping a character and expanding through the wrong paradigm.

The pt-PT and Québécois greps were case-sensitive against sentence-cased copy,
which let a leading "Actualmente…" through the guard added to catch it.

Note: this changes what the expander produces, but fr.dic.gz and pt-PT.dic.gz
are vendored and were built with the old behaviour. Both want regenerating on a
box that can fetch the upstream .deb, and BUILD_PLAN Phase 24's "pt-PT rebuild
is byte-identical" claim re-checked — if those bytes move, the NEEDAFFIX gap
was live in the Portuguese list too.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 17:07:30 -07:00
prosolis 071ea7b835 Petal learns French, and the pack that shipped was misspelling itself
Phase 24, the fr half: langpack, Hunspell dictionary, Piper voice, and the
lexicon coverage that turned out to have been measured already (63.1%, better
than pt-PT's 62.1%). No migration; not deployed.

The plan recorded that build_ptpt_dictionary.py "generalizes" to French. It
did not. It handled single-character flags and plain PFX/SFX and stopped on
everything else, and fr.aff uses four of the things it stopped on. FLAG long
is the dangerous one: French flags are two characters, so the old reader's
set(flagstr) yields a bag of unrelated letters and expands every entry through
the wrong paradigm without ever erroring. Plus continuation flags (French
really does affix an affixed form), NEEDAFFIX on 68,075 of 84,140 stems, and
FULLSTRIP. Renamed build_hunspell_dictionary.py with a per-language profile,
asserting that CIRCUMFIX and FORBIDDENWORD are still unused rather than
assuming it — and it rebuilds pt-PT byte-identical to the shipped asset, which
is the only thing that makes "generalized" a claim rather than a hope.

Elision was decided by building both halves and measuring. Keeping l'arbre and
its thirty-three siblings: 3,159,832 forms, 8.25 MB gzipped. Dropping them:
473,326 and 1.19 MB. They are not new words, but the tokenizer keeps internal
apostrophes, so they genuinely would have been underlined — so they moved out
of the dictionary into withElision, which splits at a known clitic and still
requires the remainder to be a word (l'zzzz stays flagged). Real nspell: 369 ms
and 74 MB, against pt-PT's 842 ms and 139 MB, on the larger language.

Where the regional trap lives is the mirror image of Portuguese's: every fr_*
Piper voice is fr_FR and Debian's fr_FR/fr_CA/fr_BE dictionaries are one shared
word list, so nothing can be quietly wrong about the country and the whole
decision sits in the copy. What French has instead is the 1990 reform, packaged
three ways; comprehensive ships, because Petal never corrects her French and
coût and cout are both correct.

Then the interim review pass, at the user's suggestion and explicitly "for
now": four models read each Latin pack independently, and only findings at
least two of them reached on their own were applied — five per pack. It earned
its keep on the pack that was already live. pt-PT was carrying pre-Acordo
spellings (adjectivos, actualmente) in a file whose own header commits to
post-Acordo, plus Brazilian decepção, because the Phase 21 greps checked for
Brazilian vocabulary and never checked the pack against its own spelling
policy. That grep now exists and was confirmed to fail on the old text before
being kept. Where reviewers agreed a line was wrong but split on the fix, the
wording is mine and the reasoning is in BUILD_PLAN rather than averaged away.

Still owed, and both packs now say so precisely: a quorum of models agreeing is
agreement, not authority. No native speaker has read either pack, and none of
this has been seen in a browser.

go build/vet/test clean, tsc, vite build, vitest 190/190.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 16:19:26 -07:00
121 changed files with 11055 additions and 713 deletions
+14 -1
View File
@@ -44,6 +44,12 @@ TTS_AUDIO_FORMAT=mp3 # mp3 | opus | wav — mp3/opus transcode Pipe
# them commented out for local development and Petal runs as the single
# hardcoded `local` user, exactly as it did before auth landed.
#
# That fallback is scoped to development on purpose. With a BASE_URL naming
# anything but localhost, Petal refuses to start rather than run open — see
# PETAL_REQUIRE_AUTH — because the fallback on a reachable host means every
# anonymous visitor is the `local` user, with full read and write over every
# document in the database.
#
# AUTHENTIK_URL is the issuer of the Petal provider in Authentik (the value of
# its "OpenID Configuration Issuer" field). The redirect URI to register there
# is BASE_URL + /auth/callback.
@@ -52,8 +58,15 @@ TTS_AUDIO_FORMAT=mp3 # mp3 | opus | wav — mp3/opus transcode Pipe
# AUTHENTIK_CLIENT_SECRET=
#
# Who may sign in: comma-separated OIDC subject ids and/or email addresses.
# Empty = anyone Authentik authenticates.
# Empty = anyone Authentik authenticates, which is right for a single-household
# instance and wrong the moment the IdP serves a wider audience than Petal. An
# empty list is warned about at every boot rather than assumed either way.
# PETAL_ALLOWED_SUBS=her@example.com,me@example.com
#
# Whether a missing OIDC configuration is fatal. Defaults to false for a
# loopback BASE_URL and true for anything else, so neither a laptop nor a
# deployment normally has to name it.
# PETAL_REQUIRE_AUTH=true
# --- Deferred (not wired in the local-dev build) ---
+4
View File
@@ -10,6 +10,10 @@ web/dist/*
!web/dist/.gitkeep
*.log
# Python
__pycache__/
*.pyc
# Local env & data
.env
*.db
+121 -9
View File
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large Load Diff
+72
View File
@@ -0,0 +1,72 @@
package main
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
)
// The baseline policy has to actually permit the frontend Vite builds. These
// are the allowances dist/index.html needs; if a future build starts emitting
// an inline script or pulling from a new host, that shows up here rather than
// as a blank page in production.
func TestBaselineCSPCoversTheBuiltFrontend(t *testing.T) {
required := []string{
"script-src 'self'", // Vite emits no inline script
"'unsafe-inline' https://fonts.googleapis.com", // React style={{…}} + the font link
"https://fonts.gstatic.com", // the font files themselves
"blob:", // read-aloud plays an object URL
"object-src 'none'",
"base-uri 'self'",
"frame-ancestors 'self'",
}
for _, want := range required {
if !strings.Contains(contentSecurityPolicy, want) {
t.Errorf("baseline CSP is missing %q:\n%s", want, contentSecurityPolicy)
}
}
}
// The middleware is a floor, not a ceiling: it runs *before* the handler
// precisely so a route serving untrusted bytes can overwrite the policy with a
// stricter one. This is the contract the image store depends on, and the reason
// the policy no longer lives in the Traefik labels — customresponseheaders
// would overwrite it in the other direction.
func TestRouteMayTightenTheBaselineCSP(t *testing.T) {
strict := "default-src 'none'; sandbox"
handler := securityHeaders(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Security-Policy", strict)
w.WriteHeader(http.StatusOK)
}))
rec := httptest.NewRecorder()
handler.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/images/x.svg", nil))
if got := rec.Header().Get("Content-Security-Policy"); got != strict {
t.Fatalf("handler's policy was not honoured: got %q, want %q", got, strict)
}
// The headers it didn't touch still stand.
if rec.Header().Get("X-Content-Type-Options") != "nosniff" {
t.Error("baseline nosniff was lost")
}
}
// Every ordinary response carries the baseline.
func TestBaselineHeadersOnAPlainResponse(t *testing.T) {
handler := securityHeaders(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
}))
rec := httptest.NewRecorder()
handler.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/", nil))
for header, want := range map[string]string{
"Content-Security-Policy": contentSecurityPolicy,
"X-Content-Type-Options": "nosniff",
"Referrer-Policy": "same-origin",
} {
if got := rec.Header().Get(header); got != want {
t.Errorf("%s = %q, want %q", header, got, want)
}
}
}
+70 -1
View File
@@ -64,21 +64,45 @@ func main() {
sessions := auth.NewSessionStore(database.DB)
users := auth.NewUserStore(database.DB)
// …and the fallback is exactly what must not happen quietly on a public
// host. Refuse to start rather than serve someone's journals to the open
// internet because one environment variable was misspelled. See
// config.RequireAuth for why this defaults on for any non-loopback BASE_URL.
if !cfg.AuthEnabled() && cfg.RequireAuth {
log.Fatalf("auth: refusing to start unauthenticated at %s.\n"+
" Petal would resolve every anonymous request to the single %q user, with full\n"+
" read and write over every document in the database.\n"+
" Set AUTHENTIK_URL, AUTHENTIK_CLIENT_ID and AUTHENTIK_CLIENT_SECRET, or set\n"+
" PETAL_REQUIRE_AUTH=false if this really is a trusted private network.",
cfg.BaseURL, db.LocalUserID)
}
var resolver auth.Resolver = auth.StaticResolver(db.LocalUserID)
var oidcClient *auth.OIDC
if cfg.AuthEnabled() {
allowed := auth.ParseAllowlist(cfg.AllowedSubs)
oidcClient = auth.NewOIDC(context.Background(), auth.Options{
IssuerURL: cfg.AuthentikURL,
ClientID: cfg.AuthentikClientID,
ClientSecret: cfg.AuthentikClientSecret,
BaseURL: cfg.BaseURL,
Allowed: auth.ParseAllowlist(cfg.AllowedSubs),
Allowed: allowed,
}, sessions, users)
resolver = sessions
if n, err := sessions.Prune(); err == nil && n > 0 {
log.Printf("auth: pruned %d expired session(s)", n)
}
log.Printf("auth: OIDC enabled (issuer=%s, redirect=%s)", cfg.AuthentikURL, oidcClient.RedirectURI())
// An empty allowlist is a legitimate choice for a single-household
// instance and a wide-open door in front of an IdP that fronts anything
// else. Petal cannot tell which it is, so it says so every boot rather
// than assuming.
if len(allowed) == 0 {
log.Printf("auth: WARNING — PETAL_ALLOWED_SUBS is empty, so EVERY account %s "+
"authenticates may sign in and start writing here. Set it to the "+
"comma-separated emails (or subject ids) that belong in this Petal.",
cfg.AuthentikURL)
}
} else {
log.Printf("auth: OIDC not configured — running as the single %q user", db.LocalUserID)
}
@@ -106,6 +130,7 @@ func main() {
r.Use(middleware.RealIP)
r.Use(middleware.Logger)
r.Use(middleware.Recoverer)
r.Use(securityHeaders)
// Build version: a hash of the embedded SPA shell. Vite rewrites index.html
// with content-hashed asset names on every build, so this string changes
@@ -182,6 +207,9 @@ func main() {
lex := lexicon.NewHandler(database.DB, lexSet)
pr.Mount("/word", lex.Routes())
pr.Mount("/gloss", lex.GlossRoutes())
// The same lookup pointing the other way: a Chinese word to its pinyin
// and English senses, for an account whose direction is learning_pair.
pr.Mount("/hanzi", lex.HanziRoutes())
// Vocabulary garden: words the writer looks up are captured here and
// surfaced for gentle spaced-repetition review.
@@ -231,6 +259,47 @@ func main() {
}
}
// contentSecurityPolicy is the default policy for everything Petal serves.
//
// It lives here rather than in the Traefik labels, and that move is the point:
// Traefik's customResponseHeaders *sets* a header, overwriting whatever the
// application chose, so a policy declared at the edge silently replaces the
// stricter one an individual route needs. Stored images need exactly that (an
// uploaded SVG is a document that can carry script — see internal/images), and
// a rule the edge can quietly undo is not a rule.
//
// The allowances are what the built frontend actually uses, no more: script
// only from Petal itself (Vite emits no inline script — this policy is checked
// against dist/index.html), inline *styles* because React's style={{…}} props
// compile to style attributes, and Google's font hosts because index.html links
// them. object-src and base-uri close the two attribute-injection routes that
// survive HTML escaping.
const contentSecurityPolicy = "default-src 'self'; " +
"script-src 'self'; " +
"style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; " +
"font-src 'self' data: https://fonts.gstatic.com; " +
"img-src 'self' data: blob:; " +
"media-src 'self' data: blob:; " +
"connect-src 'self'; " +
"object-src 'none'; " +
"base-uri 'self'; " +
"form-action 'self'; " +
"frame-ancestors 'self'"
// securityHeaders lays down the baseline response headers before the handler
// runs, so a route that needs something stricter — the image store — simply
// overwrites its own copy on the way past. Ordering is the mechanism: this is a
// floor, not a ceiling.
func securityHeaders(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
h := w.Header()
h.Set("Content-Security-Policy", contentSecurityPolicy)
h.Set("X-Content-Type-Options", "nosniff")
h.Set("Referrer-Policy", "same-origin")
next.ServeHTTP(w, r)
})
}
// maxAPIBodyBytes caps a JSON API request body at 2 MiB. That's far above any
// real document save (the body is text plus lightweight marks; images upload
// separately by reference) while still bounding abuse. Exceeding it makes the
+53 -9
View File
@@ -23,11 +23,13 @@ runs them.
- **petal** — the single Go binary with the frontend embedded. Publishes no host
port; Traefik is the only way in.
- **piper-en** / **piper-zh** — read-aloud. Each Piper HTTP server loads exactly
one voice, so English and Chinese are separate containers off one image, with
the models cached in a shared volume. They sit on an internal network with no
published ports, so only Petal can reach them. Adding pt-PT in Phase 21 is a
fourth service, not a new image.
- **piper-en** / **piper-zh** / **piper-pt** / **piper-fr** — read-aloud. Each
Piper HTTP server loads exactly one voice, so every language is its own
container off one image, with the models cached in a shared volume. They sit
on an internal network with no published ports, so only Petal can reach them.
Adding pt-PT in Phase 21 was a third service and fr in Phase 24 a fourth —
never a new image, and since Phase 21 never any Go either (the languages are
discovered from `TTS_ENDPOINT_<LANG>`/`TTS_VOICE_<LANG>`).
They run as containers rather than the host systemd units millenia uses because
Piper was never actually installed on the VPS, and the `reala` account has no
@@ -231,8 +233,21 @@ unauthenticated `/api/health` router that existed only to escape it: every `/api
route now answers 401 without a session, and the only thing an anonymous visitor
gets is the app shell and a redirect to sign in.
If you ever run this stack *without* `AUTHENTIK_*` configured — Petal then falls
back to the single `local` user — put the gate back before pointing DNS at it:
Removing that gate meant a missing `AUTHENTIK_*` variable stopped being a
nuisance and became an exposure: Petal would fall back to the single `local`
user and hand every anonymous visitor full read and write over the database,
saying so only in a log line. So **it now refuses to start instead**:
```
auth: refusing to start unauthenticated at https://petal.parodia.dev.
Petal would resolve every anonymous request to the single "local" user, ...
```
The guard defaults on for any `BASE_URL` that isn't loopback, so a laptop
checkout still runs open and a deployment cannot. If you genuinely want an
unauthenticated instance on a trusted private network, say so out loud with
`PETAL_REQUIRE_AUTH=false` — and if it is reachable from anywhere else, put a
gate back in front of it first:
```yaml
traefik.http.routers.petal.middlewares: compression@file,petal-headers,petal-auth
@@ -241,6 +256,28 @@ traefik.http.middlewares.petal-auth.basicauth.users: ${PETAL_BASIC_AUTH:?}
with `htpasswd -nbB petal 'your-password'` in `.env` as `PETAL_BASIC_AUTH`.
### Who gets in
`PETAL_ALLOWED_SUBS` is a comma-separated list of emails and/or OIDC subject
ids. **Set it.** Empty means everyone authentik authenticates, and authentik on
this host fronts several applications — a valid account there is not the same as
belonging in someone's private journal. An empty list is legal (a single-
household instance may want it) and warns at every boot:
```
auth: WARNING — PETAL_ALLOWED_SUBS is empty, so EVERY account ... may sign in
```
### Response headers
`Content-Security-Policy`, `X-Content-Type-Options` and `Referrer-Policy` are set
by the binary, not by the Traefik labels. Traefik's `customresponseheaders`
*overwrites*, which would silently replace the stricter policy an individual
route picks for itself — the image store serves stored uploads under
`default-src 'none'; sandbox` so that an SVG someone pasted into a document
cannot run as a page on Petal's own origin. Only HSTS stays at the edge, where
TLS is actually terminated.
---
## 4a. Moving an account (`scripts/migrate_local_user.py`)
@@ -332,8 +369,8 @@ commonest English words; ECDICT covers essentially all of them and is in daily
use by a real writer. `lexicon.Set.For` is where that decision lives — one
`switch`, changed the day a comparison on her actual lookups says otherwise.
For pt-PT and French the same measurement reads 62%, which is why they use
DreamDict: there is no alternative source for them at all.
For pt-PT and French the same measurement reads 62% and 63%, which is why they
use DreamDict: there is no alternative source for them at all.
### Rebuilding it
@@ -604,6 +641,13 @@ d=json.load(urllib.request.urlopen('https://huggingface.co/rhasspy/piper-voices/
print([k for k in d if k.startswith('pt')])"
```
**French: the opposite situation, and worth knowing it is.** Every `fr_*` voice
in the catalogue is `fr_FR`, so there is no wrong country to land on by default
and no Québec voice to choose instead; `fr_FR-siwis-medium` is picked to match
the register of the other three rather than to avoid anything. The name is also
plain ASCII, so the entrypoint's percent-encoded download fallback — which
exists only because `tugão` broke `piper.download_voices` — never fires here.
**Slow replay.** `POST /api/tts` takes `slow: true`, which raises Piper's
`length_scale` to about 4/3 (≈0.75× pace). It is a separate cache entry, not a
playback-rate trick, so the slow clip is synthesized once and then instant.
+41 -18
View File
@@ -20,13 +20,9 @@ TZ=Europe/Lisbon
PETAL_UID=1001
PETAL_GID=1001
# --- Interim edge gate (delete when Phase 16 auth lands) ---------------------
# Petal has no authentication of its own yet — StaticResolver hands every
# request the same local user — so Traefik holds the door with basic auth until
# the OIDC flow exists. user:bcrypt-hash, as produced by:
# htpasswd -nbB petal 'your-password'
# /api/health is deliberately exempt (its own router) so monitoring still works.
PETAL_BASIC_AUTH=
# (The interim PETAL_BASIC_AUTH edge gate is gone: Petal authenticates for
# itself now, and the auth block at the bottom of this file is what holds the
# door. A second password in front of a real login is one more thing to lose.)
# --- LLM (millenia, over headscale) ------------------------------------------
# The only cross-VPN dependency. Petal degrades warmly when it's unreachable:
@@ -53,27 +49,54 @@ LLM_TIMEOUT=90s
# A language is routable only when both halves are set — a TTS_ENDPOINT_XX with
# no TTS_VOICE_XX reads as "no voice for this language" and the browser's own
# synthesizer takes over, rather than as an instance that errors on every
# request. Adding fr or es is a compose service plus a pair of lines here.
# request. Adding es is a compose service plus a pair of lines here.
#
# pt_PT-tugão-medium is the only European Portuguese voice Piper ships; every
# other pt model in the catalogue is Brazilian.
# other pt model in the catalogue is Brazilian. French has the opposite
# property — every fr voice in the catalogue is fr_FR — so there is no wrong
# country to land on and no non-ASCII name to trip the downloader.
TTS_VOICE_EN=en_US-amy-medium
TTS_VOICE_ZH=zh_CN-huayan-medium
TTS_VOICE_PT=pt_PT-tugão-medium
TTS_VOICE_FR=fr_FR-siwis-medium
# Mexican, not peninsular — the es pack is written in neutral Latin American
# Spanish, and es_ES-davefx-medium would read it in the accent it avoids.
TTS_VOICE_ES=es_MX-ald-medium
TTS_AUDIO_FORMAT=mp3
TTS_TIMEOUT=15s
# --- Auth (Authentik OIDC) ---------------------------------------------------
# Authentik already runs on this host. Set all three and Petal authenticates
# for itself; leave any unset and it falls back to the single `local` user
# (which on a public host means the Traefik basic-auth gate must stay).
# NOT OPTIONAL HERE. Authentik already runs on this host; set all three and
# Petal authenticates for itself.
#
# Leave any of them unset and Petal REFUSES TO START, because the alternative is
# worse: it would otherwise fall back to resolving every anonymous request to
# the single `local` user, handing the open internet full read and write over
# every document in the database. That fallback is right on a laptop and a
# catastrophe on this host, so the guard is on for any non-loopback BASE_URL.
# See PETAL_REQUIRE_AUTH below.
#
# AUTHENTIK_URL is the provider's issuer, and the redirect URI to register in
# Authentik is https://petal.parodia.dev/auth/callback.
# AUTHENTIK_URL=https://auth.parodia.dev/application/o/petal/
# AUTHENTIK_CLIENT_ID=petal
# AUTHENTIK_CLIENT_SECRET=
AUTHENTIK_URL=https://auth.parodia.dev/application/o/petal/
AUTHENTIK_CLIENT_ID=petal
AUTHENTIK_CLIENT_SECRET=
# Who may sign in: comma-separated subject ids and/or emails.
#
# Who may sign in: comma-separated subject ids and/or emails. Empty = anyone
# Authentik authenticates, which is wider than this instance wants.
# PETAL_ALLOWED_SUBS=
# SET THIS. Empty means everyone Authentik authenticates, and Authentik on this
# host fronts half a dozen applications — being a valid user there is not the
# same as belonging in someone's private journal. An empty value is legal (a
# single-household instance may genuinely want it) and says so loudly in the
# startup log every boot.
#
# An email is knowable in advance; a subject id is an opaque uuid nobody can
# know before that person's first login. Use emails to invite, subject ids to
# pin.
PETAL_ALLOWED_SUBS=her@example.com,me@example.com
# The guard itself. Defaulted from BASE_URL — loopback origins run open, real
# ones demand a login — so it does not normally need setting. Set it to false
# only for a deployment genuinely reachable from nowhere but a trusted network,
# and understand that it means anyone who reaches Petal is the `local` user.
# PETAL_REQUIRE_AUTH=true
+54 -4
View File
@@ -51,6 +51,8 @@ services:
# change. <LANG> is the base tag — an env var name can't hold pt-PT's
# hyphen, and there is one Portuguese voice loaded either way.
TTS_ENDPOINT_PT: http://piper-pt:5000
TTS_ENDPOINT_FR: http://piper-fr:5000
TTS_ENDPOINT_ES: http://piper-es:5000
# The sidecars run piper-tts 1.6.0, which serves synthesis on
# /synthesize; millenia's older server keeps the default "/".
TTS_PATH: /synthesize
@@ -97,11 +99,18 @@ services:
# second password in front of a real login is just one more thing to lose.
traefik.http.routers.petal.middlewares: compression@file,petal-headers
traefik.http.services.petal.loadbalancer.server.port: "8080"
# Petal is a private writing space: no framing, no sniffing, HSTS on.
traefik.http.middlewares.petal-headers.headers.customresponseheaders.Content-Security-Policy: frame-ancestors 'self'
# HSTS is the edge's business — it is a statement about the TLS
# termination, which happens here and not in the container.
#
# The Content-Security-Policy that used to sit alongside it has moved into
# the app (see securityHeaders in cmd/server/main.go). customresponseheaders
# *overwrites*, so a policy set here would silently replace the stricter
# one an individual route chooses for itself — which is exactly what the
# image store does to keep an uploaded SVG from running as a page. A rule
# the edge can quietly undo is not a rule. X-Content-Type-Options and
# Referrer-Policy moved with it for the same reason: one place to read,
# and no dependence on this file being deployed alongside the binary.
traefik.http.middlewares.petal-headers.headers.customresponseheaders.Strict-Transport-Security: max-age=31536000; includeSubDomains
traefik.http.middlewares.petal-headers.headers.customresponseheaders.X-Content-Type-Options: nosniff
traefik.http.middlewares.petal-headers.headers.customresponseheaders.Referrer-Policy: same-origin
piper-en:
build:
@@ -147,6 +156,47 @@ services:
networks:
- internal
# French, for the fr pair. The opposite situation to Portuguese: every French
# voice Piper ships is fr_FR, so there is no wrong country to land on by
# default, and the name is plain ASCII so the entrypoint's percent-encoded
# fallback (added for tugão) never has to fire. siwis-medium to match the
# register of the other three.
piper-fr:
build:
context: deploy/piper
image: petal-piper:local
container_name: petal-piper-fr
restart: unless-stopped
environment:
PIPER_VOICE: ${TTS_VOICE_FR:-fr_FR-siwis-medium}
volumes:
- piper-voices:/voices
networks:
- internal
# Spanish, for the es pair — and the Portuguese trap rather than the French
# one. Piper's catalogue has nine Spanish voices, six of them es_ES, and the
# obvious pick (es_ES-davefx-medium, which the build plan itself named) is
# peninsular. The es pack is written in neutral Latin American Spanish, so a
# Castilian voice would read it aloud in the accent the copy was written to
# avoid — the same wrong-country default that pt-PT hit through packaging,
# arriving here through the voice list. Only two Latin American voices exist,
# es_AR-daniela-high and es_MX; Mexican is the neutral broadcast standard and
# ald-medium matches the register of the other four. ASCII, so the
# percent-encoded download fallback added for tugão never has to fire.
piper-es:
build:
context: deploy/piper
image: petal-piper:local
container_name: petal-piper-es
restart: unless-stopped
environment:
PIPER_VOICE: ${TTS_VOICE_ES:-es_MX-ald-medium}
volumes:
- piper-voices:/voices
networks:
- internal
networks:
# Created and owned by the host's Traefik stack.
traefik:
+6 -3
View File
@@ -129,7 +129,10 @@ func (o *OIDC) Routes() chi.Router {
r := chi.NewRouter()
r.Get("/login", o.login)
r.Get("/callback", o.callback)
r.Get("/logout", o.logout)
// POST only. Signing out is a state change, and SameSite=Lax deliberately
// *does* send the session cookie on a top-level cross-site GET — so a GET
// route here means any page on the internet can sign her out mid-draft by
// linking to it, or embedding it as an image. Small harm, free to remove.
r.Post("/logout", o.logout)
return r
}
@@ -275,8 +278,8 @@ func (o *OIDC) callback(w http.ResponseWriter, r *http.Request) {
// matters: clearing only the cookie leaves a token that still works if it was
// ever captured.
func (o *OIDC) logout(w http.ResponseWriter, r *http.Request) {
if c, err := r.Cookie(SessionCookie); err == nil && c.Value != "" {
if err := o.sessions.Revoke(c.Value); err != nil {
if token := SessionToken(r); token != "" {
if err := o.sessions.Revoke(token); err != nil {
log.Printf("auth: revoke failed: %v", err)
}
}
+14 -1
View File
@@ -234,7 +234,7 @@ func TestLoginRoundTrip(t *testing.T) {
// Signing out revokes server-side, not just in the browser.
out := httptest.NewRecorder()
flow.ServeHTTP(out, jar.attach(httptest.NewRequest(http.MethodGet, "/logout", nil)))
flow.ServeHTTP(out, jar.attach(httptest.NewRequest(http.MethodPost, "/logout", nil)))
if out.Code != http.StatusFound {
t.Fatalf("logout status=%d", out.Code)
}
@@ -243,6 +243,19 @@ func TestLoginRoundTrip(t *testing.T) {
}
}
// Signing out is a state change, so it must not be reachable by GET: with
// SameSite=Lax the session cookie *is* sent on a top-level cross-site
// navigation, which would let any page on the internet sign her out mid-draft.
func TestLogoutRejectsGET(t *testing.T) {
_, flow, _, _ := newFlow(t, nil)
out := httptest.NewRecorder()
flow.ServeHTTP(out, httptest.NewRequest(http.MethodGet, "/logout", nil))
if out.Code != http.StatusMethodNotAllowed {
t.Fatalf("GET /logout status=%d, want 405", out.Code)
}
}
// A callback whose state doesn't match the cookie is a forged one.
func TestCallbackRejectsBadState(t *testing.T) {
idp, flow, sessions, _ := newFlow(t, nil)
+143 -7
View File
@@ -24,16 +24,33 @@ func patchMe(t *testing.T, users *UserStore, id, body string) *httptest.Response
func TestSetPairLang(t *testing.T) {
_, users, _ := newStores(t)
if err := users.SetPairLang("bob", "pt-PT"); err != nil {
if err := users.SetPair("bob", "pt-PT", DirectionLearningEn); err != nil {
t.Fatalf("set pt-PT: %v", err)
}
if u, _ := users.Get("bob"); u.PairLang != "pt-PT" {
t.Fatalf("pair_lang = %q, want pt-PT", u.PairLang)
}
// Every pair with a langpack, not just the first one: this list and the
// frontend's PACKS are two copies of the same fact, and the day they
// disagree is the day she can pick a pair the app cannot render.
if err := users.SetPair("bob", "fr", DirectionLearningEn); err != nil {
t.Fatalf("set fr: %v", err)
}
if u, _ := users.Get("bob"); u.PairLang != "fr" {
t.Fatalf("pair_lang = %q, want fr", u.PairLang)
}
if err := users.SetPair("bob", "es", DirectionLearningEn); err != nil {
t.Fatalf("set es: %v", err)
}
if u, _ := users.Get("bob"); u.PairLang != "es" {
t.Fatalf("pair_lang = %q, want es", u.PairLang)
}
// And back — a writer who tries a pair and doesn't like it must be able to
// return, which is the whole reason the picker exists.
if err := users.SetPairLang("bob", "zh"); err != nil {
if err := users.SetPair("bob", "zh", DirectionLearningEn); err != nil {
t.Fatalf("set zh: %v", err)
}
if u, _ := users.Get("bob"); u.PairLang != "zh" {
@@ -46,8 +63,14 @@ func TestSetPairLang(t *testing.T) {
func TestSetPairLangRejectsUnshippedPairs(t *testing.T) {
_, users, _ := newStores(t)
for _, lang := range []string{"fr", "es", "pt-BR", "klingon", "", " "} {
if err := users.SetPairLang("bob", lang); err == nil {
// The near-misses are the ones that matter, and there are two of them now.
// "pt-BR" must not be quietly served European copy and a European voice;
// "es-ES" is the same mistake pointing the other way, because the es pack is
// deliberately Latin American and reads itself aloud in a Mexican voice. A
// regional code Petal has not decided about is refused rather than rounded
// to the nearest pack it happens to have.
for _, lang := range []string{"es-ES", "pt-BR", "fr-CA", "de", "klingon", "", " "} {
if err := users.SetPair("bob", lang, DirectionLearningEn); err == nil {
t.Fatalf("stored unshipped pair %q", lang)
}
}
@@ -58,7 +81,7 @@ func TestSetPairLangRejectsUnshippedPairs(t *testing.T) {
func TestSetPairLangUnknownUser(t *testing.T) {
_, users, _ := newStores(t)
if err := users.SetPairLang("nobody", "pt-PT"); err == nil {
if err := users.SetPair("nobody", "pt-PT", DirectionLearningEn); err == nil {
t.Fatal("set a pair language on an account that does not exist")
}
}
@@ -85,8 +108,8 @@ func TestUpdateMeHandlerRejects(t *testing.T) {
_, users, _ := newStores(t)
for name, body := range map[string]string{
"unshipped pair": `{"pair_lang":"fr"}`,
"missing field": `{}`,
"unshipped pair": `{"pair_lang":"es-ES"}`,
"unknown direction": `{"direction":"learning_klingon"}`,
"not json": `pt-PT`,
} {
if w := patchMe(t, users, "bob", body); w.Code != http.StatusBadRequest {
@@ -103,3 +126,116 @@ func TestUpdateMeHandlerRejects(t *testing.T) {
t.Fatalf("unknown user: status = %d, want 401", w.Code)
}
}
// An empty body used to be a 400, back when pair_lang was the only field and a
// request that named none of it could only be a client bug. With two optional
// fields it is an ordinary PATCH that changes nothing, and it has to be: the
// picker sends one field without knowing the other, and "omitted" has to mean
// "leave it alone" for that to be safe.
func TestUpdateMeHandlerEmptyBodyChangesNothing(t *testing.T) {
_, users, _ := newStores(t)
if err := users.SetPair("bob", "zh", DirectionLearningPair); err != nil {
t.Fatalf("set up: %v", err)
}
w := patchMe(t, users, "bob", `{}`)
if w.Code != http.StatusOK {
t.Fatalf("status = %d, want 200 (%s)", w.Code, w.Body.String())
}
u, _ := users.Get("bob")
if u.PairLang != "zh" || u.Direction != DirectionLearningPair {
t.Fatalf("empty PATCH moved the account to %q/%q", u.PairLang, u.Direction)
}
}
// The direction axis: an account can be turned around and turned back, and the
// default every existing row already carries is the one it had before the column
// existed.
func TestDirectionRoundTrip(t *testing.T) {
_, users, _ := newStores(t)
if u, _ := users.Get("bob"); u.Direction != DirectionLearningEn {
t.Fatalf("a fresh account starts at %q, want %q", u.Direction, DirectionLearningEn)
}
w := patchMe(t, users, "bob", `{"direction":"learning_pair"}`)
if w.Code != http.StatusOK {
t.Fatalf("turn around: status = %d (%s)", w.Code, w.Body.String())
}
var got db.User
if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil {
t.Fatalf("decode: %v", err)
}
// The response carries the direction, not just the pair — the client reads
// its whole state back from here rather than assuming the write took.
if got.Direction != DirectionLearningPair || got.PairLang != "zh" {
t.Fatalf("response = %+v, want bob learning zh", got)
}
if w := patchMe(t, users, "bob", `{"direction":"learning_en"}`); w.Code != http.StatusOK {
t.Fatalf("turn back: status = %d (%s)", w.Code, w.Body.String())
}
if u, _ := users.Get("bob"); u.Direction != DirectionLearningEn {
t.Fatalf("direction = %q after turning back", u.Direction)
}
}
// The refusal this axis exists to make: a pair with no word list cannot be
// learned toward, however good its langpack is. fr, es and pt-PT all have copy,
// voices and spelling dictionaries — and nothing that could segment a sentence
// or read from that language into English, which is what a learner needs.
func TestLearnerDirectionRefusedForPairsWithoutData(t *testing.T) {
_, users, _ := newStores(t)
for _, lang := range []string{"pt-PT", "fr", "es"} {
if err := users.SetPair("bob", lang, DirectionLearningEn); err != nil {
t.Fatalf("set %s: %v", lang, err)
}
w := patchMe(t, users, "bob", `{"direction":"learning_pair"}`)
if w.Code != http.StatusBadRequest {
t.Fatalf("%s: status = %d, want 400", lang, w.Code)
}
if u, _ := users.Get("bob"); u.Direction != DirectionLearningEn {
t.Fatalf("%s: a refused write still moved direction to %q", lang, u.Direction)
}
}
}
// The two-field combination the handler validates as one decision. An account
// already learning Chinese that asks only to change pair is asking for a state
// neither field names on its own — French with segmentation — and it must not
// arrive by leaving one field out.
func TestPairChangeCannotStrandTheLearnerDirection(t *testing.T) {
_, users, _ := newStores(t)
if err := users.SetPair("bob", "zh", DirectionLearningPair); err != nil {
t.Fatalf("set up: %v", err)
}
if w := patchMe(t, users, "bob", `{"pair_lang":"fr"}`); w.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want 400", w.Code)
}
u, _ := users.Get("bob")
if u.PairLang != "zh" || u.Direction != DirectionLearningPair {
t.Fatalf("refused write left the account at %q/%q", u.PairLang, u.Direction)
}
// Naming both at once is how that move is actually made, and it works.
if w := patchMe(t, users, "bob", `{"pair_lang":"fr","direction":"learning_en"}`); w.Code != http.StatusOK {
t.Fatalf("both fields: status = %d (%s)", w.Code, w.Body.String())
}
if u, _ := users.Get("bob"); u.PairLang != "fr" || u.Direction != DirectionLearningEn {
t.Fatalf("account = %q/%q, want fr/learning_en", u.PairLang, u.Direction)
}
}
// The CHECK constraint is the last line, below the handler and below SetPair:
// a direction that reaches the column by any other route is still refused.
func TestDirectionCheckConstraint(t *testing.T) {
_, users, database := newStores(t)
if _, err := database.Exec(`UPDATE users SET direction = 'sideways' WHERE id = 'bob'`); err == nil {
t.Fatal("the users.direction CHECK accepted 'sideways'")
}
if u, _ := users.Get("bob"); u.Direction != DirectionLearningEn {
t.Fatalf("direction = %q after a refused UPDATE", u.Direction)
}
}
+57 -7
View File
@@ -11,8 +11,31 @@ import (
"time"
)
// SessionCookie is the cookie carrying the opaque session token.
const SessionCookie = "petal_session"
// The cookie carrying the opaque session token, in its two spellings.
//
// Over https the name takes the __Host- prefix, which is not decoration: the
// browser will only accept such a cookie if it is Secure, Path=/, and carries
// no Domain attribute — and, crucially, refuses to let any other host set it.
// Without the prefix, anything that can write cookies for a sibling name under
// parodia.dev (another service on the box, a subdomain takeover) can plant a
// session cookie in her browser that Petal will then read as hers.
//
// The prefix is impossible over plain http, because it requires Secure and a
// browser drops a Secure cookie on an insecure origin. So local development
// keeps the bare name, and the name in use follows the same `secure` flag the
// rest of the cookie does.
const (
SessionCookie = "petal_session"
HostSessionCookie = "__Host-petal_session"
)
// sessionCookieName is the name to *write* under this scheme.
func sessionCookieName(secure bool) string {
if secure {
return HostSessionCookie
}
return SessionCookie
}
const (
// sessionTTL is how long a session lives without use. Thirty days, sliding:
@@ -78,11 +101,28 @@ func (s *SessionStore) Create(userID, userAgent string) (string, error) {
// Resolve implements [Resolver]: it reads the session cookie, validates it, and
// returns the user it belongs to — extending the session's life while it does.
func (s *SessionStore) Resolve(r *http.Request) (string, error) {
c, err := r.Cookie(SessionCookie)
if err != nil || c.Value == "" {
token := SessionToken(r)
if token == "" {
return "", ErrNoSession
}
return s.userFor(c.Value)
return s.userFor(token)
}
// SessionToken pulls the raw session token out of a request, preferring the
// __Host- spelling.
//
// Both are read because a deployment that was signing people in before the
// prefix existed has browsers holding the old name; those sessions stay valid
// and quietly re-issue under the new name at the next sign-in. The prefixed one
// wins where both are present, since it is the one another host could not have
// planted.
func SessionToken(r *http.Request) string {
for _, name := range []string{HostSessionCookie, SessionCookie} {
if c, err := r.Cookie(name); err == nil && c.Value != "" {
return c.Value
}
}
return ""
}
// userFor validates a raw token and slides its expiry forward.
@@ -146,7 +186,7 @@ func hashToken(token string) string {
// browser drop the cookie and silently break local login.
func SetSessionCookie(w http.ResponseWriter, token string, secure bool) {
http.SetCookie(w, &http.Cookie{
Name: SessionCookie,
Name: sessionCookieName(secure),
Value: token,
Path: "/",
HttpOnly: true,
@@ -158,9 +198,18 @@ func SetSessionCookie(w http.ResponseWriter, token string, secure bool) {
// ClearSessionCookie expires the session cookie in the browser. The matching
// server-side row must be revoked separately — that's the half that counts.
//
// Both spellings are expired, not just the one currently written: a browser
// carrying a pre-prefix cookie must not be left holding it after signing out,
// which is precisely the case where "clear the cookie" is the part the user can
// see working.
func ClearSessionCookie(w http.ResponseWriter, secure bool) {
for _, name := range []string{HostSessionCookie, SessionCookie} {
if name == HostSessionCookie && !secure {
continue // the browser would reject a non-Secure __Host- cookie
}
http.SetCookie(w, &http.Cookie{
Name: SessionCookie,
Name: name,
Value: "",
Path: "/",
HttpOnly: true,
@@ -168,4 +217,5 @@ func ClearSessionCookie(w http.ResponseWriter, secure bool) {
SameSite: http.SameSiteLaxMode,
MaxAge: -1,
})
}
}
+64
View File
@@ -309,3 +309,67 @@ func TestAllowlist(t *testing.T) {
}
}
}
// Over https the cookie takes the __Host- prefix, which the browser will only
// accept from the exact host that set it — closing the door on a sibling
// service under the same registrable domain planting a session in her browser.
// Over plain http it cannot: the prefix requires Secure, and a browser drops a
// Secure cookie on an insecure origin, so local development would silently stop
// logging in.
func TestSessionCookieNamePerScheme(t *testing.T) {
secure := httptest.NewRecorder()
SetSessionCookie(secure, "tok", true)
c := secure.Result().Cookies()[0]
if c.Name != HostSessionCookie {
t.Fatalf("https cookie name=%q, want %q", c.Name, HostSessionCookie)
}
// The prefix is a promise about these three attributes; a browser rejects
// the cookie outright if any is wrong.
if !c.Secure || c.Path != "/" || c.Domain != "" {
t.Fatalf("__Host- cookie violates its own contract: %+v", c)
}
insecure := httptest.NewRecorder()
SetSessionCookie(insecure, "tok", false)
if name := insecure.Result().Cookies()[0].Name; name != SessionCookie {
t.Fatalf("http cookie name=%q, want %q", name, SessionCookie)
}
}
// A browser holding a cookie issued before the prefix existed must stay signed
// in — and start using the new name at its next sign-in, not be logged out to
// get there.
func TestResolveAcceptsEitherCookieName(t *testing.T) {
store, _, _ := newStores(t)
token, err := store.Create("bob", "test-agent")
if err != nil {
t.Fatal(err)
}
for _, name := range []string{SessionCookie, HostSessionCookie} {
r := httptest.NewRequest(http.MethodGet, "/", nil)
r.AddCookie(&http.Cookie{Name: name, Value: token})
got, err := store.Resolve(r)
if err != nil || got != "bob" {
t.Fatalf("%s: resolved to %q (err=%v)", name, got, err)
}
}
}
// Signing out must not leave the browser holding either spelling.
func TestClearSessionCookieExpiresBothNames(t *testing.T) {
rec := httptest.NewRecorder()
ClearSessionCookie(rec, true)
cleared := map[string]bool{}
for _, c := range rec.Result().Cookies() {
if c.MaxAge < 0 {
cleared[c.Name] = true
}
}
for _, name := range []string{SessionCookie, HostSessionCookie} {
if !cleared[name] {
t.Errorf("%s was left in the browser after signing out", name)
}
}
}
+113 -14
View File
@@ -49,9 +49,9 @@ func (u *UserStore) Upsert(sub, email, displayName string) error {
func (u *UserStore) Get(id string) (db.User, error) {
var user db.User
err := u.db.QueryRow(
`SELECT id, email, COALESCE(display_name, ''), created_at, pair_lang
`SELECT id, email, COALESCE(display_name, ''), created_at, pair_lang, direction
FROM users WHERE id = ?`, id,
).Scan(&user.ID, &user.Email, &user.DisplayName, &user.CreatedAt, &user.PairLang)
).Scan(&user.ID, &user.Email, &user.DisplayName, &user.CreatedAt, &user.PairLang, &user.Direction)
return user, err
}
@@ -75,9 +75,14 @@ func (u *UserStore) MeHandler() http.HandlerFunc {
// every pair the prompts know how to talk about, which is a cheap thing to add;
// this one names the pairs Petal can render itself in, which requires a langpack
// on the frontend. Accepting a code with no pack would leave her looking at
// Chinese with no way back except another guess, so the server refuses it. fr
// and es join this list on the day their packs land, not before.
var shippedPairs = []string{"zh", "pt-PT"}
// Chinese with no way back except another guess, so the server refuses it. es
// joined on the day its pack landed, not before.
//
// These four are now every pair PairLang names on the frontend, which makes the
// two lists look redundant. They are not: the next pair will exist in the type
// and in the prompts long before it has copy, and this list is the one that
// says a writer may actually be sent there.
var shippedPairs = []string{"zh", "pt-PT", "fr", "es"}
func pairIsShipped(lang string) bool {
for _, p := range shippedPairs {
@@ -88,12 +93,63 @@ func pairIsShipped(lang string) bool {
return false
}
// SetPairLang moves an account to another (English + X) pair.
func (u *UserStore) SetPairLang(id, lang string) error {
// The two directions a pair can be travelled in. `DirectionLearningEn` is the
// original assumption made explicit: the writer is native in X and practising
// English. `DirectionLearningPair` is the other way round.
const (
DirectionLearningEn = "learning_en"
DirectionLearningPair = "learning_pair"
)
// The pairs whose *learner* direction Petal can actually serve, which is a
// narrower thing than a shipped pair and narrower again than a langpack.
//
// Turning a pair around needs data no langpack carries: a word list to segment
// with, and a dictionary that reads from the pair language into English. Chinese
// has both as of Phase 26 (CC-CEDICT + jieba); French, Spanish and Portuguese
// have neither yet, and — unlike a missing pack, which leaves a writer looking
// at copy she cannot read — a missing word list would leave her looking at an
// editor that silently does nothing when she hovers. Both are bad; only one is
// legible as a bug. So the server refuses, for the same reason and by the same
// mechanism as `shippedPairs`.
//
// This list is expected to grow one pair at a time and never to be inferred:
// segmentation is a property of a writing system, and there is no rule that
// derives "has a word list" from a language code.
var learnerPairs = []string{"zh"}
// SupportsLearnerDirection reports whether a pair can be turned around.
func SupportsLearnerDirection(lang string) bool {
for _, p := range learnerPairs {
if p == lang {
return true
}
}
return false
}
func directionIsKnown(d string) bool {
return d == DirectionLearningEn || d == DirectionLearningPair
}
// SetPair moves an account to another (English + X) pair, in a given direction.
//
// The two are written together because they constrain each other: a direction is
// only meaningful for a pair that can be travelled in it, and validating them a
// field at a time would let a two-step change pass through a state that neither
// step is allowed to leave behind.
func (u *UserStore) SetPair(id, lang, direction string) error {
if !pairIsShipped(lang) {
return errors.New("auth: unshipped pair language " + lang)
}
res, err := u.db.Exec(`UPDATE users SET pair_lang = ? WHERE id = ?`, lang, id)
if !directionIsKnown(direction) {
return errors.New("auth: unknown direction " + direction)
}
if direction == DirectionLearningPair && !SupportsLearnerDirection(lang) {
return errors.New("auth: no learner direction for " + lang)
}
res, err := u.db.Exec(
`UPDATE users SET pair_lang = ?, direction = ? WHERE id = ?`, lang, direction, id)
if err != nil {
return err
}
@@ -103,8 +159,8 @@ func (u *UserStore) SetPairLang(id, lang string) error {
return nil
}
// UpdateMeHandler changes the caller's own settings — today, the one setting
// there is: which language Petal speaks alongside her English.
// UpdateMeHandler changes the caller's own settings: which language Petal
// speaks alongside her English, and which of the two she is learning.
//
// It answers with the whole updated user rather than an empty 204 so the client
// has one shape to trust: /api/me and this return the same thing, and the app
@@ -114,24 +170,67 @@ func (u *UserStore) SetPairLang(id, lang string) error {
// dictionary, her read-aloud voice, which word-lookup provider answers, and the
// language the prompts ask the model to explain in. All of those read
// `users.pair_lang` at use time, so all of them follow from this one write.
//
// Both fields are optional and each defaults to what the account already has, so
// the picker can send one without knowing the other. That matters for the
// combination this endpoint exists to prevent: a client that sent only
// `pair_lang: "fr"` while the account sat on `learning_pair` would otherwise ask
// for French-with-segmentation, which does not exist. Here it is one decision
// with one validation, and the answer carries whatever actually landed.
func (u *UserStore) UpdateMeHandler() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
var body struct {
PairLang string `json:"pair_lang"`
PairLang *string `json:"pair_lang"`
Direction *string `json:"direction"`
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
httputil.BadRequest(w, "invalid request body")
return
}
lang := strings.TrimSpace(body.PairLang)
id := UserID(r.Context())
current, err := u.Get(id)
if err != nil {
// Only a missing row means "not signed in". A dictionary-file or
// SQLite fault answered as 401 would trip the client's session
// interceptor and throw a writer out of an app she is still signed
// in to — the same distinction SetPair's error branch makes below.
if errors.Is(err, sql.ErrNoRows) {
httputil.ErrorJSON(w, http.StatusUnauthorized, "not signed in")
return
}
httputil.ServerError(w, err)
return
}
lang, direction := current.PairLang, current.Direction
if body.PairLang != nil {
lang = strings.TrimSpace(*body.PairLang)
}
if body.Direction != nil {
direction = strings.TrimSpace(*body.Direction)
}
if !pairIsShipped(lang) {
// Name the ones that work. A writer who lands here has picked from a
// stale client, and "not a language" tells her nothing.
httputil.BadRequest(w, "unsupported language pair — Petal speaks "+strings.Join(shippedPairs, ", "))
return
}
id := UserID(r.Context())
if err := u.SetPairLang(id, lang); err != nil {
if !directionIsKnown(direction) {
httputil.BadRequest(w, "unknown direction — expected "+DirectionLearningEn+" or "+DirectionLearningPair)
return
}
if direction == DirectionLearningPair && !SupportsLearnerDirection(lang) {
// Refused rather than quietly downgraded to learning_en. A silent
// downgrade would leave the writer looking at an editor that behaves
// like the one she just tried to leave, with nothing to read as an
// explanation — and the caller cannot tell the two outcomes apart
// without diffing the response it was given.
httputil.BadRequest(w, "Petal can only be learned toward "+strings.Join(learnerPairs, ", ")+" so far")
return
}
if err := u.SetPair(id, lang, direction); err != nil {
if errors.Is(err, sql.ErrNoRows) {
httputil.ErrorJSON(w, http.StatusUnauthorized, "not signed in")
return
+49 -1
View File
@@ -1,7 +1,9 @@
package config
import (
"net/url"
"os"
"strconv"
"strings"
"time"
)
@@ -61,6 +63,22 @@ type Config struct {
// authenticates — right for a single-household instance, wrong the moment
// the IdP serves an audience wider than Petal's.
AllowedSubs string
// RequireAuth refuses to start when OIDC isn't configured, instead of
// falling back to the single local user.
//
// The fallback is the right behaviour on a laptop and a catastrophe on a
// public host: a typo in AUTHENTIK_CLIENT_SECRET turns every anonymous
// visitor into the `local` user, with full read and write over someone's
// private journals, and says so only in a log line nobody is reading. The
// Traefik basic-auth gate that used to stand behind that mistake was
// removed when Petal learned to authenticate for itself, so nothing catches
// it now.
//
// Defaulted from BASE_URL rather than declared: a Petal that knows itself by
// a real public origin has no business running open, and one on localhost
// has no business demanding an IdP. Set PETAL_REQUIRE_AUTH explicitly to
// override in either direction.
RequireAuth bool
}
// TTSVoice is one Piper instance and the single voice it has loaded.
@@ -77,9 +95,10 @@ func (c *Config) AuthEnabled() bool {
// Load reads configuration from the environment, applying sane local-dev defaults.
func Load() *Config {
baseURL := env("BASE_URL", "http://localhost:8080")
return &Config{
Port: env("PORT", "8080"),
BaseURL: env("BASE_URL", "http://localhost:8080"),
BaseURL: baseURL,
DatabasePath: env("DATABASE_PATH", "./data/petal.db"),
ImageDir: env("IMAGE_DIR", "./data/images"),
DictPath: env("DICT_PATH", "./data/dict.db"),
@@ -101,9 +120,26 @@ func Load() *Config {
AuthentikClientID: env("AUTHENTIK_CLIENT_ID", ""),
AuthentikClientSecret: env("AUTHENTIK_CLIENT_SECRET", ""),
AllowedSubs: env("PETAL_ALLOWED_SUBS", ""),
RequireAuth: envBool("PETAL_REQUIRE_AUTH", !isLoopbackOrigin(baseURL)),
}
}
// isLoopbackOrigin reports whether a base URL names this machine — the shape a
// development checkout has, and the only shape where running without a login is
// a reasonable default. Anything else (a hostname, a public origin) is a
// deployment, however small.
func isLoopbackOrigin(baseURL string) bool {
u, err := url.Parse(strings.TrimSpace(baseURL))
if err != nil {
return false
}
switch strings.ToLower(u.Hostname()) {
case "localhost", "127.0.0.1", "::1", "":
return true
}
return false
}
// ttsVoices reads the Piper instances out of an environment slice (as returned
// by os.Environ) into a map keyed by base language tag.
//
@@ -169,6 +205,18 @@ func env(key, fallback string) string {
return fallback
}
// envBool reads a boolean knob. Anything unparseable keeps the default rather
// than silently reading as false — a mistyped PETAL_REQUIRE_AUTH must not be the
// thing that turns the guard off.
func envBool(key string, fallback bool) bool {
if v := strings.TrimSpace(os.Getenv(key)); v != "" {
if b, err := strconv.ParseBool(v); err == nil {
return b
}
}
return fallback
}
func envDuration(key string, fallback time.Duration) time.Duration {
if v := os.Getenv(key); v != "" {
if d, err := time.ParseDuration(v); err == nil {
+59
View File
@@ -14,6 +14,10 @@ func TestTTSVoicesDiscovery(t *testing.T) {
"TTS_VOICE_ZH=zh_CN-huayan-medium",
"TTS_ENDPOINT_PT=http://piper-pt:5000",
"TTS_VOICE_PT=pt_PT-tugão-medium",
"TTS_ENDPOINT_FR=http://piper-fr:5000",
"TTS_VOICE_FR=fr_FR-siwis-medium",
"TTS_ENDPOINT_ES=http://piper-es:5000",
"TTS_VOICE_ES=es_MX-ald-medium",
// Noise that must not become a language.
"TTS_PATH=/synthesize",
"PATH=/usr/bin",
@@ -25,6 +29,13 @@ func TestTTSVoicesDiscovery(t *testing.T) {
// cleanly rather than producing a double slash at every call site.
"zh": {"http://piper-zh:5000", "zh_CN-huayan-medium"},
"pt": {"http://piper-pt:5000", "pt_PT-tugão-medium"},
// Phase 24's whole TTS change: a fourth language costs two lines here
// and a compose service, and no Go at all.
"fr": {"http://piper-fr:5000", "fr_FR-siwis-medium"},
// And a fifth cost exactly the same, which is the claim actually being
// tested. The voice is Mexican on purpose: the es pack is Latin
// American, and es_ES-davefx-medium would read it in the wrong accent.
"es": {"http://piper-es:5000", "es_MX-ald-medium"},
}
if len(voices) != len(want) {
t.Fatalf("discovered %v, want %v", voices, want)
@@ -81,3 +92,51 @@ func TestTTSVoicesEmptyWithoutEndpoints(t *testing.T) {
t.Errorf("discovered %v, want none", voices)
}
}
// The fallback to the single local user is right on a laptop and a catastrophe
// on a public host, so it is defaulted from the origin Petal knows itself by
// rather than left to be remembered.
func TestRequireAuthDefaultsFromBaseURL(t *testing.T) {
cases := []struct {
baseURL string
want bool
}{
{"http://localhost:8080", false},
{"http://127.0.0.1:8080", false},
{"http://[::1]:8080", false},
{"", false}, // no BASE_URL set at all: the local-dev default
{"https://petal.parodia.dev", true},
{"http://petal.parodia.dev", true},
{"https://petal.example.com/", true},
}
for _, c := range cases {
t.Setenv("BASE_URL", c.baseURL)
t.Setenv("PETAL_REQUIRE_AUTH", "")
if got := Load().RequireAuth; got != c.want {
t.Errorf("BASE_URL=%q: RequireAuth=%v, want %v", c.baseURL, got, c.want)
}
}
}
// The default is a default, not a rule: a trusted private network is a real
// deployment shape, and so is wanting the guard on locally.
func TestRequireAuthExplicitOverride(t *testing.T) {
t.Setenv("BASE_URL", "https://petal.parodia.dev")
t.Setenv("PETAL_REQUIRE_AUTH", "false")
if Load().RequireAuth {
t.Error("an explicit false must be honoured on a public origin")
}
t.Setenv("BASE_URL", "http://localhost:8080")
t.Setenv("PETAL_REQUIRE_AUTH", "true")
if !Load().RequireAuth {
t.Error("an explicit true must be honoured on localhost")
}
// A typo must not be the thing that disables the guard.
t.Setenv("BASE_URL", "https://petal.parodia.dev")
t.Setenv("PETAL_REQUIRE_AUTH", "nope")
if !Load().RequireAuth {
t.Error("an unparseable value must keep the default, not read as false")
}
}
+92
View File
@@ -497,6 +497,98 @@ CREATE INDEX idx_suggestions_resolved ON suggestions(status, resolved_at);
stmt: `
ALTER TABLE suggestions ADD COLUMN source TEXT NOT NULL DEFAULT 'llm';
UPDATE suggestions SET source = 'local' WHERE type = 'mechanics';
`,
},
{
// Sentence-level identity, so a re-check stops regenerating the world.
// Every pass used to delete its whole family and re-insert it, which
// meant accepting one edit gave every other card a new id and a newly
// worded explanation — the rail visibly emptied and refilled, and the
// model was asked again about sentences nobody had touched.
//
// `chunk_hash` records which sentence a suggestion belongs to, and
// checked_chunks records which sentences a family has already read. A
// re-check then asks only about the difference and keeps the rest of
// the rows exactly as they are, id and wording included.
//
// Existing rows get '' — "sentence unknown", which reads as in-play, so
// they are simply reconciled on the next pass like any fresh finding.
name: "0014_suggestion_chunk_hash",
stmt: `
ALTER TABLE suggestions ADD COLUMN chunk_hash TEXT NOT NULL DEFAULT '';
CREATE TABLE checked_chunks (
doc_id TEXT NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
family TEXT NOT NULL,
hash TEXT NOT NULL,
PRIMARY KEY (doc_id, family, hash)
);
`,
},
{
// A sentence she wrote in her own language gets its own type. Petal already
// detected such spans and already rendered them into English — it just
// filed the result under 'clarity', so the pair model's flagship moment
// read as tidying up her Chinese. As with 0005 and 0008, the `type` CHECK
// can't be ALTERed in place, so rebuild the table with the extended
// constraint, copy every row across, and recreate both indexes.
//
// Existing rows are left on whatever type they have. A card she is already
// reading keeps the label she has already read (the same rule reconcile.go
// follows for a re-proposed edit); new findings get the new label.
name: "0015_translate_suggestion_type",
stmt: `
CREATE TABLE suggestions_new (
id TEXT PRIMARY KEY DEFAULT (lower(hex(randomblob(16)))),
doc_id TEXT NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
from_pos INTEGER NOT NULL,
to_pos INTEGER NOT NULL,
original TEXT NOT NULL,
replacement TEXT NOT NULL,
explanation TEXT NOT NULL,
type TEXT NOT NULL CHECK(type IN ('grammar','phrasing','idiom','clarity','translate','voice','collocation','mechanics')),
status TEXT NOT NULL DEFAULT 'pending' CHECK(status IN ('pending','accepted','rejected')),
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
resolved_at DATETIME,
source TEXT NOT NULL DEFAULT 'llm',
chunk_hash TEXT NOT NULL DEFAULT ''
);
INSERT INTO suggestions_new (id, doc_id, from_pos, to_pos, original, replacement, explanation, type, status, created_at, resolved_at, source, chunk_hash)
SELECT id, doc_id, from_pos, to_pos, original, replacement, explanation, type, status, created_at, resolved_at, source, chunk_hash FROM suggestions;
DROP TABLE suggestions;
ALTER TABLE suggestions_new RENAME TO suggestions;
CREATE INDEX idx_suggestions_doc_id ON suggestions(doc_id);
CREATE INDEX idx_suggestions_resolved ON suggestions(status, resolved_at);
`,
},
{
// Which half of the pair is being learned.
//
// `pair_lang` (0010) has always answered "which two languages", and every
// surface built on it assumed the answer to a second question nobody had
// asked: that English is the language being *learned*. That assumption is
// load-bearing in a dozen places — CJK is deliberately never tokenized,
// never spell-checked, never glossed; the prompts explain English in her
// language; the vocabulary garden captures English words. All correct for
// a Mandarin native practising English, and all backwards for an English
// native practising Mandarin.
//
// A second pair code ('zh-learner') was the cheaper option and is the
// wrong shape: it would make the two directions of one pair look like two
// unrelated languages to every query, and it would have to be repeated for
// fr, es and pt-PT before any of them could turn around. A column keeps
// the two questions separate, which is what they are.
//
// 'learning_en' is the default and is what every existing row means — the
// backfill is the DEFAULT itself, and it is right rather than merely
// convenient: all three accounts today are Mandarin natives writing
// English.
name: "0016_user_direction",
stmt: `
ALTER TABLE users ADD COLUMN direction TEXT NOT NULL DEFAULT 'learning_en'
CHECK(direction IN ('learning_en','learning_pair'));
`,
},
}
+137
View File
@@ -3,6 +3,7 @@ package db
import (
"path/filepath"
"testing"
"time"
)
func TestOpenMigratesAndSeeds(t *testing.T) {
@@ -227,3 +228,139 @@ func TestSuggestionSourceBackfill(t *testing.T) {
t.Errorf("default source = %q, want %q", fresh, SuggestionSourceLLM)
}
}
// TestTranslateTypeMigrationPreservesRows runs migration 0015 against a database
// that predates it. Unlike the two backfills above, 0015 *rebuilds the table* —
// SQLite can't ALTER a CHECK constraint — so it copies every row across by hand,
// and a column left out of that copy list silently loses her data. Every test
// elsewhere starts from a fresh database and would never notice; the live box has
// years of rows in it.
func TestTranslateTypeMigrationPreservesRows(t *testing.T) {
path := filepath.Join(t.TempDir(), "old.db")
d, err := Open(path)
if err != nil {
t.Fatalf("open: %v", err)
}
// Rewind to the pre-0015 table: the same shape, minus 'translate' in the CHECK.
if _, err := d.Exec(`
CREATE TABLE suggestions_old (
id TEXT PRIMARY KEY DEFAULT (lower(hex(randomblob(16)))),
doc_id TEXT NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
from_pos INTEGER NOT NULL,
to_pos INTEGER NOT NULL,
original TEXT NOT NULL,
replacement TEXT NOT NULL,
explanation TEXT NOT NULL,
type TEXT NOT NULL CHECK(type IN ('grammar','phrasing','idiom','clarity','voice','collocation','mechanics')),
status TEXT NOT NULL DEFAULT 'pending' CHECK(status IN ('pending','accepted','rejected')),
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
resolved_at DATETIME,
source TEXT NOT NULL DEFAULT 'llm',
chunk_hash TEXT NOT NULL DEFAULT ''
);
DROP TABLE suggestions;
ALTER TABLE suggestions_old RENAME TO suggestions;
CREATE INDEX idx_suggestions_doc_id ON suggestions(doc_id);
CREATE INDEX idx_suggestions_resolved ON suggestions(status, resolved_at);
DELETE FROM schema_migrations WHERE name = '0015_translate_suggestion_type';
`); err != nil {
t.Fatalf("rewind schema: %v", err)
}
if _, err := d.Exec(`INSERT INTO documents (id, user_id) VALUES ('d1', ?)`, LocalUserID); err != nil {
t.Fatalf("insert document: %v", err)
}
// One row with every column carrying a distinguishable value, so a dropped
// column shows up as a changed value rather than as a passing test.
if _, err := d.Exec(
`INSERT INTO suggestions (id, doc_id, from_pos, to_pos, original, replacement, explanation, type, status, created_at, resolved_at, source, chunk_hash)
VALUES ('s-1', 'd1', 7, 11, 'by foots', 'on foot', 'idiom advice she has read', 'idiom', 'accepted', '2026-01-02 03:04:05', '2026-01-02 03:05:00', 'local', 'abc123')`,
); err != nil {
t.Fatalf("seed row: %v", err)
}
d.Close()
d2, err := Open(path)
if err != nil {
t.Fatalf("reopen (migrate): %v", err)
}
defer d2.Close()
var (
docID, original, replacement, explanation string
typ, status, source, chunkHash string
from, to int
// Scanned as instants, not strings: the driver renders a DATETIME column in
// its own format, so the claim is "the same moment", not the same text.
createdAt, resolvedAt time.Time
)
if err := d2.QueryRow(
`SELECT doc_id, from_pos, to_pos, original, replacement, explanation, type, status, created_at, resolved_at, source, chunk_hash
FROM suggestions WHERE id = 's-1'`,
).Scan(&docID, &from, &to, &original, &replacement, &explanation,
&typ, &status, &createdAt, &resolvedAt, &source, &chunkHash); err != nil {
t.Fatalf("read migrated row: %v", err)
}
for _, c := range []struct{ name, got, want string }{
{"doc_id", docID, "d1"},
{"original", original, "by foots"},
{"replacement", replacement, "on foot"},
{"explanation", explanation, "idiom advice she has read"},
{"type", typ, SuggestionTypeIdiom},
{"status", status, SuggestionStatusAccepted},
{"source", source, SuggestionSourceLocal},
{"chunk_hash", chunkHash, "abc123"},
} {
if c.got != c.want {
t.Errorf("%s = %q, want %q", c.name, c.got, c.want)
}
}
if from != 7 || to != 11 {
t.Errorf("offsets = (%d, %d), want (7, 11)", from, to)
}
// created_at and resolved_at must survive: the rail's arrival chime keys on
// created_at, and the growth journal counts by resolved_at. A rebuild that
// reset either would re-chime her whole document and rewrite her history.
for _, c := range []struct {
name string
got time.Time
want string
}{
{"created_at", createdAt, "2026-01-02 03:04:05"},
{"resolved_at", resolvedAt, "2026-01-02 03:05:00"},
} {
want, err := time.Parse("2006-01-02 15:04:05", c.want)
if err != nil {
t.Fatalf("parse want: %v", err)
}
if !c.got.Equal(want) {
t.Errorf("%s = %v, want the original instant %v", c.name, c.got, want)
}
}
// The point of the rebuild: the new type is now insertable, and a bogus one
// still isn't.
if _, err := d2.Exec(
`INSERT INTO suggestions (id, doc_id, from_pos, to_pos, original, replacement, explanation, type)
VALUES ('s-2', 'd1', 0, 3, '苹果', 'apple', 'x', ?)`, SuggestionTypeTranslate,
); err != nil {
t.Fatalf("insert translate row: %v", err)
}
if _, err := d2.Exec(
`INSERT INTO suggestions (id, doc_id, from_pos, to_pos, original, replacement, explanation, type)
VALUES ('s-3', 'd1', 0, 3, 'x', 'y', 'x', 'nonsense')`,
); err == nil {
t.Error("CHECK constraint accepted an unknown type after the rebuild")
}
// Both indexes must come back, or every document load starts table-scanning.
for _, idx := range []string{"idx_suggestions_doc_id", "idx_suggestions_resolved"} {
var name string
if err := d2.QueryRow(
`SELECT name FROM sqlite_master WHERE type = 'index' AND name = ?`, idx,
).Scan(&name); err != nil {
t.Errorf("index %s missing after rebuild: %v", idx, err)
}
}
}
+21 -1
View File
@@ -15,6 +15,19 @@ type User struct {
// today, "pt-PT"/"fr"/"es" once the langpacks land. It selects the UI copy
// and dictionary set, not the language they may type in.
PairLang string `json:"pair_lang"`
// Direction says which half of the pair is being *learned*. Every pair until
// now assumed one answer: the writer is native in X and practising English,
// so hanzi is never tokenized and English is what gets underlined. Turn it
// around — a native English speaker learning Chinese — and the same pair
// wants the opposite of nearly every default.
//
// It is a separate column from PairLang rather than a second pair code
// ("zh-learner") because it is a genuinely separate question: the pair says
// *which two languages*, this says *which way round*. Keeping them apart is
// what lets fr, es and pt-PT inherit the learner direction later without a
// second langpack each.
Direction string `json:"direction"`
}
// Document is a single piece of writing. `Content` is the Tiptap JSON document
@@ -104,7 +117,7 @@ type Suggestion struct {
Original string `json:"original"`
Replacement string `json:"replacement"`
Explanation string `json:"explanation"`
Type string `json:"type"` // grammar | phrasing | idiom | clarity | voice | collocation
Type string `json:"type"` // grammar | phrasing | idiom | clarity | translate | voice | collocation
Status string `json:"status"` // pending | accepted | rejected
// Source names the engine that proposed the edit, not its family: an offline
// rule and the model can both propose a collocation, and the writer is never
@@ -119,6 +132,13 @@ const (
SuggestionTypePhrasing = "phrasing"
SuggestionTypeIdiom = "idiom"
SuggestionTypeClarity = "clarity"
// A span she wrote in her own language, rendered into English. Not a
// correction — nothing was wrong with it — which is why it is its own type
// rather than a clarity fix: the card is the pair model's flagship moment
// (SUGGESTIONS §1), and labelling it "Clarity" reads as a tidy-up of her
// first language. The model isn't asked for this label; it is derived from the
// span itself (see suggestions/language.go), so it can't drift.
SuggestionTypeTranslate = "translate"
SuggestionTypeVoice = "voice"
SuggestionTypeCollocation = "collocation"
SuggestionTypeMechanics = "mechanics" // deterministic rule-based pass (no LLM)
+71 -4
View File
@@ -287,7 +287,14 @@ func mdBlock(n pmNode, depth int) string {
return "---"
case "image":
alt := n.attrStr("alt")
return fmt.Sprintf("![%s](%s)", alt, n.attrStr("src"))
src := safeURL(n.attrStr("src"))
if src == "" {
// Nowhere safe to point. Keep the alt text as plain prose — it is
// the part that carries meaning — rather than emitting an image
// whose destination was rejected. See safeURL.
return alt
}
return fmt.Sprintf("![%s](%s)", alt, src)
case "table":
return mdTable(n)
case "bulletList", "orderedList":
@@ -396,7 +403,9 @@ func applyMdMarks(n pmNode) string {
if n.hasMark("underline") {
t = "<u>" + t + "</u>"
}
if href := n.markAttr("link", "href"); href != "" {
// Same rule as the HTML export: plenty of Markdown renderers pass a
// `javascript:` destination straight through into an <a href>. See safeURL.
if href := safeURL(n.markAttr("link", "href")); href != "" {
t = "[" + t + "](" + href + ")"
}
return t
@@ -518,7 +527,16 @@ func htmlBlock(n pmNode) string {
return "<hr>\n"
case "image":
alt := htmlEscape(n.attrStr("alt"))
return fmt.Sprintf("<p><img src=\"%s\" alt=\"%s\"></p>\n", htmlEscape(n.attrStr("src")), alt)
src := safeURL(n.attrStr("src"))
if src == "" {
// Nowhere safe to point: keep the alt text, which is the part that
// carries meaning, rather than emitting a broken image.
if alt == "" {
return ""
}
return "<p>" + alt + "</p>\n"
}
return fmt.Sprintf("<p><img src=\"%s\" alt=\"%s\"></p>\n", htmlEscape(src), alt)
case "table":
return htmlTable(n)
case "bulletList", "orderedList":
@@ -617,7 +635,9 @@ func applyHTMLMarks(n pmNode) string {
if n.hasMark("highlight") {
t = "<mark>" + t + "</mark>"
}
if href := n.markAttr("link", "href"); href != "" {
// An unsafe href is dropped, not the link: the words stay, they just stop
// being clickable. See safeURL.
if href := safeURL(n.markAttr("link", "href")); href != "" {
t = fmt.Sprintf("<a href=\"%s\">%s</a>", htmlEscape(href), t)
}
return t
@@ -858,6 +878,53 @@ func htmlEscape(s string) string {
return r.Replace(s)
}
// safeURLSchemes are the schemes an exported document may point at. Escaping
// makes a URL safe to sit inside an attribute; it says nothing about what
// happens when the attribute is followed, and `javascript:` survives it
// untouched.
//
// The toolbar can't produce one — it prefixes anything it doesn't recognise
// with https:// — but the toolbar is not the only way in: PUT /api/docs/{id}
// stores whatever Tiptap JSON it is given. And an export is the one artifact
// here that is *meant* to leave: the passport and the .html backup are files a
// writer hands to a teacher or an editor, opened on a machine that has no
// reason to trust them. A link that runs code when clicked is not something to
// ship inside one.
//
// Relative and fragment links pass through: they're how a document refers to
// its own headings, and they can't reach anything.
var safeURLSchemes = map[string]bool{
"http": true, "https": true, "mailto": true, "tel": true, "ftp": true,
}
// safeURL returns u if it is safe to follow from an exported file, and "" if it
// isn't. A dropped href leaves the link text in place — the reader loses a
// destination, never the writing.
func safeURL(u string) string {
trimmed := strings.TrimSpace(u)
if trimmed == "" {
return ""
}
// A scheme is everything before the first ':', but only when no '/', '?' or
// '#' comes first — otherwise "notes/a:b" would read as the "notes/a" scheme.
// Nothing before a colon means a relative or fragment link, which is fine.
if i := strings.IndexAny(trimmed, ":/?#"); i >= 0 && trimmed[i] == ':' {
// Control characters and whitespace are stripped by browsers *before*
// the scheme is read, so "java\nscript:" is javascript:. Fold them out
// before deciding rather than after.
scheme := strings.Map(func(r rune) rune {
if r <= ' ' || r == 0x7f {
return -1
}
return r
}, trimmed[:i])
if !safeURLSchemes[strings.ToLower(scheme)] {
return ""
}
}
return trimmed
}
func xmlEscape(s string) string {
r := strings.NewReplacer("&", "&amp;", "<", "&lt;", ">", "&gt;", `"`, "&quot;")
return r.Replace(s)
+72
View File
@@ -8,6 +8,8 @@ import (
"net/http"
"strings"
"testing"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// richDocJSON is a Tiptap document exercising headings, marks, and a list —
@@ -243,3 +245,73 @@ func TestExportUnsupportedFormat(t *testing.T) {
t.Fatalf("expected 400 for unsupported format, got %d", rec.Code)
}
}
// Escaping makes a URL safe to sit inside an attribute; it says nothing about
// what happens when the attribute is followed. An export is the one artifact
// here meant to leave — the file handed to a teacher, opened on a machine with
// no reason to trust it — so a destination that runs code is dropped.
func TestExportDropsUnsafeLinkSchemes(t *testing.T) {
unsafe := []string{
"javascript:alert(1)",
"JaVaScRiPt:alert(1)",
"java\nscript:alert(1)", // browsers strip control characters first
" javascript:alert(1)",
"data:text/html;base64,PHNjcmlwdD5hbGVydCgxKTwvc2NyaXB0Pg==",
"vbscript:msgbox(1)",
}
for _, href := range unsafe {
if got := safeURL(href); got != "" {
t.Errorf("safeURL(%q) = %q, want it dropped", href, got)
}
}
safe := []string{
"https://example.com/a?b=1#c",
"http://example.com",
"mailto:her@example.com",
"/api/images/abc.png",
"#a-heading",
"notes/chapter:one.md", // a colon that isn't a scheme
}
for _, href := range safe {
if got := safeURL(href); got != href {
t.Errorf("safeURL(%q) = %q, want it kept", href, got)
}
}
}
// End to end through the renderers: an unsafe href loses its destination, never
// its words.
func TestRenderedExportsCarryNoScriptURLs(t *testing.T) {
doc := db.Document{
Title: "Notes",
Content: `{"type":"doc","content":[{"type":"paragraph","content":[
{"type":"text","text":"click me","marks":[{"type":"link","attrs":{"href":"javascript:alert(1)"}}]}]},
{"type":"image","attrs":{"src":"javascript:alert(2)","alt":"a drawing"}}]}`,
}
html, err := renderHTMLFile(doc)
if err != nil {
t.Fatal(err)
}
if strings.Contains(strings.ToLower(string(html)), "javascript:") {
t.Fatalf("html export carried a javascript: URL:\n%s", html)
}
if !strings.Contains(string(html), "click me") {
t.Fatal("html export dropped the link text along with the href")
}
if !strings.Contains(string(html), "a drawing") {
t.Fatal("html export dropped the alt text of the rejected image")
}
md, err := renderMarkdown(doc)
if err != nil {
t.Fatal(err)
}
if strings.Contains(strings.ToLower(string(md)), "javascript:") {
t.Fatalf("markdown export carried a javascript: URL:\n%s", md)
}
if !strings.Contains(string(md), "click me") {
t.Fatal("markdown export dropped the link text along with the href")
}
}
+15
View File
@@ -35,3 +35,18 @@ func ServerError(w http.ResponseWriter, err error) {
log.Printf("internal error: %v", err)
ErrorJSON(w, http.StatusInternalServerError, "something went wrong")
}
// UpstreamError is ServerError's counterpart for a dependency Petal calls out
// to — the model, chiefly. Same discipline, and for a sharper reason: a dial
// failure's error text contains the endpoint it failed to dial, so relaying it
// hands anyone who can reach Petal the address of the inference box on the far
// side of the VPN, along with which backend is running there.
//
// `what` names the pass for the operator's log ("checkpoint", "chat"). The
// browser is told only that the helper is unreachable, which is all the client
// ever did anything with: every LLM route's 502 renders as the same warm
// "小助手在休息 · Petal's helper is resting".
func UpstreamError(w http.ResponseWriter, what string, err error) {
log.Printf("upstream error (%s): %v", what, err)
ErrorJSON(w, http.StatusBadGateway, "Petal's helper is out of reach right now")
}
+62 -1
View File
@@ -37,6 +37,17 @@ import (
// small enough to keep a careless paste from filling the disk.
const maxUploadBytes = 10 << 20
// maxUserBytes caps what one account may keep stored, at 1 GiB. The per-upload
// limit bounds a single careless paste; nothing bounded ten thousand of them,
// and Petal's data directory is an 8 GiB encrypted volume shared with the
// database, the backups and the TTS cache — the disk filling is the database
// losing writes, not just images failing.
//
// A tenth of the volume per writer is far past any real use: a heavily
// illustrated journal is tens of megabytes. It is a runaway backstop, and it is
// deliberately generous enough that nobody writing normally will ever meet it.
const maxUserBytes = 1 << 30
// extByContentType maps the image types we accept to a canonical extension. The
// allowlist doubles as validation: anything not here is rejected.
var extByContentType = map[string]string{
@@ -177,6 +188,19 @@ func (h *Handler) upload(w http.ResponseWriter, r *http.Request) {
name := hex.EncodeToString(sum[:])[:32] + ext
path := filepath.Join(h.dir, name)
userID := auth.UserID(r.Context())
within, err := h.withinQuota(userID, name, int64(len(data)))
if err != nil {
log.Printf("images: quota check failed for %s: %v", userID, err)
http.Error(w, "could not store image", http.StatusInternalServerError)
return
}
if !within {
http.Error(w, "you've filled Petal's picture store — delete a few images and try again",
http.StatusInsufficientStorage)
return
}
// Skip the write if this exact content is already stored.
if _, statErr := os.Stat(path); errors.Is(statErr, os.ErrNotExist) {
if err := os.WriteFile(path, data, 0o644); err != nil {
@@ -190,7 +214,7 @@ func (h *Handler) upload(w http.ResponseWriter, r *http.Request) {
if _, err := h.db.Exec(
`INSERT INTO images (name, user_id, content_type, size) VALUES (?, ?, ?, ?)
ON CONFLICT (name, user_id) DO NOTHING`,
name, auth.UserID(r.Context()), ct, len(data),
name, userID, ct, len(data),
); err != nil {
log.Printf("images: could not record ownership of %s: %v", name, err)
http.Error(w, "could not store image", http.StatusInternalServerError)
@@ -221,6 +245,20 @@ func (h *Handler) serve(w http.ResponseWriter, r *http.Request) {
}
// Private: a shared cache must never hand one writer's image to another.
w.Header().Set("Cache-Control", "private, max-age=31536000, immutable")
// SVG is a document format wearing an image's name: it can carry <script>,
// and this route serves it from Petal's own origin. Rendered through an
// <img> — the only way the editor ever shows one — that script never runs.
// Navigated to directly, which is one "open image in new tab" away, it does,
// and it runs with the API of whoever opened it.
//
// So every stored image answers with a CSP that permits nothing at all
// except the inline styles an illustration legitimately carries. It costs
// pasted SVGs nothing (an <img> was already a script-free context) and
// leaves the direct-navigation case inert. nosniff is set at the edge, but
// repeated here so the guarantee doesn't depend on Traefik's config.
w.Header().Set("Content-Security-Policy", "default-src 'none'; style-src 'unsafe-inline'; sandbox")
w.Header().Set("X-Content-Type-Options", "nosniff")
http.ServeFile(w, r, path)
}
@@ -261,6 +299,29 @@ func (h *Handler) remove(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNoContent)
}
// withinQuota reports whether userID may store one more image of size bytes.
//
// An image the caller already owns is free: content addressing means re-pasting
// the same picture stores nothing new, and charging for it would let a document
// that merely repeats one illustration walk into the limit. Deduplication
// across *accounts* is not credited the same way — two people each keep their
// own claim on a shared file, because either of them deleting it must not
// depend on what the other did.
func (h *Handler) withinQuota(userID, name string, size int64) (bool, error) {
var used, already sql.NullInt64
if err := h.db.QueryRow(
`SELECT (SELECT COALESCE(SUM(size), 0) FROM images WHERE user_id = ?),
(SELECT size FROM images WHERE user_id = ? AND name = ?)`,
userID, userID, name,
).Scan(&used, &already); err != nil {
return false, err
}
if already.Valid {
return true, nil // already stored for this account — costs nothing more
}
return used.Int64+size <= maxUserBytes, nil
}
// owns reports whether userID has a claim on a stored image.
func (h *Handler) owns(name, userID string) bool {
var ok bool
+82
View File
@@ -240,3 +240,85 @@ func TestServeMissing(t *testing.T) {
t.Fatalf("expected 404, got %d", rec.Code)
}
}
// An SVG is a document, not a picture: it can carry <script>, and this route
// serves it from Petal's own origin. Rendered through an <img> that script
// never runs, but "open image in new tab" is one click away, and there it
// would — with the API of whoever opened it. Every stored image therefore
// answers with a CSP that permits nothing.
func TestStoredImagesAreServedInert(t *testing.T) {
_, alice, _ := newStore(t)
svg := []byte(`<svg xmlns="http://www.w3.org/2000/svg"><script>fetch('/api/docs')</script></svg>`)
rec := httptest.NewRecorder()
alice.ServeHTTP(rec, uploadReq(t, "image", svg))
if rec.Code != http.StatusOK {
t.Fatalf("svg upload code=%d body=%s", rec.Code, rec.Body)
}
var resp struct{ URL string }
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
t.Fatal(err)
}
name := strings.TrimPrefix(resp.URL, "/api/images/")
got := get(t, alice, name)
if got.Code != http.StatusOK {
t.Fatalf("serve code=%d", got.Code)
}
csp := got.Header().Get("Content-Security-Policy")
if !strings.Contains(csp, "default-src 'none'") || !strings.Contains(csp, "sandbox") {
t.Fatalf("CSP %q does not neutralize the response", csp)
}
if got.Header().Get("X-Content-Type-Options") != "nosniff" {
t.Fatal("stored images must be served nosniff")
}
}
// A per-upload cap bounds one careless paste; nothing bounded ten thousand of
// them, on the same volume the database lives on.
func TestUploadQuota(t *testing.T) {
database, err := db.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatalf("open db: %v", err)
}
t.Cleanup(func() { database.Close() })
if _, err := database.Exec(
`INSERT INTO users (id, email, display_name) VALUES (?, ?, ?)`,
"bob", "bob@petal.local", "Bob",
); err != nil {
t.Fatalf("seed second user: %v", err)
}
h, err := New(t.TempDir(), database.DB, db.LocalUserID)
if err != nil {
t.Fatalf("new store: %v", err)
}
alice := auth.Middleware(auth.StaticResolver(db.LocalUserID))(h.Routes())
bob := auth.Middleware(auth.StaticResolver("bob"))(h.Routes())
// Fill Alice's allowance by hand — uploading a gibibyte in a test would be
// absurd, and what's under test is the accounting, not the arithmetic.
name := upload(t, alice, pngBytes)
if _, err := database.Exec(
`UPDATE images SET size = ? WHERE user_id = ? AND name = ?`,
int64(maxUserBytes), db.LocalUserID, name,
); err != nil {
t.Fatal(err)
}
// Re-storing something she already has costs nothing, so it still works.
if again := upload(t, alice, pngBytes); again != name {
t.Fatalf("a re-upload of an owned image should dedupe, got %q", again)
}
// Anything new does not.
rec := httptest.NewRecorder()
alice.ServeHTTP(rec, uploadReq(t, "image", otherPNG))
if rec.Code != http.StatusInsufficientStorage {
t.Fatalf("over-quota upload code=%d, want 507", rec.Code)
}
// And it is *her* allowance, not the store's: Bob is unaffected.
if got := upload(t, bob, otherPNG); got == "" {
t.Fatal("one writer's quota must not stop another writing")
}
}
+10
View File
@@ -36,3 +36,13 @@ var glossGz []byte
//
//go:embed data/phonetic.json.gz
var phoneticGz []byte
// hanziGz is the gzipped Chinese→English map: simplified headword → [[pinyin,
// senses], …]. Built from CC-CEDICT (scripts/build_cedict.py), unfiltered — the
// word a learner stops on is the one they do not know, so this is the one
// dataset here with no frequency gate. Loaded on its own sync.Once (see
// hanzi.go), not with the four above, because only a learner-direction account
// ever asks for it.
//
//go:embed data/hanzi.json.gz
var hanziGz []byte
Binary file not shown.
+37 -1
View File
@@ -4,6 +4,7 @@ import (
"context"
"database/sql"
"encoding/json"
"log"
"net/http"
"net/url"
@@ -41,6 +42,36 @@ func (h *Handler) GlossRoutes() chi.Router {
return r
}
// HanziRoutes returns the router mounted at /api/hanzi — a Chinese word to its
// pinyin and English senses, for a writer going the other way through the zh
// pair (`users.direction = 'learning_pair'`).
//
// It does not go through [Handler.providerFor], and that is not an oversight.
// providerFor picks a dictionary by the writer's *pair*, to answer "what does
// this English word mean in her language" — a question whose answer differs per
// pair. This endpoint asks the opposite question of exactly one language, and
// [auth.SupportsLearnerDirection] already guarantees that language is Chinese.
// Routing it through the pair would add a database read per hover to choose
// between one option and itself.
func (h *Handler) HanziRoutes() chi.Router {
r := chi.NewRouter()
r.Get("/{word}", h.hanzi)
return r
}
// hanzi answers a Chinese word lookup. Like the other two, a miss is a 200 with
// empty lists — a hover that lands on a word the dictionary has never heard of
// is an ordinary thing to happen while reading, and the tooltip simply doesn't
// open.
func (h *Handler) hanzi(w http.ResponseWriter, r *http.Request) {
res, err := h.Set.Hanzi(pathWord(r))
if err != nil {
writeLookupErr(w, err)
return
}
writeLookup(w, res)
}
// providerFor returns the provider for the caller's language pair.
//
// The pair language is read here rather than threaded down because a word
@@ -95,10 +126,15 @@ func (h *Handler) gloss(w http.ResponseWriter, r *http.Request) {
writeLookup(w, res)
}
// writeLookupErr answers a failed lookup. The real error is a dictionary or
// database fault — a file path, a SQLite message — and belongs in the log, not
// in a tooltip. The client treats any non-200 the same way, so nothing is lost
// by saying less.
func writeLookupErr(w http.ResponseWriter, err error) {
log.Printf("lexicon: lookup failed: %v", err)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusInternalServerError)
_ = json.NewEncoder(w).Encode(map[string]string{"error": err.Error()})
_ = json.NewEncoder(w).Encode(map[string]string{"error": "lookup failed"})
}
func writeLookup(w http.ResponseWriter, v any) {
+144
View File
@@ -0,0 +1,144 @@
package lexicon
import (
"fmt"
"strings"
"sync"
"unicode"
)
// The Chinese half of the lexicon: a word written in hanzi to its pinyin and
// English senses. This is the mirror image of `gloss` — that one reads English
// and answers in Chinese, for a Mandarin native practising English; this one
// reads Chinese and answers in English, for the other direction of the same
// pair (`users.direction = 'learning_pair'`).
//
// It is deliberately not folded into [Lexicon.load]. That method reads four
// datasets on the first lookup of any kind, and this one is 3.1 MB gzipped that
// only a learner-direction account will ever ask for — every other writer would
// pay the decompression and the resident memory for a map they never touch. Its
// own sync.Once means the cost lands on the first Chinese hover and nowhere
// else.
// HanziReading is one pronunciation of a word and the senses it carries in that
// pronunciation. A word usually has one; the ones that have two are why this is
// a list rather than a pair of strings. 得 is dé, "to obtain", *and* de, the
// particle that makes 说得很好 mean "speaks well" — a learner shown only the
// first has been told something false about the sentence in front of them.
type HanziReading struct {
Pinyin string `json:"pinyin"`
Senses string `json:"senses"`
}
// HanziChar is one character of a word that the dictionary could not answer as
// a whole. See [Lexicon.Hanzi].
type HanziChar struct {
Char string `json:"char"`
Pinyin string `json:"pinyin"`
Senses string `json:"senses"`
}
// HanziResult is what a Chinese word lookup answers. Readings is empty for a
// word the dictionary does not have, in which case Chars may carry the
// character-by-character reading instead.
type HanziResult struct {
Word string `json:"word"`
Readings []HanziReading `json:"readings"`
Chars []HanziChar `json:"chars"`
}
type hanziStore struct {
once sync.Once
err error
// word → [[pinyin, senses], …], exactly as scripts/build_cedict.py writes it.
entries map[string][][]string
}
var hanzi hanziStore
func (h *hanziStore) load() {
h.once.Do(func() {
if err := gunzipJSON(hanziGz, &h.entries); err != nil {
h.err = fmt.Errorf("load hanzi: %w", err)
}
})
}
// maxHanziChars caps the per-character fallback. A run longer than this is
// almost certainly a phrase the segmenter split badly rather than a word, and
// spelling out eight characters one at a time is a wall, not a hint.
const maxHanziChars = 6
// Hanzi returns the pinyin and English senses of a Chinese word.
//
// There is no de-inflection walk here, and its absence is a fact about the
// language rather than an omission: Chinese words do not inflect, so the
// candidate forms [lookupGloss] tries for "running" → "run" have no analogue.
// A lookup either hits the headword or it does not.
//
// What it does instead is fall back to the characters. The segmentation word
// list is a superset of this dictionary — every glossable word can be
// segmented, but jieba knows ordinary compounds CC-CEDICT has no entry for — so
// a hover really can land on a word with nothing to say about it. Chinese
// compounds are usually transparent from their parts (电脑 is "electric brain"),
// which makes the character reading a genuinely useful second answer rather
// than a consolation prize. It is returned as its own field so the surface can
// say which of the two it is showing; a caller that only wants whole words can
// ignore it.
func (l *Lexicon) Hanzi(word string) (HanziResult, error) {
hanzi.load()
if hanzi.err != nil {
return HanziResult{}, hanzi.err
}
norm := strings.TrimSpace(word)
res := HanziResult{Word: word, Readings: []HanziReading{}, Chars: []HanziChar{}}
if norm == "" {
return res, nil
}
if rows, ok := hanzi.entries[norm]; ok {
res.Readings = toReadings(rows)
return res, nil
}
chars := []rune(norm)
if len(chars) < 2 || len(chars) > maxHanziChars {
// A single character that missed has no parts to fall back to, and a long
// run is not a word. Either way the honest answer is nothing.
return res, nil
}
for _, r := range chars {
if !unicode.Is(unicode.Han, r) {
// Mixed input (a stray letter or digit inside the run) is not something
// the character reading can explain, and guessing at the hanzi parts of
// it would be worse than silence.
return HanziResult{Word: word, Readings: []HanziReading{}, Chars: []HanziChar{}}, nil
}
rows, ok := hanzi.entries[string(r)]
if !ok {
continue
}
first := toReadings(rows)
if len(first) == 0 {
continue
}
res.Chars = append(res.Chars, HanziChar{
Char: string(r),
Pinyin: first[0].Pinyin,
Senses: first[0].Senses,
})
}
return res, nil
}
func toReadings(rows [][]string) []HanziReading {
out := make([]HanziReading, 0, len(rows))
for _, row := range rows {
if len(row) < 2 {
continue
}
out = append(out, HanziReading{Pinyin: row[0], Senses: row[1]})
}
return out
}
+150
View File
@@ -0,0 +1,150 @@
package lexicon
import (
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/go-chi/chi/v5"
)
// The Chinese direction of the lexicon, against the real embedded asset — not a
// fixture. The dataset is built by scripts/build_cedict.py, which asserts its
// own invariants at build time; what these assert is that the *lookup* over it
// behaves, including on the entries the build script goes out of its way to keep.
func TestHanziLookup(t *testing.T) {
l := New()
res, err := l.Hanzi("公园")
if err != nil {
t.Fatalf("lookup 公园: %v", err)
}
if len(res.Readings) == 0 {
t.Fatal("公园 has no readings")
}
// Tone marks, not the numbered pinyin CC-CEDICT stores. The number is the
// storage format; the marks are what a learner reads.
if got := res.Readings[0].Pinyin; got != "gōngyuán" {
t.Errorf("公园 pinyin = %q, want gōngyuán", got)
}
if !strings.Contains(res.Readings[0].Senses, "park") {
t.Errorf("公园 senses = %q, want something about a park", res.Readings[0].Senses)
}
// A word answered whole says nothing about its characters — the fallback is
// the other branch, and sending both would double the payload of the common
// case to no purpose.
if len(res.Chars) != 0 {
t.Errorf("a whole-word hit also returned %d characters", len(res.Chars))
}
}
// 得 is the reason readings are a list. Answered with only dé "to obtain", a
// learner hovering it in 说得很好 has been told something false about the
// sentence they are looking at.
func TestHanziParticleCarriesItsGrammaticalReading(t *testing.T) {
l := New()
for _, particle := range []string{"的", "地", "得"} {
res, err := l.Hanzi(particle)
if err != nil {
t.Fatalf("lookup %s: %v", particle, err)
}
var found bool
for _, r := range res.Readings {
if r.Pinyin == "de" {
found = true
}
}
if !found {
t.Errorf("%s never reads as neutral \"de\": %+v", particle, res.Readings)
}
}
}
// The fallback the segmentation gap makes necessary: jieba knows ordinary
// compounds CC-CEDICT has no headword for, so a hover can land on a real word
// with no entry. Chinese compounds are usually transparent from their parts, so
// the characters are a real second answer.
func TestHanziFallsBackToCharacters(t *testing.T) {
l := New()
// Constructed rather than borrowed from the corpus: a word that CC-CEDICT
// *does* carry would test the other branch, and which compounds it happens to
// omit is not something a test should pin.
const made = "猫书"
if _, ok := hanzi.entries[made]; ok {
t.Skipf("%s has become a real headword; pick another compound", made)
}
res, err := l.Hanzi(made)
if err != nil {
t.Fatalf("lookup %s: %v", made, err)
}
if len(res.Readings) != 0 {
t.Fatalf("%s answered as a whole word: %+v", made, res.Readings)
}
if len(res.Chars) != 2 {
t.Fatalf("character fallback gave %d entries, want 2: %+v", len(res.Chars), res.Chars)
}
if res.Chars[0].Char != "猫" || !strings.Contains(res.Chars[0].Senses, "cat") {
t.Errorf("first character = %+v, want 猫 ~ cat", res.Chars[0])
}
if res.Chars[0].Pinyin != "māo" {
t.Errorf("猫 pinyin = %q, want māo", res.Chars[0].Pinyin)
}
}
func TestHanziMisses(t *testing.T) {
l := New()
for name, word := range map[string]string{
// A single character with no entry has no parts to fall back to.
"lone unknown character": "龥",
"empty": "",
"whitespace": " ",
// Not Chinese at all: the English tokenizer owns these, and answering
// would mean guessing.
"english": "hello",
"mixed": "猫cat",
// Longer than a word: a bad segmentation, not something to spell out
// character by character.
"a whole clause": "我今天早上去公园跑步了",
} {
res, err := l.Hanzi(word)
if err != nil {
t.Fatalf("%s: %v", name, err)
}
if len(res.Readings) != 0 || len(res.Chars) != 0 {
t.Errorf("%s (%q) answered with %+v / %+v", name, word, res.Readings, res.Chars)
}
}
}
func TestHanziEndpoint(t *testing.T) {
h := NewHandler(nil, NewSet(nil))
r := chi.NewRouter()
r.Mount("/hanzi", h.HanziRoutes())
w := httptest.NewRecorder()
r.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/hanzi/"+"跑步", nil))
if w.Code != http.StatusOK {
t.Fatalf("status = %d", w.Code)
}
var got HanziResult
if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil {
t.Fatalf("decode: %v", err)
}
if got.Word != "跑步" || len(got.Readings) == 0 || got.Readings[0].Pinyin != "pǎobù" {
t.Fatalf("response = %+v", got)
}
// A miss is a 200 with empty lists, like the other two lookups — the tooltip
// quietly doesn't open rather than showing an error over her writing.
w = httptest.NewRecorder()
r.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/hanzi/zzz", nil))
if w.Code != http.StatusOK {
t.Fatalf("miss: status = %d, want 200", w.Code)
}
}
+10
View File
@@ -109,3 +109,13 @@ func (g glossless) Lookup(word string) (Result, error) {
func (g glossless) Gloss(word string) (GlossResult, error) {
return GlossResult{Word: word}, nil
}
// Hanzi answers a Chinese-word lookup from the embedded CC-CEDICT map.
//
// It is on the Set rather than on [Provider] because it is not the same
// question the other two ask. Lookup and Gloss vary by pair — which is why they
// are behind an interface with two implementations — while this one is asked of
// Chinese or not at all: the learner direction exists for exactly one pair (see
// auth.learnerPairs), and DreamDict's own CC-CEDICT would be a second copy of
// the same dictionary, chosen by a rule with one branch.
func (s *Set) Hanzi(word string) (HanziResult, error) { return s.embedded.Hanzi(word) }
+37
View File
@@ -83,3 +83,40 @@ func TestDefaultPairStillReadsAsBefore(t *testing.T) {
t.Fatalf("zh ask-petal lost its Mandarin \"why\":\n%s", got)
}
}
// UX item 6: the Ask Petal answer is bilingual, pair language first, halves
// separated by one blank line. That separator is not a stylistic preference —
// AskPetal.tsx splits on it to render the two halves the way the companion
// renders its two lines — so the instruction has to survive prompt edits.
//
// The direction the writer is learning in is deliberately not encoded: the pair
// is (English + X), and an English speaker learning French needs the same two
// halves a Mandarin speaker learning English does. The prompt asks for both and
// lets the reader choose, so there is nothing here that names one half the
// answer and the other a courtesy.
func TestAskPetalAnswersInBothLanguages(t *testing.T) {
for _, code := range []string{"zh", "pt-PT", "fr", "es"} {
lang := LangFor(code)
ask := AskPetalSystemPrompt("a", "b", "grammar", "d", "e", lang)
if !strings.Contains(ask, "BOTH languages") {
t.Fatalf("%s: ask-petal no longer asks for both languages:\n%s", code, ask)
}
if !strings.Contains(ask, "single blank line") {
t.Fatalf("%s: ask-petal lost the blank-line separator the client splits on:\n%s", code, ask)
}
// Order matters to the rendering: the pair language is the prominent
// half, English the muted one beneath it.
if !strings.Contains(ask, "first the whole answer in "+lang.Name) {
t.Fatalf("%s: ask-petal doesn't put %s first:\n%s", code, lang.Name, ask)
}
// The instruction it replaced. Left in place it directly contradicts the
// new one, and a model given both will pick one at random.
if strings.Contains(ask, "Never mix languages") {
t.Fatalf("%s: ask-petal still forbids the bilingual reply it now asks for:\n%s", code, ask)
}
if strings.Contains(ask, "%!") {
t.Fatalf("%s: ask-petal prompt has a formatting error:\n%s", code, ask)
}
}
}
+29 -4
View File
@@ -144,6 +144,25 @@ func CollocationMessages(contentText, tone string, lang Lang) []Message {
// askPetalSystemTemplate is the Ask Petal tutor prompt. The suggestion context
// is interpolated in; the user's own messages are appended after this system
// turn by the caller.
//
// The reply is bilingual, the pair language first. Until UX item 6 it mirrored
// the language of the question instead — self-consistent, but it meant asking in
// one language cost you the other, and the writer doesn't always know which one
// the answer will be clearer in. Which half is the safety net and which is the
// lesson depends on who is writing: the pair is (English + X) either way, and an
// English speaker learning French wants the French half for the same reason a
// Mandarin speaker learning English wants the English one. Petal cannot tell
// them apart from a chat message, and doesn't need to — every other explanation
// surface already gives both (the card's English body, the seeded bubble in the
// pair language). The answer that goes deepest into the "why" was the one place
// that didn't.
//
// The blank line between the halves is a contract with the client: AskPetal.tsx
// splits on the first one to render her language prominently and the English
// beneath it, mirroring the companion's bubble. A model that ignores the
// instruction and writes one language degrades to a single plain block — the
// answer is still readable, which is why the split is a rendering nicety and
// never a parse the reply depends on.
const askPetalSystemTemplate = `You are Petal, a warm and patient English writing tutor helping someone who is learning English ` +
`as a second language. You are currently discussing a specific writing suggestion.
@@ -154,15 +173,21 @@ Suggestion context:
- Initial explanation: "%[4]s"
- Surrounding paragraph: "%[5]s"
The user wants to understand this suggestion better. Detect the language of the user's message ` +
`and respond in that same language. If they write in %[6]s, respond entirely in ` +
`%[6]s. If they write in English, respond in English. Never mix languages in a single response.
The user wants to understand this suggestion better. Answer in BOTH languages, every time, ` +
`whichever language they asked their question in: first the whole answer in %[6]s, then the ` +
`same answer again in English. Separate the two with a single blank line. Do not label them, ` +
`do not use a blank line anywhere else, and do not mix the two languages within one half — ` +
`each half is complete on its own.
One of those two languages is the one they are surest in and the other is the one they are ` +
`working in — you do not know which way round, so give both and let them choose. Both halves ` +
`say the same thing: do not put a point in one that is missing from the other.
Explain clearly and kindly. Use simple language appropriate to the user's message. Give examples ` +
`when helpful. If they ask "why" (or "%[7]s"), explain the grammar rule or idiom behind it. ` +
`If they suggest an alternative phrasing, evaluate it honestly.
Keep responses concise (2-4 sentences). This is a chat, not an essay. Be encouraging — ` +
Keep each half concise (2-3 sentences). This is a chat, not an essay. Be encouraging — ` +
`learning a language is hard and they're doing great.`
// AskPetalSystemPrompt fills the tutor prompt with one suggestion's context and
+1 -1
View File
@@ -75,7 +75,7 @@ func (h *Handler) chat(w http.ResponseWriter, r *http.Request) {
if err != nil {
// The stream never opened (e.g. LLM unreachable) — a normal JSON error is
// still appropriate since we haven't written SSE headers yet.
httputil.ErrorJSON(w, http.StatusBadGateway, "chat failed: "+err.Error())
httputil.UpstreamError(w, "chat", err)
return
}
+145
View File
@@ -0,0 +1,145 @@
package suggestions
import (
"crypto/sha256"
"encoding/hex"
"strings"
"unicode"
)
// Chunking splits a document into sentence-sized units so a re-check can ask the
// model only about the sentences that actually changed. Accepting one edit used
// to re-run the whole document: every card vanished, came back with a new id and
// a freshly-worded explanation, and spans re-merged into different shapes. The
// sentences she didn't touch have nothing new to say about themselves, so their
// suggestions are simply kept (see reconcilePending).
//
// A chunk's identity is its hash, not its position — she inserts a paragraph at
// the top and every sentence below keeps its suggestions.
// chunk is one sentence of the document, with the hash that identifies it.
type chunk struct {
text string
hash string
}
// asciiTerminators end a sentence only when whitespace (or the end of the text)
// follows, so "3.5" and "Ms." don't split mid-word — a wrong split costs only a
// slightly smaller chunk, but a split inside a number would churn its hash on
// every keystroke around it.
const asciiTerminators = ".!?"
// cjkTerminators end a sentence outright: Chinese runs sentences together with
// no space after 。, and she writes in both languages in one document.
const cjkTerminators = "。!?"
// closers are swallowed into the sentence they close, so the quote mark travels
// with the sentence rather than opening the next one.
const closers = `)]}"'’”」』`
// splitChunks divides text into sentences, dropping whitespace-only runs.
// Newlines always break a chunk, so a list or a line of dialogue is its own unit.
//
// `salt` distinguishes two *readings* of the same sentence. The grammar
// checkpoint's advice depends on the document's tone — the same line gets
// different notes as an academic essay than as a journal entry — so switching
// tone must re-open every sentence rather than serve back advice written for the
// old register.
func splitChunks(text, salt string) []chunk {
var out []chunk
runes := []rune(text)
start := 0
add := func(end int) {
if s := string(runes[start:end]); strings.TrimSpace(s) != "" {
out = append(out, chunk{text: s, hash: hashChunk(s, salt)})
}
start = end
}
for i := 0; i < len(runes); i++ {
r := runes[i]
if r == '\n' {
add(i + 1)
continue
}
cjk := strings.ContainsRune(cjkTerminators, r)
if !cjk && !strings.ContainsRune(asciiTerminators, r) {
continue
}
// Swallow a run of terminators ("?!", "…") and any closing punctuation.
j := i + 1
for j < len(runes) && (strings.ContainsRune(asciiTerminators+cjkTerminators+closers, runes[j])) {
j++
}
if cjk || j >= len(runes) || unicode.IsSpace(runes[j]) {
add(j)
i = j - 1
}
}
if start < len(runes) {
add(len(runes))
}
return out
}
// hashChunk identifies a sentence by its content under the same normalization
// the suppression logic uses: quote style and whitespace runs churn constantly
// (the editor rewrites quotes as she types, a paragraph reflows) and none of
// that changes what the sentence says, so none of it should cost a re-check.
func hashChunk(s, salt string) string {
sum := sha256.Sum256([]byte(salt + "\x00" + normalizeForDedup(s)))
return hex.EncodeToString(sum[:])[:16]
}
// hashSet indexes chunks by hash — "is this sentence in the document?"
func hashSet(chunks []chunk) map[string]bool {
out := make(map[string]bool, len(chunks))
for _, c := range chunks {
out[c.hash] = true
}
return out
}
// changedChunks returns the chunks whose hash wasn't in the last checked set,
// in document order and deduplicated — a sentence repeated verbatim is one
// question, not two.
func changedChunks(chunks []chunk, checked map[string]bool) []chunk {
seen := make(map[string]bool, len(chunks))
var out []chunk
for _, c := range chunks {
if checked[c.hash] || seen[c.hash] {
continue
}
seen[c.hash] = true
out = append(out, c)
}
return out
}
// joinChunks renders a chunk set as the text to hand the model: one sentence per
// line, so two sentences pulled from opposite ends of the document don't read as
// one run-on.
func joinChunks(chunks []chunk) string {
parts := make([]string, 0, len(chunks))
for _, c := range chunks {
parts = append(parts, strings.TrimSpace(c.text))
}
return strings.Join(parts, "\n")
}
// chunkFor names the sentence a suggestion belongs to: the first chunk whose
// text contains the flagged span. Returns "" when the span straddles a sentence
// boundary or the model paraphrased what it quoted — such a row is re-examined
// on every pass rather than cached, which is the safe direction.
func chunkFor(original string, chunks []chunk) string {
o := normalizeForDedup(original)
if o == "" {
return ""
}
for _, c := range chunks {
if strings.Contains(normalizeForDedup(c.text), o) {
return c.hash
}
}
return ""
}
+118
View File
@@ -0,0 +1,118 @@
package suggestions
import (
"strings"
"testing"
)
func texts(chunks []chunk) []string {
out := make([]string, 0, len(chunks))
for _, c := range chunks {
out = append(out, strings.TrimSpace(c.text))
}
return out
}
func TestSplitChunks(t *testing.T) {
cases := []struct {
name string
in string
want []string
}{
{
name: "plain sentences",
in: "I has two apple. She go to market yesterday! Why?",
want: []string{"I has two apple.", "She go to market yesterday!", "Why?"},
},
{
// A decimal must not split, or the sentence's identity would churn
// while she types the number.
name: "decimals stay whole",
in: "It costs 3.50 today. Tomorrow, more.",
want: []string{"It costs 3.50 today.", "Tomorrow, more."},
},
{
name: "closing quote travels with its sentence",
in: `He said "early," and left. She stayed.`,
want: []string{`He said "early," and left.`, "She stayed."},
},
{
// Chinese runs sentences together with no space after 。 — she writes
// in both languages in one document.
name: "cjk terminators split without a space",
in: "我想说这句话。但是不知道用英语怎么说。",
want: []string{"我想说这句话。", "但是不知道用英语怎么说。"},
},
{
name: "newlines break chunks",
in: "A list item\nAnother item\n",
want: []string{"A list item", "Another item"},
},
{
name: "blank runs are dropped",
in: "\n\n \nOnly this.\n\n",
want: []string{"Only this."},
},
{
name: "trailing fragment is its own chunk",
in: "Done. Still writing",
want: []string{"Done.", "Still writing"},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := texts(splitChunks(tc.in, ""))
if len(got) != len(tc.want) {
t.Fatalf("want %q, got %q", tc.want, got)
}
for i := range got {
if got[i] != tc.want[i] {
t.Fatalf("chunk %d: want %q, got %q", i, tc.want[i], got[i])
}
}
})
}
}
// A sentence's identity survives the churn that doesn't change what it says:
// the editor rewrites quotes as she types, and a paragraph reflows.
func TestChunkIdentityIgnoresCosmeticChurn(t *testing.T) {
a := splitChunks(`She said "hello" softly.`, "")
b := splitChunks("She said “hello” softly.", "")
if len(a) != 1 || len(b) != 1 {
t.Fatalf("want one chunk each, got %d and %d", len(a), len(b))
}
if a[0].hash != b[0].hash {
t.Fatalf("quote/whitespace churn changed the sentence's identity")
}
if same := splitChunks(`She said "hello" softly.`, "academic"); same[0].hash == a[0].hash {
t.Fatalf("a different tone must be a different reading of the sentence")
}
}
func TestChangedChunksAndLookup(t *testing.T) {
chunks := splitChunks("One thing. Another thing. One thing.", "")
if len(chunks) != 3 {
t.Fatalf("want 3 chunks, got %d", len(chunks))
}
// A repeated sentence is one question, not two.
if got := changedChunks(chunks, nil); len(got) != 2 {
t.Fatalf("want 2 distinct changed chunks, got %d", len(got))
}
checked := hashSet(chunks[:1])
changed := changedChunks(chunks, checked)
if len(changed) != 1 || strings.TrimSpace(changed[0].text) != "Another thing." {
t.Fatalf("want only the unread sentence, got %q", texts(changed))
}
if chunkFor("Another", chunks) != chunks[1].hash {
t.Fatalf("span was attributed to the wrong sentence")
}
// A span the document doesn't contain has no sentence, so it is never cached.
if chunkFor("nowhere in here", chunks) != "" {
t.Fatalf("unanchorable span should have no chunk")
}
}
+203 -78
View File
@@ -56,6 +56,7 @@ func (h *Handler) RegisterDocRoutes(r chi.Router) {
r.Post("/{id}/collocation", h.collocation)
r.Post("/{id}/rewrite", h.rewrite)
r.Get("/{id}/suggestions", h.listForDoc)
r.Get("/{id}/settled", h.listSettled)
}
// Routes returns the router mounted at /api/suggestions for per-suggestion
@@ -160,16 +161,21 @@ func (h *Handler) mechanics(w http.ResponseWriter, r *http.Request) {
httputil.WriteJSON(w, http.StatusOK, out)
}
// replaceMechanics swaps the document's pending offline rows for the supplied
// findings in one transaction, leaving the LLM families and actioned rows
// untouched. Findings the user already accepted or dismissed are suppressed (the
// detector has no memory between runs), and malformed spans are skipped.
// replaceMechanics brings the document's pending offline rows in line with the
// supplied findings in one transaction, leaving the LLM families and actioned
// rows untouched. Findings the user already accepted or dismissed are suppressed
// (the detector has no memory between runs), and malformed spans are skipped.
//
// The DELETE is scoped by *source*, not by type: the rule pack owns both the
// mechanics family and its share of the collocation family, and every run is a
// full recompute of the document, so everything it wrote last time goes. Scoping
// by type instead would strand offline collocations the current text no longer
// warrants — the one row nobody would ever replace.
// A finding the detector still reports keeps its existing row — same id, same
// created_at — and only its offsets move. This pass fires 250 ms after a
// keystroke, so deleting and re-inserting the family would hand every card a new
// identity several times a sentence: the rail would remount, a card expanded for
// Ask Petal would collapse under her, and the arrival chime would re-fire.
//
// The scope is *source*, not type: the rule pack owns both the mechanics family
// and its share of the collocation family, and every run is a full recompute of
// the document. Scoping by type instead would strand offline collocations the
// current text no longer warrants — the one row nobody would ever replace.
func (h *Handler) replaceMechanics(docID string, findings []mechanicsFinding) error {
tx, err := h.DB.Begin()
if err != nil {
@@ -177,18 +183,18 @@ func (h *Handler) replaceMechanics(docID string, findings []mechanicsFinding) er
}
defer tx.Rollback()
if _, err := tx.Exec(
`DELETE FROM suggestions WHERE doc_id = ? AND status = ? AND source = ?`,
docID, db.SuggestionStatusPending, db.SuggestionSourceLocal,
); err != nil {
existing, err := loadPending(tx, docID, "source = '"+db.SuggestionSourceLocal+"'")
if err != nil {
return err
}
index := indexByEdit(existing)
sup, err := buildSuppressor(tx, docID)
if err != nil {
return err
}
kept := make(map[string]bool, len(existing))
for _, f := range findings {
if f.From < 0 || f.To <= f.From || strings.TrimSpace(f.Original) == "" {
continue // malformed span — the client re-anchors by string anyway
@@ -196,16 +202,34 @@ func (h *Handler) replaceMechanics(docID string, findings []mechanicsFinding) er
if sup.suppressed(f.Original, f.Replacement) {
continue
}
typ := localType(f.Type)
if row, ok := index.take(f.Original, f.Replacement, f.From); ok {
kept[row.id] = true
if err := reposition(tx, row, f.From, f.To, ""); err != nil {
return err
}
continue
}
if _, err := tx.Exec(
`INSERT INTO suggestions (doc_id, from_pos, to_pos, original, replacement, explanation, type, source)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
docID, f.From, f.To, f.Original, f.Replacement, f.Explanation,
localType(f.Type), db.SuggestionSourceLocal,
typ, db.SuggestionSourceLocal,
); err != nil {
return err
}
}
// Whatever the detector no longer reports, she has fixed.
for _, row := range existing {
if kept[row.id] {
continue
}
if _, err := tx.Exec(`DELETE FROM suggestions WHERE id = ?`, row.id); err != nil {
return err
}
}
return tx.Commit()
}
@@ -258,11 +282,64 @@ func (h *Handler) runPass(w http.ResponseWriter, r *http.Request, limiter *llm.R
return
}
// Nothing to analyze on an empty document — skip the LLM round-trip.
// Nothing to analyze on an empty document — skip the LLM round-trip. The
// family's rows go with the text they were about.
if strings.TrimSpace(contentText) == "" {
httputil.WriteJSON(w, http.StatusOK, []db.Suggestion{})
if err := h.reconcilePending(docID, contentText, pairLang, nil, scope, nil, nil, false); err != nil {
httputil.ServerError(w, err)
return
}
out, err := h.fetchPending(userID, docID)
if err != nil {
httputil.ServerError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, out)
return
}
// Decide what to ask about before spending anything: a chunked pass asks only
// about the sentences that changed since it last read the document, and when
// none did it doesn't call the model at all — nor consume its rate-limit slot,
// so the next real edit isn't throttled by a check that had nothing to do.
//
// Only a chunked pass consults that record, so only it needs the tone folded
// into a sentence's identity.
salt := ""
if scope.chunked {
salt = tone
}
chunks := splitChunks(contentText, salt)
askText, fresh := contentText, chunks
if scope.chunked {
checked, err := h.checkedChunks(docID, scope.family)
if err != nil {
httputil.ServerError(w, err)
return
}
changed := changedChunks(chunks, checked)
if len(changed) == 0 {
// Every sentence has already been read. Drop the rows whose sentence is
// gone, keep the rest exactly as they are, and answer immediately.
if err := h.reconcilePending(docID, contentText, pairLang, nil, scope, chunks, nil, false); err != nil {
httputil.ServerError(w, err)
return
}
out, err := h.fetchPending(userID, docID)
if err != nil {
httputil.ServerError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, out)
return
}
// When every sentence is new — a first pass, a paste, a tone switch — hand
// over the document verbatim so the model reads it with its paragraphing
// intact. Otherwise send just the delta, one sentence per line.
if len(changed) < len(hashSet(chunks)) {
askText, fresh = joinChunks(changed), changed
}
}
ok, _, slotAt := limiter.Allow(docID)
if !ok {
@@ -277,17 +354,19 @@ func (h *Handler) runPass(w http.ResponseWriter, r *http.Request, limiter *llm.R
return
}
raw, err := run(r.Context(), h.Client, contentText, tone, llm.LangFor(pairLang))
raw, err := run(r.Context(), h.Client, askText, tone, llm.LangFor(pairLang))
if err != nil {
// Allow ran before the model call, so a failed pass would otherwise hold
// the per-document slot for the full interval — stranding the frontend's
// auto-retry on the throttle path. Release it so a retry can re-run.
limiter.Release(docID, slotAt)
httputil.ErrorJSON(w, http.StatusBadGateway, "llm pass failed: "+err.Error())
httputil.UpstreamError(w, "pass", err)
return
}
if err := h.replacePending(docID, contentText, raw, scope); err != nil {
// A whole-document pass re-read everything, so every one of its rows is up for
// re-proposal; a chunked pass only puts the sentences it asked about in play.
if err := h.reconcilePending(docID, contentText, pairLang, raw, scope, chunks, fresh, !scope.chunked); err != nil {
httputil.ServerError(w, err)
return
}
@@ -308,78 +387,49 @@ func (h *Handler) runPass(w http.ResponseWriter, r *http.Request, limiter *llm.R
// inserts. The grammar checkpoint and voice pass each own a disjoint family, so
// running one never disturbs the other's pending flags.
type pendingScope struct {
deleteWhere string // extra WHERE clause scoping the DELETE to this family
deleteWhere string // extra WHERE clause scoping this pass to its own family
forceType string // if set, every inserted row gets this type; else normalizeType
// family keys the sentences this pass has already read (see checked_chunks).
family string
// chunked passes re-read only the sentences that changed. True for the typing-
// cadence grammar checkpoint, which fires constantly and must feel still;
// false for the explicit whole-document passes, where she pressed a button
// asking for a fresh read of everything.
chunked bool
}
// Every scope below is confined to source='llm'. The offline rule pack replaces
// its own rows wholesale on each edit (see replaceMechanics) and its findings
// Every scope below is confined to source='llm'. The offline rule pack owns its
// own rows and recomputes them on each edit (see replaceMechanics); its findings
// must survive all three model passes — including the collocation coach, which
// now shares the collocation family with it.
var (
// grammarScope owns the grammar/phrasing/idiom/clarity flags — everything but
// the other self-owned families (voice, collocation), which run on their own
// cadence/pass and must survive a grammar checkpoint. Notably the offline pass
// writes its rows in the same /check request just before this DELETE runs, so
// the source clause is also what keeps them alive.
grammarScope = pendingScope{deleteWhere: "source = 'llm' AND type NOT IN ('voice','collocation')", forceType: ""}
// voiceScope owns the model's voice flags only.
voiceScope = pendingScope{deleteWhere: "source = 'llm' AND type = 'voice'", forceType: db.SuggestionTypeVoice}
// writes its rows in the same /check request just before this pass reconciles,
// so the source clause is also what keeps them alive.
grammarScope = pendingScope{
deleteWhere: "source = 'llm' AND type NOT IN ('voice','collocation')",
family: "grammar",
chunked: true,
}
// voiceScope owns the model's voice flags only. Voice is a property of the
// document as a whole — a sentence isn't inconsistent with itself — so this
// pass always reads everything.
voiceScope = pendingScope{
deleteWhere: "source = 'llm' AND type = 'voice'",
forceType: db.SuggestionTypeVoice,
family: "voice",
}
// collocationScope owns the model's collocation flags only — the rule pack's
// share of the same family is left standing.
collocationScope = pendingScope{deleteWhere: "source = 'llm' AND type = 'collocation'", forceType: db.SuggestionTypeCollocation}
collocationScope = pendingScope{
deleteWhere: "source = 'llm' AND type = 'collocation'",
forceType: db.SuggestionTypeCollocation,
family: "collocation",
}
)
// replacePending swaps a document's pending suggestions within one family for a
// fresh batch in a single transaction. Accepted/rejected suggestions and the
// other family's pending rows are left untouched.
//
// Suggestions touching a sentence the user already settled are suppressed from
// the fresh batch (see suppressor): not just the identical edit re-proposed, but
// reversals and re-polishing of the model's own just-accepted output — the
// "fickle, keeps going back and forth on a few sentences" behavior. The model has
// no memory between passes, so without this it re-opens resolved sentences every
// checkpoint.
func (h *Handler) replacePending(docID, contentText string, raw []llm.RawSuggestion, scope pendingScope) error {
tx, err := h.DB.Begin()
if err != nil {
return err
}
defer tx.Rollback()
if _, err := tx.Exec(
`DELETE FROM suggestions WHERE doc_id = ? AND status = ? AND `+scope.deleteWhere,
docID, db.SuggestionStatusPending,
); err != nil {
return err
}
sup, err := buildSuppressor(tx, docID)
if err != nil {
return err
}
for _, s := range raw {
if sup.suppressed(s.Original, s.Replacement) {
continue
}
typ := scope.forceType
if typ == "" {
typ = normalizeType(s.Type)
}
from, to := locate(contentText, s.Original)
if _, err := tx.Exec(
`INSERT INTO suggestions (doc_id, from_pos, to_pos, original, replacement, explanation, type, source)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
docID, from, to, s.Original, s.Replacement, s.Explanation, typ, db.SuggestionSourceLLM,
); err != nil {
return err
}
}
return tx.Commit()
}
// dedupQuoteReplacer folds every straight/curly single- and double-quote variant
// (and backtick/acute accent) onto one canonical character. The editor and the
// model both rewrite quotes between passes — a sentence accepted with "…" comes
@@ -518,6 +568,76 @@ func (h *Handler) listForDoc(w http.ResponseWriter, r *http.Request) {
httputil.WriteJSON(w, http.StatusOK, out)
}
// listSettled returns the normalized originals of every edit the user has
// already accepted or dismissed on this document — the same spans buildSuppressor
// drops on the server, handed to the client so its instant rule-pack pass can
// drop them too.
//
// Without this the offline half of the loop has no memory. The rule pack detects
// from the text alone and re-runs 250 ms after a keystroke, so a dismissed "the
// the" comes straight back the moment she types anywhere in the document; the
// server's reply then removes it again. That flicker is the visible symptom, but
// the real one is worse: with the server unreachable — the case the rule pack
// exists for — the reply never comes and a card she dismissed simply stays.
//
// Only the originals are sent. Replacements are the model's words, not hers, and
// the client only needs to answer "has she settled this span?"
func (h *Handler) listSettled(w http.ResponseWriter, r *http.Request) {
out, err := h.fetchSettled(auth.UserID(r.Context()), chi.URLParam(r, "id"))
if err != nil {
httputil.ServerError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, settledResponse{Originals: out})
}
// settledResponse wraps the list so the endpoint can grow a second field without
// breaking a client that reads a bare array.
type settledResponse struct {
Originals []string `json:"originals"`
}
// fetchSettled loads the distinct normalized originals of the document's actioned
// rows. Scoped through documents for the same reason fetchPending is: an original
// is a quotation of her writing.
func (h *Handler) fetchSettled(userID, docID string) ([]string, error) {
rows, err := h.DB.Query(
`SELECT DISTINCT s.original
FROM suggestions s
JOIN documents d ON d.id = s.doc_id
WHERE s.doc_id = ? AND d.user_id = ? AND s.status IN (?, ?)`,
docID, userID, db.SuggestionStatusAccepted, db.SuggestionStatusRejected,
)
if err != nil {
return nil, err
}
defer rows.Close()
// DISTINCT is on the raw text; normalizing can collapse two rows into one, so
// dedupe again on this side to keep the payload honest.
seen := map[string]struct{}{}
out := []string{}
for rows.Next() {
var original string
if err := rows.Scan(&original); err != nil {
return nil, err
}
norm := normalizeForDedup(original)
if norm == "" {
continue
}
if _, dup := seen[norm]; dup {
continue
}
seen[norm] = struct{}{}
out = append(out, norm)
}
if err := rows.Err(); err != nil {
return nil, err
}
return out, nil
}
// fetchPending loads a document's pending suggestions, joined through documents
// so the rows are only reachable by the document's owner. A suggestion quotes the
// sentence it corrects, so an unscoped read here would leak document text to
@@ -707,6 +827,11 @@ func locate(contentText, original string) (int, int) {
// normalizeType maps the model's type string onto a valid suggestion type,
// defaulting unknown values to grammar so a stray label never trips the CHECK.
//
// 'translate' is absent on purpose, and stays absent even though the type now
// exists: it is decided from the span (see language.go), never taken from the
// model. A model that volunteers the label anyway lands on grammar here and is
// then promoted — or not — on the evidence.
func normalizeType(t string) string {
switch strings.ToLower(strings.TrimSpace(t)) {
case db.SuggestionTypeGrammar, db.SuggestionTypePhrasing, db.SuggestionTypeIdiom, db.SuggestionTypeClarity, db.SuggestionTypeCollocation:
+27 -2
View File
@@ -20,10 +20,17 @@ import (
type stubClient struct {
response string
calls int
// The full prompt of the most recent call, so a test can assert which
// sentences a chunked pass actually asked about.
lastPrompt string
}
func (s *stubClient) Complete(_ context.Context, _ llm.CompletionRequest) (string, error) {
func (s *stubClient) Complete(_ context.Context, req llm.CompletionRequest) (string, error) {
s.calls++
s.lastPrompt = ""
for _, m := range req.Messages {
s.lastPrompt += m.Content + "\n"
}
return s.response, nil
}
@@ -64,6 +71,17 @@ func newTestServer(t *testing.T, client llm.LLMClient) (http.Handler, string, *H
return authed, docID, h
}
// setDocText rewrites the seeded document, standing in for the writer editing.
// The grammar checkpoint only asks the model about sentences that changed since
// it last read the document, so a test that wants a second real pass has to
// change something first — as she always has.
func setDocText(t *testing.T, h *Handler, docID, text string) {
t.Helper()
if _, err := h.DB.Exec(`UPDATE documents SET content_text = ? WHERE id = ?`, text, docID); err != nil {
t.Fatalf("update doc text: %v", err)
}
}
func do(t *testing.T, srv http.Handler, method, path, body string) *httptest.ResponseRecorder {
t.Helper()
var r *http.Request
@@ -183,6 +201,7 @@ func TestFickleEditsSuppressed(t *testing.T) {
]}`}
srv, docID, h := newTestServer(t, client)
h.Limit = llm.NewRateLimiter(0)
setDocText(t, h, docID, `He left "early," because of the rain. The cat always have a calm face.`)
rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "")
var got []db.Suggestion
@@ -194,6 +213,10 @@ func TestFickleEditsSuppressed(t *testing.T) {
do(t, srv, http.MethodPost, "/suggestions/"+s.ID+"/accept", "")
}
// Both edits are now in the document, which is what re-opens those sentences
// for a second reading.
setDocText(t, h, docID, `He left "early," due to the rain. The cat always has a calm face.`)
// Reversal of the first accept (note the " → ' quote churn) and a re-polish of
// the second accept must both be dropped; only the unrelated edit survives.
client.response = `{"suggestions":[
@@ -314,7 +337,9 @@ func TestCollocationPassCoexists(t *testing.T) {
t.Fatalf("collocation response should carry all three families, got %+v", got)
}
// A grammar checkpoint must NOT wipe the voice or collocation flags.
// A grammar checkpoint must NOT wipe the voice or collocation flags. She fixes
// the flagged sentence, so its own grammar row goes and nothing replaces it.
setDocText(t, h, docID, "I have two apples.")
client.response = `{"suggestions":[]}`
rec = do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "")
if rec.Code != http.StatusOK {
+10
View File
@@ -116,4 +116,14 @@ func TestSuggestionIsolation(t *testing.T) {
if rec.Code != http.StatusNoContent {
t.Fatalf("owner accept = %d, want 204 (body: %s)", rec.Code, rec.Body)
}
// That accept created a settled span, which is the other read of this table.
// It carries originals only — but an original is a verbatim quotation of her
// sentence, so it is the same leak as the pending list through a smaller hole.
if got := getSettled(t, owner, docID); len(got) != 1 {
t.Fatalf("owner should see their own settled span, got %v", got)
}
if got := getSettled(t, stranger, docID); len(got) != 0 {
t.Fatalf("stranger read %d settled span(s) (leaking %q)", len(got), got[0])
}
}
+174
View File
@@ -0,0 +1,174 @@
package suggestions
import (
"strings"
"unicode"
)
// Telling her language from English, well enough to label a card.
//
// When the checkpoint quotes a span she wrote in her own language and hands back
// an English rendering, that is not a correction — nothing was wrong with what
// she wrote — and it should not be filed under 'clarity'. The label is derived
// here rather than asked of the model: a type is structural, and a model that
// re-reasons every pass would drift between labels for the same sentence.
//
// The failure mode is deliberately cheap. Getting this wrong changes a card's
// coloured pill and nothing else — the replacement, the explanation and the
// Accept button are identical either way — so a heuristic is the right tool. It
// is written to under-claim: a span it isn't sure about stays whatever the model
// called it.
//
// The two pair families need genuinely different tests, and pretending otherwise
// would be the bug:
//
// - zh is a different script. Counting Han runes is close to certain.
// - pt-PT, fr and es share the Latin alphabet with English, where no such
// signal exists. Those fall back to function words — the short, extremely
// common words a sentence in that language can hardly avoid and an English
// sentence has no reason to contain.
// isTranslation reports whether this edit is her own language rendered into
// English, rather than a correction to her English. Both halves must hold: the
// quoted span reads as the pair language, and what Petal offers back reads as
// English. The second half matters — a Chinese span rewritten into different
// Chinese is something else entirely, and Petal has no business calling it a
// translation.
func isTranslation(original, replacement, pairLang string) bool {
if strings.TrimSpace(original) == "" || strings.TrimSpace(replacement) == "" {
return false
}
return readsAsPairLang(original, pairLang) && readsAsEnglish(replacement)
}
// readsAsPairLang reports whether s is predominantly in the writer's language.
func readsAsPairLang(s, pairLang string) bool {
switch normalizePairLang(pairLang) {
case "zh":
han, latin := scriptCounts(s)
// Predominantly, not merely partly: one Chinese word inside an English
// sentence is a vocabulary question, and the sentence around it is still
// English prose with its own grammar to correct. Two runes is the floor
// because a single Han character is as likely to be a stray keystroke.
return han >= 2 && han > latin
case "pt-PT", "fr", "es":
return distinctMarkers(s, latinMarkers[normalizePairLang(pairLang)]) >= 2
}
// A pair Petal has no test for. Say no: an unlabelled card is a card that
// reads as it did yesterday, and a wrongly-labelled one is a new defect.
return false
}
// readsAsEnglish reports whether s is English prose rather than more of her own
// language. It is not a language identifier — it only has to separate "English"
// from "the pair language", and it is only ever asked about text Petal itself
// generated, so the bar is low on purpose: Latin letters present, and not
// swamped by another script.
func readsAsEnglish(s string) bool {
han, latin := scriptCounts(s)
return latin > 0 && latin > han
}
// normalizePairLang folds the stored `users.pair_lang` into the codes below.
// Empty (a document whose owner has no pair recorded) falls through to no test.
func normalizePairLang(pairLang string) string {
switch p := strings.ToLower(strings.TrimSpace(pairLang)); p {
case "zh", "zh-cn", "zh-hans":
return "zh"
case "pt", "pt-pt":
return "pt-PT"
case "fr", "fr-fr":
return "fr"
case "es", "es-es":
return "es"
default:
return p
}
}
// scriptCounts counts Han runes and ASCII letters. Everything else — digits,
// punctuation, spaces, emoji — is ignored, so trailing 。or a stray comma
// changes nothing.
func scriptCounts(s string) (han, latin int) {
for _, r := range s {
switch {
case unicode.Is(unicode.Han, r):
han++
case r < unicode.MaxASCII && unicode.IsLetter(r):
latin++
}
}
return han, latin
}
// distinctMarkers counts how many *different* marker words appear in s. Distinct
// rather than total: "que ... que" is one writer's habit, while "eu quero" is two
// independent pieces of evidence.
func distinctMarkers(s string, markers map[string]bool) int {
if len(markers) == 0 {
return 0
}
seen := map[string]bool{}
for _, w := range strings.FieldsFunc(strings.ToLower(s), func(r rune) bool {
// Split on anything that isn't a letter, so punctuation and digits are
// separators. Apostrophes included: French elision (j'ai, n'est) should
// yield its parts.
return !unicode.IsLetter(r)
}) {
if markers[w] {
seen[w] = true
}
}
return len(seen)
}
// Function words that a sentence in each Latin pair can hardly avoid.
//
// Curated against English, not for coverage: every entry here is a word an
// English sentence has essentially no reason to contain, which is why the lists
// omit plenty of far more common words. Deliberately absent — each of them a
// false positive waiting to happen — is anything that is *also* an English word:
// the pan-Romance shorts (a, o, e, as, no, on, en, de, se, na, mi, son, era,
// plus, pour, si, ma, ce, ne), Portuguese "do", Spanish "con", "ya" and "todo".
// Dropping "con" costs the Spanish list one of its commonest words, and that is
// the right trade — a marker that fires on English corroborates the wrong
// answer, which is worse than a sentence Petal declines to label.
//
// A single marker is not enough (see readsAsPairLang), so these lists are read
// as evidence to be corroborated rather than as a decision.
var latinMarkers = map[string]map[string]bool{
"fr": words(
"je", "tu", "il", "elle", "ils", "elles", "nous", "vous", "est", "sont",
"était", "étais", "une", "des", "les", "du", "dans", "avec", "que", "qui",
"mais", "très", "être", "avoir", "pas", "cette", "cet", "ces", "mon",
"mes", "notre", "votre", "leur", "aussi", "alors", "parce", "comme",
"beaucoup", "toujours", "jamais", "quand", "bien", "chose", "temps",
"moi", "toi", "lui", "peux", "veux", "sais", "faire", "dit", "aujourd",
"hui", "quelque", "chez", "tout", "tous", "rien", "déjà", "encore",
),
"pt-PT": words(
"eu", "você", "ele", "ela", "eles", "elas", "nós", "são", "uma", "os",
"da", "dos", "das", "com", "que", "mas", "muito", "não", "meu",
"minha", "seu", "sua", "isso", "este", "esta", "está", "estou", "quero",
"também", "quando", "porque", "coisa", "tempo", "fazer", "sempre",
"nunca", "bem", "obrigado", "obrigada", "gosto", "tenho", "tem", "foi",
"ser", "ter", "mais", "já", "ainda", "aqui", "ali", "nada", "tudo",
"todos", "para", "pela", "pelo", "sobre", "assim",
),
"es": words(
"yo", "él", "ella", "ellos", "ellas", "nosotros", "una", "los", "las",
"del", "que", "pero", "muy", "esto", "esta", "este", "está",
"estoy", "quiero", "también", "cuando", "porque", "cosa", "tiempo",
"hacer", "siempre", "nunca", "bien", "gracias", "tengo", "tiene", "fue",
"ser", "tener", "más", "aquí", "allí", "nada", "todos",
"para", "sobre", "así", "hola", "señor", "usted", "muchas",
),
}
func words(list ...string) map[string]bool {
out := make(map[string]bool, len(list))
for _, w := range list {
out[w] = true
}
return out
}
+176
View File
@@ -0,0 +1,176 @@
package suggestions
import "testing"
// The flagship case, and the ones next to it that must NOT become translations.
func TestIsTranslation(t *testing.T) {
cases := []struct {
name string
original string
replacement string
pairLang string
want bool
}{
{
// The sentence from the UX review, verbatim.
name: "whole Chinese sentence rendered into English",
original: "我想说这句话但是不知道用英语怎么说。",
replacement: "I want to say this but I don't know how to say it in English.",
pairLang: "zh",
want: true,
},
{
name: "ordinary English correction is not a translation",
original: "She goes to market yesterday",
replacement: "She went to the market yesterday",
pairLang: "zh",
want: false,
},
{
// One Chinese word inside English prose. The sentence around it is
// still English with its own grammar to fix, and calling the card a
// translation would mislabel a grammar fix.
name: "single Chinese word inside an English sentence",
original: "I bought a 苹果 at the store",
replacement: "I bought an apple at the store",
pairLang: "zh",
want: false,
},
{
name: "a lone stray Han rune is not a sentence",
original: "的",
replacement: "of",
pairLang: "zh",
want: false,
},
{
// Chinese in, Chinese out: whatever this is, Petal is not translating.
name: "Chinese rewritten as Chinese",
original: "我想说这句话",
replacement: "我要说这句话",
pairLang: "zh",
want: false,
},
{
// The same Chinese span, but the writer is on the French pair. Petal
// has no business offering to translate a language she never claimed.
name: "Chinese span on a non-zh pair",
original: "我想说这句话但是不知道用英语怎么说。",
replacement: "I want to say this in English.",
pairLang: "fr",
want: false,
},
{
name: "French sentence rendered into English",
original: "Je ne sais pas comment le dire en anglais.",
replacement: "I don't know how to say it in English.",
pairLang: "fr",
want: true,
},
{
name: "Portuguese sentence rendered into English",
original: "Eu quero dizer isso mas não sei como.",
replacement: "I want to say this but I don't know how.",
pairLang: "pt-PT",
want: true,
},
{
name: "Spanish sentence rendered into English",
original: "Yo quiero decir esto pero no sé cómo.",
replacement: "I want to say this but I don't know how.",
pairLang: "es",
want: true,
},
{
// A single marker is not evidence. "Que" appears in English writing
// about other languages, in names, in quoted phrases.
name: "one Latin marker is not enough",
original: "The word que confused me",
replacement: "The word que confuses me",
pairLang: "pt-PT",
want: false,
},
{
// The words most likely to sink this heuristic: English function words
// that are also Romance function words. They are kept out of the lists
// precisely so this sentence stays a grammar fix.
name: "English full of pan-Romance lookalikes",
original: "I do not know if a con man on the plus side as no era",
replacement: "I do not know whether a con man, on the plus side, is no era",
pairLang: "es",
want: false,
},
{
name: "English with a borrowed French phrase stays English",
original: "It was a pas de deux, more or less",
replacement: "It was a pas de deux, more or less.",
pairLang: "fr",
want: false,
},
{
name: "empty replacement (an awareness-only finding)",
original: "我想说这句话但是不知道用英语怎么说。",
replacement: "",
pairLang: "zh",
want: false,
},
{
// A document whose owner has no pair recorded. No test, no label.
name: "no pair language",
original: "我想说这句话但是不知道用英语怎么说。",
replacement: "I want to say this in English.",
pairLang: "",
want: false,
},
{
// An unshipped pair. Same rule: decline rather than guess.
name: "unknown pair language",
original: "Ich weiß nicht wie man das sagt.",
replacement: "I don't know how to say that.",
pairLang: "de",
want: false,
},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
if got := isTranslation(c.original, c.replacement, c.pairLang); got != c.want {
t.Errorf("isTranslation(%q, %q, %q) = %v, want %v",
c.original, c.replacement, c.pairLang, got, c.want)
}
})
}
}
// pair_lang is stored as the pack code, but a stored value has drifted before
// (see the picker's history), so the fold is tested rather than assumed.
func TestNormalizePairLang(t *testing.T) {
for in, want := range map[string]string{
"zh": "zh", "zh-CN": "zh", "ZH": "zh",
"pt": "pt-PT", "pt-PT": "pt-PT", "pt-pt": "pt-PT",
"fr": "fr", "fr-FR": "fr",
"es": "es", "es-ES": "es",
" zh ": "zh",
"": "",
"de": "de",
} {
if got := normalizePairLang(in); got != want {
t.Errorf("normalizePairLang(%q) = %q, want %q", in, got, want)
}
}
}
// French elision must yield its parts, or "j'ai" and "n'est" — two of the
// commonest shapes in the language — count for nothing.
func TestElisionYieldsMarkers(t *testing.T) {
if n := distinctMarkers("Je n'est pas", latinMarkers["fr"]); n < 3 {
t.Errorf("elided French: got %d markers, want >= 3 (je, est, pas)", n)
}
}
// Distinct, not total: one word repeated is one piece of evidence.
func TestRepeatedMarkerCountsOnce(t *testing.T) {
if n := distinctMarkers("que que que", latinMarkers["pt-PT"]); n != 1 {
t.Errorf("repeated marker: got %d, want 1", n)
}
}
+33
View File
@@ -201,3 +201,36 @@ func TestOfflineCardWinsSpanCollision(t *testing.T) {
t.Errorf("the exact offline card should own the span, got %+v", got[0])
}
}
// TestOfflineHanziFindingStaysMechanics: a 错别字 the Chinese rule pack found —
// both halves written in hanzi — files as an ordinary mechanics row.
//
// The check is worth its own test because there is a rule one layer over that
// would plausibly claim it. `isTranslation` re-labels an edit whose original
// reads as the writer's language and whose replacement reads as English, which
// is exactly how a zh-pair writer's quoted Chinese becomes a 'translate' card.
// A wrong-character fix looks like the first half of that and nothing like the
// second: 己经 → 已经 never leaves Chinese. It must stay a tidy-up in her own
// sentence, on the same rail as a doubled word, with no rendering-into-English
// implied anywhere.
func TestOfflineHanziFindingStaysMechanics(t *testing.T) {
srv, docID, _ := newTestServer(t, &stubClient{response: `{"suggestions":[]}`})
got := postMechanics(t, srv, docID, `[
{"from":1,"to":3,"original":"己经","replacement":"已经","explanation":"已经 (already) takes 已","type":"mechanics"}
]`)
if len(got) != 1 {
t.Fatalf("want the one finding, got %+v", got)
}
if got[0].Type != db.SuggestionTypeMechanics {
t.Errorf("hanzi fix filed as %q, want %q", got[0].Type, db.SuggestionTypeMechanics)
}
if got[0].Source != db.SuggestionSourceLocal {
t.Errorf("source = %q, want %q", got[0].Source, db.SuggestionSourceLocal)
}
// The characters survive the round trip intact — a mangled span here would
// replace the wrong characters in her document.
if got[0].Original != "己经" || got[0].Replacement != "已经" {
t.Errorf("round-tripped as %q → %q", got[0].Original, got[0].Replacement)
}
}
+303
View File
@@ -0,0 +1,303 @@
package suggestions
import (
"database/sql"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/llm"
)
// Reconciliation replaces the old "delete the family, insert the new batch"
// shape of every pass. A suggestion the pass proposes again is the *same*
// suggestion: it keeps its row, and therefore its id, its created_at and — most
// visibly — the explanation it was first given. The model re-words its reasoning
// every time it is asked, so re-inserting meant one unchanged mistake carried
// three different explanations in a single sitting.
//
// The id is what the frontend keys its cards on, so a stable id is also what
// keeps the rail from emptying and refilling, a card from collapsing mid-read,
// and the arrival chime from re-firing for advice she has already seen.
// pendingRow is the part of an existing pending suggestion reconciliation cares
// about.
type pendingRow struct {
id string
original string
replacement string
chunkHash string
from int
}
// loadPending reads the pending rows a pass owns. `where` is the pass's own
// scoping clause (by source, and for the model passes by family) — the same
// fragment that used to scope its DELETE.
func loadPending(tx *sql.Tx, docID, where string) ([]pendingRow, error) {
rows, err := tx.Query(
`SELECT id, original, replacement, chunk_hash, from_pos FROM suggestions
WHERE doc_id = ? AND status = ? AND `+where,
docID, db.SuggestionStatusPending,
)
if err != nil {
return nil, err
}
defer rows.Close()
var out []pendingRow
for rows.Next() {
var r pendingRow
if err := rows.Scan(&r.id, &r.original, &r.replacement, &r.chunkHash, &r.from); err != nil {
return nil, err
}
out = append(out, r)
}
return out, rows.Err()
}
// editKey identifies an edit by what it proposes, not where: "this exact change
// to this exact text". Normalized like the suppression comparisons, so the
// editor's quote rewriting and a reflowed paragraph don't read as a new edit.
func editKey(original, replacement string) string {
return normalizeForDedup(original) + "\x00" + normalizeForDedup(replacement)
}
// editIndex matches freshly proposed edits against the rows already standing.
type editIndex struct {
rows []pendingRow
used []bool
byKey map[string][]int
}
func indexByEdit(rows []pendingRow) *editIndex {
idx := &editIndex{rows: rows, used: make([]bool, len(rows)), byKey: map[string][]int{}}
for i, r := range rows {
k := editKey(r.original, r.replacement)
idx.byKey[k] = append(idx.byKey[k], i)
}
return idx
}
// take claims the standing row for this edit, if there is one. When a document
// repeats the same mistake, `near` (the fresh span's start) picks the closest
// standing row, so two identical cards keep their own identities instead of
// trading them whenever the text between them grows.
func (i *editIndex) take(original, replacement string, near int) (pendingRow, bool) {
best, bestDist := -1, 0
for _, n := range i.byKey[editKey(original, replacement)] {
if i.used[n] {
continue
}
d := i.rows[n].from - near
if d < 0 {
d = -d
}
if best < 0 || d < bestDist {
best, bestDist = n, d
}
}
if best < 0 {
return pendingRow{}, false
}
i.used[best] = true
return i.rows[best], true
}
// reposition updates the advisory offsets (and the sentence a row belongs to)
// without touching anything the writer can see. The frontend re-anchors by
// string at render time, so these only matter for the local-vs-model span
// arbitration in dedupeSpans.
func reposition(tx *sql.Tx, row pendingRow, from, to int, chunkHash string) error {
if row.from == from && row.chunkHash == chunkHash {
return nil
}
_, err := tx.Exec(
`UPDATE suggestions SET from_pos = ?, to_pos = ?, chunk_hash = ? WHERE id = ?`,
from, to, chunkHash, row.id,
)
return err
}
// reconcilePending brings a model pass's family in line with what it just
// proposed, sentence by sentence:
//
// - A row on a sentence this pass didn't ask about is kept untouched — that
// is the whole point of chunking. Only its offsets are refreshed.
// - A row on a sentence that no longer exists in the document is dropped: she
// rewrote or deleted it.
// - A row on a sentence the pass *did* ask about survives only if the model
// proposed the same edit again, in which case it keeps its identity.
//
// `fresh` names the sentences the model was asked about (nil when it wasn't
// called at all). inPlayAll marks the whole-document passes — voice and the
// collocation coach — where every row is up for re-proposal because the model
// just re-read everything.
//
// `pairLang` is the writer's own language, needed only to type a finding that
// turns out to be her language rendered into English (see language.go).
func (h *Handler) reconcilePending(
docID, contentText, pairLang string,
raw []llm.RawSuggestion,
scope pendingScope,
chunks, fresh []chunk,
inPlayAll bool,
) error {
tx, err := h.DB.Begin()
if err != nil {
return err
}
defer tx.Rollback()
existing, err := loadPending(tx, docID, scope.deleteWhere)
if err != nil {
return err
}
present := hashSet(chunks)
asked := hashSet(fresh)
modelRan := inPlayAll || fresh != nil
// Sentences to hand back to the model next time, because a row we were
// caching on them turned out to be unanchorable (see below).
reopen := map[string]bool{}
var inPlay []pendingRow
for _, r := range existing {
switch {
// A row whose sentence we can't name is never cached — it is re-examined
// whenever the model speaks, and left alone when it doesn't.
case inPlayAll, r.chunkHash == "" && modelRan, asked[r.chunkHash]:
inPlay = append(inPlay, r)
case r.chunkHash != "" && !present[r.chunkHash]:
if _, err := tx.Exec(`DELETE FROM suggestions WHERE id = ?`, r.id); err != nil {
return err
}
default:
// Untouched sentence: keep the card exactly as she last saw it.
from, to := locate(contentText, r.original)
if from < 0 {
// The sentence is unchanged in substance but the quoted span no
// longer matches byte for byte — a quote mark the editor rewrote
// inside it, say. The frontend anchors by that string, so this card
// can't be shown; drop it and let the sentence be read again rather
// than cache advice nobody can see.
reopen[r.chunkHash] = true
if _, err := tx.Exec(`DELETE FROM suggestions WHERE id = ?`, r.id); err != nil {
return err
}
continue
}
if err := reposition(tx, r, from, to, r.chunkHash); err != nil {
return err
}
}
}
for h := range reopen {
delete(present, h)
}
sup, err := buildSuppressor(tx, docID)
if err != nil {
return err
}
index := indexByEdit(inPlay)
kept := make(map[string]bool, len(inPlay))
for _, s := range raw {
if sup.suppressed(s.Original, s.Replacement) {
continue
}
from, to := locate(contentText, s.Original)
// Attribute the finding to a sentence the model was actually shown before
// falling back to the whole document: a short span ("the the") can occur in
// two sentences, and crediting it to the cached one would drop it as advice
// we already have.
hash := chunkFor(s.Original, fresh)
if hash == "" {
hash = chunkFor(s.Original, chunks)
}
// A sentence we didn't ask about already has whatever advice it deserves.
// The model can't normally quote one — it was only shown the delta — but if
// it wanders there anyway, the cached card stands rather than gaining a
// twin.
if !inPlayAll && hash != "" && present[hash] && !asked[hash] {
continue
}
if row, ok := index.take(s.Original, s.Replacement, from); ok {
kept[row.id] = true
if err := reposition(tx, row, from, to, hash); err != nil {
return err
}
continue
}
// A pass with a forced type owns its family outright and is never asked
// about translation: voice reads whole paragraphs for tone, and the
// collocation coach is about English word pairings. Only the open-typed
// grammar checkpoint can turn out to have been handed her own language.
typ := scope.forceType
if typ == "" {
typ = normalizeType(s.Type)
if isTranslation(s.Original, s.Replacement, pairLang) {
typ = db.SuggestionTypeTranslate
}
}
if _, err := tx.Exec(
`INSERT INTO suggestions (doc_id, from_pos, to_pos, original, replacement, explanation, type, source, chunk_hash)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
docID, from, to, s.Original, s.Replacement, s.Explanation, typ, db.SuggestionSourceLLM, hash,
); err != nil {
return err
}
}
// Asked about and not proposed again: the model has changed its mind, or she
// has fixed it.
for _, r := range inPlay {
if kept[r.id] {
continue
}
if _, err := tx.Exec(`DELETE FROM suggestions WHERE id = ?`, r.id); err != nil {
return err
}
}
// Record the sentences this family has now read. Every sentence still in the
// document has been read by *some* pass: the ones just asked about now, the
// rest in an earlier round.
if scope.chunked {
if _, err := tx.Exec(
`DELETE FROM checked_chunks WHERE doc_id = ? AND family = ?`, docID, scope.family,
); err != nil {
return err
}
for h := range present {
if _, err := tx.Exec(
`INSERT INTO checked_chunks (doc_id, family, hash) VALUES (?, ?, ?)`,
docID, scope.family, h,
); err != nil {
return err
}
}
}
return tx.Commit()
}
// checkedChunks loads the sentences a family read on its last pass.
func (h *Handler) checkedChunks(docID, family string) (map[string]bool, error) {
rows, err := h.DB.Query(
`SELECT hash FROM checked_chunks WHERE doc_id = ? AND family = ?`, docID, family,
)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]bool{}
for rows.Next() {
var hash string
if err := rows.Scan(&hash); err != nil {
return nil, err
}
out[hash] = true
}
return out, rows.Err()
}
+1 -1
View File
@@ -69,7 +69,7 @@ func (h *Handler) rewrite(w http.ResponseWriter, r *http.Request) {
out, err := llm.RunRewrite(r.Context(), h.Client, text, body.Style)
if err != nil {
httputil.ErrorJSON(w, http.StatusBadGateway, "rewrite failed: "+err.Error())
httputil.UpstreamError(w, "rewrite", err)
return
}
+142
View File
@@ -0,0 +1,142 @@
package suggestions
import (
"encoding/json"
"net/http"
"testing"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// The settled endpoint exists for the offline half of the loop. The rule pack
// detects from the text alone, 250 ms after a keystroke, and has no memory
// between runs — so without the document's record of what she has already
// answered, a dismissed finding is re-detected and re-rendered on the next
// keystroke, and stays there for as long as the server can't be reached.
func getSettled(t *testing.T, srv http.Handler, docID string) []string {
t.Helper()
rec := do(t, srv, http.MethodGet, "/docs/"+docID+"/settled", "")
if rec.Code != http.StatusOK {
t.Fatalf("settled: code=%d body=%s", rec.Code, rec.Body)
}
var out struct {
Originals []string `json:"originals"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatalf("decode: %v", err)
}
return out.Originals
}
func contains(list []string, want string) bool {
for _, s := range list {
if s == want {
return true
}
}
return false
}
// TestSettledListsActionedSpans proves the endpoint reports exactly the spans the
// suppressor would drop: accepted and dismissed, never pending. A pending row
// leaking in would be the damaging direction — the client would hide a card she
// has never been shown an answer to.
func TestSettledListsActionedSpans(t *testing.T) {
client := &stubClient{response: `{"suggestions":[
{"original":"I has","replacement":"I have","explanation":"agreement","type":"grammar"},
{"original":"two apple","replacement":"two apples","explanation":"plural","type":"grammar"}
]}`}
srv, docID, _ := newTestServer(t, client)
if got := getSettled(t, srv, docID); len(got) != 0 {
t.Fatalf("nothing actioned yet, got %v", got)
}
rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "")
var got []db.Suggestion
_ = json.Unmarshal(rec.Body.Bytes(), &got)
if len(got) != 2 {
t.Fatalf("first pass: want 2, got %d", len(got))
}
// One accepted, one still pending: only the accepted span is settled.
do(t, srv, http.MethodPost, "/suggestions/"+got[0].ID+"/accept", "")
settled := getSettled(t, srv, docID)
if len(settled) != 1 || settled[0] != got[0].Original {
t.Fatalf("want just %q settled, got %v", got[0].Original, settled)
}
// A dismissal settles a span just as an accept does — the whole point of the
// item: "you already decided about this one" doesn't mean "you agreed".
do(t, srv, http.MethodPost, "/suggestions/"+got[1].ID+"/dismiss", "")
settled = getSettled(t, srv, docID)
if len(settled) != 2 || !contains(settled, got[1].Original) {
t.Fatalf("dismissed span missing from %v", settled)
}
}
// TestSettledNormalizesAndDedupes proves the payload is normalized server-side
// and collapsed. The client compares its freshly-detected findings against these
// strings, so the two sides have to agree on what "the same span" is — the
// editor's quote churn is the case that breaks a byte-exact match, and it is why
// normalizeForDedup exists at all.
func TestSettledNormalizesAndDedupes(t *testing.T) {
client := &stubClient{response: `{"suggestions":[]}`}
srv, docID, h := newTestServer(t, client)
// The same span twice, differing only in quote style and line breaks — one
// accepted, one dismissed. Distinct rows; one settled span.
a := seedSuggestion(t, h, docID, "text", db.SuggestionTypeGrammar,
"She said \"hello\"\n to me", "She said 'hello' to me", "quotes")
b := seedSuggestion(t, h, docID, "text", db.SuggestionTypeGrammar,
"She said “hello” to me", "She said 'hello' to me", "quotes")
do(t, srv, http.MethodPost, "/suggestions/"+a+"/accept", "")
do(t, srv, http.MethodPost, "/suggestions/"+b+"/dismiss", "")
settled := getSettled(t, srv, docID)
if len(settled) != 1 {
t.Fatalf("two spellings of one span should collapse to one, got %v", settled)
}
if want := "She said 'hello' to me"; settled[0] != want {
t.Fatalf("settled[0] = %q, want normalized %q", settled[0], want)
}
}
// TestNormalizeMatchesTheClient is the Go half of a pair. Every case here also
// appears in web/src/lib/settled.test.ts, asserted against the TypeScript
// reimplementation of this function. The two are compared across a network
// boundary — the server normalizes what it sends, the client normalizes what it
// checks against it — so they have to fold the same characters the same way, and
// nothing but a shared list of cases can say so. Add to both or neither.
func TestNormalizeMatchesTheClient(t *testing.T) {
cases := []struct{ in, want string }{
{"She said “hello”", "She said 'hello'"},
{"She said \"hello\"", "She said 'hello'"},
{"its", "it's"},
{"its", "it's"},
{"`code´", "'code'"},
{" a apple\n here ", "a apple here"},
{"a\tapple", "a apple"},
{" \n ", ""},
{"我想说这句话", "我想说这句话"},
}
for _, c := range cases {
if got := normalizeForDedup(c.in); got != c.want {
t.Errorf("normalizeForDedup(%q) = %q, want %q", c.in, got, c.want)
}
}
}
// TestSettledEmptyIsAList guards the shape rather than the content: the client
// spreads this array into its settled set, and a null would throw there. Go
// marshals a nil slice as null, so this is one `[]string{}` away from breaking.
func TestSettledEmptyIsAList(t *testing.T) {
client := &stubClient{response: `{"suggestions":[]}`}
srv, docID, _ := newTestServer(t, client)
rec := do(t, srv, http.MethodGet, "/docs/"+docID+"/settled", "")
if body := rec.Body.String(); body != "{\"originals\":[]}\n" && body != "{\"originals\":[]}" {
t.Fatalf("empty settled body = %q, want an empty list", body)
}
}
+191
View File
@@ -0,0 +1,191 @@
package suggestions
import (
"encoding/json"
"net/http"
"strings"
"testing"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/llm"
)
// byOriginal indexes a pending set by the text each card flags.
func byOriginal(in []db.Suggestion) map[string]db.Suggestion {
out := map[string]db.Suggestion{}
for _, s := range in {
out[s.Original] = s
}
return out
}
// TestUntouchedSentencesKeepTheirCards is the heart of the stability work: she
// edits one sentence, and the cards on every other sentence stay exactly as they
// were — same id (so the rail keeps the card instead of remounting it), same
// explanation (the model re-words its reasoning every time it is asked, and one
// unchanged mistake used to carry three different explanations in a sitting).
// The model is only asked about the sentence that changed.
func TestUntouchedSentencesKeepTheirCards(t *testing.T) {
client := &stubClient{response: `{"suggestions":[
{"original":"I has two apple","replacement":"I have two apples","explanation":"first wording","type":"grammar"},
{"original":"She go to market","replacement":"She goes to market","explanation":"agreement","type":"grammar"}
]}`}
srv, docID, h := newTestServer(t, client)
h.Limit = llm.NewRateLimiter(0)
setDocText(t, h, docID, "I has two apple. She go to market yesterday.")
rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "")
var first []db.Suggestion
if err := json.Unmarshal(rec.Body.Bytes(), &first); err != nil {
t.Fatalf("decode: %v", err)
}
if len(first) != 2 {
t.Fatalf("first pass: want 2, got %d: %+v", len(first), first)
}
kept := byOriginal(first)["I has two apple"]
// She fixes only the second sentence. The model, asked again, re-words its
// reasoning about the first — which it must never get the chance to do.
setDocText(t, h, docID, "I has two apple. She goes to market yesterday.")
client.response = `{"suggestions":[
{"original":"I has two apple","replacement":"I have two apples","explanation":"REWORDED","type":"grammar"}
]}`
rec = do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "")
var second []db.Suggestion
if err := json.Unmarshal(rec.Body.Bytes(), &second); err != nil {
t.Fatalf("decode: %v", err)
}
if strings.Contains(client.lastPrompt, "I has two apple") {
t.Fatalf("untouched sentence was sent to the model:\n%s", client.lastPrompt)
}
if !strings.Contains(client.lastPrompt, "She goes to market") {
t.Fatalf("edited sentence was not sent to the model:\n%s", client.lastPrompt)
}
now := byOriginal(second)["I has two apple"]
if now.ID != kept.ID {
t.Fatalf("card was remounted: id %q became %q", kept.ID, now.ID)
}
if now.Explanation != "first wording" {
t.Fatalf("explanation drifted: %q", now.Explanation)
}
// The fixed sentence's card is gone, and the model's stray re-proposal for the
// cached sentence did not become a second card.
if len(second) != 1 {
t.Fatalf("want exactly one card left, got %d: %+v", len(second), second)
}
}
// TestUnchangedDocumentSkipsTheModel proves a check with nothing new to read
// costs nothing: no model call, and every card left standing untouched. This is
// the doc-open and tone-less re-check path.
func TestUnchangedDocumentSkipsTheModel(t *testing.T) {
client := &stubClient{response: `{"suggestions":[
{"original":"I has","replacement":"I have","explanation":"agreement","type":"grammar"}
]}`}
srv, docID, h := newTestServer(t, client)
h.Limit = llm.NewRateLimiter(0)
rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "")
var first []db.Suggestion
_ = json.Unmarshal(rec.Body.Bytes(), &first)
if len(first) != 1 || client.calls != 1 {
t.Fatalf("first pass: %d cards, %d calls", len(first), client.calls)
}
rec = do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "")
var second []db.Suggestion
if err := json.Unmarshal(rec.Body.Bytes(), &second); err != nil {
t.Fatalf("decode: %v", err)
}
if client.calls != 1 {
t.Fatalf("re-checking an unedited document called the model %d times", client.calls)
}
if len(second) != 1 || second[0].ID != first[0].ID {
t.Fatalf("card did not survive an idle re-check: %+v", second)
}
}
// TestDeletedSentenceDropsItsCard covers the other half of the skip path: she
// removes a flagged sentence outright, so nothing changed that the model could
// be asked about — but its card must still go.
func TestDeletedSentenceDropsItsCard(t *testing.T) {
client := &stubClient{response: `{"suggestions":[
{"original":"I has","replacement":"I have","explanation":"agreement","type":"grammar"}
]}`}
srv, docID, h := newTestServer(t, client)
h.Limit = llm.NewRateLimiter(0)
do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "")
setDocText(t, h, docID, "")
rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "")
var got []db.Suggestion
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("decode: %v", err)
}
if len(got) != 0 {
t.Fatalf("card outlived its sentence: %+v", got)
}
}
// TestToneChangeReopensEverySentence: the checkpoint's advice is written for the
// document's tone, so switching from a journal to an academic essay has to
// re-read sentences that haven't changed a character.
func TestToneChangeReopensEverySentence(t *testing.T) {
client := &stubClient{response: `{"suggestions":[
{"original":"I has","replacement":"I have","explanation":"agreement","type":"grammar"}
]}`}
srv, docID, h := newTestServer(t, client)
h.Limit = llm.NewRateLimiter(0)
do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "")
if _, err := h.DB.Exec(`UPDATE documents SET tone = 'academic' WHERE id = ?`, docID); err != nil {
t.Fatalf("set tone: %v", err)
}
do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "")
if client.calls != 2 {
t.Fatalf("tone change did not re-read the document: %d model calls", client.calls)
}
}
// TestMechanicsFindingsKeepTheirRows: the rule pack re-runs 250 ms after every
// keystroke. A finding it still reports must keep its row, or the rail would
// remount several times a sentence — collapsing a card she has open, and
// re-firing the arrival chime for advice she is already reading.
func TestMechanicsFindingsKeepTheirRows(t *testing.T) {
srv, docID, _ := newTestServer(t, &stubClient{response: `{"suggestions":[]}`})
body := `{"findings":[
{"from":0,"to":5,"original":"I has","replacement":"I have","explanation":"agreement"},
{"from":6,"to":15,"original":"two apple","replacement":"two apples","explanation":"plural"}
]}`
rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/mechanics", body)
var first []db.Suggestion
if err := json.Unmarshal(rec.Body.Bytes(), &first); err != nil {
t.Fatalf("decode: %v", err)
}
if len(first) != 2 {
t.Fatalf("want 2 rows, got %d", len(first))
}
// She types elsewhere: same findings, shifted spans, one of them now fixed.
rec = do(t, srv, http.MethodPost, "/docs/"+docID+"/mechanics", `{"findings":[
{"from":20,"to":25,"original":"I has","replacement":"I have","explanation":"agreement"}
]}`)
var second []db.Suggestion
if err := json.Unmarshal(rec.Body.Bytes(), &second); err != nil {
t.Fatalf("decode: %v", err)
}
if len(second) != 1 {
t.Fatalf("want 1 row, got %d: %+v", len(second), second)
}
if second[0].ID != byOriginal(first)["I has"].ID {
t.Fatalf("surviving finding was given a new identity: %+v", second[0])
}
if second[0].FromPos != 20 {
t.Fatalf("span did not follow the text: %+v", second[0])
}
}
+1 -1
View File
@@ -51,7 +51,7 @@ func (h *Handler) translate(w http.ResponseWriter, r *http.Request) {
out, err := llm.RunTranslate(r.Context(), h.Client, explanation, llm.LangFor(pairLang))
if err != nil {
httputil.ErrorJSON(w, http.StatusBadGateway, "translate failed: "+err.Error())
httputil.UpstreamError(w, "translate", err)
return
}
+107
View File
@@ -0,0 +1,107 @@
package suggestions
import (
"encoding/json"
"net/http"
"testing"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// The pair model's flagship moment, end to end: she reaches for a sentence in
// her own language mid-document, and the card that comes back is labelled as a
// translation rather than as a tidy-up of her Chinese.
//
// The label is asserted through the real /check path rather than against
// isTranslation directly, because the point of the item was never the detector —
// Petal already found these spans and already rendered them into English. What
// was wrong was the type that reached the rail.
func TestChineseSpanBecomesATranslateCard(t *testing.T) {
// Note the model calls it "clarity", as the live build did. The type it
// volunteers is not consulted.
client := &stubClient{response: `{"suggestions":[
{"original":"我想说这句话但是不知道用英语怎么说。","replacement":"I want to say this but I don't know how to say it in English.","explanation":"这是英文说法 · Here is how to say it in English","type":"clarity"}
]}`}
srv, docID, database := newPairServer(t, client, "zh")
setDocTextDB(t, database, docID, "My weekend was good. 我想说这句话但是不知道用英语怎么说。")
var out []db.Suggestion
rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "")
if rec.Code != http.StatusOK {
t.Fatalf("check: code=%d body=%s", rec.Code, rec.Body)
}
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatalf("decode: %v", err)
}
if len(out) != 1 {
t.Fatalf("want 1 card, got %d: %+v", len(out), out)
}
if out[0].Type != db.SuggestionTypeTranslate {
t.Fatalf("card type = %q, want %q", out[0].Type, db.SuggestionTypeTranslate)
}
// The rendering and the reasoning are the model's, untouched — only the label
// is Petal's.
if out[0].Replacement != "I want to say this but I don't know how to say it in English." {
t.Fatalf("replacement was rewritten: %q", out[0].Replacement)
}
}
// The other half of the same claim: an ordinary English correction on the same
// writer's document keeps the type the model gave it. A relabel that fired on
// everything would be no better than the label it replaced.
func TestEnglishCorrectionKeepsItsType(t *testing.T) {
client := &stubClient{response: `{"suggestions":[
{"original":"My weekend was very good","replacement":"My weekend was wonderful","explanation":"stronger wording","type":"phrasing"}
]}`}
srv, docID, database := newPairServer(t, client, "zh")
setDocTextDB(t, database, docID, "My weekend was very good.")
var out []db.Suggestion
rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/check", "")
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatalf("decode: %v", err)
}
if len(out) != 1 {
t.Fatalf("want 1 card, got %d: %+v", len(out), out)
}
if out[0].Type != db.SuggestionTypePhrasing {
t.Fatalf("card type = %q, want %q", out[0].Type, db.SuggestionTypePhrasing)
}
}
// The voice pass reads whole paragraphs for tone and stamps its own family. A
// Chinese paragraph must not be able to smuggle a translate row into it — voice
// rows carry no replacement to accept, so a "translation" there would be a card
// offering nothing.
func TestVoicePassCannotProduceATranslateCard(t *testing.T) {
client := &stubClient{response: `{"suggestions":[
{"original":"我想说这句话但是不知道用英语怎么说。","replacement":"I want to say this in English.","explanation":"tone","type":"clarity"}
]}`}
srv, docID, database := newPairServer(t, client, "zh")
setDocTextDB(t, database, docID, "A first paragraph.\n\n我想说这句话但是不知道用英语怎么说。")
var out []db.Suggestion
rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/voice", "")
if rec.Code != http.StatusOK {
t.Fatalf("voice: code=%d body=%s", rec.Code, rec.Body)
}
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatalf("decode: %v", err)
}
for _, s := range out {
if s.Type == db.SuggestionTypeTranslate {
t.Fatalf("voice pass produced a translate card: %+v", s)
}
}
}
// setDocTextDB is setDocText for the pair harness, which hands back the DB
// rather than the Handler.
func setDocTextDB(t *testing.T, database *db.DB, docID, text string) {
t.Helper()
if _, err := database.Exec(
`UPDATE documents SET content_text = ? WHERE id = ?`, text, docID,
); err != nil {
t.Fatalf("update doc text: %v", err)
}
}
+77
View File
@@ -21,6 +21,7 @@ import (
"path/filepath"
"sort"
"strings"
"sync/atomic"
"time"
"unicode/utf8"
@@ -97,6 +98,7 @@ type Handler struct {
cacheDir string
format audioFormat
client *http.Client
writes atomic.Uint64 // cache writes since boot; drives the prune throttle
}
// New builds a Handler from config. It returns (nil, false) when TTS_ENDPOINT is
@@ -233,6 +235,7 @@ func (h *Handler) synth(w http.ResponseWriter, r *http.Request) {
if err := os.WriteFile(tmp, audio, 0o644); err == nil {
_ = os.Rename(tmp, path)
}
h.pruneCache()
}
w.Header().Set("Content-Type", h.format.contentType)
@@ -240,6 +243,80 @@ func (h *Handler) synth(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write(audio)
}
// maxCacheBytes bounds the whole clip cache at 512 MiB.
//
// Each clip is small, so nothing about ordinary reading approaches this — a
// year of tapping words is tens of megabytes. What it bounds is the shape of
// the endpoint: the cache key is the *text*, so a client asking for four
// thousand distinct characters at a time writes a new file every request, for
// as long as it cares to. That is an authenticated writer filling the same
// encrypted volume the database lives on, and a full disk is SQLite failing to
// write, not merely read-aloud getting slower.
const maxCacheBytes = 512 << 20
// pruneEvery throttles the sweep: checking the directory on every synthesis
// would stat the whole cache for each new word. Synthesis is already the slow
// path and misses are rare once a writer settles, so one sweep per this many
// cache writes keeps the cost invisible while still converging long before the
// limit means anything.
const pruneEvery = 64
// pruneCache trims the cache back under maxCacheBytes, oldest-first, and is a
// no-op the great majority of the time it is called.
//
// Oldest by modification time is a fair approximation of least-recently-useful
// here: a clip is written once and only ever read afterwards, so its age is how
// long ago someone wanted it. Evicting one costs a re-synthesis, never data —
// which is why this can be as approximate as it likes, and why every error
// along the way is simply given up on.
func (h *Handler) pruneCache() {
if n := h.writes.Add(1); n%pruneEvery != 0 {
return
}
entries, err := os.ReadDir(h.cacheDir)
if err != nil {
return
}
type clip struct {
path string
size int64
mod time.Time
}
var clips []clip
var total int64
for _, e := range entries {
if e.IsDir() {
continue
}
info, err := e.Info()
if err != nil {
continue
}
clips = append(clips, clip{filepath.Join(h.cacheDir, e.Name()), info.Size(), info.ModTime()})
total += info.Size()
}
if total <= maxCacheBytes {
return
}
sort.Slice(clips, func(i, j int) bool { return clips[i].mod.Before(clips[j].mod) })
// Drop to 80% rather than exactly to the line, so the next few hundred
// clips don't each trigger another sweep.
target := int64(maxCacheBytes / 100 * 80)
removed := 0
for _, c := range clips {
if total <= target {
break
}
if os.Remove(c.path) == nil {
total -= c.size
removed++
}
}
fmt.Fprintf(os.Stderr, "tts: cache over %d bytes — evicted %d oldest clip(s)\n", int64(maxCacheBytes), removed)
}
// serve streams a cached clip with a long-lived immutable cache header (the URL
// is content-addressed, so the bytes never change for a given request).
func (h *Handler) serve(w http.ResponseWriter, r *http.Request, path string) {
+322
View File
@@ -0,0 +1,322 @@
#!/usr/bin/env python3
"""Build the two Chinese assets the learner direction of the zh pair needs.
Why two, and why they are split the way they are
------------------------------------------------
Every other pair Petal ships needs one asset: a word list the browser loads so
it can underline. Chinese needs two, because the browser and the server want
different halves of the same dictionary and for different reasons.
* **The browser needs a word list, and it needs it offline.** Chinese is
written without spaces, so there is no such thing as "the word under the
cursor" until something segments the sentence. Every ESL surface Petal
already has the hover gloss, the right-click lookup, Ctrl/Cmd+D, the
vocabulary garden capture is built on `wordAt`, and `wordAt` is a regex
over Latin letters. Segmentation is what replaces that regex, it runs on
every hover, and a round-trip per hover is not a hover. So the word list
ships to the browser: `web/public/dictionaries/zh/words.txt`.
* **The server holds the whole dictionary.** Pinyin and English senses are
only ever wanted one word at a time, in answer to a hover or a click, which
is exactly what `/api/gloss/{word}` already does for the other direction. So
the readings stay in the binary `internal/lexicon/data/hanzi.json.gz`
where their size costs a browser nothing.
That split is what makes the coverage decisions below come out *opposite* to
each other, and both are deliberate.
Two sources, because neither one has both halves
------------------------------------------------
* **CC-CEDICT** (CC BY-SA 4.0, https://www.mdbg.net/) has the headwords,
pinyin and English senses, and no frequency information at all.
* **jieba's `dict.txt`** (MIT, https://github.com/fxsjy/jieba) has ~349k
headwords with corpus frequencies, and no definitions.
Segmentation needs the frequencies: the standard algorithm is a shortest-path
walk over log-probabilities, not longest-match, and without frequencies the
classic ambiguities go the wrong way. The client list therefore carries
`word freq` per line; the gloss map carries readings.
The size decision is the client list, and it is a size decision only
--------------------------------------------------------------------
Measured on ordinary learner prose, the segmentation produced by the full jieba
dictionary (381,886 hanzi headwords once CC-CEDICT is unioned in) and by a
frequency-gated one is **identical**, including on the textbook ambiguities
(研究生命的起源, 乒乓球拍卖完了, 南京市长江大桥). What the long tail contains is
rare proper nouns, and the max-probability walk almost never chooses one: a
freq-3 name loses to two common words every time. The cases where a missing word
does change the answer degrade *gracefully* the sentence splits into smaller
real words, which is a slightly clumsier gloss, not a wrong underline.
So the gate is set where the size is, at **freq >= 5**: 188,522 words, ~0.97 MB
gzipped over the wire, in line with fr (1.19 MB) and es (1.74 MB) rather than in
excess of them. Every CC-CEDICT headword is unioned back in regardless of
frequency, so the segmenter can always see a word the server can explain.
The gloss map is gated by nothing, for the opposite reason
-----------------------------------------------------------
The es phase settled that a *spelling* dictionary should hold the union of every
variety, because its only power is to underline and it must not underline
correct writing. This asset's only power is to **explain**, and the word a
learner stops on is precisely the one they do not know which is to say, the
rare one. Trimming this by frequency would remove exactly the entries it exists
for. All 113,637 glossable headwords ship, ~3.1 MB gzipped, which is less than
half of what `synonyms.json.gz` has embedded since Phase 9.
Simplified only, and said out loud
-----------------------------------
The zh langpack is written in simplified characters and jieba's frequencies are
counted over simplified text, so the traditional headword in each CC-CEDICT line
is dropped and simplified is what both assets are keyed by. Glossing traditional
would be nearly free *here* and useless in the app: nothing would segment it, so
nothing would ever ask. Traditional support is a real feature and it starts with
a traditional word list, not with this file.
Usage:
curl -sL https://www.mdbg.net/chinese/export/cedict/cedict_1_0_ts_utf-8_mdbg.txt.gz | gunzip > cedict.txt
curl -sL https://raw.githubusercontent.com/fxsjy/jieba/master/jieba/dict.txt -o jieba.txt
python3 scripts/build_cedict.py cedict.txt jieba.txt \
web/public/dictionaries/zh/words.txt.gz \
internal/lexicon/data/hanzi.json.gz
"""
import gzip
import json
import re
import sys
# Frequency gate for the *client* list only (see the module docstring). Words
# below it survive if CC-CEDICT knows them, so "segmentable" is always a superset
# of "glossable" and a hover can never land on a word the server cannot explain.
MIN_FREQ = 5
# A CC-CEDICT headword we keep must be nothing but han characters. This drops the
# entries that are really English or numerals with a Chinese gloss attached
# ("AA制", "PM2.5", "11区"): the segmenter walks runs of hanzi, so a mixed
# headword can never be matched anyway, and a Latin one would collide with the
# English tokenizer that is still running on the same paragraph.
HANZI_ONLY = re.compile(r'^[一-鿿]+$')
CEDICT_LINE = re.compile(r'^(\S+) (\S+) \[(.*?)\] /(.*)/$')
# At most this many readings per word, and this many senses per reading. Two
# readings is not an arbitrary cap: it is what the particles need. 得 is dé "to
# obtain" *and* de, the complement marker — and a learner who hovers 得 in
# 说得很好 and is told only "to obtain" has been actively misinformed. Beyond two
# the tail is dialect and surnames, which crowd out the sense actually wanted.
MAX_READINGS = 2
MAX_SENSES = 3
MAX_SENSE_CHARS = 110
# Senses that describe the *dictionary* rather than the word. A learner hovering
# a word wants to know what it means, not that it is an orthographic variant of
# another headword they also do not know.
SKIP_SENSE_PREFIXES = ('variant of', 'old variant', 'see ', 'used in', 'abbr. for')
# ── pinyin: numbered syllables to tone marks ────────────────────────────────
# CC-CEDICT stores "gong1 yuan2". A learner reading their own writing back wants
# gōngyuán: the tone mark is the part that is hard to remember and the part that
# changes the word. The placement rule is the standard one — a/o/e take the mark
# if present, otherwise the last vowel of the final — and it is small enough to
# do here rather than to take a dependency for.
TONE_VOWELS = {
'a': 'āáǎà',
'e': 'ēéěè',
'i': 'īíǐì',
'o': 'ōóǒò',
'u': 'ūúǔù',
'ü': 'ǖǘǚǜ',
}
SYLLABLE = re.compile(r'^([a-zA-Zü:]+)([1-5])$')
def tone_mark(syllable: str) -> str:
"""One numbered pinyin syllable to its tone-marked form."""
m = SYLLABLE.match(syllable)
if not m:
# Punctuation, a bare letter (CC-CEDICT writes "X" for unknown), or an
# already-marked syllable: pass it through rather than mangling it.
return syllable
body, tone = m.group(1), int(m.group(2))
# CC-CEDICT writes ü as "u:" and, in a few entries, as "v".
body = body.replace('u:', 'ü').replace('U:', 'Ü').replace('v', 'ü').replace('V', 'Ü')
if tone == 5: # neutral tone carries no mark
return body
low = body.lower()
idx = -1
for vowel in ('a', 'o', 'e'):
idx = low.find(vowel)
if idx >= 0:
break
if idx < 0:
# No a/o/e: the mark goes on the last of i/u/ü (liú, guǐ, nǚ).
idx = max(low.rfind('i'), low.rfind('u'), low.rfind('ü'))
if idx < 0:
return body
marked = TONE_VOWELS[low[idx]][tone - 1]
if body[idx].isupper():
marked = marked.upper()
return body[:idx] + marked + body[idx + 1:]
def pinyin(numbered: str) -> str:
"""A whole CC-CEDICT pinyin field to tone marks, syllables joined up.
Joined rather than spaced because that is how a word is written when it is
being read as a word (gōngyuán, not gōng yuán); the spaces in the source are
a storage convention, not orthography.
"""
return ''.join(tone_mark(s) for s in numbered.split())
def clean_senses(raw: list[str]) -> list[str]:
"""Strip the apparatus CC-CEDICT carries for lexicographers, not learners."""
out = []
for sense in raw:
# "CL:座[zuo4]" is the measure-word field, useful and not a definition.
sense = re.sub(r'\s*CL:.*$', '', sense).strip()
# Bracketed pinyin cross-references ("abbr. for 的士[di1 shi4]").
sense = re.sub(r'\[[a-zA-Z0-9: ]+\]', '', sense).strip()
# Both edits cut inside parentheses — "cat (CL:只)" loses its closing
# bracket and leaves "cat (" on the card. Drop a dangling opener rather
# than trying to rebalance: what it introduced is gone.
if sense.count('(') > sense.count(')'):
sense = re.sub(r'\s*\([^()]*$', '', sense).strip()
if not sense or sense.startswith(SKIP_SENSE_PREFIXES):
continue
out.append(sense)
return out
def read_cedict(path: str) -> dict[str, list[tuple[str, list[str]]]]:
entries: dict[str, list[tuple[str, list[str]]]] = {}
for line in open(path, encoding='utf-8'):
if line.startswith('#'):
continue
m = CEDICT_LINE.match(line.strip())
if not m:
continue
_traditional, simplified, py, defs = m.groups()
if not HANZI_ONLY.match(simplified):
continue
entries.setdefault(simplified, []).append((py, defs.split('/')))
return entries
def read_jieba(path: str) -> dict[str, int]:
freqs: dict[str, int] = {}
for line in open(path, encoding='utf-8'):
parts = line.split()
if len(parts) >= 2 and HANZI_ONLY.match(parts[0]):
freqs[parts[0]] = int(parts[1])
return freqs
# ── the assertions ──────────────────────────────────────────────────────────
# The es phase's lesson, in the place it applies here: a check that every
# plausible input would pass is not a check. The Spanish MUST_ACCEPT list
# asserted vocabulary that all twenty-four builds carried, so it could not tell
# them apart. These assert the things that actually go wrong in *this* build —
# a mis-parsed pinyin field, a missing particle reading, a word list gated so
# hard the segmenter can no longer see a word the server can explain.
# Tone marking, including the three cases the placement rule exists for.
MUST_MARK = {
'gong1 yuan2': 'gōngyuán', # a/o/e rule, first syllable
'pao3 bu4': 'pǎobù',
'liu2': 'liú', # no a/o/e: mark the *last* of i/u
'gui3': 'guǐ',
'nu:3': '', # u: is ü
'lu:e4': 'lüè', # ü and an e in the same syllable: e wins
'de5': 'de', # neutral tone takes no mark at all
'Zhong1 wen2': 'Zhōngwén', # capitalised headword keeps its capital
}
# The particles the 错别字 rules are about must each carry the *grammatical*
# reading, not only the lexical one. 的/地/得 are the single most confused triple
# in written Chinese and all three are neutral-tone "de" in the use that matters;
# an entry that only knows 得 as dé is worse than no entry.
MUST_READ_DE = ('', '', '')
# Words the segmenter must be able to see. 图书馆 and 乒乓球 are ordinary
# vocabulary; 我 and 的 are the two commonest words in the language and a gate
# that dropped either would be visibly broken; 的士 is a CC-CEDICT headword rare
# enough to fall below the frequency gate, and is here to prove the union.
MUST_SEGMENT = ('', '', '图书馆', '乒乓球', '公园', '的士')
def check(words: dict[str, int], gloss: dict[str, list[list[str]]]) -> None:
for numbered, want in MUST_MARK.items():
got = pinyin(numbered)
assert got == want, f'pinyin({numbered!r}) = {got!r}, want {want!r}'
for particle in MUST_READ_DE:
readings = gloss.get(particle)
assert readings, f'{particle} has no gloss entry at all'
assert any(r[0] == 'de' for r in readings), \
f'{particle} never reads as neutral "de": {readings}'
for word in MUST_SEGMENT:
assert word in words, f'{word} missing from the segmentation list'
# The invariant the two gates exist to keep: everything the server can
# explain, the browser can find.
missing = [w for w in gloss if w not in words]
assert not missing, f'{len(missing)} glossable words are unsegmentable, e.g. {missing[:5]}'
# Nothing Latin leaked into either asset (see HANZI_ONLY).
for name, keys in (('words', words), ('gloss', gloss)):
bad = [k for k in keys if not HANZI_ONLY.match(k)]
assert not bad, f'non-hanzi headwords in {name}: {bad[:5]}'
def main() -> None:
if len(sys.argv) != 5:
sys.exit(__doc__.strip().rsplit('Usage:', 1)[-1].strip())
cedict_path, jieba_path, words_out, gloss_out = sys.argv[1:]
entries = read_cedict(cedict_path)
freqs = read_jieba(jieba_path)
# The client list: frequency-gated, then unioned with every glossable word.
# A CC-CEDICT word jieba has never seen gets frequency 1 — real, and rare
# enough that the max-probability walk will only choose it when nothing else
# fits, which is exactly the standing it should have.
words = {w: f for w, f in freqs.items() if f >= MIN_FREQ}
for w in entries:
words.setdefault(w, 1)
gloss: dict[str, list[list[str]]] = {}
for word, rows in entries.items():
readings: list[list[str]] = []
for numbered, defs in rows:
senses = clean_senses(defs)
if not senses:
continue
readings.append([pinyin(numbered), '; '.join(senses[:MAX_SENSES])[:MAX_SENSE_CHARS]])
if len(readings) == MAX_READINGS:
break
if readings:
gloss[word] = readings
check(words, gloss)
# Gzipped on disk, like the pt-PT/fr/es word lists: the browser inflates it
# with DecompressionStream (see useSpellChecker.fetchText), which costs no
# bundle bytes, and 0.97 MB over the wire rather than 2.23 MB is the whole
# difference between this and the biggest asset Petal ships.
body = ('\n'.join(f'{w} {words[w]}' for w in sorted(words)) + '\n').encode('utf-8')
with gzip.open(words_out, 'wb', compresslevel=9) as fh:
fh.write(body)
payload = json.dumps(gloss, ensure_ascii=False, separators=(',', ':')).encode('utf-8')
with gzip.open(gloss_out, 'wb', compresslevel=9) as fh:
fh.write(payload)
print(f'{words_out}: {len(words)} words, {len(body) / 1e6:.2f} MB raw, '
f'{len(gzip.compress(body, 9)) / 1e6:.2f} MB gzipped')
print(f'{gloss_out}: {len(gloss)} entries, {len(gzip.compress(payload, 9)) / 1e6:.2f} MB gzipped')
if __name__ == '__main__':
main()
+543
View File
@@ -0,0 +1,543 @@
#!/usr/bin/env python3
"""Build one of Petal's browser spelling dictionaries from a Hunspell one.
Why this script exists at all
----------------------------
English is vendored the obvious way: `dictionary-en`'s `en.aff` + `en.dic` go
into web/public/dictionaries/en and nspell reads them in the browser. The plan
for Phase 21 said "Hunspell pt-PT vendored like en-US", and that turns out not to
work, for a measured reason.
nspell expands affixes **eagerly at load time** it materialises every surface
form into a hash the moment you construct it. English gets away with this: ~50k
stems and a small rule set. European Portuguese does not. `pt_PT.aff` carries
1,340 affix rules (the full verb paradigm: six persons x a dozen tenses, plus
diminutives, plus productive prefixes) over 44,257 stems. Measured on this
machine, nspell needed ~340 MB of heap for the first 12,000 entries alone and had
not returned after three minutes on the whole file; extrapolated, it wants well
over a gigabyte. That is not something to hand a browser, still less a tablet.
French is larger again: 5,600 affix rules over 84,140 stems.
So the expansion happens **here**, once, at build time, and the browser gets a
flat word list it can load with no affix machinery at all. The runtime code path
is then *identical* to English same nspell, same interface which is the real
prize. The aff shipped alongside keeps only the suggestion-shaping directives
(TRY/KEY/REP/MAP), so corrections still know that "cao" wants "ção" and that a
missing acute accent is a near miss.
What Phase 24 had to add
------------------------
The build plan recorded that the pt-PT version of this script "generalizes" to
French. It did not. It handled single-character flags and plain PFX/SFX and
stopped on anything else the right call, because `fr.aff` uses four of the
things it stopped on, and getting any of them wrong changes which words are
accepted:
* **`FLAG long`** French flags are *two characters* (`S.`, `L'`, `Um`). The
pt-PT reader took `set(flagstr)`, one flag per character, which on a French
entry yields a bag of unrelated single letters: every entry would have been
expanded through the wrong paradigm. This is the one that fails silently.
* **Continuation flags** pt-PT's affixes append plain text, so that script
dropped anything after a `/` and asserted the drop was safe. French really
does affix an affixed form: `PFX Um 0 0/S.` says the prefixed form then takes
the plural suffix, and the elision prefixes arrive the same way from the
other side (`SFX ... ait/n'q'l'm't's'`).
* **`NEEDAFFIX`** French marks thousands of stems "not a word on its own"
(`Allemagne/S.()`), the bare form arriving instead through a zero-append
rule. Ignoring the flag accepts stems the real dictionary rejects.
* **`FULLSTRIP`** a rule may strip the whole stem.
`CIRCUMFIX` and `FORBIDDENWORD` are *declared* in `fr.aff` and used by nothing,
which this script asserts rather than assumes: an upstream release that started
using either would otherwise change what is accepted without changing this file.
`KEEPCASE` and `NOSUGGEST` are honoured by being ignored on purpose they shape
casing and suggestions, not membership, and a NOSUGGEST word is still a word.
Elision is handled at lookup, not here and that is the size decision
----------------------------------------------------------------------
Most of French's affix machinery by volume is elision: `l'`, `d'`, `qu'`, `j'`,
`n'`, `s'`, `jusqu'`, `puisqu'`. Hunspell treats `l'arbre` as one word, so a
faithful expansion carries much of the language thirty-four times over and
Petal's tokenizer keeps internal apostrophes, so `l'arbre` really does arrive at
the dictionary as one token and really would be underlined if it were absent.
Both halves were built and measured. Keeping the elided forms: **3,159,832 forms,
8.25 MB gzipped**, ~45 MB of text for nspell to hash on a tablet. Dropping them:
**473,326 forms, 1.19 MB gzipped**. The elided seven-eighths are not new words
they are thirteen little words glued to words already in the list so the third
option is the one taken: rules whose append carries an apostrophe are skipped
here (the count is printed), and `withElision` in `useSpellChecker.ts` splits a
token at a *known clitic* and checks the remainder. `l'arbre` costs one extra
lookup instead of seven megabytes, and `zzz'arbre` is still flagged because
`zzz` is not one of the thirteen.
Stems that carry an apostrophe of their own `aujourd'hui`, `quelqu'un`,
`presqu'île`, `prud'homme` are dictionary entries rather than affixed forms,
so they are kept verbatim and matched directly. `entr'aide` and `grand'mère` are
absent for the same reason they are absent from Dicollecte: modern French spells
them `entraide` and `grand-mère`.
Choosing the source
-------------------
Both languages have a trap here, and they are different traps.
**pt-PT: the wrong country.** npm's `dictionary-pt` is not European Portuguese.
Both it and `dictionary-pt-br` package VERO ("Verificador Ortográfico Livre",
Brasil), so vendoring the obvious npm name would have shipped Brazilian spellings
under a pt-PT label the pt-BR drift SUGGESTIONS.md §3 warns about, arriving
through the packaging rather than through the model. The authentic dictionary is
the Projecto Natura one (Universidade do Minho) that LibreOffice ships and Debian
packages as `hunspell-pt-pt`; its aff declares `LANG pt_PT`.
**es: the wrong country again, hidden one layer further down.** Spanish looked
like it would repeat the pt trap `hunspell-es` installs twenty country codes,
`es_AR` through `es_VE` and then looked like it did not, because every one of
them is a symlink to a single `es_ES.aff`/`es_ES.dic`. Both readings were wrong.
Debian collapses the twenty because it ships **one** of upstream's builds, and
the one it ships is the **peninsular** `es_ES`. RLA (Santiago Bosio's project,
`sbosio/rla-es`) publishes twenty-four dictionaries per release: one per country,
plus a **generic `es`** that is the union of all of them. Debian packages neither
the generic one nor a choice it packages Spain, under a name that reads like
"Spanish".
Measured against the v2.9 release: Debian's file is 659,085 expanded forms and
upstream `es_ES` is 659,018; the generic `es` is **717,640**. The 58,622-form
difference is almost entirely **voseo** `vení`, `tenés`, `querés`, `sabés`,
`andá` the present tense of most of Latin America, which Debian's package
rejects as misspellings. Petal ships the **generic** build.
**Vocabulary cannot detect this and morphology can.** The first version of the es
profile asserted the pan-Hispanic lexicon *computadora* and *ordenador*, *papa*
and *patata* and passed happily on the peninsular file, because **every** RLA
variant carries the full pan-Hispanic vocabulary; only the verb paradigms are
localised. The `REP` table is no help either: its `ll``y` and `ás``az` entries
look like evidence of yeísmo and seseo, but they are shared by all twenty-four
builds. What separates them is exactly two things, and the profile now demands
both at once: **voseo** (absent from `es_ES`) and **vosotros** (largely absent
from `es_MX`). Only the generic build has both, so only the generic build passes.
This is the same decision fr made between `-classical` and `-revised`, arriving
by a different road. The only thing this dictionary can do is underline
something, and *tienes* and *tenés* are both correct Spanish taught in different
countries so Petal takes the build that accepts every variety rather than one
that makes a writer wrong for where she is from. Nothing is generated to get
there: the forms come from a real upstream package, which is what lets the
MUST_ACCEPT list prove which package it was.
Licensing note: RLA is tri-licensed GPL-3+ / LGPL-3+ / MPL-1.1+; Petal
redistributes under the MPL. The upstream README and LICENSE are vendored beside
the output.
**fr: the wrong side of an argument the French have not settled.** The regional
question turns out to be a non-question Debian's `fr_FR`, `fr_CA`, `fr_BE`,
`fr_CH`, `fr_LU` and `fr_MC` are all symlinks to one `fr.dic`, so unlike pt there
is no country here to get wrong. What there is instead is the 1990 spelling
reform, packaged three ways: `hunspell-fr-classical` (traditional), `-revised`
(reform only) and `-comprehensive` (both). Petal ships **comprehensive**, because
Petal never corrects her French the only thing this dictionary can do is
underline something. *coût* and *cout* are both correct French, taught in
different decades to different people, and a writing companion has no business
underlining one of them to take a side. The `fr` MUST_ACCEPT list is written to
*prove* which package was used: classical rejects `cout`, revised rejects `coût`,
and only comprehensive accepts both.
Licensing: pt-PT is GPL-2 or LGPL-2.1 or MPL-1.1, (c) José João de Almeida, Rui
Vilela, Alberto Simões. fr is MPL-2.0, (c) 2007-2018 the Dicollecte contributors
(grammalecte.net). The upstream copyright file is vendored beside each output.
Usage
-----
apt-get download hunspell-fr-comprehensive # or hunspell-pt-pt
dpkg-deb -x hunspell-fr-comprehensive_*.deb src
python3 scripts/build_hunspell_dictionary.py fr \\
src/usr/share/hunspell/fr.aff \\
src/usr/share/hunspell/fr.dic \\
web/public/dictionaries/fr
Spanish does not come from Debian see below; `hunspell-es` is the peninsular
build. Take the generic dictionary from an upstream release instead:
curl -LO https://github.com/sbosio/rla-es/releases/download/v2.9/es.oxt
unzip -d src es.oxt # an .oxt is a zip
python3 scripts/build_hunspell_dictionary.py es \\
src/es.aff src/es.dic web/public/dictionaries/es
"""
import gzip
import os
import re
import sys
import unicodedata
# Directives worth keeping in the shipped aff. These shape *suggestions*, not
# membership: TRY orders the alphabet the corrector tries, KEY knows which keys
# are adjacent, REP holds the language's own confusions (cao/ção, ss/ç), and MAP
# says an accented vowel and its bare form are the same letter for scoring —
# which is most of what an ESL writer gets wrong in either language.
KEEP_DIRECTIVES = ("SET", "TRY", "KEY", "REP", "MAP", "WORDCHARS")
# Directives that would change which words are *accepted* and that this expander
# does not implement. If a future upstream release starts using one, the output
# would silently disagree with the real dictionary, so the build stops instead.
UNSUPPORTED = (
"COMPOUNDFLAG", "COMPOUNDMIN", "COMPOUNDRULE", "COMPOUNDBEGIN",
"ONLYINCOMPOUND", "PSEUDOROOT", "AF", "AM",
)
APOSTROPHES = "'"
class Aff:
"""The parts of an .aff file that decide which words exist."""
def __init__(self):
self.pfx = {} # flag -> [(strip, append, condition, continuation)]
self.sfx = {}
# Cross-product, per flag — and kept per table, because PFX and SFX are
# separate flag namespaces in hunspell: the same flag may name a prefix
# table and a suffix table, with different cross-product settings. One
# shared dict let the second block silently overwrite the first.
self.cross_pfx = {} # flag -> bool
self.cross_sfx = {}
self.flag_kind = "char"
self.needaffix = None
self.circumfix = None
self.forbidden = None
self.dropped_apostrophe_rules = 0
def parse_flags(raw, kind):
"""Split a flag string into flags, per the aff's FLAG declaration."""
raw = raw.strip()
if not raw:
return set()
if kind == "long":
# Two characters per flag, exactly. An odd length is a malformed flag
# string, and silently dropping the trailing character would quietly
# expand an entry through the wrong paradigm — the failure mode this
# whole FLAG-aware rewrite exists to avoid.
if len(raw) % 2:
raise SystemExit(f"odd-length long flag string {raw!r}")
return {raw[i:i + 2] for i in range(0, len(raw), 2)}
if kind == "num":
return {f for f in raw.split(",") if f}
return set(raw)
def parse_aff(path):
with open(path, encoding="utf-8") as fh:
lines = fh.read().splitlines()
aff = Aff()
# FLAG has to be known before anything containing a flag is read, and it can
# sit anywhere in the file. So: header pass first, rules second.
for line in lines:
parts = line.split()
if not parts:
continue
head = parts[0]
if head in UNSUPPORTED:
raise SystemExit(
f"{path}: unsupported directive {head!r} — this expander handles "
"PFX/SFX affixation with continuation flags, and honouring "
f"{head} would change which words are accepted. Extend the "
"script before shipping."
)
if len(parts) < 2:
continue
if head == "FLAG":
aff.flag_kind = parts[1]
if aff.flag_kind not in ("long", "num", "UTF-8"):
raise SystemExit(f"{path}: unknown FLAG type {aff.flag_kind!r}")
elif head == "NEEDAFFIX":
aff.needaffix = parts[1]
elif head == "CIRCUMFIX":
aff.circumfix = parts[1]
elif head == "FORBIDDENWORD":
aff.forbidden = parts[1]
i = 0
while i < len(lines):
parts = lines[i].split()
if parts and parts[0] in ("PFX", "SFX"):
kind, flag, cross_flag, count = parts[0], parts[1], parts[2], int(parts[3])
table = aff.pfx if kind == "PFX" else aff.sfx
cross = aff.cross_pfx if kind == "PFX" else aff.cross_sfx
cross[flag] = cross_flag == "Y"
rules = table.setdefault(flag, [])
for j in range(1, count + 1):
p = lines[i + j].split()
strip = "" if p[2] == "0" else p[2]
append, _, cont_raw = p[3].partition("/")
if append == "0":
append = ""
# Elision. See the header: these forms are `l'` and its twelve
# siblings glued to words already in the list, they multiply the
# download by seven, and `withElision` reconstructs them at
# lookup for the cost of one extra hash probe.
if any(a in append for a in APOSTROPHES):
aff.dropped_apostrophe_rules += 1
continue
cont = parse_flags(cont_raw, aff.flag_kind)
if aff.circumfix and aff.circumfix in cont:
raise SystemExit(
f"{path}: CIRCUMFIX is used by a {kind} {flag} rule. It "
"was declared-but-unused when this expander was written "
"and is not implemented; honouring it would change which "
"words are accepted."
)
cond = p[4] if len(p) > 4 else "."
anchored = ("^" + cond) if kind == "PFX" else (cond + "$")
rules.append((strip, append, re.compile(anchored), cont))
i += count + 1
continue
i += 1
return aff
def apply_suffix(word, rules):
"""Every (form, continuation flags) a suffix table yields for `word`."""
out = []
for strip, append, cond, cont in rules:
if strip and not word.endswith(strip):
continue
if not cond.search(word):
continue
stem = word[: len(word) - len(strip)] if strip else word
out.append((stem + append, cont))
return out
def apply_prefix(word, rules):
out = []
for strip, append, cond, cont in rules:
if strip and not word.startswith(strip):
continue
if not cond.search(word):
continue
stem = word[len(strip):] if strip else word
out.append((append + stem, cont))
return out
def expand_entry(word, flags, aff, out):
"""Add every surface form of one dictionary entry to `out`.
Hunspell's model without compounding: a form is the stem plus at most one
prefix and at most one suffix. A flag reaches an affix either from the stem's
own flags or from the continuation flags of the affix applied on the other
side; when both sides apply, both rules must be declared cross-product.
NEEDAFFIX is why the bare form is not simply added: the flag says *this* form
is not a word, only whatever can be built from it and it arrives both on
stems and on continuations.
"""
def is_word(carried):
return not (aff.needaffix and aff.needaffix in carried)
if is_word(flags):
out.add(word)
suffixed = [] # (form, flag, continuation flags)
for f in flags:
if f in aff.sfx:
for form, cont in apply_suffix(word, aff.sfx[f]):
suffixed.append((form, f, cont))
if is_word(cont):
out.add(form)
prefixed = []
for f in flags:
if f in aff.pfx:
for form, cont in apply_prefix(word, aff.pfx[f]):
prefixed.append((form, f, cont))
if is_word(cont):
out.add(form)
# Prefix then suffix. The suffix flag may come from the stem or from the
# prefix's own continuation (`PFX Um 0 0/S.`), and the suffix condition is
# matched against the whole prefixed word, which is what hunspell does.
# NEEDAFFIX is checked here too, exactly as on the single-affix paths above:
# a doubly-affixed form whose last continuation still carries the flag is
# "not a word on its own", and without compounding there is no third affix
# left to make it one.
for form, pf, pcont in prefixed:
if not aff.cross_pfx.get(pf):
continue
for f in flags | pcont:
if f in aff.sfx and aff.cross_sfx.get(f):
for full, fcont in apply_suffix(form, aff.sfx[f]):
if is_word(fcont):
out.add(full)
# Suffix then prefix — the same pair reached from the other side, which is
# how the elision prefixes arrive in French. Only the flags the suffix hands
# forward are new here; the stem's own were covered above.
for form, sf, scont in suffixed:
if not aff.cross_sfx.get(sf):
continue
for f in scont:
if f in aff.pfx and aff.cross_pfx.get(f):
for full, fcont in apply_prefix(form, aff.pfx[f]):
if is_word(fcont):
out.add(full)
def expand(aff_path, dic_path):
aff = parse_aff(aff_path)
forms = set()
needaffix_stems = 0
with open(dic_path, encoding="utf-8") as fh:
fh.readline() # leading entry count, not a word
for raw in fh:
# Morphological fields (po:nom is:fem) follow the entry, separated by
# a tab in pt-PT and by a space in fr.
entry = raw.strip().split("\t")[0].split(" ")[0]
if not entry:
continue
word, _, flagstr = entry.partition("/")
word = word.strip()
if not word:
continue
flags = parse_flags(flagstr, aff.flag_kind)
if aff.forbidden and aff.forbidden in flags:
raise SystemExit(
f"{dic_path}: FORBIDDENWORD is in use ({word!r}). It was "
"declared-but-unused when this expander was written; the "
"forms it removes would be wrongly accepted."
)
if aff.needaffix and aff.needaffix in flags:
needaffix_stems += 1
expand_entry(word, flags, aff, forms)
# NFC, because the aff's own ICONV table normalises decomposed accents on the
# way in and the browser hands nspell whatever the keyboard produced.
forms = {unicodedata.normalize("NFC", f) for f in forms}
return forms, aff, needaffix_stems
def shipped_aff(aff_path):
keep = []
for line in open(aff_path, encoding="utf-8").read().splitlines():
head = line.split()[0] if line.split() else ""
if head in KEEP_DIRECTIVES:
keep.append(line)
return "\n".join(keep) + "\n"
# Words the built list must accept, and must reject, before it is written. Each
# set is chosen to fail loudly on the *specific* wrong source that language has a
# packaged, plausible way of reaching — not to spot-check spelling in general.
PROFILES = {
# The pt-PT/pt-BR fault lines: post-Acordo spellings, the European lexicon,
# and the first-person-plural preterite accent that only pt-PT writes.
"pt-PT": {
"accept": ("receção", "húmido", "telemóvel", "autocarro", "comboio",
"ótimo", "pensámos", "escrevêssemos", "jardim"),
"reject": ("recepção", "úmido", "ônibus", "óptimo"),
"wrong": "this does not look like European Portuguese",
},
# Which of the three 1990-reform packagings this is. `coût`/`cout` and
# `paraître`/`paraitre` are each accepted by exactly one of classical and
# revised, so a build accepting all four is comprehensive and one that drops
# any of them is not. `Allemagne` is a NEEDAFFIX stem reachable only through
# a zero-append rule and `km` only through a prefix continuation, so between
# them they also check that this expander honoured the two features the
# pt-PT one refused.
"fr": {
"accept": ("coût", "cout", "paraître", "paraitre", "nénuphar", "nénufar",
"oignon", "ognon", "événement", "évènement", "jardin",
"Allemagne", "écrivissions", "km"),
"reject": ("jardinn", "écrivaitz", "xyzzyque"),
"wrong": "this does not look like the comprehensive French dictionary",
},
# The generic RLA build, and the accept list is written to reject the four
# neighbouring builds rather than to describe this one.
#
# The first version of this profile demanded *computadora* and *ordenador*,
# *papa* and *patata*, and passed — on the peninsular file, because **every**
# RLA variant carries the whole pan-Hispanic vocabulary. Vocabulary does not
# discriminate here at all; only morphology does, and it discriminates
# completely:
#
# * **voseo** (`vení`, `tenés`, `querés`) is in `es` and `es_AR` and not in
# `es_ES` or Debian's package. Demanding it rejects the peninsular build.
# * **vosotros** (`tenéis`, `escribid`) is in `es`, `es_AR` and `es_ES`, and
# largely absent from `es_MX`. Demanding it rejects the Mexican build.
#
# Requiring both at once leaves exactly one package standing: the generic
# `es`, which is the only one that accepts every variety of Spanish. That is
# the same reason fr ships `-comprehensive` — the only thing this dictionary
# can do is underline something, and *tienes* and *tenés* are both correct
# Spanish taught in different countries.
#
# The rest are shape checks: `escribiésemos` is the -se imperfect subjunctive,
# `dámelo` proves the enclitic pronoun rules ran, and `jardín`/`niño` prove
# FLAG UTF-8 was read as characters rather than bytes.
"es": {
"accept": (
# Rejects es_ES and Debian's hunspell-es.
"vení", "tenés", "querés", "sabés", "andá",
# Rejects es_MX.
"tenéis", "escribid",
# Rejects es_AR, which has both voseo and vosotros and would
# otherwise pass. Caribbean and Andean everyday words: the generic
# build is the union of all twenty-four, so it is the only one that
# holds another region's vocabulary as well as its own.
"arepa", "chévere", "bacán",
# Pan-Hispanic vocabulary. These pass on every RLA build, so they
# prove nothing on their own — kept because a source that stopped
# being RLA at all would fail them.
"computadora", "ordenador", "papa", "patata", "jugo", "zumo",
# Morphology and encoding.
"escribiéramos", "escribiésemos", "escríbeme", "dámelo",
"jardín", "niño", "corazón",
),
"reject": ("jardinn", "escribiz", "xyzzyque", "haiga"),
"wrong": "this is not the generic RLA build (a per-country one accepts "
"only some of these)",
},
}
def main(lang, aff_path, dic_path, out_dir):
profile = PROFILES.get(lang)
if profile is None:
raise SystemExit(f"no profile for {lang!r}; known: {', '.join(PROFILES)}")
forms, aff, needaffix_stems = expand(aff_path, dic_path)
missing = [w for w in profile["accept"] if w not in forms]
present = [w for w in profile["reject"] if w in forms]
if missing or present:
raise SystemExit(
f"{profile['wrong']}: missing {missing}, unexpectedly present {present}"
)
os.makedirs(out_dir, exist_ok=True)
ordered = sorted(forms)
body = f"{len(ordered)}\n" + "\n".join(ordered) + "\n"
dic_out = os.path.join(out_dir, f"{lang}.dic.gz")
# mtime=0 so rebuilding identical input produces an identical file — a
# vendored asset that changes on every build is noise in the diff.
with gzip.GzipFile(dic_out, "wb", compresslevel=9, mtime=0) as fh:
fh.write(body.encode("utf-8"))
aff_out = os.path.join(out_dir, f"{lang}.aff")
with open(aff_out, "w", encoding="utf-8") as fh:
fh.write(shipped_aff(aff_path))
print(f"{len(ordered)} forms -> {dic_out} "
f"({os.path.getsize(dic_out) / 1e6:.2f} MB gzipped); "
f"{needaffix_stems} NEEDAFFIX stems, "
f"{aff.dropped_apostrophe_rules} elision rules skipped", file=sys.stderr)
if __name__ == "__main__":
if len(sys.argv) != 5:
raise SystemExit(
"usage: build_hunspell_dictionary.py <lang> <aff> <dic> <out-dir>\n"
f" lang is one of: {', '.join(PROFILES)}"
)
main(*sys.argv[1:5])
-232
View File
@@ -1,232 +0,0 @@
#!/usr/bin/env python3
"""Build Petal's browser pt-PT spelling dictionary from the LibreOffice Hunspell one.
Why this script exists at all
----------------------------
English is vendored the obvious way: `dictionary-en`'s `en.aff` + `en.dic` go
into web/public/dictionaries/en and nspell reads them in the browser. The plan
for Phase 21 said "Hunspell pt-PT vendored like en-US", and that turns out not to
work, for a measured reason.
nspell expands affixes **eagerly at load time** it materialises every surface
form into a hash the moment you construct it. English gets away with this: ~50k
stems and a small rule set. European Portuguese does not. `pt_PT.aff` carries
1,340 affix rules (the full verb paradigm: six persons x a dozen tenses, plus
diminutives, plus productive prefixes) over 44,257 stems. Measured on this
machine, nspell needed ~340 MB of heap for the first 12,000 entries alone and had
not returned after three minutes on the whole file; extrapolated, it wants well
over a gigabyte. That is not something to hand a browser, still less a tablet.
So the expansion happens **here**, once, at build time, and the browser gets a
flat word list it can load with no affix machinery at all: 1,039,058 forms, 15 MB
of text, 2.7 MB gzipped, which nspell reads in ~0.8s using ~120 MB. The runtime
code path is then *identical* to English same nspell, same interface which is
the real prize. The aff we ship alongside keeps only the suggestion-shaping
directives (TRY/KEY/REP/MAP), so corrections still know that "cao" wants "ção"
and that a missing acute accent is a near miss.
Choosing the source
-------------------
npm's `dictionary-pt` is **not** European Portuguese. Both it and
`dictionary-pt-br` package VERO ("Verificador Ortográfico Livre", Brasil), so
vendoring the obvious npm name would have shipped Brazilian spellings under a
pt-PT label exactly the pt-BR drift SUGGESTIONS.md §3 warns about, arriving
through the packaging rather than through the model.
The authentic dictionary is the Projecto Natura one (Universidade do Minho) that
LibreOffice ships and Debian packages as `hunspell-pt-pt`. Its aff declares
`LANG pt_PT`. Spot-checked against the built list, it accepts `receção`,
`húmido`, `telemóvel`, `autocarro`, `comboio`, `ótimo` and `pensámos`, and
rejects `recepção`, `úmido`, `ônibus` and `óptimo` post-Acordo European
Portuguese, which is what a pt-PT writer should be held to.
Licensing: GPL-2 or LGPL-2.1 or MPL-1.1, (c) José João de Almeida, Rui Vilela,
Alberto Simões. The upstream copyright file is vendored beside the output.
Usage
-----
apt-get download hunspell-pt-pt # or take pt_PT.aff/.dic from LibreOffice
dpkg-deb -x hunspell-pt-pt_*.deb ptpt
python3 scripts/build_ptpt_dictionary.py \
ptpt/usr/share/hunspell/pt_PT.aff \
ptpt/usr/share/hunspell/pt_PT.dic \
web/public/dictionaries/pt-PT
Only the directives pt_PT.aff actually uses are implemented: PFX/SFX with
single-character flags, strip/append/condition, and cross-product. It carries no
compounding, no flag aliases and no NEEDAFFIX, so there is nothing else to
honour the script asserts that rather than assuming it.
"""
import gzip
import os
import re
import sys
# Directives worth keeping in the shipped aff. These shape *suggestions*, not
# membership: TRY orders the alphabet the corrector tries, KEY knows which keys
# are adjacent, REP holds Portuguese-specific confusions (cao/ção, ss/ç), and MAP
# says an accented vowel and its bare form are the same letter for scoring —
# which is most of what an ESL writer gets wrong in Portuguese.
KEEP_DIRECTIVES = ("SET", "TRY", "KEY", "REP", "MAP", "WORDCHARS")
# Directives that would change which words are *accepted*. If a future upstream
# release starts using one, this script's output would silently disagree with
# the real dictionary, so it stops instead.
UNSUPPORTED = (
"COMPOUNDFLAG", "COMPOUNDMIN", "COMPOUNDRULE", "COMPOUNDBEGIN",
"ONLYINCOMPOUND", "NEEDAFFIX", "PSEUDOROOT", "CIRCUMFIX",
"FORBIDDENWORD", "AF", "AM", "FLAG",
)
def parse_aff(path):
"""Return (prefix rules, suffix rules, cross-product flags) keyed by flag."""
with open(path, encoding="utf-8") as fh:
lines = fh.read().splitlines()
for line in lines:
head = line.split()[0] if line.split() else ""
if head in UNSUPPORTED:
raise SystemExit(
f"{path}: unsupported directive {head!r} — this expander only "
"handles plain PFX/SFX affixation, and honouring it would change "
"which words are accepted. Extend the script before shipping."
)
pfx, sfx, cross = {}, {}, {}
i = 0
while i < len(lines):
parts = lines[i].split()
if parts and parts[0] in ("PFX", "SFX"):
kind, flag, cross_flag, count = parts[0], parts[1], parts[2], int(parts[3])
table = pfx if kind == "PFX" else sfx
cross[flag] = cross_flag == "Y"
rules = table.setdefault(flag, [])
for j in range(1, count + 1):
p = lines[i + j].split()
strip = "" if p[2] == "0" else p[2]
# The appended text may carry its own continuation flags after a
# slash (append/FLAGS). We drop them: honouring them would mean
# affixing an affixed form, which pt_PT.aff does not do.
append = "" if p[3] == "0" else p[3].split("/")[0]
cond = p[4] if len(p) > 4 else "."
anchored = ("^" + cond) if kind == "PFX" else (cond + "$")
rules.append((strip, append, re.compile(anchored)))
i += count + 1
continue
i += 1
return pfx, sfx, cross
def apply_suffix(word, rules):
out = []
for strip, append, cond in rules:
if strip and not word.endswith(strip):
continue
if not cond.search(word):
continue
stem = word[: len(word) - len(strip)] if strip else word
out.append(stem + append)
return out
def apply_prefix(word, rules):
out = []
for strip, append, cond in rules:
if strip and not word.startswith(strip):
continue
if not cond.search(word):
continue
stem = word[len(strip):] if strip else word
out.append(append + stem)
return out
def expand(aff_path, dic_path):
pfx, sfx, cross = parse_aff(aff_path)
forms = set()
with open(dic_path, encoding="utf-8") as fh:
fh.readline() # leading entry count, not a word
for raw in fh:
entry = raw.strip().split("\t")[0] # drop morphological fields
if not entry:
continue
word, _, flagstr = entry.partition("/")
word = word.strip()
if not word:
continue
flags = set(flagstr.strip())
forms.add(word)
for f in flags:
if f in sfx:
forms.update(apply_suffix(word, sfx[f]))
prefixed = []
for f in flags:
if f in pfx:
prefixed.extend(apply_prefix(word, pfx[f]))
forms.update(prefixed)
# Cross-product: a prefixed form may also take a suffix, but only
# when both rules are declared cross-product ("Y").
for f in flags:
if f in pfx and cross.get(f):
for base in apply_prefix(word, pfx[f]):
for g in flags:
if g in sfx and cross.get(g):
forms.update(apply_suffix(base, sfx[g]))
return forms
def shipped_aff(aff_path):
keep = []
for line in open(aff_path, encoding="utf-8").read().splitlines():
head = line.split()[0] if line.split() else ""
if head in KEEP_DIRECTIVES:
keep.append(line)
return "\n".join(keep) + "\n"
# Words the built list must accept, and must reject, before it is written. These
# are the pt-PT/pt-BR fault lines: post-Acordo spellings, the European lexicon,
# and the first-person-plural preterite accent that only pt-PT writes. A source
# dictionary that fails these is not the one this script is for.
MUST_ACCEPT = ("receção", "húmido", "telemóvel", "autocarro", "comboio",
"ótimo", "pensámos", "escrevêssemos", "jardim")
MUST_REJECT = ("recepção", "úmido", "ônibus", "óptimo")
def main(aff_path, dic_path, out_dir):
forms = expand(aff_path, dic_path)
missing = [w for w in MUST_ACCEPT if w not in forms]
present = [w for w in MUST_REJECT if w in forms]
if missing or present:
raise SystemExit(
"this does not look like European Portuguese: "
f"missing {missing}, unexpectedly present {present}"
)
os.makedirs(out_dir, exist_ok=True)
ordered = sorted(forms)
body = f"{len(ordered)}\n" + "\n".join(ordered) + "\n"
dic_out = os.path.join(out_dir, "pt-PT.dic.gz")
# mtime=0 so rebuilding identical input produces an identical file — a
# vendored asset that changes on every build is noise in the diff.
with gzip.GzipFile(dic_out, "wb", compresslevel=9, mtime=0) as fh:
fh.write(body.encode("utf-8"))
aff_out = os.path.join(out_dir, "pt-PT.aff")
with open(aff_out, "w", encoding="utf-8") as fh:
fh.write(shipped_aff(aff_path))
print(f"{len(ordered)} forms -> {dic_out} "
f"({os.path.getsize(dic_out) / 1e6:.2f} MB gzipped)", file=sys.stderr)
if __name__ == "__main__":
if len(sys.argv) != 4:
raise SystemExit(__doc__.strip().splitlines()[-1])
main(*sys.argv[1:4])
+28 -10
View File
@@ -30,6 +30,7 @@
},
"devDependencies": {
"@tailwindcss/vite": "^4.0.0",
"@types/node": "^26.1.2",
"@types/react": "^19.1.0",
"@types/react-dom": "^19.1.0",
"@vitejs/plugin-react": "^4.3.4",
@@ -2107,6 +2108,16 @@
"integrity": "sha512-RGdgjQUZba5p6QEFAVx2OGb8rQDL/cPRG7GiedRzMcJ1tYnUANBncjbSB1NRGwbvjcPeikRABz2nshyPk1bhWg==",
"license": "MIT"
},
"node_modules/@types/node": {
"version": "26.1.2",
"resolved": "https://registry.npmjs.org/@types/node/-/node-26.1.2.tgz",
"integrity": "sha512-Vu4a5UFA9rIIFJ7rB/Vaafh9lrCQszopTCx6KjFboXTGQbPNasehVR5TEiithSDGyd1DEiUByggTZsg8jukeIg==",
"dev": true,
"license": "MIT",
"dependencies": {
"undici-types": "~8.3.0"
}
},
"node_modules/@types/react": {
"version": "19.2.17",
"resolved": "https://registry.npmjs.org/@types/react/-/react-19.2.17.tgz",
@@ -2928,9 +2939,9 @@
}
},
"node_modules/linkify-it": {
"version": "5.0.1",
"resolved": "https://registry.npmjs.org/linkify-it/-/linkify-it-5.0.1.tgz",
"integrity": "sha512-wVoTjP4Q6R0NW5hiZkVJaFZPWgtXfoGF+6LucL3/FtiNjmcHhYjEr5f1Kqjirc1nBW07J/ZuRFumqr2oqccEWg==",
"version": "5.0.2",
"resolved": "https://registry.npmjs.org/linkify-it/-/linkify-it-5.0.2.tgz",
"integrity": "sha512-ONTm2jCMAVZjgQa/Fy1kScXsuOoF5NPTsoFBdE1KVIZ2vAh/r9+Bqo+0jINCBYnavTPQZz38QzFTme79ENoN3Q==",
"funding": [
{
"type": "github",
@@ -3019,9 +3030,9 @@
"license": "MIT"
},
"node_modules/nanoid": {
"version": "3.3.15",
"resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.15.tgz",
"integrity": "sha512-y7Wygv/7mEOvxTuEQDB8StXdMRBWf1kR/tlhAzBRUFkB2jfcLOAxO/SHmOO2zgz1pVgK29/kyupn059/bCHdjA==",
"version": "3.3.16",
"resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.16.tgz",
"integrity": "sha512-bzlKTyNJ7+LdGIIwy8ijFpIqEQIvafahV7eYykJ8Cvh42EdJeODoJ6gUJXpQJvej1BddH8OqTXZNE/KfbWAu8Q==",
"dev": true,
"funding": [
{
@@ -3104,9 +3115,9 @@
}
},
"node_modules/postcss": {
"version": "8.5.15",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.15.tgz",
"integrity": "sha512-FfR8sjd4em2T6fb3I2MwAJU7HWVMr9zba+enmQeeWFfCbm+UOC/0X4DS8XtpUTMwWMGbjKYP7xjfNekzyGmB3A==",
"version": "8.5.23",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.23.tgz",
"integrity": "sha512-g50586zr4bZmwFiTlflMu8E0bDTb5I5gertgwAKmsdUlTQIhZtunzUlD1WSzwcVWPoAVpsrA6vlfCD7oXvRwgg==",
"dev": true,
"funding": [
{
@@ -3124,7 +3135,7 @@
],
"license": "MIT",
"dependencies": {
"nanoid": "^3.3.12",
"nanoid": "^3.3.16",
"picocolors": "^1.1.1",
"source-map-js": "^1.2.1"
},
@@ -3559,6 +3570,13 @@
"integrity": "sha512-ARDJmphmdvUk6Glw7y9DQ2bFkKBHwQHLi2lsaH6PPmz/Ka9sFOBsBluozhDltWmnv9u/cF6Rt87znRTPV+yp/A==",
"license": "MIT"
},
"node_modules/undici-types": {
"version": "8.3.0",
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.3.0.tgz",
"integrity": "sha512-j375ScV60dom+YkPFIfTLcOiPxkN/buHz5GobjLhixFuANaNs3C9l4GmrWqejgXWJ7BbJcFYpTEUkS1Ge8bpZQ==",
"dev": true,
"license": "MIT"
},
"node_modules/update-browserslist-db": {
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.2.3.tgz",
+1
View File
@@ -33,6 +33,7 @@
},
"devDependencies": {
"@tailwindcss/vite": "^4.0.0",
"@types/node": "^26.1.2",
"@types/react": "^19.1.0",
"@types/react-dom": "^19.1.0",
"@vitejs/plugin-react": "^4.3.4",
+68
View File
@@ -0,0 +1,68 @@
Spanish spelling dictionary
===========================
The word list in `es.dic.gz` and the suggestion directives in `es.aff` are
derived from the **generic** Spanish Hunspell dictionary published by the RLA-ES
project ("Recursos Lingüísticos Abiertos del Español"), release v2.9.
Copyright (C) Santiago Bosio and the RLA-ES contributors
License: GPL-3+ or LGPL-3+ or MPL-1.1+
Tri-licensed; you may choose freely among the three. Petal
redistributes under the MPL. Full texts:
https://www.gnu.org/licenses/gpl-3.0.en.html
https://www.gnu.org/licenses/lgpl-3.0.en.html
https://www.mozilla.org/en-US/MPL/1.1/
Upstream: https://github.com/sbosio/rla-es
Source: https://github.com/sbosio/rla-es/releases/download/v2.9/es.oxt
(an .oxt is a zip; es.aff and es.dic are at its root)
Not the Debian package, and that is the point
---------------------------------------------
`hunspell-es` looks like the obvious source and is the wrong one. It installs
twenty country codes — `es_AR` through `es_VE` — all symlinked to a single file,
which reads like "one pan-Hispanic dictionary". It is not. RLA publishes
twenty-four dictionaries per release: one per country, plus a **generic `es`**
that is the union of all of them, and Debian ships the **peninsular `es_ES`**
build under the collapsed name.
Measured against v2.9, expanded to surface forms:
Debian hunspell-es 659,085 forms voseo: no vosotros: yes
upstream es_ES 659,018 forms voseo: no vosotros: yes
upstream es_MX 554,923 forms voseo: no vosotros: no
upstream es_AR 669,605 forms voseo: yes vosotros: yes
upstream es (generic) 717,640 forms voseo: yes vosotros: yes <-- this
The 58,622-form gap between Debian's file and the generic one is essentially the
**voseo** paradigm — `vení`, `tenés`, `querés`, `sabés`, `andá` — the ordinary
present tense of Argentina, Uruguay, Paraguay and much of Central America. Under
the Debian package, a writer using it would have had her own verbs underlined as
misspellings.
Why the generic build rather than one country
---------------------------------------------
The only thing this dictionary can do is underline something. *Tienes* and
*tenés* are both correct Spanish, taught in different countries, and a writing
companion has no business marking one of them wrong — the same reasoning that
makes the French dictionary here the `-comprehensive` packaging rather than
`-classical` or `-revised`. The generic build accepts every variety, so Petal
underlines only what no Spanish speaker anywhere would write.
How the build proves it got this file
-------------------------------------
Vocabulary cannot tell these builds apart: *every* RLA variant carries the full
pan-Hispanic lexicon, so *computadora* alongside *ordenador* passes on the
peninsular file too. (The `REP` table is likewise no evidence — its `ll`/`y` and
`ás`/`az` entries look like yeísmo and seseo but are shared by all builds.) Only
the verb paradigms are localised, so the `es` profile in
`scripts/build_hunspell_dictionary.py` demands, all at once:
* **voseo** (`vení`, `tenés`, `querés`) — rejects `es_ES` and Debian's package;
* **vosotros** (`tenéis`, `escribid`) — rejects `es_MX`;
* **another region's everyday words** (`arepa`, `chévere`, `bacán`) — rejects
`es_AR`, which has both paradigms and would otherwise pass.
Only the generic build satisfies all three. Each of the four neighbouring builds
was run through the profile and confirmed to fail.
+29
View File
@@ -0,0 +1,29 @@
SET UTF-8
TRY aeroinsctldumpbgfvhzóíjáqéñxyúükwAEROINSCTLDUMPBGFVHZÓÍJÁQÉÑXYÚÜKW
REP 19
REP ás az
REP az ás
REP cc x
REP és ez
REP ez és
REP güe hue
REP güi hui
REP hue güe
REP hui güi
REP ís iz
REP ío ido
REP ke que
REP ki qui
REP ll y
REP mb nv
REP nv mb
REP seci cesi
REP x cc
REP y ll
MAP 6
MAP aáAÁ
MAP eéEÉ
MAP iíIÍ
MAP oóOÓ
MAP uúüUÚÜ
MAP nñNÑ
Binary file not shown.
+49
View File
@@ -0,0 +1,49 @@
French spelling dictionary
==========================
The word list in `fr.dic.gz` and the suggestion directives in `fr.aff` are
derived from the Dicollecte / Grammalecte Hunspell dictionary for French
(`fr.aff` / `fr.dic`), as packaged by Debian/Ubuntu in
`hunspell-fr-comprehensive`.
Copyright (C) 2007-2018 the Dicollecte contributors
(full list at
https://grammalecte.net/members.php?prj=fr)
Dictionary author: Olivier R.
License: MPL-2.0
This Source Code Form is subject to the terms of the Mozilla
Public License, v. 2.0. If a copy of the MPL was not distributed
with this file, You can obtain one at
http://mozilla.org/MPL/2.0/.
Upstream: https://grammalecte.net/home.php?prj=fr
Which of the three
------------------
Debian packages this dictionary three ways, by how it treats the 1990 spelling
reform: `hunspell-fr-classical` (traditional spellings), `hunspell-fr-revised`
(reform spellings) and `hunspell-fr-comprehensive` (both). Petal ships the
**comprehensive** one, because Petal never corrects her French — the only thing
this dictionary can do is underline something, and *coût* and *cout* are both
correct French. The regional packages (`fr_FR`, `fr_CA`, `fr_BE`, `fr_CH`,
`fr_LU`, `fr_MC`) are all symlinks to the same word list, so there is no
regional choice being made here.
What Petal changed
------------------
`scripts/build_hunspell_dictionary.py` applies the upstream affix rules ahead of
time — Hunspell's PFX/SFX expansion run once at build time instead of once per
browser — and writes the resulting 473,326 surface forms as a flat word list.
The shipped `.aff` keeps only upstream's TRY/KEY/REP/MAP/WORDCHARS lines, which
shape *corrections* rather than membership.
One thing about membership did change, and it is reversible at lookup rather
than lost: the elided forms (`l'arbre`, `qu'elle`, `jusqu'ici`) are **not** in
the word list. Expanding them costs 8.25 MB gzipped against 1.19 MB, and they
are not new words — they are thirteen clitics glued to words already present —
so `withElision` in `web/src/hooks/useSpellChecker.ts` splits the token and
checks the remainder instead. Stems that carry an apostrophe of their own
(`aujourd'hui`, `quelqu'un`, `presqu'île`, `prud'homme`) are kept verbatim.
+141
View File
@@ -0,0 +1,141 @@
SET UTF-8
WORDCHARS -'1234567890.
TRY esntiarulodcpmévqfgbhàxèjyêMILzACçôîPâùJFSûBVœRDGNETHXkïOwKWYUëQÉZŒüãÎáöóÈíæÅñäśńÿ
MAP 25
MAP aàâäAÀÂÄ
MAP eéèêëEÉÈÊË
MAP iîïyIÎÏY
MAP oôöOÔÖ
MAP uùûüUÙÛÜ
MAP cçCÇ
MAP bB
MAP dD
MAP fF
MAP gG
MAP hH
MAP jJ
MAP kK
MAP lL
MAP mM
MAP nN
MAP pP
MAP qQ
MAP rR
MAP sS
MAP tT
MAP vV
MAP wW
MAP xX
MAP zZ
REP 110
REP a â
REP â a
REP e é
REP é e
REP e ê
REP ê e
REP e è
REP è e
REP i î
REP î i
REP o ô
REP ô o
REP u û
REP û u
REP A Â
REP Â A
REP E É
REP É E
REP E Ê
REP Ê E
REP E È
REP È E
REP I Î
REP Î I
REP O Ô
REP Ô O
REP U Û
REP Û U
REP ^Ca$ Ça
REP ^l l'
REP ^d d'
REP ^n n'
REP ^s s'
REP ^j j'
REP ^m m'
REP ^t t'
REP ^c c'
REP f ph
REP ph f
REP c qu
REP qu c
REP k qu
REP qu k
REP x ct
REP ct x
REP bb b
REP b bb
REP cc c
REP c cc
REP ff f
REP f ff
REP ll l
REP l ll
REP mm m
REP m mm
REP nn n
REP n nn
REP pp p
REP p pp
REP rr r
REP r rr
REP ss s
REP s ss
REP ss c
REP c ss
REP ss ç
REP ç ss
REP tt t
REP t tt
REP œ oe
REP oe œ
REP æ ae
REP ae æ
REP ai é
REP é ai
REP ai è
REP è ai
REP ai ê
REP ê ai
REP ei é
REP é ei
REP ei è
REP è ei
REP ei ê
REP ê ei
REP o au
REP au o
REP o eau
REP eau o
REP ett èt
REP èt ett
REP ell èl
REP èl ell
REP t th
REP th t
REP ième$ e
REP ème$ e
REP è$ e
REP mn$ min
REP ogue$ ogiste
REP ogiste$ ogue
REP disez$ dites
REP fesez$ faites
REP faisez$ faites
REP puit puits
REP sanctionnable punissable
REP questionnable discutable
REP antitartre détartrant
REP email courriel
REP construirent construisirent
KEY azertyuiop|qsdfghjklmù|wxcvbn|aéz|yèu|iço|oàp|aqz|zse|edr|rft|tgy|yhu|uji|iko|olpm|qws|sxd|dcf|fvg|gbh|hnj
Binary file not shown.
+1 -1
View File
@@ -20,7 +20,7 @@ https://git.libreoffice.org/dictionaries/+/refs/heads/master/pt_PT
What Petal changed
------------------
Nothing about which words are correct. `scripts/build_ptpt_dictionary.py`
Nothing about which words are correct. `scripts/build_hunspell_dictionary.py`
applies the upstream affix rules ahead of time — Hunspell's PFX/SFX expansion
run once at build time instead of once per browser — and writes the resulting
1,039,058 surface forms as a flat word list. The shipped `.aff` keeps only
+59
View File
@@ -0,0 +1,59 @@
Chinese word list (segmentation)
================================
`words.txt.gz` is not a spelling dictionary — Chinese has no spelling to check
in the Hunspell sense. It is the word list Petal's segmenter walks, so that a
sentence written without spaces has words in it to hover, look up and capture.
Each line is `word frequency`. See scripts/build_cedict.py for how it is built
and why it is gated where it is.
It is derived from two upstream sources, both redistributable, both credited
here because the file itself has no room for a header.
CC-CEDICT — the headwords
-------------------------
Community maintained free Chinese-English dictionary, published by MDBG.
https://www.mdbg.net/chinese/dictionary?page=cedict
Licensed under the Creative Commons Attribution-ShareAlike 4.0 International
License — https://creativecommons.org/licenses/by-sa/4.0/
Referenced works:
CEDICT — Copyright (C) 1997, 1998 Paul Andrew Denisowski
CC-CEDICT is also the source of `internal/lexicon/data/hanzi.json.gz`, the
pinyin and English senses embedded in the Petal binary. The same attribution and
the same ShareAlike terms apply to that file; it is named here because it has
nowhere of its own to say so.
jieba — the frequencies
-----------------------
"结巴" Chinese word segmentation, by Sun Junyi.
https://github.com/fxsjy/jieba
MIT License
Copyright (c) 2013 Sun Junyi
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Only the word/frequency columns are used; jieba's part-of-speech tags and its
algorithm are not (Petal's segmenter is its own, in web/src/lib/segment.ts).
Binary file not shown.
+104 -25
View File
@@ -1,12 +1,14 @@
import { useCallback, useEffect, useRef, useState } from 'react'
import { api, type DocSummary, type DocUpdate, type Document, type Suggestion, type Tag, type TagColor } from './api/client'
import { useAutoSave } from './hooks/useAutoSave'
import { useCheckpoint } from './hooks/useCheckpoint'
import { findingKey, useCheckpoint } from './hooks/useCheckpoint'
import { useSpellChecker } from './hooks/useSpellChecker'
import { useSegmenter } from './hooks/useSegmenter'
import { useTags } from './hooks/useTags'
import { DocList } from './components/DocList/DocList'
import { EditorCore, type EditorChange } from './components/Editor/EditorCore'
import { ToneSelect } from './components/Editor/ToneSelect'
import { ChromeStrip } from './components/Editor/ChromeStrip'
import { ExportMenu } from './components/Export/ExportMenu'
import { HistoryPanel } from './components/History/HistoryPanel'
import { GardenPanel } from './components/Garden/GardenPanel'
@@ -21,6 +23,7 @@ import { PetalFall } from './effects/PetalFall'
import { usePack } from './i18n'
import { useNightMode } from './hooks/useNightMode'
import { playSuggestionSound } from './audio/sounds'
import { fromIME } from './lib/ime'
export default function App() {
const updateAvailable = useVersionWatch()
@@ -30,7 +33,7 @@ export default function App() {
const night = useNightMode()
// Who's writing, and whether the server still recognises them. `signedOut`
// flips the moment any call comes back 401.
const { me, signedOut } = useSession()
const { me, signedOut, setDirection, setPair } = useSession()
const t = usePack()
// A real account to sign out of, as opposed to the hardcoded local user a
// build without auth configured runs as.
@@ -79,6 +82,13 @@ export default function App() {
}, [])
const { status, schedule, saveNow } = useAutoSave(currentDoc?.id ?? null)
// The Chinese word list, for a writer going the other way through the zh pair.
// Gated on the account's own setting rather than on anything in the text: a
// Mandarin native drafting English quotes Chinese constantly, and none of that
// is what segmentation is for. Declared above the checkpoint because the
// offline 错别字 pass reads it.
const segmenter = useSegmenter(me?.direction === 'learning_pair')
const {
suggestions,
checking,
@@ -89,7 +99,8 @@ export default function App() {
runVoice,
runCollocation,
removeSuggestion,
} = useCheckpoint(currentDoc?.id ?? null)
resolveServerId,
} = useCheckpoint(currentDoc?.id ?? null, segmenter)
// Browser-side spell checker — loads the en-US dictionary once per session.
const { checker: spellChecker, addWord } = useSpellChecker()
// The tag roster (with counts). Assignments live on the doc summaries below.
@@ -321,14 +332,25 @@ export default function App() {
const handleEditorChange = useCallback(
(change: EditorChange) => {
const { composing, ...patch } = change
setWordCount(change.word_count)
setDocText(change.content_text)
setEditTick((n) => n + 1)
if (currentDoc) {
patchSummary(currentDoc.id, { word_count: change.word_count })
schedule(change)
scheduleCheckpoint(change.content_text)
// The save is never held: see EditorChange.composing. The flag itself
// stays out of the patch — it describes the keyboard, not the document,
// and the stashed draft a signed-out save leaves behind should be the
// document alone.
schedule(patch)
}
// Everything below reads the text as prose. While an IME composition is
// in flight it is not prose yet — it is the pinyin she is converting — so
// the checkpoint, the rule pack and the companion all wait for the word
// to commit. EditorCore emits one more change the moment it does, so
// nothing is skipped, only deferred by the length of a word.
if (composing) return
setDocText(change.content_text)
setEditTick((n) => n + 1)
if (currentDoc) scheduleCheckpoint(change.content_text)
},
[currentDoc, patchSummary, schedule, scheduleCheckpoint],
)
@@ -349,17 +371,44 @@ export default function App() {
// Accept applies the replacement in the editor (handled in EditorCore) and
// marks the suggestion accepted; dismiss just rejects it. Both drop it locally.
// A rule-pack card can be accepted before its row exists — the edit has already
// landed either way, so a missing id just means there's nothing to file.
const handleAccept = useCallback(
async (s: Suggestion) => {
removeSuggestion(s.id)
setAcceptTick((n) => n + 1)
try {
await api.acceptSuggestion(s.id)
const id = await resolveServerId(s)
if (id) await api.acceptSuggestion(id)
} catch (err) {
console.error('accept failed', err)
}
},
[removeSuggestion],
[removeSuggestion, resolveServerId],
)
// Accept-all: EditorCore has already applied the whole category in one editor
// transaction, so this is only the bookkeeping. Each row is filed individually
// (there's no batch endpoint, and each accept plants its own word in the
// garden), but the kitten cheers once — five cheers for one click would read as
// five separate congratulations for a decision she made once.
const handleAcceptMany = useCallback(
async (list: Suggestion[]) => {
if (list.length === 0) return
for (const s of list) removeSuggestion(s.id)
setAcceptTick((n) => n + 1)
await Promise.all(
list.map(async (s) => {
try {
const id = await resolveServerId(s)
if (id) await api.acceptSuggestion(id)
} catch (err) {
console.error('accept failed', err)
}
}),
)
},
[removeSuggestion, resolveServerId],
)
// After restoring a version, swap the restored doc into the editor. Bumping
@@ -377,11 +426,14 @@ export default function App() {
[patchSummary],
)
// Escape always restores the sidebar while in distraction-free mode.
// Escape always restores the sidebar while in distraction-free mode — unless
// it belongs to an IME, where it cancels a candidate and never reaches Petal
// at all. This is the writer typing Chinese in the very mode built for
// uninterrupted writing, so it is the one worth getting right.
useEffect(() => {
if (!focusMode) return
const onKey = (e: KeyboardEvent) => {
if (e.key === 'Escape') setFocusMode(false)
if (e.key === 'Escape' && !fromIME(e)) setFocusMode(false)
}
window.addEventListener('keydown', onKey)
return () => window.removeEventListener('keydown', onKey)
@@ -397,12 +449,13 @@ export default function App() {
async (s: Suggestion) => {
removeSuggestion(s.id)
try {
await api.dismissSuggestion(s.id)
const id = await resolveServerId(s)
if (id) await api.dismissSuggestion(id)
} catch (err) {
console.error('dismiss failed', err)
}
},
[removeSuggestion],
[removeSuggestion, resolveServerId],
)
// Play a soft sound when freshly-checked suggestions arrive — one per distinct
@@ -410,10 +463,14 @@ export default function App() {
// a pile-up. We track which ids we've already chimed for, and only chime for
// recently-created suggestions so opening a doc with old pending advice stays
// silent (the existing set was created in a past session).
// Rule-pack findings are chimed by their wording, not their id: the same fix
// appears first as a provisional card and then as its persisted row, and the
// writer should hear it once.
const chimedRef = useRef<Set<string>>(new Set())
useEffect(() => {
const fresh = suggestions.filter((s) => !chimedRef.current.has(s.id))
fresh.forEach((s) => chimedRef.current.add(s.id))
const key = (s: Suggestion) => (s.source === 'local' ? `local:${findingKey(s)}` : s.id)
const fresh = suggestions.filter((s) => !chimedRef.current.has(key(s)))
fresh.forEach((s) => chimedRef.current.add(key(s)))
const justMade = fresh.filter(
(s) => Date.now() - new Date(s.created_at).getTime() < 12_000,
)
@@ -435,7 +492,7 @@ export default function App() {
<PetalFall night={night} />
<header
onMouseDown={handleChromeDown}
className="petal-no-print flex h-12 shrink-0 items-center gap-2 px-5"
className="petal-no-print flex h-12 shrink-0 items-center gap-2 px-3 md:px-5"
style={{ borderBottom: '1px solid var(--color-border)' }}
>
<button
@@ -451,7 +508,15 @@ export default function App() {
</button>
<span className="text-xl">🌸</span>
<span className="text-lg font-extrabold text-plum">Petal</span>
{/* The wordmark is the first thing to go on a small phone. The header
holds a hamburger, a name and the garden button, and the garden
button's label is a langpack string: · Garden is 130px
where Jardim de palavras · Garden is nearly 200, which is the
difference between fitting a 320px screen and scrolling the whole
app sideways. Of everything in this row, the one that can be spared
is the app's own name she is already inside the app, and the
blossom stays. */}
<span className="petal-wordmark text-lg font-extrabold text-plum">Petal</span>
<button
type="button"
onClick={() => setGardenOpen(true)}
@@ -486,6 +551,9 @@ export default function App() {
onToggleTag={handleToggleTag}
onCreateTag={handleCreateTag}
account={account}
direction={me?.direction}
onDirection={setDirection}
onPair={setPair}
/>
</div>
@@ -500,27 +568,36 @@ export default function App() {
<>
<div
onMouseDown={handleChromeDown}
className="flex flex-1 flex-col overflow-y-auto px-6 py-8"
// `petal-scrollport` marks this as the editor's scrolling
// ancestor; EditorCore measures it to pin the text column while
// the suggestion rail's overhang is scrolled.
className="petal-scrollport flex flex-1 flex-col overflow-y-auto px-6 py-8"
>
<div ref={canvasRef} className="mx-auto flex w-full max-w-[720px] flex-1 flex-col">
<div className="mb-5 flex items-center gap-3">
{/* Title, then the three chrome pills. Their labels are
bilingual and don't shrink, so how much width this row
wants is a property of the langpack is two glyphs
where Historique is ten and on a phone no language's
version of it fits. The pills therefore live in a strip
that scrolls itself; the title takes its own line below
the drawer breakpoint so it keeps its full width. */}
<div className="mb-5 flex flex-wrap items-center gap-2 md:gap-3">
<input
value={title}
onChange={(e) => handleTitleChange(e.target.value)}
placeholder="Untitled"
aria-label="Document title"
className="min-w-0 flex-1 bg-transparent text-3xl font-extrabold text-plum focus:outline-none"
className="min-w-0 flex-1 basis-full bg-transparent text-3xl font-extrabold text-plum focus:outline-none md:basis-0"
style={{ fontFamily: 'var(--font-ui)' }}
/>
<div className="petal-no-print shrink-0">
<ChromeStrip className="petal-no-print flex items-center gap-2 py-0.5 md:gap-3">
<ToneSelect value={tone} onChange={handleToneChange} />
</div>
<button
type="button"
onClick={() => setHistoryOpen(true)}
aria-label="Version history"
title="Browse and restore earlier versions"
className="petal-no-print inline-flex h-9 shrink-0 items-center gap-1.5 whitespace-nowrap px-3 text-sm font-bold"
className="inline-flex h-9 shrink-0 items-center gap-1.5 whitespace-nowrap px-3 text-sm font-bold"
style={{
borderRadius: 'var(--radius-pill)',
background: 'var(--color-surface)',
@@ -532,17 +609,18 @@ export default function App() {
<span>{t.app.history}</span>
<span style={{ color: 'var(--color-muted)' }}>· History</span>
</button>
<div className="petal-no-print">
<ExportMenu docId={currentDoc.id} />
</div>
</ChromeStrip>
</div>
<EditorCore
key={`${currentDoc.id}:${editorEpoch}`}
docId={currentDoc.id}
initialContent={currentDoc.content}
onChange={handleEditorChange}
segmenter={segmenter}
suggestions={suggestions}
onAccept={handleAccept}
onAcceptMany={handleAcceptMany}
onDismiss={handleDismiss}
onVoiceCheck={runVoice}
voicing={voicing}
@@ -563,6 +641,7 @@ export default function App() {
voicing={voicing}
collocating={collocating}
llmDown={llmDown}
suggestionCount={suggestions.length}
/>
</div>
</>
+53 -1
View File
@@ -101,7 +101,18 @@ export interface Gloss {
reverse?: string
}
export type SuggestionType = 'grammar' | 'phrasing' | 'idiom' | 'clarity' | 'voice' | 'collocation' | 'mechanics'
// 'translate' is a span she wrote in her own language, rendered into English —
// not a correction. The server decides the label from the span itself, never from
// the model, so the client can trust it (see suggestions/language.go).
export type SuggestionType =
| 'grammar'
| 'phrasing'
| 'idiom'
| 'clarity'
| 'translate'
| 'voice'
| 'collocation'
| 'mechanics'
// One word in the vocabulary garden: a looked-up word with its gloss/phonetic,
// the sentence it was met in, and its spaced-repetition state. `reps` drives how
@@ -199,6 +210,21 @@ export interface PersonalWords {
words: string[]
}
// One pronunciation of a Chinese word, and what it means in that pronunciation.
// A list, because 得 is dé "to obtain" and also the particle in 说得很好.
export interface HanziReading {
pinyin: string
senses: string
}
// A Chinese word lookup. `readings` is empty for a word with no headword, in
// which case `chars` may carry the character-by-character reading.
export interface HanziInfo {
word: string
readings: HanziReading[]
chars: { char: string; pinyin: string; senses: string }[]
}
// Who's writing. Mirrors the backend db.User.
export interface Me {
id: string
@@ -206,6 +232,11 @@ export interface Me {
display_name: string
created_at: string
pair_lang: string
// Which half of the pair is being learned: 'learning_en' (the writer is
// native in pair_lang and practising English) or 'learning_pair' (the other
// way round). Mirrors users.direction; the server refuses 'learning_pair' for
// a pair it has no word list for.
direction: string
}
// Thrown when the server says the session is gone. Callers can tell it apart
@@ -258,6 +289,15 @@ export const api = {
setPairLang: (lang: string) =>
req<Me>('/me', { method: 'PATCH', body: JSON.stringify({ pair_lang: lang }) }),
// Turn the pair around. Same endpoint, same contract, and deliberately a
// separate call: the two fields are validated together server-side, so a
// client that wants to change both says both in one request rather than
// sending two that each pass on their own.
setDirection: (direction: string) =>
req<Me>('/me', { method: 'PATCH', body: JSON.stringify({ direction }) }),
setPair: (lang: string, direction: string) =>
req<Me>('/me', { method: 'PATCH', body: JSON.stringify({ pair_lang: lang, direction }) }),
listDocs: () => req<DocSummary[]>('/docs'),
createDoc: () => req<Document>('/docs', { method: 'POST' }),
getDoc: (id: string) => req<Document>(`/docs/${id}`),
@@ -287,6 +327,12 @@ export const api = {
}),
// Pending suggestions for a doc, loaded when the editor opens it.
listSuggestions: (id: string) => req<Suggestion[]>(`/docs/${id}/suggestions`),
// The spans she has already accepted or dismissed on this doc, normalized. The
// server suppresses these itself; the client needs them so the instant rule-pack
// pass doesn't hand back a dismissed card before the server can say otherwise —
// or, with the server unreachable, at all. See lib/settled.ts.
listSettled: (id: string) =>
req<{ originals: string[] }>(`/docs/${id}/settled`),
acceptSuggestion: (id: string) =>
req<void>(`/suggestions/${id}/accept`, { method: 'POST' }),
dismissSuggestion: (id: string) =>
@@ -331,6 +377,12 @@ export const api = {
// Lightweight Chinese-only gloss for the inline hover/select tooltip — instant
// and offline, so it fires on hover without spinning up the heavier lookup.
glossWord: (word: string) => req<Gloss>(`/gloss/${encodeURIComponent(word)}`),
// The same lookup pointing the other way: a Chinese word to its pinyin and
// English senses, for an account learning the pair language rather than
// English. A word the dictionary has no headword for comes back with empty
// readings and — when its characters are known — a per-character reading
// instead, which is a real second answer for a compound.
hanziWord: (word: string) => req<HanziInfo>(`/hanzi/${encodeURIComponent(word)}`),
// Tone-rewrite: rewrites a selected passage in the given style ('natural',
// 'academic', …) and returns the rewritten text for an in-editor preview. Not
// persisted — the editor applies it directly on accept.
+11
View File
@@ -61,6 +61,8 @@ describe('nativeLang', () => {
expect(nativeLang()).toBe('zh-CN')
setPackLang('pt-PT')
expect(nativeLang()).toBe('pt-PT')
setPackLang('fr')
expect(nativeLang()).toBe('fr-FR')
})
it('names a European Portuguese voice, never a Brazilian one', () => {
@@ -73,4 +75,13 @@ describe('nativeLang', () => {
speak('comum', nativeLang())
expect(bodies[0]).toMatchObject({ text: 'comum', lang: 'pt-PT' })
})
it('speaks the French reading of a collision in French', () => {
// "chat" is the sharpest case in the fr pair: an English word, a French
// word, and the companion's own animal. Nothing about the letters says
// which — only the pack does.
setPackLang('fr')
speak('chat', nativeLang())
expect(bodies.at(-1)).toMatchObject({ text: 'chat', lang: 'fr-FR' })
})
})
@@ -3,6 +3,7 @@ import type { SaveStatus } from '../../hooks/useAutoSave'
import { useCompanion, type Mood } from './useCompanion'
import { LottiePlayer } from './LottiePlayer'
import { COMPANIONS, DEFAULT_COMPANION } from './companions'
import { useCardOverlap } from './useCardOverlap'
import { onPrefsScopeChange, readPref, writePref } from '../../lib/prefs'
import { usePack } from '../../i18n'
@@ -86,6 +87,15 @@ export function PetalCompanion({
const companion = COMPANIONS.find((c) => c.id === companionId) ?? COMPANIONS[0]
const [pickerOpen, setPickerOpen] = useState(false)
const rootRef = useRef<HTMLDivElement>(null)
const badgeRef = useRef<HTMLButtonElement>(null)
// When suggestion cards stack down into the corner, the kitten fades to
// translucent and shrinks a step so the card stays readable and clickable.
// It wakes back up whenever it has something to say (bubble) or is being
// interacted with (picker open) — except under an open History or Garden
// panel, where even a cheer would cover the controls she just reached for.
const crowded = useCardOverlap(badgeRef)
const faded = crowded.modal || (crowded.cards && !pickerOpen && !bubble)
// Awake companions (no sleeping clip) don't visibly nap — when the engine
// dozes them, keep their normal idle pose instead of a sleepy face. Only a
@@ -176,7 +186,10 @@ export function PetalCompanion({
</div>
)}
{bubble && !pickerOpen && (
{/* The bubble is its own layer, so fading the badge doesn't hide it
hold it back explicitly while a panel is open. useCompanion keeps the
bubble in state, so it reappears when she closes the panel. */}
{bubble && !pickerOpen && !crowded.modal && (
<div
role="status"
onClick={dismiss}
@@ -239,11 +252,14 @@ export function PetalCompanion({
)}
<button
ref={badgeRef}
type="button"
onClick={() => setPickerOpen((o) => !o)}
title="Choose a companion"
aria-label="Choose a companion"
className={`petal-companion pointer-events-auto select-none ${napping ? 'petal-companion-sleep' : ''}`}
className={`petal-companion select-none ${faded ? 'petal-companion-faded' : 'pointer-events-auto'}${
napping ? ' petal-companion-sleep' : ''
}`}
style={{
// Size scales with the viewport — see --petal-companion-size in index.css.
width: 'var(--petal-companion-size)',
+141
View File
@@ -0,0 +1,141 @@
import { readFileSync } from 'node:fs'
import { gunzipSync } from 'node:zlib'
import { describe, expect, it } from 'vitest'
import { CONFUSION_PAIRS, hanziFindings } from './hanzi'
import { buildSegmenter } from '../../lib/segment'
// The 错别字 pack, held to the bar Phase 22 set for the English rule pack: every
// rule pinned in *two* directions — the mistake it must catch, and the correct
// writing next to it that it must leave alone.
//
// Here the second direction is the one that matters, and it is unusually easy to
// get wrong. Chinese has no spaces, so every one of these rules is a substring
// match on running text, and for most of them there exists an ordinary correct
// sentence that contains the substring across a word boundary. Those sentences
// are the real test.
const raw = gunzipSync(readFileSync(new URL('../../../public/dictionaries/zh/words.txt.gz', import.meta.url)))
const seg = buildSegmenter(raw.toString('utf8'))
const flagged = (text: string) => hanziFindings(text, seg).map((f) => `${f.original}${f.replacement}`)
describe('the gate that admits a rule', () => {
// The pack's own claim about itself, checked against the shipped dictionary
// rather than asserted in a comment. A pair whose wrong form is a real word
// cannot be decided mechanically and does not belong here.
it('every wrong form is not a word, and every right form is', () => {
for (const { wrong, right } of CONFUSION_PAIRS) {
expect(seg.has(wrong), `${wrong} is a dictionary word and must not be flagged`).toBe(false)
expect(seg.has(right), `${right} is not a dictionary word`).toBe(true)
}
})
// The errors this pack deliberately refuses, and why — each is a genuine
// mistake by a modern standard whose wrong form is itself a headword. If a
// dictionary rebuild ever drops one of these, this test fails and the pair
// becomes admissible; that is the intended way to find out.
it('refuses the well-known errors it cannot decide', () => {
for (const undecidable of ['自已', '好象', '倒底', '帐号', '部份']) {
expect(seg.has(undecidable), `${undecidable} is no longer a word — reconsider the rule`).toBe(true)
expect(flagged(`这是${undecidable}的例子`)).toEqual([])
}
})
})
describe('the mistakes it catches', () => {
it('已 / 己 / 以', () => {
expect(flagged('我己经写完了作业')).toEqual(['己经→已经'])
expect(flagged('我以经吃过饭了')).toEqual(['以经→已经'])
expect(flagged('下课已后我们去公园')).toEqual(['已后→以后'])
})
it('在 / 再', () => {
expect(flagged('明天在见')).toEqual(['在见→再见'])
expect(flagged('他正再看书')).toEqual(['正再→正在'])
expect(flagged('现再几点了')).toEqual(['现再→现在'])
})
it('做 / 作', () => {
expect(flagged('我的工做很忙')).toEqual(['工做→工作'])
expect(flagged('老师给我们很多做业')).toEqual(['做业→作业'])
expect(flagged('这本书的做者是谁')).toEqual(['做者→作者'])
})
it('the rest', () => {
expect(flagged('我觉的这个很好')).toEqual(['觉的→觉得'])
expect(flagged('你因该早点睡')).toEqual(['因该→应该'])
expect(flagged('即然你来了就坐下吧')).toEqual(['即然→既然'])
expect(flagged('你知到吗')).toEqual(['知到→知道'])
expect(flagged('请输入你的蜜码')).toEqual(['蜜码→密码'])
})
it('reports an exact span, so the card replaces the right characters', () => {
const text = '我己经到了'
const [f] = hanziFindings(text, seg)
expect(text.slice(f.from, f.to)).toBe('己经')
expect(text.slice(0, f.from) + f.replacement + text.slice(f.to)).toBe('我已经到了')
})
it('finds every occurrence, in document order', () => {
expect(flagged('我己经吃了,他也己经吃了')).toEqual(['己经→已经', '己经→已经'])
expect(flagged('我的工做很忙,所以我觉的很累')).toEqual(['工做→工作', '觉的→觉得'])
})
})
// ── the direction that matters ──────────────────────────────────────────────
describe('the correct writing it must not touch', () => {
// Each of these is an ordinary sentence containing a flagged substring across
// a word boundary. Without the boundary gate, every one would be corrupted —
// and corrupted silently, into text that is still made of real characters.
it('leaves two real words alone where they happen to abut', () => {
// 自己 + 经常. The substring is 己经.
expect(flagged('他自己经常做饭')).toEqual([])
// 睡觉 + 的. The substring is 觉的.
expect(flagged('睡觉的时候不要看手机')).toEqual([])
// 感觉 + 的.
expect(flagged('这是我感觉的方向')).toEqual([])
// 不知 + 到底.
expect(flagged('我不知到底该怎么办')).toEqual([])
// 因 + 位置.
expect(flagged('因位置不好我们换了座位')).toEqual([])
// 已 + 后悔.
expect(flagged('他已后悔了')).toEqual([])
})
it('leaves ordinary correct prose entirely alone', () => {
for (const good of [
'我今天早上去公园跑步了',
'他的中文说得很好',
'我已经完成了我的作业',
'现在几点了,我们再见面吧',
'我觉得这个工作很有意思',
'既然你已经知道了,就按照计划做',
]) {
expect(flagged(good), good).toEqual([])
}
})
// Where the gate costs the pack a real catch, and the trade it is making.
// 不知 is itself a word, so 我不知到他在哪里 — which really is 知到 for 知道 —
// reads to the segmenter as 不知 + 到 and is left alone. That is the gate
// preferring a missed error to a corrupted sentence, which is the whole
// premise: 我不知到底该怎么办 is the same three characters and is correct.
it('declines a real error rather than risk the sentence beside it', () => {
expect(flagged('我不知到他在哪里')).toEqual([])
expect(flagged('你知到吗')).toEqual(['知到→知道'])
})
it('says nothing about English, or about nothing', () => {
expect(flagged('I already finished my homework')).toEqual([])
expect(flagged('')).toEqual([])
})
// The direction gate. The word list is loaded only for an account learning
// Chinese, so without one this pack is silent — a writer practising English
// must never be told her own quoted Chinese is wrong.
it('is silent without a segmenter, which is how the direction gate works', () => {
expect(hanziFindings('我己经写完了', null)).toEqual([])
})
})
+149
View File
@@ -0,0 +1,149 @@
import type { MechanicsFinding } from '../../api/client'
import type { Segmenter } from '../../lib/segment'
// 错别字 — wrong-character detection, the Chinese counterpart of the spell
// checker, and a different problem from the one Hunspell solves.
//
// Chinese has no misspellings in the English sense: every character a writer can
// type is a real character, correctly formed, and an IME will not offer one that
// is not. What it *will* offer is the wrong one. Typing pinyin `yijing` and
// taking the first candidate gives 已经 or 己经 depending on the moment, and both
// are made of real characters. So the unit of error is not a malformed word but
// a **substituted character inside a correct-looking one** — which is why this
// is a rule pack over confusable pairs rather than a dictionary membership test.
//
// The discipline is Phase 22's, and the bar is the same: **precision over
// recall**. A wrong nudge costs more trust than a missed one earns, and it costs
// double here, because a learner has no way to know the tool is wrong. Two
// mechanical gates enforce it, and both are checked in the tests rather than
// asserted in prose.
// A confusable pair: `wrong` is never a word, `right` is what was meant.
//
// **Gate one — the pair must be decidable by the dictionary.** Each entry is
// admitted only if `wrong` is absent from the 188k-word list *and* `right` is
// present. That is what makes the correction a fact rather than a preference,
// and it is checked against the shipped asset in hanzi.test.ts.
//
// It is also the gate that keeps out errors everyone knows are errors. 自已 for
// 自己 is among the commonest slips in written Chinese, and 自已 is itself a
// dictionary headword — so this pack does not flag it, exactly as Phase 22's
// English pack left out `married with`. The same fate for 好象 (an older form of
// 好像, still in the dictionary), 倒底, 帐号 and 部份: all real errors by a modern
// standard, none of them decidable here.
interface Confusion {
wrong: string
right: string
// The note on the card. English, because this pack only ever runs for a writer
// whose English is the language they think in — see the direction gate below.
why: string
}
const CONFUSIONS: Confusion[] = [
// 已 / 己 / 以 — three characters that differ by one stroke and share a
// syllable. The most productive source of 错别字 there is.
{ wrong: '己经', right: '已经', why: '已经 (already) — 己 is the "self" character; the one you want is 已.' },
{ wrong: '以经', right: '已经', why: '已经 (already) — 以 is a different word; 已 is the one that means "already".' },
{ wrong: '已后', right: '以后', why: '以后 (afterwards) takes 以, not 已.' },
// 在 / 再 — same pinyin (zài), completely different jobs: one is location and
// ongoing action, the other is repetition.
{ wrong: '在见', right: '再见', why: '再见 (goodbye) — 再 is "again", which is what "see you again" needs.' },
{ wrong: '正再', right: '正在', why: '正在 (in the middle of doing) takes 在, the one about being somewhere.' },
{ wrong: '现再', right: '现在', why: '现在 (now) takes 在.' },
// 做 / 作 — both zuò, both "to do", and which one a compound takes is simply
// fixed by convention. A learner cannot reason it out, which is what makes a
// reminder worth having.
{ wrong: '工做', right: '工作', why: '工作 (work) is written with 作.' },
{ wrong: '做业', right: '作业', why: '作业 (homework) is written with 作.' },
{ wrong: '做者', right: '作者', why: '作者 (author) is written with 作.' },
{ wrong: '做文', right: '作文', why: '作文 (an essay) is written with 作.' },
{ wrong: '做用', right: '作用', why: '作用 (effect, function) is written with 作.' },
// 得 / 的 — the pair everyone knows about. Only the fixed compound is flagged:
// deciding 的 against 地 against 得 in the general case needs to know whether
// the next word is a verb or a noun, which nothing here can tell.
{ wrong: '觉的', right: '觉得', why: '觉得 (to feel, to think) ends in 得.' },
// 即 / 既 — one stroke apart, opposite meanings ("namely" against "since").
{ wrong: '即然', right: '既然', why: '既然 (since, given that) takes 既.' },
{ wrong: '既使', right: '即使', why: '即使 (even if) takes 即.' },
// The rest: ordinary IME slips where the wrong character is a homophone.
{ wrong: '因该', right: '应该', why: '应该 (should) — 因 means "because"; the word you want starts with 应.' },
{ wrong: '因位', right: '因为', why: '因为 (because) ends in 为.' },
{ wrong: '知到', right: '知道', why: '知道 (to know) ends in 道.' },
{ wrong: '安照', right: '按照', why: '按照 (according to) takes 按.' },
{ wrong: '蜜码', right: '密码', why: '密码 (password) takes 密 — 蜜 is honey.' },
{ wrong: '犹其', right: '尤其', why: '尤其 (especially) takes 尤.' },
{ wrong: '甘净', right: '干净', why: '干净 (clean) takes 干.' },
{ wrong: '什末', right: '什么', why: '什么 (what) ends in 么.' },
{ wrong: '一像', right: '一样', why: '一样 (the same) ends in 样 — 像 is "to resemble".' },
{ wrong: '必须品', right: '必需品', why: '必需品 (a necessity) takes 需. 必须 is "must", which is a different word.' },
]
// **Gate two — the characters must not already belong to two different words.**
//
// This is the gate that stops the pack from destroying correct writing, and
// without it every rule above is dangerous. 自己经常 ("oneself, often") contains
// the string 己经. 睡觉的时候 ("when sleeping") contains 觉的. 不知到底 contains 知到.
// A substring match would corrupt all three.
//
// The segmenter already knows the difference, so the test is: split the text,
// and if the two characters land in different tokens *and* either token is a
// real multi-character word, this is a word boundary and not an error. Two
// adjacent single-character tokens is what the walk produces when it has nothing
// better to offer — which is exactly what a mistyped compound looks like.
function isWordBoundary(tokens: { word: string; from: number; to: number }[], at: number): boolean {
const left = tokens.find((t) => at >= t.from && at < t.to)
const right = tokens.find((t) => at + 1 >= t.from && at + 1 < t.to)
if (!left || !right || left === right) return false
return left.word.length > 1 || right.word.length > 1
}
// hanziFindings returns the 错别字 in a piece of text, as ordinary mechanics
// findings — the same shape, the same rail, the same cards, the same accept.
//
// It needs the segmenter and does nothing without one, which is also the
// direction gate: the word list is loaded only for an account learning Chinese
// (useSegmenter), so a writer practising English can never be told her quoted
// Chinese is wrong. That is not a nicety. Petal deliberately never corrects the
// pair language — the fr and es dictionaries are chosen to hold every variety
// precisely so they cannot underline correct writing — and a Mandarin native
// does not need her own language checked by a rule pack of two dozen entries.
export function hanziFindings(text: string, segmenter: Segmenter | null): MechanicsFinding[] {
if (!segmenter || !text) return []
// One segmentation for the whole text, shared by every rule. The walk is
// linear, but running it two dozen times over a long document would not be.
const tokens = segmenter.segment(text)
const found: MechanicsFinding[] = []
for (const c of CONFUSIONS) {
let from = text.indexOf(c.wrong)
while (from !== -1) {
// The boundary test is asked at the seam the substitution sits on: the
// gap between the first two characters, which is where a mistyped
// compound and two adjacent words look different from each other.
if (!isWordBoundary(tokens, from)) {
found.push({
from,
to: from + c.wrong.length,
original: c.wrong,
replacement: c.right,
explanation: c.why,
type: 'mechanics',
})
}
from = text.indexOf(c.wrong, from + 1)
}
}
// Document order, so the rail reads down the page rather than down this file.
return found.sort((a, b) => a.from - b.from)
}
// Exported for the tests, which check every pair against the shipped word list.
// A pack whose own gate is only described in a comment is a pack whose gate can
// rot; this is how the description is made to stay true.
export const CONFUSION_PAIRS = CONFUSIONS.map((c) => ({ wrong: c.wrong, right: c.right }))
@@ -0,0 +1,79 @@
import { useEffect, useState, type RefObject } from 'react'
// How often to re-measure outside of scroll/resize events. Cards re-pack when
// suggestions arrive, expand, or get accepted — none of which fire an event we
// can hear from here, so a slow poll picks those up.
const POLL_MS = 500
// What the mascot yields to. Suggestion cards stack down into its corner; the
// History and Garden drawers cover it outright, and their footer controls sat
// under the halo. Matching on the modal role rather than each panel's own class
// means a future drawer is covered the day it's written, without a list to keep
// in sync. Anything that doesn't actually reach the corner still won't trip the
// rect test below.
const CARD = '.petal-rail-card'
const MODAL = '[role="dialog"][aria-modal="true"]'
export interface CardOverlap {
// A suggestion card reaches the mascot. It should get out of the way, but may
// still wake up when it has something to say.
cards: boolean
// A modal panel (History, Garden) covers the mascot's corner. Nothing the
// kitten wants to say is worth covering a dialog the writer opened on
// purpose, so this yields unconditionally.
modal: boolean
}
// Reports what, if anything, the mascot should yield to at the given element.
// Used to fade the corner mascot out of the way when a card or panel reaches
// into its corner, so nothing is ever hidden (or made unclickable) by the
// kitten.
export function useCardOverlap(ref: RefObject<HTMLElement | null>): CardOverlap {
const [overlap, setOverlap] = useState<CardOverlap>({ cards: false, modal: false })
useEffect(() => {
let raf = 0
const check = () => {
const el = ref.current
if (!el) return
const r = el.getBoundingClientRect()
const hits = (selector: string) => {
for (const other of document.querySelectorAll(selector)) {
const b = other.getBoundingClientRect()
if (b.left < r.right && b.right > r.left && b.top < r.bottom && b.bottom > r.top) {
return true
}
}
return false
}
const next = { cards: hits(CARD), modal: hits(MODAL) }
// Same-value object identity would re-render on every poll tick.
setOverlap((prev) =>
prev.cards === next.cards && prev.modal === next.modal ? prev : next,
)
}
const schedule = () => {
if (raf) return
raf = requestAnimationFrame(() => {
raf = 0
check()
})
}
check()
// Capture phase so scrolls inside nested scrollers (History panel, rail) count.
window.addEventListener('scroll', schedule, true)
window.addEventListener('resize', schedule)
const timer = window.setInterval(check, POLL_MS)
return () => {
window.removeEventListener('scroll', schedule, true)
window.removeEventListener('resize', schedule)
window.clearInterval(timer)
cancelAnimationFrame(raf)
}
}, [ref])
return overlap
}
+31 -5
View File
@@ -5,6 +5,7 @@ import { SearchBox } from './SearchBox'
import { TagChip } from './TagChip'
import { LanguagePicker } from './LanguagePicker'
import { usePack, type Pack } from '../../i18n'
import { forgetAllDrafts } from '../../lib/drafts'
interface Props {
docs: DocSummary[]
@@ -19,6 +20,12 @@ interface Props {
// The signed-in writer, when there is real auth to sign out of. Null in a
// local-dev build, where there is nothing to leave.
account: { name: string } | null
// The account's learner direction and the way to change it, passed straight
// through to the language picker in the footer — the sidebar is the drawer,
// and the drawer is the only chrome always one tap away on a phone.
direction?: string
onDirection?: (direction: string) => Promise<void>
onPair?: (lang: string, direction: string) => Promise<void>
}
// Sidebar sort orders. 'recent' keeps the server's updated_at-desc ordering.
@@ -42,6 +49,9 @@ export function DocList({
onToggleTag,
onCreateTag,
account,
direction,
onDirection,
onPair,
}: Props) {
const t = usePack()
// Active tag filter (null = show all). Cleared automatically if the tag
@@ -160,7 +170,7 @@ export function DocList({
{/* The pair Petal speaks. Unlike the rows above it this is not about any
document, and unlike sign-out it is offered whether or not there is an
account behind the session a local-dev build still has a langpack. */}
<LanguagePicker />
<LanguagePicker direction={direction} onDirection={onDirection} onPair={onPair} />
{/* Who's writing, and the way out. Shown only when there's a real account
behind the session a local-dev build has nobody to sign out as. */}
@@ -172,13 +182,29 @@ export function DocList({
<span className="min-w-0 flex-1 truncate" title={account.name}>
🌸 {account.name}
</span>
<a
href="/auth/logout"
className="shrink-0 font-bold hover:underline"
{/* A form, not a link: /auth/logout is POST-only, because with
SameSite=Lax a plain GET route would let any page on the internet
sign her out of her own draft. The browser still does a normal
navigation and follows the redirect home, so this behaves exactly
as the link did.
onSubmit fires before the navigation and clears the drafts this
browser is holding for her signing out of a shared machine
should not leave her unsaved sentences behind in it. */}
<form
method="post"
action="/auth/logout"
className="shrink-0"
onSubmit={() => forgetAllDrafts()}
>
<button
type="submit"
className="font-bold hover:underline"
style={{ color: 'var(--color-accent-hover)' }}
>
{t.docs.signOut}
</a>
</button>
</form>
</div>
)}
</aside>
+109 -6
View File
@@ -15,26 +15,73 @@ import { setPackLang, shippedPacks, usePack } from '../../i18n'
// read a label that says "Portuguese" in Chinese, so the buttons say 中文 and
// Português and nothing else — the one place in Petal where bilingual copy would
// actively get in the way.
export function LanguagePicker() {
interface Props {
// The account's current direction ('learning_en' | 'learning_pair'), and the
// way to change it. Owned by App rather than here, because turning the pair
// around changes what the *editor* does — it is what loads the word list —
// and this control is only where the writer says so.
direction?: string
onDirection?: (direction: string) => Promise<void>
// Move the pair itself. Owned by App for the same reason: the answer carries
// the direction too, and the account's direction is what loads the word list.
onPair?: (lang: string, direction: string) => Promise<void>
}
export function LanguagePicker({ direction, onDirection, onPair }: Props = {}) {
const t = usePack()
const packs = shippedPacks()
const [saving, setSaving] = useState<string | null>(null)
const [failed, setFailed] = useState(false)
const [turning, setTurning] = useState(false)
const [turnFailed, setTurnFailed] = useState(false)
// Nothing to choose between — a deployment with one pack shows no picker
// rather than a single button that does nothing.
if (packs.length < 2) return null
// rather than a single button that does nothing. The direction control is
// still worth rendering in that case, so it is checked separately below.
const showPacks = packs.length >= 2
// `t.learner` is the pack's own statement that this pair can be learned
// toward, and the server keeps the matching list (auth.learnerPairs). A pack
// without it renders nothing here, which is the same failure mode as a pair
// without copy: absent rather than broken.
const learner = t.learner
if (!showPacks && !learner) return null
const turn = async (next: string) => {
if (!onDirection || next === (direction ?? 'learning_en') || turning) return
setTurning(true)
setTurnFailed(false)
try {
await onDirection(next)
} catch {
setTurnFailed(true)
} finally {
setTurning(false)
}
}
const choose = async (code: string) => {
if (code === t.code || saving) return
setSaving(code)
setFailed(false)
try {
// Name the direction alongside the pair. The server validates the two as
// one decision and refuses a learner direction for a pair it has no word
// list for, so an account that is learning Chinese cannot move to French
// by naming only the pair — that request is rejected outright, and the
// writer is left on a picker whose buttons all fail. A pair with no
// learner side can only be travelled toward English; saying so is how the
// move is actually made.
const target = packs.find((p) => p.code === code)
const next = target?.learner ? (direction ?? 'learning_en') : 'learning_en'
if (onPair) {
await onPair(code, next)
} else {
const me = await api.setPairLang(code)
// The server's answer, not the code we asked for. Everything downstream
// her dictionary, the read-aloud voice, the word lookups — follows the
// The server's answer, not the code we asked for. Everything downstream
// her dictionary, the read-aloud voice, the word lookups — follows the
// pack, so it must follow what was actually stored.
setPackLang(me.pair_lang)
}
} catch {
// A 401 has already surfaced as the sign-in overlay through the client's
// interceptor; anything else leaves her on the pair she was already on,
@@ -47,7 +94,17 @@ export function LanguagePicker() {
return (
<div className="flex flex-col gap-1 px-1">
<div className="flex items-center gap-2 text-xs" style={{ color: 'var(--color-muted)' }}>
{showPacks && (
<>
{/* Label and buttons wrap as a pair: the label is itself bilingual
("Langue · Language"), and three self-naming buttons beside it need
more than the drawer is wide in every language Petal ships. When they
don't fit the buttons drop to their own line instead of spilling out
of the drawer. */}
<div
className="flex flex-wrap items-center gap-x-2 gap-y-1 text-xs"
style={{ color: 'var(--color-muted)' }}
>
<span className="shrink-0 font-semibold">{t.docs.language}</span>
<div className="ml-auto flex shrink-0 gap-1">
{packs.map((p) => {
@@ -81,6 +138,52 @@ export function LanguagePicker() {
{t.docs.languageFailed}
</span>
)}
</>
)}
{/* Which way round the pair is being learned. Below the language buttons
because it only makes sense once the language is settled, and rendered
at all only for a pair Petal has the learner-side data for. */}
{learner && onDirection && (
<div
className="flex flex-wrap items-center gap-x-2 gap-y-1 text-xs"
style={{ color: 'var(--color-muted)' }}
>
<span className="shrink-0 font-semibold">{learner.label}</span>
<div className="ml-auto flex shrink-0 gap-1">
{[
{ code: 'learning_en', text: learner.toEn, en: `learning English` },
{ code: 'learning_pair', text: learner.toPair, en: `learning ${t.nativeName}` },
].map((opt) => {
const active = (direction ?? 'learning_en') === opt.code
return (
<button
key={opt.code}
type="button"
onClick={() => void turn(opt.code)}
disabled={turning}
aria-pressed={active}
aria-label={`I am ${opt.en}`}
className="petal-tap-sm px-2.5 py-1 text-xs font-bold transition-colors disabled:opacity-60"
style={{
borderRadius: 'var(--radius-pill)',
background: active ? 'var(--color-accent)' : 'var(--color-surface)',
color: active ? '#fff' : 'var(--color-plum)',
boxShadow: active ? 'none' : 'var(--shadow-soft)',
}}
>
{opt.text}
</button>
)
})}
</div>
</div>
)}
{turnFailed && learner && (
<span className="text-[0.7rem]" style={{ color: 'var(--color-accent)' }}>
{learner.failed}
</span>
)}
</div>
)
}
+5 -2
View File
@@ -1,6 +1,7 @@
import { useEffect, useRef, useState } from 'react'
import { tagColorVar, type Tag, type TagColor } from '../../api/client'
import { usePack } from '../../i18n'
import { fromIME } from '../../lib/ime'
const COLORS: TagColor[] = ['rose', 'mint', 'peach', 'lavender', 'sky', 'honey']
@@ -26,7 +27,9 @@ export function TagPicker({ roster, assignedIds, onToggle, onCreate, onClose }:
if (!ref.current?.contains(e.target as Node)) onClose()
}
const onKey = (e: KeyboardEvent) => {
if (e.key === 'Escape') onClose()
// Not while an IME is open: a tag named in Chinese is composed in this
// very field, and Escape there means "wrong candidate", not "close".
if (e.key === 'Escape' && !fromIME(e)) onClose()
}
// Defer so the opening click doesn't immediately close it.
const id = setTimeout(() => document.addEventListener('pointerdown', onDown), 0)
@@ -114,7 +117,7 @@ export function TagPicker({ roster, assignedIds, onToggle, onCreate, onClose }:
value={name}
onChange={(e) => setName(e.target.value)}
onKeyDown={(e) => {
if (e.key === 'Enter') submit()
if (e.key === 'Enter' && !fromIME(e)) submit()
}}
placeholder={t.docs.newTagPlaceholder}
aria-label="New tag name"
+88 -14
View File
@@ -1,27 +1,52 @@
import { useEffect, useRef, useState } from 'react'
import { api, streamSuggestionChat, type ChatMessage } from '../../api/client'
import { usePack } from '../../i18n'
import { splitBilingual } from './bilingualReply'
import { fromIME } from '../../lib/ime'
interface Props {
suggestionId: string
// The English explanation (shown in the card body). Petal's opening bubble is
// its Simplified-Chinese translation, fetched on open — so the panel doesn't
// just repeat the same English text twice. Falls back to this on failure.
// its translation into the pair language, fetched on open — so the panel
// doesn't just repeat the same English text twice. Falls back to this on
// failure.
explanation: string
}
// CJK fallback stack — Nunito has no Chinese glyphs, and the user asks questions
// in Mandarin (spec Note #17). Applied to the bubbles specifically, not the
// serif editor body.
// CJK fallback stack — Nunito has no Chinese glyphs, and on the zh pair both
// the questions and half of every answer are in Mandarin (spec Note #17). The
// Latin pairs fall through to Nunito as before. Applied to the bubbles
// specifically, not the serif editor body.
const CHAT_FONT = "'Nunito', 'PingFang SC', 'Microsoft YaHei', 'Noto Sans CJK SC', sans-serif"
// How tall the conversation may grow (UX item 6: "room to read"). A bilingual
// three-paragraph answer in a 220px box was a scrollbar with a sentence in it.
//
// An earlier version of this took the smaller of half the viewport and the room
// left below the card, so the card could never overhang the screen. Measured, it
// gave 176px against a 442px answer — the card's own pill, diff, explanation and
// action row already spend ~290px of an 810px screen, so "fits below the word"
// and "room to read" are simply not both available.
//
// So this is the flat ceiling, and the overhang is made navigable instead —
// item 4's answer to the same conflict, and its words for it: "the answer is to
// make the overhang navigable, not to shrink what each card says". Both surfaces
// that host this panel report their reach to the editor wrapper (SuggestionCard
// via onExtent, the rail via its own measureTick), which grows the column, so a
// conversation that runs past the fold has real page under it and the Accept
// button below it can be scrolled to.
const CHAT_MAX_FRACTION = 0.5
// Below this a max-height stops being a reading area and becomes a peephole —
// the floor for a very short window, where half of it is not worth having.
const CHAT_MIN_PX = 160
// AskPetal is the mini chat panel inside an expanded SuggestionCard. The whole
// conversation lives in this component's state — nothing is persisted; closing
// the card (unmounting) clears it. Each send streams Petal's reply token-by-
// token into the latest assistant bubble.
export function AskPetal({ suggestionId, explanation }: Props) {
const t = usePack()
// Opening bubble starts empty (caret-only) and fills with the Mandarin
// Opening bubble starts empty (caret-only) and fills with the pair-language
// translation once it lands; `seeding` drives that loading caret.
const [messages, setMessages] = useState<ChatMessage[]>([{ role: 'assistant', content: '' }])
const [seeding, setSeeding] = useState(true)
@@ -36,6 +61,19 @@ export function AskPetal({ suggestionId, explanation }: Props) {
if (el) el.scrollTop = el.scrollHeight
}, [messages])
// How tall the conversation may grow. A share of the window, so a laptop and a
// large monitor both give the answer a sensible amount of themselves — and a
// window she resizes mid-conversation is answered live.
const [maxHeight, setMaxHeight] = useState(() =>
Math.max(CHAT_MIN_PX, window.innerHeight * CHAT_MAX_FRACTION),
)
useEffect(() => {
const onResize = () =>
setMaxHeight(Math.max(CHAT_MIN_PX, window.innerHeight * CHAT_MAX_FRACTION))
window.addEventListener('resize', onResize)
return () => window.removeEventListener('resize', onResize)
}, [])
// Focus the input when the panel opens. preventScroll: the card is already on
// screen as an absolutely-positioned overlay, and a default focus() would make
// the browser scroll its ancestor to "reveal" the input — jumping the document
@@ -44,7 +82,8 @@ export function AskPetal({ suggestionId, explanation }: Props) {
inputRef.current?.focus({ preventScroll: true })
}, [])
// Fetch the Chinese translation of the explanation to seed the first bubble.
// Fetch the pair-language translation of the explanation to seed the first
// bubble.
// Only replaces the seed bubble if the user hasn't started chatting yet (the
// conversation always opens with this one assistant turn). Falls back to the
// English explanation if the translation can't be fetched.
@@ -91,10 +130,10 @@ export function AskPetal({ suggestionId, explanation }: Props) {
} catch (err) {
setMessages((prev) => {
const next = prev.slice()
next[next.length - 1] = {
role: 'assistant',
content: 'Sorry, I had trouble responding just now. Please try again. 🌸',
}
// Bilingual, from the pack, and blank-line separated like a real reply —
// so the one message Petal writes without the model still renders
// through the same two-half bubble as every message with it.
next[next.length - 1] = { role: 'assistant', content: t.editor.chatFailed }
return next
})
console.error('Ask Petal chat failed:', err)
@@ -116,7 +155,7 @@ export function AskPetal({ suggestionId, explanation }: Props) {
<div
ref={scrollRef}
className="flex flex-col gap-2 overflow-y-auto pr-1"
style={{ maxHeight: 220 }}
style={{ maxHeight }}
>
{messages.map((m, i) => (
<Bubble
@@ -139,6 +178,13 @@ export function AskPetal({ suggestionId, explanation }: Props) {
ref={inputRef}
value={input}
onChange={(e) => setInput(e.target.value)}
// She asks Petal in Mandarin, so the Enter that commits an IME
// candidate lands in this field constantly. Most browsers already
// withhold implicit form submission during a composition; the ones
// that don't would send her half-typed question. Cheap to be certain.
onKeyDown={(e) => {
if (e.key === 'Enter' && fromIME(e)) e.preventDefault()
}}
placeholder={t.editor.askPlaceholder}
className="min-w-0 flex-1 rounded-full px-3 py-1.5 text-xs focus:outline-none"
style={{
@@ -164,6 +210,19 @@ export function AskPetal({ suggestionId, explanation }: Props) {
// Bubble renders one chat turn: Petal rose-tinted and left-aligned, the user
// lavender and right-aligned. A trailing caret marks the actively streaming
// reply until its first token lands.
//
// Petal's turns are bilingual (see bilingualReply.ts) and are laid out the way
// the companion lays out its own two lines: the pair language first and plainly
// readable, the English beneath it in the muted tone. That order is the pack's
// order everywhere else in the UI, and it holds whichever direction the writer
// is learning in — the muted half is the one they can already read, and which
// half that is isn't Petal's to decide. The writer's own turns are their own
// words in whichever language they typed them, so they are never split.
//
// Petal's bubble also takes the full width the card offers rather than the 85%
// a chat normally reserves to show who is talking — the alignment and the
// tint already say that, and two languages in a 4/5-width column wraps a
// sentence-length answer into a paragraph-shaped one.
function Bubble({
role,
content,
@@ -174,10 +233,11 @@ function Bubble({
streaming: boolean
}) {
const isPetal = role === 'assistant'
const reply = isPetal ? splitBilingual(content) : null
return (
<div className={`flex ${isPetal ? 'justify-start' : 'justify-end'}`}>
<div
className="max-w-[85%] rounded-2xl px-3 py-1.5 text-xs leading-snug"
className={`${isPetal ? 'w-full' : 'max-w-[85%]'} rounded-2xl px-3 py-2 leading-snug`}
style={{
background: isPetal ? 'var(--color-surface-alt)' : 'var(--color-lavender)',
color: 'var(--color-plum)',
@@ -185,7 +245,21 @@ function Bubble({
whiteSpace: 'pre-wrap',
}}
>
{content}
{reply ? (
<>
<span className="text-[0.8rem]">{reply.native}</span>
{reply.en !== '' && (
<span
className="mt-1.5 block text-xs"
style={{ color: 'var(--color-muted)' }}
>
{reply.en}
</span>
)}
</>
) : (
<span className="text-xs">{content}</span>
)}
{streaming && content === '' && (
<span className="petal-chat-caret" aria-hidden>
+62
View File
@@ -0,0 +1,62 @@
import { useCallback, useEffect, useRef, useState } from 'react'
// ChromeStrip is the row of document pills (tone, history, export) on a screen
// too narrow to hold them. It scrolls within itself rather than letting the
// overflow escape into the editor pane, and it says which way the rest of the
// row is.
//
// Saying so matters more here than in most scrollers: the pills are the only
// way to reach three features, and a pill that has scrolled out of sight is
// indistinguishable from a pill that doesn't exist. So each edge with more
// behind it fades — the same "there is more this way" language .petal-toolbar
// speaks with its clipped right edge, except this row can be scrolled from
// either end and has to point in the right direction.
//
// The fade is a mask rather than a gradient overlay so it works on whatever is
// behind it (the cream page, the night theme, a falling petal) without knowing
// the background colour.
type Edge = 'none' | 'left' | 'right' | 'both'
interface Props {
className?: string
children: React.ReactNode
}
export function ChromeStrip({ className = '', children }: Props) {
const ref = useRef<HTMLDivElement>(null)
const [edge, setEdge] = useState<Edge>('none')
// A pixel of slack: scrollLeft is fractional under browser zoom and on
// high-DPI screens, so an exactly-scrolled-to-the-end strip can report
// something like 0.5px remaining and fade an edge that has nothing behind it.
const measure = useCallback(() => {
const el = ref.current
if (!el) return
const more = el.scrollWidth - el.clientWidth - el.scrollLeft > 1
const less = el.scrollLeft > 1
setEdge(less && more ? 'both' : less ? 'left' : more ? 'right' : 'none')
}, [])
useEffect(() => {
const el = ref.current
if (!el) return
measure()
el.addEventListener('scroll', measure, { passive: true })
// Both halves of "does it fit" can change without a scroll: the window
// resizes, or the labels themselves change when she switches her pair
// language and every pill in the row grows or shrinks at once.
const ro = new ResizeObserver(measure)
ro.observe(el)
for (const child of Array.from(el.children)) ro.observe(child)
return () => {
el.removeEventListener('scroll', measure)
ro.disconnect()
}
}, [measure])
return (
<div ref={ref} data-edge={edge} className={`petal-chrome-strip ${className}`}>
{children}
</div>
)
}
@@ -0,0 +1,235 @@
import { describe, it, expect } from 'vitest'
import { EditorState, TextSelection } from '@tiptap/pm/state'
import type { Transaction } from '@tiptap/pm/state'
import { Schema } from '@tiptap/pm/model'
import { compositionKey, compositionPlugin, isComposing, holdRedraw } from './Composition'
import { suggestionPlugin, suggestionPluginKey, setSuggestions } from './SuggestionHighlight'
import { spellPlugin, spellPluginKey, setSpellChecker } from './SpellCheck'
import { searchPlugin, searchPluginKey, setSearch } from './SearchHighlight'
import type { Suggestion } from '../../api/client'
import type { SpellChecker } from '../../hooks/useSpellChecker'
// These tests are about one moment: she is typing 公园 with a pinyin IME, so the
// document briefly contains "gongyuan" and a candidate window sits over it. Every
// decoration layer wants to recompute, and recomputing rewrites the DOM around
// the node the browser is composing in — which is what eats half-typed input.
//
// Nothing here needs a real EditorView: composition is tracked in plugin state
// by the compositionstart/compositionend handlers, so a plain EditorState with
// the same plugins reproduces exactly the decisions the layers make.
const schema = new Schema({
nodes: {
doc: { content: 'block+' },
paragraph: { group: 'block', content: 'inline*', toDOM: () => ['p', 0] },
text: { group: 'inline' },
},
})
const doc = (text: string) =>
schema.node('doc', null, [schema.node('paragraph', null, text ? [schema.text(text)] : [])])
// A dictionary that knows ordinary English and nothing else — so the pinyin run
// an IME leaves in the document mid-composition is a misspelling to it, which is
// precisely the risk this guard exists for.
const english: SpellChecker = {
correct: (w) => ['the', 'park', 'went', 'to', 'today'].includes(w.toLowerCase()),
suggest: () => [],
extendedAlphabet: false,
}
const suggestion = (original: string, replacement: string): Suggestion => ({
id: `s-${original}`,
doc_id: 'd',
from_pos: 0,
to_pos: 0,
original,
replacement,
explanation: '',
type: 'grammar',
status: 'pending',
source: 'llm',
created_at: new Date().toISOString(),
})
function harness(text: string) {
let state = EditorState.create({
schema,
doc: doc(text),
plugins: [compositionPlugin({ onEnd: null }), suggestionPlugin(), spellPlugin(), searchPlugin()],
})
const api = {
get state() {
return state
},
tr: (f: (tr: Transaction) => Transaction) => {
state = state.apply(f(state.tr))
},
dispatch: (tr: Transaction) => {
state = state.apply(tr)
},
// The two ends of a composition, as the DOM handlers dispatch them.
startComposing: () => api.tr((tr) => tr.setMeta(compositionKey, true)),
endComposing: () => api.tr((tr) => tr.setMeta(compositionKey, false)),
// Typing, whether by keystroke or by an IME writing into the document.
type: (at: number, text: string) =>
api.tr((tr) => tr.insertText(text, at).setSelection(TextSelection.create(tr.doc, at + text.length))),
// Replace a span, the way committing an IME candidate does.
commit: (from: number, to: number, text: string) => api.tr((tr) => tr.insertText(text, from, to)),
spans: (key: typeof suggestionPluginKey | typeof spellPluginKey | typeof searchPluginKey) => {
const deco = (key.getState(state) as { decorations: import('@tiptap/pm/view').DecorationSet }).decorations
return deco.find().map((d) => [d.from, d.to] as const)
},
}
return api
}
describe('composition tracking', () => {
it('is off until a composition starts, and off again once it ends', () => {
const h = harness('I went to the ')
expect(isComposing(h.state)).toBe(false)
h.startComposing()
expect(isComposing(h.state)).toBe(true)
h.endComposing()
expect(isComposing(h.state)).toBe(false)
})
it('releases the redraw on the very transaction that ends the composition', () => {
const h = harness('hello')
h.startComposing()
const before = h.state
expect(holdRedraw(before.tr, before)).toBe(true)
// The end transaction is dispatched while composing is still true; if it
// held its own redraw like any other, nothing would ever release it.
expect(holdRedraw(before.tr.setMeta(compositionKey, false), before)).toBe(false)
})
})
describe('spell underlines during composition', () => {
it('does not underline the pinyin she is part-way through converting', () => {
const h = harness('I went to the ')
h.dispatch(h.state.tr.setMeta(spellPluginKey, english))
expect(h.spans(spellPluginKey)).toEqual([])
h.startComposing()
// The IME writes its buffer into the document one letter at a time. The
// caret sits inside the run, so the caret exemption would cover "gongyuan"
// on its own — but not a second word, and not once she moves back to fix a
// syllable. The guard is what makes that irrelevant.
h.type(15, 'gong')
h.type(19, 'yuan')
expect(h.spans(spellPluginKey)).toEqual([])
// And the caret has moved away, which normally forces a rebuild.
h.tr((tr) => tr.setSelection(TextSelection.create(tr.doc, 1)))
expect(h.spans(spellPluginKey)).toEqual([])
})
it('re-checks the moment the candidate is committed', () => {
const h = harness('I went to the ')
h.dispatch(h.state.tr.setMeta(spellPluginKey, english))
h.startComposing()
h.type(15, 'gongyuan')
h.commit(15, 23, '公园') // she picks 公园; the pinyin is gone
h.endComposing()
// Nothing to flag: the pinyin never existed by the time anyone looked, and
// CJK is not tokenized at all.
expect(h.spans(spellPluginKey)).toEqual([])
// A real misspelling typed afterwards still underlines, so the layer is
// released rather than switched off. (The caret moves off it first: a word
// under the cursor is exempt, mid-typing, IME or no IME.)
h.type(17, ' parc')
h.tr((tr) => tr.setSelection(TextSelection.create(tr.doc, 1)))
expect(h.spans(spellPluginKey).length).toBe(1)
})
it('underlines the same text immediately when no IME is involved', () => {
const h = harness('I went to the ')
h.dispatch(h.state.tr.setMeta(spellPluginKey, english))
h.type(15, 'gongyuan')
h.tr((tr) => tr.setSelection(TextSelection.create(tr.doc, 1)))
expect(h.spans(spellPluginKey).length).toBe(1)
})
})
describe('suggestion highlights during composition', () => {
it('carries existing highlights along with the text instead of re-anchoring', () => {
const h = harness('I went to the park today')
setSuggestions(h.state, h.dispatch, [suggestion('went to', 'go to')])
expect(h.spans(suggestionPluginKey)).toEqual([[3, 10]])
h.startComposing()
h.type(1, 'x') // insert before the highlight: it has to move with the text
expect(h.spans(suggestionPluginKey)).toEqual([[4, 11]])
})
it('holds a freshly arrived suggestion list until the composition ends', () => {
const h = harness('I went to the park today')
h.startComposing()
setSuggestions(h.state, h.dispatch, [suggestion('the park', 'a park')])
// The list is stored, but the page is not repainted under the IME.
expect(h.spans(suggestionPluginKey)).toEqual([])
h.endComposing()
expect(h.spans(suggestionPluginKey)).toEqual([[11, 19]])
})
it('re-anchors against the committed text, not the pinyin it replaced', () => {
const h = harness('I went to ')
setSuggestions(h.state, h.dispatch, [suggestion('公园', '花园')])
expect(h.spans(suggestionPluginKey)).toEqual([]) // not there yet
h.startComposing()
h.type(11, 'gongyuan')
h.commit(11, 19, '公园')
h.endComposing()
expect(h.spans(suggestionPluginKey)).toEqual([[11, 13]])
})
})
describe('find-and-replace highlights during composition', () => {
it('holds the match set, then refreshes it against the committed text', () => {
const h = harness('公园 and 公园')
setSearch(h.state, h.dispatch, '公园', false)
expect(h.spans(searchPluginKey).length).toBe(2)
h.startComposing()
h.type(10, ' gongyuan') // at the end of the text, where the caret is
expect(h.spans(searchPluginKey).length).toBe(2) // still two, not three
h.commit(11, 19, '公园')
h.endComposing()
expect(h.spans(searchPluginKey).length).toBe(3)
})
it('closing the bar clears immediately — a composition never holds a removal', () => {
const h = harness('公园 and 公园')
setSearch(h.state, h.dispatch, '公园', false)
h.startComposing()
h.dispatch(h.state.tr.setMeta(searchPluginKey, { kind: 'clear' }))
expect(h.spans(searchPluginKey)).toEqual([])
})
})
describe('the layers are only paused, never left stale', () => {
it('rebuilds even if the composition ends on a transaction of its own', () => {
// The end signal is dispatched on a timer, after ProseMirror has flushed the
// composition's last document change — so the releasing transaction usually
// carries no document change at all. That must still be enough.
const h = harness('I went to the ')
h.dispatch(h.state.tr.setMeta(spellPluginKey, english))
h.startComposing()
h.type(15, 'parc')
expect(h.spans(spellPluginKey)).toEqual([])
h.tr((tr) => tr.setSelection(TextSelection.create(tr.doc, 1)))
h.endComposing() // no doc change, no selection change
expect(h.spans(spellPluginKey).length).toBe(1)
})
it('a checker arriving mid-composition is applied once it ends', () => {
const h = harness('公园 parc')
h.startComposing()
setSpellChecker(h.state, h.dispatch, english)
expect(h.spans(spellPluginKey)).toEqual([])
h.tr((tr) => tr.setSelection(TextSelection.create(tr.doc, 1)))
h.endComposing()
expect(h.spans(spellPluginKey).length).toBe(1)
})
})
+112
View File
@@ -0,0 +1,112 @@
import { Extension } from '@tiptap/core'
import { Plugin, PluginKey } from '@tiptap/pm/state'
import type { EditorState, Transaction } from '@tiptap/pm/state'
import type { EditorView } from '@tiptap/pm/view'
// Composition tracks whether an IME composition is in flight, and is the one
// place the rest of the editor asks.
//
// Why it exists: typing Chinese (or Japanese, or Korean) does not produce
// characters a keystroke at a time. The IME opens a *composition* — the pinyin
// she types goes into the document as it is typed, a candidate window sits over
// it, and only when she picks a candidate is the run replaced with hanzi.
// Petal's three decoration layers (SuggestionHighlight, SpellCheck,
// SearchHighlight) all recompute from the live document on every change, so
// mid-composition they would recompute over half-typed pinyin — and rebuilding
// decorations means rewriting the DOM around the node the IME is composing in.
// That is the classic bug that eats half-typed input: the composition is
// abandoned by the browser and the letters vanish or double.
//
// The fix is to hold the redraws, not to skip them. Decorations that are due
// while a composition is in flight are kept (mapped through the transaction, so
// they follow the text that moved) and rebuilt the moment the composition ends.
// Nothing is lost — the pause is measured in the length of one word.
//
// Input rules need no guard here: Tiptap's own input-rule plugin already returns
// early while `view.composing` is true, which matters because pinyin uses an
// apostrophe as a syllable separator (xi'an → 西安) and Typography.ts rewrites
// every ' into a curly .
export const compositionKey = new PluginKey<boolean>('petalComposition')
// isComposing answers "was an IME composition in flight as of this state?".
// Decoration plugins ask it of the state *before* the transaction they are
// applying, which is what makes the answer independent of plugin ordering: the
// flag was set by an earlier transaction (compositionstart), not by this one.
export function isComposing(state: EditorState): boolean {
return compositionKey.getState(state) === true
}
// holdRedraw is the question every decoration layer asks in its `apply`: should
// this rebuild wait? Yes while composing — except on the transaction that ends
// the composition, which is precisely the one that releases the held redraws.
export function holdRedraw(tr: Transaction, stateBefore: EditorState): boolean {
if (tr.getMeta(compositionKey) === false) return false
return isComposing(stateBefore)
}
function setComposing(view: EditorView, composing: boolean) {
if (compositionKey.getState(view.state) === composing) return
view.dispatch(view.state.tr.setMeta(compositionKey, composing))
}
export interface CompositionOptions {
// Called once after a composition has ended and the document has settled.
// EditorCore uses it to re-report the committed text, since the analysis
// passes were told to ignore everything typed while composing.
onEnd: (() => void) | null
}
export function compositionPlugin(options: CompositionOptions): Plugin<boolean> {
return new Plugin<boolean>({
key: compositionKey,
state: {
init: () => false,
apply(tr, value) {
const meta = tr.getMeta(compositionKey)
return typeof meta === 'boolean' ? meta : value
},
},
props: {
handleDOMEvents: {
compositionstart: (view) => {
setComposing(view, true)
return false
},
// A custom handleDOMEvents handler runs *before* ProseMirror's own, and
// ProseMirror's compositionend queues the composition's final DOM
// changes as a microtask. Ending on a macrotask puts us after both, so
// the rebuild we release sees the committed hanzi rather than the pinyin
// it replaced. (If a transaction from that flush arrives first it
// rebuilds anyway — by then `composing` is false. Both orders land.)
compositionend: (view) => {
setTimeout(() => {
if (view.isDestroyed) return
setComposing(view, false)
options.onEnd?.()
}, 0)
return false
},
// Clicking away mid-candidate abandons the composition without a
// compositionend in some browsers. Without this the layers would stay
// held — silently, and until she typed again.
blur: (view) => {
setComposing(view, false)
return false
},
},
},
})
}
export const Composition = Extension.create<CompositionOptions>({
name: 'composition',
addOptions() {
return { onEnd: null }
},
addProseMirrorPlugins() {
return [compositionPlugin(this.options)]
},
})
+315 -43
View File
@@ -15,10 +15,11 @@ import TableHeader from '@tiptap/extension-table-header'
import TableCell from '@tiptap/extension-table-cell'
import { FontSize } from './FontSize'
import type { EditorView } from '@tiptap/pm/view'
import { useCallback, useEffect, useRef, useState } from 'react'
import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
import { Toolbar } from '../Toolbar/Toolbar'
import { SuggestionCard } from './SuggestionCard'
import { SuggestionRail, type RailItem } from './SuggestionRail'
import { railFitsBeside } from './railFit'
import { SuggestionHighlight, setSuggestions, setActiveSuggestion, findRange } from './SuggestionHighlight'
import { SpellCheck, setSpellChecker, wordAt } from './SpellCheck'
import { MisspellCard } from './MisspellCard'
@@ -28,16 +29,35 @@ import { SelectionBubble } from './SelectionBubble'
import { SearchHighlight } from './SearchHighlight'
import { FindReplace } from './FindReplace'
import { Typography } from './Typography'
import { Composition } from './Composition'
import { RewritePreview, type RewriteStatus } from './RewritePreview'
import { api, type Suggestion, type WordInfo } from '../../api/client'
import { planBatch } from './acceptBatch'
import { api, type Suggestion, type SuggestionType, type WordInfo } from '../../api/client'
import { speak, speechSupported } from '../../audio/speech'
import type { SpellChecker } from '../../hooks/useSpellChecker'
import type { Segmenter } from '../../lib/segment'
import { hanziWordAt, hanziToWordInfo, hanziPinyin } from './hanziWord'
import { usePack } from '../../i18n'
// Breathing room left below the last suggestion card when the rail's stack is what
// defines the column's height, so the bottom card doesn't sit flush on the edge.
const RAIL_TAIL = 24
export interface EditorChange {
content: string // Tiptap JSON, stringified
content_text: string // flattened plain text for the LLM
word_count: number
// True while an IME composition is in flight: this text contains the pinyin
// she is part-way through converting, not the sentence she is writing.
//
// The save is deliberately NOT gated on it — a tablet keyboard can hold one
// composition open for a whole sentence, and Petal never makes writing wait
// for anything. Saving an intermediate state costs nothing: the next change
// supersedes it, and one always arrives (this component emits a final change
// once the composition commits). What it gates is *analysis* — asking the
// rule pack or the model to read half-typed pinyin can only produce advice
// about text that is about to stop existing.
composing: boolean
}
interface Props {
@@ -49,6 +69,9 @@ interface Props {
// call + state removal (accept's text replacement happens here in the editor).
suggestions: Suggestion[]
onAccept: (s: Suggestion) => void
// A whole category accepted at once. The text replacement happens here, in one
// transaction; the parent files each row and cheers once for the batch.
onAcceptMany: (list: Suggestion[]) => void
onDismiss: (s: Suggestion) => void
// Triggers the whole-document voice-consistency pass; `voicing` is true while
// it runs (drives the toolbar button's loading state).
@@ -64,6 +87,11 @@ interface Props {
// to the personal dictionary is bubbled up so it persists app-wide.
spellChecker: SpellChecker | null
onAddWord: (word: string) => void
// The Chinese word list, non-null only for a writer learning the pair
// language (users.direction = 'learning_pair'). Its presence is what turns on
// every Chinese-side behaviour here: hanzi stops being text the editor steps
// over and becomes words it can point at.
segmenter: Segmenter | null
}
interface MisspellState {
@@ -86,6 +114,12 @@ interface WordInfoState {
left: number
loading: boolean
info: WordInfo | null
// The word's own pinyin, for a Chinese lookup. Kept beside `info` rather than
// inside it because WordInfo is the English dictionary's shape and `phonetic`
// there means IPA — printing pinyin between the slashes that say "this is
// IPA" would be a small lie in the one place a learner is looking for the
// truth about pronunciation.
pinyin: string
// Garden state: the captured word's id (null until the auto-capture returns or
// after it's removed) and whether it's currently in the garden.
vocabId: string | null
@@ -179,6 +213,10 @@ interface GlossState {
gloss: string
// The other reading, when the token is a word in her language too.
reverse?: string
// A line shown *above* the meaning rather than below it: pinyin, for a
// Chinese word. Above because it is read first — the meaning of 公园 may
// already be clear to someone who cannot yet say it.
lead?: string
from: number
to: number
top: number
@@ -216,6 +254,7 @@ export function EditorCore({
onChange,
suggestions,
onAccept,
onAcceptMany,
onDismiss,
onVoiceCheck,
voicing,
@@ -224,11 +263,15 @@ export function EditorCore({
onFocusMode,
spellChecker,
onAddWord,
segmenter,
}: Props) {
// Her pair's copy — the hover tip labels the second reading with the language's
// own name, so it says "português" rather than "pt-PT".
const pack = usePack()
const wrapperRef = useRef<HTMLDivElement>(null)
// The text column itself, measured separately from its wrapper: the wrapper is
// grown to cover the card stack, so only this reports the height of the prose.
const contentRef = useRef<HTMLDivElement>(null)
const [hover, setHover] = useState<HoverState | null>(null)
// The open spelling popover (click a red-underlined word), or null.
const [misspell, setMisspell] = useState<MisspellState | null>(null)
@@ -266,12 +309,46 @@ export function EditorCore({
// `activeId` is the suggestion currently emphasized (hovered text or card).
const [railItems, setRailItems] = useState<RailItem[]>([])
const [railEnabled, setRailEnabled] = useState(false)
// How far the resolved card stack reaches below the wrapper's top, reported by
// the rail. Cards are absolutely positioned and so contribute no layout height:
// without this the column below the last line of text isn't scrollable and any
// card that lands there is unreachable, not merely far from its sentence.
const [railExtent, setRailExtent] = useState(0)
// The same report from the anchored card, which is absolutely positioned for
// the same reason and so has the same problem: an open Ask Petal conversation
// can reach well past the last line of a short document, and its Accept button
// goes with it. 0 whenever no card is open.
const [cardExtent, setCardExtent] = useState(0)
// How far down the column has to reach to cover its floating surfaces. Both
// reach past the prose for the same reason and are answered the same way, so
// they resolve to one number: whichever is lower wins, and 0 means the text
// alone decides the height.
//
// The rail's extent is conditional on the rail being mounted — a stale measure
// from a rail that has since been dismissed would leave a document padded with
// blank scroll. The card's is not: it reports 0 as it unmounts.
const overhang = Math.max(
railEnabled && railExtent > 0 ? railExtent + RAIL_TAIL : 0,
cardExtent > 0 ? cardExtent + RAIL_TAIL : 0,
)
// Sticky offset for the text column, or null when it should sit in normal flow.
// Set only while the stack overhangs the text: scrolling down to reach the lower
// cards would otherwise carry every sentence off the top of the screen.
const [stickTop, setStickTop] = useState<number | null>(null)
const [railExpandedId, setRailExpandedId] = useState<string | null>(null)
const [activeId, setActiveId] = useState<string | null>(null)
// A stable handle to the latest recompute so the editor's onUpdate (captured
// once at construction) can trigger a re-measure without stale closures.
const recomputeRailRef = useRef<() => void>(() => {})
// Re-report the document once an IME composition commits. Everything typed
// while composing was reported with `composing: true`, so the analysis passes
// ignored it; without this nudge the committed sentence would wait for the
// next keystroke to be looked at. Held in a ref because the extension list is
// built once, at construction.
const emitCommittedRef = useRef<() => void>(() => {})
const editor = useEditor({
extensions: [
StarterKit,
@@ -289,6 +366,11 @@ export function EditorCore({
TextAlign.configure({ types: ['heading', 'paragraph'] }),
Placeholder.configure({ placeholder: 'Start writing…' }),
CharacterCount,
// First in the list so its state is settled before the layers that read
// it — not that they depend on the ordering (they read the state as of
// the transaction before), but the one that answers the question should
// come before the ones that ask it.
Composition.configure({ onEnd: () => emitCommittedRef.current() }),
SuggestionHighlight,
SpellCheck,
SearchHighlight,
@@ -333,6 +415,7 @@ export function EditorCore({
content: JSON.stringify(editor.getJSON()),
content_text: editor.getText(),
word_count: editor.storage.characterCount.words(),
composing: editor.view.composing,
})
// Edits reflow the text, so the rail anchors need re-measuring.
recomputeRailRef.current()
@@ -359,6 +442,26 @@ export function EditorCore({
},
})
// The composition-end nudge. Same payload as onUpdate's, with `composing`
// false by construction — this runs after the composition has ended and its
// final changes have been flushed, so the text here is the committed one.
//
// Written in an effect rather than during render, like recomputeRailRef
// below: a render React throws away must not be the one that leaves its
// closure behind for a DOM event to call later.
useEffect(() => {
emitCommittedRef.current = () => {
if (!editor) return
onChange({
content: JSON.stringify(editor.getJSON()),
content_text: editor.getText(),
word_count: editor.storage.characterCount.words(),
composing: false,
})
recomputeRailRef.current()
}
}, [editor, onChange])
// When the selected document changes, swap in its content without emitting an
// update (false) so loading a doc doesn't trigger a spurious save.
useEffect(() => {
@@ -381,6 +484,26 @@ export function EditorCore({
// popover would offer a definition of "cora".
const wordAlphabet = spellChecker?.extendedAlphabet ?? false
// "The word under here", for a document that may hold two writing systems at
// once — which every document in this pair does, because a learner's Chinese
// practice is full of English and her English is full of quoted Chinese.
//
// Chinese is tried first and Latin second, and the order costs nothing to get
// right: the two can never both answer, because a Han character is not a Latin
// letter and neither tokenizer will cross into the other's run. `hanzi` rides
// along because the two answers go to different dictionaries — the same
// string is a word in exactly one of them.
const resolveWord = useCallback(
(pos: number): { from: number; to: number; word: string; hanzi: boolean } | null => {
if (!editor) return null
const han = hanziWordAt(editor.state.doc, pos, segmenter)
if (han) return { ...han, hanzi: true }
const latin = wordAt(editor.state.doc, pos, wordAlphabet)
return latin ? { ...latin, hanzi: false } : null
},
[editor, segmenter, wordAlphabet],
)
// Push the spell checker into its decoration plugin once the dictionary loads
// (and again whenever the personal dictionary changes its identity).
useEffect(() => {
@@ -409,9 +532,7 @@ export function EditorCore({
const wrapper = wrapperRef.current
if (!wrapper) return
const wrapRect = wrapper.getBoundingClientRect()
// Need room for the 300px column + its 32px gutter (see .petal-rail), plus
// a little breathing space to the viewport edge.
setRailEnabled(window.innerWidth - wrapRect.right >= 348)
setRailEnabled(railFitsBeside(window.innerWidth, wrapRect.right))
const seen = new Set<string>()
const items: RailItem[] = []
wrapper.querySelectorAll<HTMLElement>('.petal-suggestion[data-suggestion-id]').forEach((el) => {
@@ -432,11 +553,24 @@ export function EditorCore({
// Re-anchor when the suggestion set changes (after the decorations repaint),
// and keep the rail in sync with viewport/editor width changes (room + reflow).
//
// The scrollport is observed as well as the wrapper, and it is not redundant:
// the wrapper is a fixed 720px column, so entering or leaving distraction-free
// mode *moves* it (the pane re-centres) without ever changing its size. A
// ResizeObserver on the wrapper alone reports nothing, no window resize fires,
// and `railEnabled` keeps whatever value it had — leaving the 300px rail
// rendered into the 266px margin a restored sidebar leaves behind, cards
// clipped mid-sentence and the page scrolling sideways. The scrollport spans
// the pane, so it resizes whenever the chrome around the editor does.
useEffect(() => {
recomputeRail()
const wrapper = wrapperRef.current
const port = wrapper?.closest('.petal-scrollport')
const ro = wrapper ? new ResizeObserver(() => recomputeRail()) : null
if (wrapper && ro) ro.observe(wrapper)
if (wrapper && ro) {
ro.observe(wrapper)
if (port) ro.observe(port)
}
window.addEventListener('resize', recomputeRail)
return () => {
ro?.disconnect()
@@ -444,6 +578,42 @@ export function EditorCore({
}
}, [recomputeRail])
// The rail only reports its extent while it's mounted, so clear it when the last
// card goes (accepted the lot, or the window narrowed past the rail's threshold)
// — otherwise the column keeps the height of a stack that no longer exists.
useEffect(() => {
if (!railEnabled || railItems.length === 0) setRailExtent(0)
}, [railEnabled, railItems.length])
// Decide whether the text column has to be pinned. The rail's cards hang off an
// absolutely-positioned column, so when several suggestions share one short
// paragraph the stack runs far past the last line of text. Growing the wrapper to
// `railExtent` makes that space scrollable (item 4: the lower cards were simply
// unreachable); pinning the prose inside it means scrolling down to read those
// cards keeps the sentences on screen instead of scrolling them away.
//
// The offset is `min(0, port - content)`: prose shorter than the viewport sticks
// at the top, taller prose sticks by its *bottom* edge, so its last lines — the
// ones the overhanging cards flag — stay visible rather than the first.
useLayoutEffect(() => {
const wrapper = wrapperRef.current
const content = contentRef.current
if (!wrapper || !content || !railEnabled || railExtent <= 0) {
setStickTop(null)
return
}
const contentH = content.offsetHeight
// Only pin when the stack actually overhangs the prose; a rail that fits
// beside its text needs nothing, and pinning it would be a change for free.
if (railExtent <= contentH) {
setStickTop(null)
return
}
const port = wrapper.closest('.petal-scrollport')
const portH = port ? port.clientHeight : window.innerHeight
setStickTop(Math.min(0, portH - contentH - RAIL_TAIL))
}, [railEnabled, railExtent, railItems])
// Emphasize the flagged text for the active suggestion, mirroring the rail
// card ↔ text link both ways. Driven through the decoration plugin (not an
// imperative DOM class) so it survives the repaints that fire on every edit.
@@ -538,12 +708,15 @@ export function EditorCore({
(e: React.MouseEvent) => {
if (!(e.target as HTMLElement).closest('.petal-suggestion')) return
if (railEnabled) {
setActiveId(null)
// A click-opened card outlives the pointer (it closes on a click away),
// so its rail card keeps the glow — otherwise the open card and the
// margin stop agreeing about which suggestion is being read.
if (!hover) setActiveId(null)
return
}
scheduleClose()
},
[scheduleClose, railEnabled],
[scheduleClose, railEnabled, hover],
)
const keepOpen = useCallback(() => clearTimeout(closeTimer.current), [])
@@ -553,18 +726,28 @@ export function EditorCore({
// anchored to the highlight itself (captured before the replacement removes it),
// so it fires in the right spot whether the accept came from the hover card or
// the margin rail.
const handleAccept = useCallback(
(s: Suggestion) => {
// Where a suggestion's highlight sits, in wrapper coordinates — captured before
// the replacement removes it, so the confetti fires over the words that changed.
const burstAt = useCallback((id: string): { top: number; left: number } | null => {
const wrapper = wrapperRef.current
const el = wrapper?.querySelector(
`.petal-suggestion[data-suggestion-id="${CSS.escape(s.id)}"]`,
`.petal-suggestion[data-suggestion-id="${CSS.escape(id)}"]`,
) as HTMLElement | null
let burst: { top: number; left: number } | null = null
if (wrapper && el) {
if (!wrapper || !el) return null
const wrapRect = wrapper.getBoundingClientRect()
const elRect = el.getBoundingClientRect()
burst = { top: elRect.top - wrapRect.top, left: elRect.right - wrapRect.left }
}
return { top: elRect.top - wrapRect.top, left: elRect.right - wrapRect.left }
}, [])
const showConfetti = useCallback((burst: { top: number; left: number }) => {
setConfetti(burst)
clearTimeout(confettiTimer.current)
confettiTimer.current = setTimeout(() => setConfetti(null), 720)
}, [])
const handleAccept = useCallback(
(s: Suggestion) => {
let burst = burstAt(s.id)
if (editor && s.replacement.trim() !== '') {
const range = findRange(editor.state.doc, s.original)
if (range) {
@@ -573,16 +756,57 @@ export function EditorCore({
}
// Fall back to the hover card's position if the highlight wasn't found.
if (!burst && hover) burst = { top: hover.top, left: hover.left + 16 }
if (burst) {
setConfetti(burst)
clearTimeout(confettiTimer.current)
confettiTimer.current = setTimeout(() => setConfetti(null), 720)
}
if (burst) showConfetti(burst)
closeCard()
setRailExpandedId(null)
onAccept(s)
},
[editor, onAccept, closeCard, hover],
[editor, onAccept, closeCard, hover, burstAt, showConfetti],
)
// How many pending cards of each type could be accepted in one go. A card
// offers "Accept all" only when it has company, so a lone Grammar card doesn't
// grow a second button saying the same thing as the first.
const batchCounts = useMemo(() => {
const counts: Partial<Record<SuggestionType, number>> = {}
for (const s of suggestions) {
if (s.replacement.trim() === '') continue // awareness-only: nothing to accept
counts[s.type] = (counts[s.type] ?? 0) + 1
}
return counts
}, [suggestions])
// Accept a whole category at once. Five tense fixes were five clicks, five
// confetti bursts and five separate undo steps; this is one of each. The single
// undo step is the reason every replacement goes into ONE chain: Tiptap applies
// a chain as a single transaction, and prosemirror-history undoes it as a
// single event, so Ctrl+Z takes back the batch rather than unpicking it.
const handleAcceptAll = useCallback(
(type: SuggestionType) => {
if (!editor) return
const family = suggestions.filter((s) => s.type === type)
const plan = planBatch(family, (original) => findRange(editor.state.doc, original))
const settled = [...plan.steps.map((step) => step.suggestion), ...plan.missing]
if (settled.length === 0) return
// The topmost span that's about to change — the last step, since steps run
// bottom-up. Read before the edit, while the highlights still exist.
const first = plan.steps[plan.steps.length - 1]
const burst = first ? burstAt(first.suggestion.id) : null
if (plan.steps.length > 0) {
let chain = editor.chain().focus()
for (const step of plan.steps) {
chain = chain.insertContentAt({ from: step.from, to: step.to }, step.suggestion.replacement)
}
chain.run()
}
if (burst) showConfetti(burst)
closeCard()
setRailExpandedId(null)
onAcceptMany(settled)
},
[editor, suggestions, onAcceptMany, closeCard, burstAt, showConfetti],
)
const handleDismiss = useCallback(
@@ -633,15 +857,15 @@ export function EditorCore({
if (suggestionEl) {
const id = suggestionEl.getAttribute('data-suggestion-id')
if (id) {
// With the rail open, a tap emphasizes and expands its margin card
// instead of opening a floating one.
if (railEnabled) {
// Clicking a highlight always opens the card at the word. Hover still
// defers to the rail (see handleMouseOver) — an unbidden floating card
// beside a margin card that already says the same thing is noise. But a
// click is her asking to deal with *this* word, and answering it 650px
// away in the periphery is the gesture item 7 is about. The rail card
// glows rather than expanding, so the suggestion is never open twice.
setActiveId(id)
setRailExpandedId(id)
} else {
openCardFor(id, suggestionEl)
}
}
return
}
if (!editor) return
@@ -692,7 +916,7 @@ export function EditorCore({
const openWordLookup = useCallback(
(pos: number) => {
if (!editor) return
const range = wordAt(editor.state.doc, pos, wordAlphabet)
const range = resolveWord(pos)
if (!range) return
const wrapper = wrapperRef.current
if (!wrapper) return
@@ -708,12 +932,18 @@ export function EditorCore({
closeCard()
setMisspell(null)
const token = ++wordReqRef.current
setWordInfo({ word: range.word, from: range.from, to: range.to, top, left, loading: true, info: null, vocabId: null, saved: false })
setWordInfo({ word: range.word, from: range.from, to: range.to, top, left, loading: true, info: null, pinyin: '', vocabId: null, saved: false })
// The sentence the word sits in, for review context in the garden.
const example = exampleAt(range.from)
api
.lookupWord(range.word)
.then((info) => {
// Two dictionaries, one card. The Chinese lookup answers in English and
// the English one answers in her language; which is wanted follows from
// which script the word is written in, so nothing here has to consult the
// account's direction a second time.
const lookup: Promise<{ info: WordInfo; pinyin: string }> = range.hanzi
? api.hanziWord(range.word).then((h) => ({ info: hanziToWordInfo(h), pinyin: hanziPinyin(h) }))
: api.lookupWord(range.word).then((info) => ({ info, pinyin: '' }))
lookup
.then(({ info, pinyin }) => {
if (token !== wordReqRef.current) return
// Auto-capture into the vocabulary garden — only words the dictionary
// actually knows (a real gloss or definition), so accidental lookups of
@@ -723,14 +953,18 @@ export function EditorCore({
// Reflect the saved state optimistically so the heart shows 💚 the
// moment a known word loads, rather than flashing 🤍 until the capture
// round-trips. vocabId is filled in when recordVocab returns.
setWordInfo((w) => (w ? { ...w, loading: false, info, saved: known } : null))
setWordInfo((w) => (w ? { ...w, loading: false, info, pinyin, saved: known } : null))
if (!known) return
api
.recordVocab({
word: range.word,
gloss: info.gloss,
definition: info.definitions[0]?.definition ?? '',
phonetic: info.phonetic,
// The garden's pronunciation field holds whichever this word has:
// IPA for an English word, pinyin for a Chinese one. Both answer
// the same question on a review card — how do I say this — and a
// second column would only be a second thing to keep in sync.
phonetic: pinyin || info.phonetic,
example,
doc_id: docId,
})
@@ -750,7 +984,7 @@ export function EditorCore({
}
})
},
[editor, closeCard, docId],
[editor, closeCard, docId, resolveWord, exampleAt],
)
// Toggle a looked-up word in/out of the vocabulary garden from the WordCard
@@ -797,13 +1031,16 @@ export function EditorCore({
if (!editor) return
const coords = editor.view.posAtCoords({ left: e.clientX, top: e.clientY })
if (!coords) return
if (!wordAt(editor.state.doc, coords.pos, wordAlphabet)) return
// Whichever script is under the pointer — the same resolver openWordLookup
// uses, so a Chinese word gets the card here too rather than falling
// through to the native menu.
if (!resolveWord(coords.pos)) return
e.preventDefault()
// A misspelled word offers corrections first; otherwise look it up.
if (openMisspellAt(coords.pos)) return
openWordLookup(coords.pos)
},
[editor, wordAlphabet, openMisspellAt, openWordLookup],
[editor, resolveWord, openMisspellAt, openWordLookup],
)
// Touch has no hover or right-click, so a long-press (~500ms without moving)
@@ -857,7 +1094,7 @@ export function EditorCore({
clear()
return
}
const range = wordAt(editor.state.doc, coords.pos, wordAlphabet)
const range = resolveWord(coords.pos)
if (!range) {
clear()
return
@@ -866,9 +1103,21 @@ export function EditorCore({
if (gloss && gloss.from === range.from && gloss.to === range.to) return
clearTimeout(glossTimer.current)
const token = ++glossReqRef.current
// The Chinese hover carries a second line the English one has no use for:
// pinyin above the meaning. It is the thing a learner most often stops to
// ask about their own writing — reading a character back is not the same
// as being able to say it — and it is why this tooltip is worth having at
// all for a script the writer can already read the meaning of half the
// time.
const ask = (): Promise<{ gloss: string; reverse?: string; lead?: string }> =>
range.hanzi
? api.hanziWord(range.word).then((h) => ({
gloss: h.readings[0]?.senses ?? h.chars.map((c) => `${c.char} ${c.senses}`).join(' · '),
lead: hanziPinyin(h),
}))
: api.glossWord(range.word).then((g) => ({ gloss: g.gloss, reverse: g.reverse }))
glossTimer.current = setTimeout(() => {
api
.glossWord(range.word)
ask()
.then((g) => {
if (token !== glossReqRef.current) return
const wrapper = wrapperRef.current
@@ -883,14 +1132,14 @@ export function EditorCore({
const wrapRect = wrapper.getBoundingClientRect()
const left = Math.max(0, Math.min(start.left - wrapRect.left, wrapper.clientWidth - 280))
const top = end.bottom - wrapRect.top + 6
setGloss({ word: range.word, gloss: g.gloss, reverse: g.reverse, from: range.from, to: range.to, top, left })
setGloss({ word: range.word, gloss: g.gloss, reverse: g.reverse, lead: g.lead, from: range.from, to: range.to, top, left })
})
.catch(() => {
if (token === glossReqRef.current) setGloss(null)
})
}, 350)
},
[editor, wordAlphabet, selection, rewrite, misspell, wordInfo, pinned, gloss],
[editor, resolveWord, selection, rewrite, misspell, wordInfo, pinned, gloss],
)
// Leaving the editor surface drops any pending/shown gloss.
@@ -1082,6 +1331,11 @@ export function EditorCore({
<div
ref={wrapperRef}
className="relative flex-1"
// Grown to cover whichever absolutely-positioned surface reaches lowest —
// the rail's card stack, or an open anchored card — so the space those
// cards occupy is actually scrollable. `minHeight` never shrinks the
// column, so a rail or card that fits beside its text changes nothing.
style={overhang > 0 ? { minHeight: overhang } : undefined}
onMouseOver={handleMouseOver}
onMouseOut={handleMouseOut}
onMouseMove={handleMouseMove}
@@ -1092,13 +1346,24 @@ export function EditorCore({
onTouchStart={handleTouchStart}
onTouchMove={cancelLongPress}
onTouchEnd={cancelLongPress}
>
{/* The prose sits in its own box so it can be measured (and pinned)
independently of the wrapper, which the rail may have grown. The box is
deliberately left at its natural height: sized to the wrapper it would
report the stack's height back as the text's own, and the pin below
could never trip. */}
<div
ref={contentRef}
style={stickTop === null ? undefined : { position: 'sticky', top: stickTop }}
>
<EditorContent editor={editor} className="h-full" />
</div>
{findOpen && editor && <FindReplace editor={editor} onClose={() => setFindOpen(false)} />}
{confetti && <Confetti top={confetti.top} left={confetti.left} />}
{gloss && (
<GlossTip
gloss={gloss.gloss}
lead={gloss.lead}
reverse={gloss.reverse}
reverseLang={pack.nativeName}
style={{ top: gloss.top, left: gloss.left }}
@@ -1131,6 +1396,7 @@ export function EditorCore({
loading={wordInfo.loading}
saved={wordInfo.saved}
onToggleSave={toggleSaveWord}
pinyin={wordInfo.pinyin}
style={{ top: wordInfo.top, left: wordInfo.left }}
onReplace={replaceWord}
/>
@@ -1144,15 +1410,18 @@ export function EditorCore({
onAdd={addMisspellingToDict}
/>
)}
{hover && !railEnabled && (
{hover && (
<SuggestionCard
suggestion={hover.suggestion}
style={{ top: hover.top, left: hover.left }}
batchCount={batchCounts[hover.suggestion.type] ?? 0}
onAccept={handleAccept}
onAcceptAll={handleAcceptAll}
onDismiss={handleDismiss}
onPointerEnter={keepOpen}
onPointerLeave={scheduleClose}
onExpandChange={setPinned}
onExtent={setCardExtent}
/>
)}
{railEnabled && railItems.length > 0 && (
@@ -1160,11 +1429,14 @@ export function EditorCore({
items={railItems}
activeId={activeId}
expandedId={railExpandedId}
batchCounts={batchCounts}
onAccept={handleAccept}
onAcceptAll={handleAcceptAll}
onDismiss={handleDismiss}
onHover={setActiveId}
onActivate={activateRailCard}
onToggleExpand={toggleRailExpand}
onExtent={setRailExtent}
/>
)}
</div>
+7 -3
View File
@@ -2,6 +2,7 @@ import { useCallback, useEffect, useRef, useState } from 'react'
import type { Editor } from '@tiptap/react'
import { clearSearch, getSearchState, setActive, setSearch } from './SearchHighlight'
import { usePack } from '../../i18n'
import { fromIME } from '../../lib/ime'
// FindReplace is the in-document search bar (Ctrl/Cmd+F). It drives the
// SearchHighlight decoration layer: typing updates the highlighted matches, the
@@ -108,7 +109,10 @@ export function FindReplace({ editor, onClose }: Props) {
role="dialog"
aria-label="Find and replace"
onKeyDown={(e) => {
if (e.key === 'Escape') {
// Both fields take Chinese, so both take an IME: Escape cancels a
// candidate and Enter commits one. A key that belongs to the composition
// is not a command here — see lib/ime.
if (e.key === 'Escape' && !fromIME(e)) {
e.preventDefault()
onClose()
}
@@ -137,7 +141,7 @@ export function FindReplace({ editor, onClose }: Props) {
value={query}
onChange={(e) => setQuery(e.target.value)}
onKeyDown={(e) => {
if (e.key === 'Enter') {
if (e.key === 'Enter' && !fromIME(e)) {
e.preventDefault()
go(e.shiftKey ? -1 : 1)
}
@@ -171,7 +175,7 @@ export function FindReplace({ editor, onClose }: Props) {
value={replacement}
onChange={(e) => setReplacement(e.target.value)}
onKeyDown={(e) => {
if (e.key === 'Enter') {
if (e.key === 'Enter' && !fromIME(e)) {
e.preventDefault()
replaceActive()
}
+11 -1
View File
@@ -7,6 +7,11 @@
interface Props {
gloss: string
// A line above the gloss, in a lighter weight: the pinyin of a Chinese word.
// It leads because it is what is actually being asked — a learner reading
// their own 公园 back may know it means a park and still not know how to say
// it, which is the one thing the character does not tell them.
lead?: string
// The English meaning of the same token read as a word of the writer's own
// language, when it is one. On a Latin-script pair "sale" is both, and the
// bubble shows the two readings stacked rather than picking one — the same
@@ -17,7 +22,7 @@ interface Props {
style: React.CSSProperties
}
export function GlossTip({ gloss, reverse, reverseLang, style }: Props) {
export function GlossTip({ gloss, lead, reverse, reverseLang, style }: Props) {
return (
<div
className="petal-gloss-tip pointer-events-none absolute z-20 px-2.5 py-1.5 text-sm"
@@ -33,6 +38,11 @@ export function GlossTip({ gloss, reverse, reverseLang, style }: Props) {
...style,
}}
>
{lead && (
<span className="mb-0.5 block font-semibold" style={{ opacity: 0.9 }}>
{lead}
</span>
)}
{gloss}
{reverse && (
<span className="mt-0.5 block" style={{ opacity: 0.72, fontSize: '0.85em' }}>
+45 -19
View File
@@ -4,6 +4,7 @@ import type { EditorState, Transaction } from '@tiptap/pm/state'
import { Decoration, DecorationSet } from '@tiptap/pm/view'
import type { Node as PMNode } from '@tiptap/pm/model'
import { mapOffset } from './SuggestionHighlight'
import { holdRedraw } from './Composition'
// SearchHighlight powers the in-document Find & Replace bar. Like the suggestion
// layer it uses ProseMirror *decorations* (not stored marks), so matches are
@@ -22,6 +23,11 @@ interface PluginState {
matches: Match[]
active: number // index into matches, or -1 when there are none
decorations: DecorationSet
// Held back while an IME composition was in flight — see Composition.ts.
// `matches` is held with the decorations rather than recomputed on its own:
// the Find bar's "3 / 7" and the wash on the page are one answer, and half of
// it moving while the other half waits would be worse than both waiting.
stale: boolean
}
export const searchPluginKey = new PluginKey<PluginState>('petalSearch')
@@ -57,7 +63,7 @@ function build(doc: PMNode, query: string, caseSensitive: boolean, preferred: nu
class: i === active ? 'petal-find-match petal-find-match-active' : 'petal-find-match',
}),
)
return { query, caseSensitive, matches, active, decorations: DecorationSet.create(doc, decos) }
return { query, caseSensitive, matches, active, decorations: DecorationSet.create(doc, decos), stale: false }
}
const EMPTY: PluginState = {
@@ -66,6 +72,7 @@ const EMPTY: PluginState = {
matches: [],
active: -1,
decorations: DecorationSet.empty,
stale: false,
}
// setSearch updates the query / case-sensitivity and recomputes matches. Passing
@@ -100,36 +107,49 @@ type Meta =
| { kind: 'active'; index: number }
| { kind: 'clear' }
export const SearchHighlight = Extension.create({
name: 'searchHighlight',
addProseMirrorPlugins() {
return [
new Plugin<PluginState>({
export function searchPlugin(): Plugin<PluginState> {
return new Plugin<PluginState>({
key: searchPluginKey,
state: {
init: () => EMPTY,
apply(tr, value, _oldState, newState): PluginState {
apply(tr, value, oldState, newState): PluginState {
const meta = tr.getMeta(searchPluginKey) as Meta | undefined
if (meta?.kind === 'search') {
return build(newState.doc, meta.query, meta.caseSensitive, value.active < 0 ? 0 : value.active)
}
// Clearing the layer is the one thing a composition never holds: it
// removes decorations rather than adding them, and it is what closing
// the Find bar does.
if (meta?.kind === 'clear') return EMPTY
const held = holdRedraw(tr, oldState)
const query = meta?.kind === 'search' ? meta.query : value.query
const caseSensitive = meta?.kind === 'search' ? meta.caseSensitive : value.caseSensitive
if (meta?.kind === 'active') {
if (value.matches.length === 0) return value
const active = ((meta.index % value.matches.length) + value.matches.length) % value.matches.length
if (held) return { ...value, active, stale: true }
const decos = value.matches.map((m, i) =>
Decoration.inline(m.from, m.to, {
class: i === active ? 'petal-find-match petal-find-match-active' : 'petal-find-match',
}),
)
return { ...value, active, decorations: DecorationSet.create(newState.doc, decos) }
return { ...value, active, stale: false, decorations: DecorationSet.create(newState.doc, decos) }
}
if (meta?.kind === 'clear') return EMPTY
// Re-anchor on any document change so highlights track edits/replaces.
if (tr.docChanged && value.query) {
return build(newState.doc, value.query, value.caseSensitive, value.active)
// A new query, or any document change: re-anchor so highlights track
// edits and replaces. Once due, it stays due until it happens.
const due = value.stale || meta?.kind === 'search' || (tr.docChanged && !!value.query)
if (!due) return value
if (held) {
return {
...value,
query,
caseSensitive,
stale: true,
decorations: tr.docChanged ? value.decorations.map(tr.mapping, tr.doc) : value.decorations,
}
return value
}
const preferred = meta?.kind === 'search' && value.active < 0 ? 0 : value.active
return build(newState.doc, query, caseSensitive, preferred)
},
},
props: {
@@ -137,7 +157,13 @@ export const SearchHighlight = Extension.create({
return searchPluginKey.getState(state)?.decorations
},
},
}),
]
})
}
export const SearchHighlight = Extension.create({
name: 'searchHighlight',
addProseMirrorPlugins() {
return [searchPlugin()]
},
})
@@ -50,6 +50,25 @@ describe('wordAt and the pair alphabet', () => {
expect(wordAt(doc, posOf(6), true)?.word).toBe('today')
})
it('keeps a curly apostrophe inside the word, because that is the only kind here', () => {
// Typography.ts rewrites every ' typed in this editor into , so a token
// that only matched the straight mark never saw an apostrophe at all.
// "aujourdhui" cut into "aujourd" + "hui" — neither one a French word, both
// underlined, and withElision never reached.
const doc = para('cest aujourdhui')
expect(wordAt(doc, posOf(8), true)?.word).toBe('aujourdhui')
// English pays the same debt: "dont" must stay whole to be looked up.
expect(wordAt(doc, posOf(1))?.word).toBe('cest')
})
it('reads œ as a letter, not as a word boundary', () => {
// U+0153 sits outside the Latin-1 ranges, so "cœur" tokenized as "c" + "ur"
// and the orphan "ur" was long enough to be underlined. 586 œ forms ship in
// the French word list and none of them were reachable.
const doc = para('mon cœur')
expect(wordAt(doc, posOf(5), true)?.word).toBe('cœur')
})
it('stops the wide alphabet at the maths symbols hiding in Latin-1', () => {
// × (U+00D7) and ÷ (U+00F7) sit inside the accented-letter block. A range
// written À-ÿ would swallow them and glue "3×4" into one token.
+47 -18
View File
@@ -4,6 +4,7 @@ import type { EditorState, Transaction } from '@tiptap/pm/state'
import { Decoration, DecorationSet } from '@tiptap/pm/view'
import type { Node as PMNode } from '@tiptap/pm/model'
import { mapOffset } from './SuggestionHighlight'
import { holdRedraw } from './Composition'
import type { SpellChecker } from '../../hooks/useSpellChecker'
// SpellCheck renders browser-side nspell misspellings as ProseMirror
@@ -18,6 +19,11 @@ export const spellPluginKey = new PluginKey<PluginState>('petalSpellCheck')
interface PluginState {
checker: SpellChecker | null
decorations: DecorationSet
// Held back while an IME composition was in flight — see Composition.ts. This
// layer is the one with the most to gain from the guard: the pinyin she is
// part-way through typing is Latin letters, so it is exactly what the
// tokenizer picks up and exactly what an underline would redraw over.
stale: boolean
}
// A word is a run of Latin letters with optional internal/edge apostrophes
@@ -32,8 +38,19 @@ interface PluginState {
// writer with no Latin second language, adding accented letters can only find
// new words to underline (the "café" and "naïve" she borrows), and finds no
// mistakes she has actually made.
const WORD_RE = /[A-Za-z][A-Za-z']*/g
const WORD_RE_LATIN = /[A-Za-zÀ-ÖØ-öø-ÿ][A-Za-zÀ-ÖØ-öø-ÿ']*/g
//
// Both apostrophes count as word characters, because Typography.ts rewrites
// every ' typed in this editor into a curly — so by the time text reaches the
// tokenizer the straight mark is only ever seen in pasted content. Matching only
// the straight one cut "aujourdhui" into "aujourd" + "hui", neither of which is
// a French word, and left the elision handling in useSpellChecker unreachable.
// The dictionaries spell theirs straight; `combine` normalises on lookup.
//
// Œ/œ are named explicitly: they sit at U+0152/U+0153, outside the Latin-1
// ranges below, so without them "cœur" tokenized as "c" + "ur" and the bare
// "ur" was underlined. Æ/æ need no help — they are inside À-Ö and Ø-ö.
const WORD_RE = /[A-Za-z][A-Za-z']*/g
const WORD_RE_LATIN = /[A-Za-zÀ-ÖØ-öø-ÿŒœ][A-Za-zÀ-ÖØ-öø-ÿŒœ']*/g
// wordRe returns a fresh matcher for the alphabet in force. Fresh because these
// are /g regexes carrying lastIndex, and two scans sharing one would interleave.
@@ -55,8 +72,8 @@ function isCheckable(word: string): boolean {
// lookup sees the bare token; returns the core plus how many chars were trimmed
// off the front (to re-anchor the decoration).
function coreOf(word: string): { core: string; lead: number } {
const lead = word.match(/^'+/)?.[0].length ?? 0
const trail = word.match(/'+$/)?.[0].length ?? 0
const lead = word.match(/^[']+/)?.[0].length ?? 0
const trail = word.match(/[']+$/)?.[0].length ?? 0
return { core: word.slice(lead, word.length - trail), lead }
}
@@ -140,25 +157,31 @@ export function setSpellChecker(
dispatch(state.tr.setMeta(spellPluginKey, checker ?? null))
}
export const SpellCheck = Extension.create({
name: 'spellCheck',
addProseMirrorPlugins() {
return [
new Plugin<PluginState>({
export function spellPlugin(): Plugin<PluginState> {
return new Plugin<PluginState>({
key: spellPluginKey,
state: {
init: () => ({ checker: null, decorations: DecorationSet.empty }),
apply(tr, value, _oldState, newState) {
init: () => ({ checker: null, decorations: DecorationSet.empty, stale: false }),
apply(tr, value, oldState, newState) {
const meta = tr.getMeta(spellPluginKey) as SpellChecker | null | undefined
const checker = meta !== undefined ? meta : value.checker
if (!checker) return { checker: null, decorations: DecorationSet.empty }
if (!checker) return { checker: null, decorations: DecorationSet.empty, stale: false }
// Rebuild on a checker swap, a doc edit, or a caret move (so the word
// you just left gets re-evaluated and the new caret word is exempt).
if (meta !== undefined || tr.docChanged || tr.selectionSet) {
return { checker, decorations: buildDecorations(newState.doc, checker, newState.selection.head) }
const due = value.stale || meta !== undefined || tr.docChanged || tr.selectionSet
if (!due) return value
if (holdRedraw(tr, oldState)) {
return {
checker,
stale: true,
decorations: tr.docChanged ? value.decorations.map(tr.mapping, tr.doc) : value.decorations,
}
}
return {
checker,
stale: false,
decorations: buildDecorations(newState.doc, checker, newState.selection.head),
}
return { checker, decorations: value.decorations }
},
},
props: {
@@ -166,7 +189,13 @@ export const SpellCheck = Extension.create({
return spellPluginKey.getState(state)?.decorations
},
},
}),
]
})
}
export const SpellCheck = Extension.create({
name: 'spellCheck',
addProseMirrorPlugins() {
return [spellPlugin()]
},
})
+52 -5
View File
@@ -1,18 +1,30 @@
import { useState } from 'react'
import type { Suggestion } from '../../api/client'
import { useEffect, useRef, useState } from 'react'
import type { Suggestion, SuggestionType } from '../../api/client'
import { usePack } from '../../i18n'
import { AskPetal } from './AskPetal'
import { TYPE_META } from './suggestionMeta'
import { TYPE_META, batchLabel, typeLabel } from './suggestionMeta'
interface Props {
suggestion: Suggestion
style: React.CSSProperties
// How many pending suggestions share this card's type (including this one).
// Two or more offers to take the whole category in one step.
batchCount: number
onAccept: (s: Suggestion) => void
onAcceptAll: (type: SuggestionType) => void
onDismiss: (s: Suggestion) => void
onPointerEnter: () => void
onPointerLeave: () => void
// Pins the card open while the Ask Petal panel is expanded, so the chat isn't
// dismissed by the hover-close timer when the pointer drifts away.
onExpandChange: (expanded: boolean) => void
// How far the card reaches below the wrapper's top, in wrapper coordinates —
// the same report the rail makes (item 4). The card is absolutely positioned
// and so adds no layout height of its own; without this, an Ask Petal
// conversation that runs past the last line of text has no scrollable page
// under it and its Accept button simply can't be reached. 0 means "nothing to
// cover", which is what an unmounted card reports on its way out.
onExtent?: (bottom: number) => void
}
// SuggestionCard is the hover panel for a single suggestion: a colored type tag,
@@ -22,15 +34,38 @@ interface Props {
export function SuggestionCard({
suggestion,
style,
batchCount,
onAccept,
onAcceptAll,
onDismiss,
onPointerEnter,
onPointerLeave,
onExpandChange,
onExtent,
}: Props) {
const pack = usePack()
const meta = TYPE_META[suggestion.type]
const label = typeLabel(suggestion.type, pack)
const hasReplacement = suggestion.replacement.trim() !== ''
const [asking, setAsking] = useState(false)
const cardRef = useRef<HTMLDivElement>(null)
// Report the card's reach while it is open, and withdraw it on the way out.
// A ResizeObserver rather than a one-shot measure because the card grows
// after it is mounted: the Ask Petal panel opens, and then the reply streams
// into it token by token.
useEffect(() => {
const el = cardRef.current
if (!el || !onExtent) return
const report = () => onExtent(el.offsetTop + el.offsetHeight)
report()
const observer = new ResizeObserver(report)
observer.observe(el)
return () => {
observer.disconnect()
onExtent(0)
}
}, [onExtent])
function toggleAsking() {
setAsking((prev) => {
@@ -42,8 +77,9 @@ export function SuggestionCard({
return (
<div
ref={cardRef}
role="dialog"
aria-label={`${meta.label} suggestion`}
aria-label={`${label} suggestion`}
onMouseEnter={onPointerEnter}
onMouseLeave={onPointerLeave}
className="petal-suggestion-card absolute z-20 p-3.5 text-sm"
@@ -60,7 +96,7 @@ export function SuggestionCard({
className="inline-flex items-center gap-1.5 rounded-full px-2.5 py-0.5 text-xs font-bold"
style={{ background: meta.color, color: 'var(--color-plum)' }}
>
{meta.label}
{label}
</span>
{hasReplacement && (
@@ -114,6 +150,17 @@ export function SuggestionCard({
Dismiss
</button>
</div>
{hasReplacement && batchCount > 1 && (
<button
type="button"
onClick={() => onAcceptAll(suggestion.type)}
className="petal-accept-all mt-2 w-full rounded-full py-1.5 text-xs font-bold"
style={{ color: 'var(--color-plum)', borderColor: meta.color }}
>
{batchLabel(suggestion.type, batchCount)}
</button>
)}
</div>
)
}
@@ -4,6 +4,7 @@ import type { EditorState, Transaction } from '@tiptap/pm/state'
import { Decoration, DecorationSet } from '@tiptap/pm/view'
import type { Node as PMNode } from '@tiptap/pm/model'
import type { Suggestion } from '../../api/client'
import { holdRedraw } from './Composition'
// SuggestionHighlight renders LLM suggestions as ProseMirror *decorations*, not
// stored marks. Decorations are ephemeral overlays recomputed from the live
@@ -21,6 +22,10 @@ interface PluginState {
// decoration repaints that fire on every document change.
activeId: string | null
decorations: DecorationSet
// A rebuild fell due while an IME composition was in flight and was held back
// (see Composition.ts). The decorations on screen are the previous ones,
// mapped forward; this says they still owe a rebuild.
stale: boolean
}
// Meta carried on a transaction to update the plugin: either a fresh suggestion
@@ -173,40 +178,37 @@ export function setActiveSuggestion(
dispatch(state.tr.setMeta(suggestionPluginKey, { activeId } satisfies SuggestionMeta))
}
export const SuggestionHighlight = Extension.create({
name: 'suggestionHighlight',
addProseMirrorPlugins() {
return [
new Plugin<PluginState>({
export function suggestionPlugin(): Plugin<PluginState> {
return new Plugin<PluginState>({
key: suggestionPluginKey,
state: {
init: () => ({ suggestions: [], activeId: null, decorations: DecorationSet.empty }),
apply(tr, value, _oldState, newState) {
init: () => ({ suggestions: [], activeId: null, decorations: DecorationSet.empty, stale: false }),
apply(tr, value, oldState, newState) {
const meta = tr.getMeta(suggestionPluginKey) as SuggestionMeta | undefined
if (meta && 'suggestions' in meta) {
const suggestions = meta && 'suggestions' in meta ? meta.suggestions : value.suggestions
const activeId = meta && 'activeId' in meta ? meta.activeId : value.activeId
// A rebuild is due on a new list, a new emphasis, or any document change
// (which is how a suggestion re-anchors by string), and stays due until
// it happens.
const due = value.stale || meta !== undefined || tr.docChanged
if (!due) return value
if (holdRedraw(tr, oldState)) {
return {
suggestions: meta.suggestions,
activeId: value.activeId,
decorations: buildDecorations(newState.doc, meta.suggestions, value.activeId),
suggestions,
activeId,
stale: true,
// Map rather than keep: the composing text is growing under these
// highlights, and an unmapped decoration would drift a character at
// a time across a word she is still typing.
decorations: tr.docChanged ? value.decorations.map(tr.mapping, tr.doc) : value.decorations,
}
}
if (meta && 'activeId' in meta) {
return {
suggestions: value.suggestions,
activeId: meta.activeId,
decorations: buildDecorations(newState.doc, value.suggestions, meta.activeId),
suggestions,
activeId,
stale: false,
decorations: buildDecorations(newState.doc, suggestions, activeId),
}
}
// On any document change, re-anchor by string against the new doc.
if (tr.docChanged) {
return {
suggestions: value.suggestions,
activeId: value.activeId,
decorations: buildDecorations(newState.doc, value.suggestions, value.activeId),
}
}
return value
},
},
props: {
@@ -214,7 +216,13 @@ export const SuggestionHighlight = Extension.create({
return suggestionPluginKey.getState(state)?.decorations
},
},
}),
]
})
}
export const SuggestionHighlight = Extension.create({
name: 'suggestionHighlight',
addProseMirrorPlugins() {
return [suggestionPlugin()]
},
})
+56 -7
View File
@@ -1,7 +1,9 @@
import { forwardRef, useLayoutEffect, useRef, useState } from 'react'
import type { Suggestion } from '../../api/client'
import type { Suggestion, SuggestionType } from '../../api/client'
import { usePack } from '../../i18n'
import { AskPetal } from './AskPetal'
import { TYPE_META } from './suggestionMeta'
import { batchLeaders } from './acceptBatch'
import { TYPE_META, batchLabel, typeLabel } from './suggestionMeta'
// Vertical breathing room kept between stacked cards when their natural anchors
// would otherwise collide.
@@ -21,13 +23,23 @@ interface Props {
activeId: string | null
// The card expanded to show the full explanation + Ask Petal, or null.
expandedId: string | null
// Pending suggestions per type. A card whose type has company offers to take
// the whole category in one step (and one undo step).
batchCounts: Partial<Record<SuggestionType, number>>
onAccept: (s: Suggestion) => void
onAcceptAll: (type: SuggestionType) => void
onDismiss: (s: Suggestion) => void
// Pointer entering/leaving a card, so the matching highlight can light up.
onHover: (id: string | null) => void
// A card's body was clicked — scroll its highlight into view and toggle expand.
onActivate: (id: string) => void
onToggleExpand: (id: string) => void
// How far down the resolved stack reaches (px below the wrapper's top). Cards
// are absolutely positioned, so they add nothing to layout height — a cluster of
// errors in one short paragraph can pile cards hundreds of px past the end of the
// text, with no scrollable space to reach them. The editor uses this to grow the
// column so every card can at least be scrolled to.
onExtent: (bottom: number) => void
}
// SuggestionRail is the right-margin "comment column": every outstanding
@@ -39,11 +51,14 @@ export function SuggestionRail({
items,
activeId,
expandedId,
batchCounts,
onAccept,
onAcceptAll,
onDismiss,
onHover,
onActivate,
onToggleExpand,
onExtent,
}: Props) {
// Measured resolved tops keyed by suggestion id (after collision avoidance).
const [tops, setTops] = useState<Record<string, number>>({})
@@ -65,13 +80,16 @@ export function SuggestionRail({
const layoutKey = ordered.map((i) => `${i.suggestion.id}:${Math.round(i.anchorTop)}`).join('|')
useLayoutEffect(() => {
let cursor = -Infinity
let bottom = 0
const next: Record<string, number> = {}
for (const { suggestion, anchorTop } of ordered) {
const h = cardRefs.current.get(suggestion.id)?.offsetHeight ?? 96
const top = Math.max(anchorTop, cursor)
next[suggestion.id] = top
cursor = top + h + CARD_GAP
bottom = top + h
}
onExtent(bottom)
setTops((prev) => {
const ids = Object.keys(next)
if (ids.length === Object.keys(prev).length && ids.every((id) => prev[id] === next[id])) return prev
@@ -82,6 +100,9 @@ export function SuggestionRail({
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [layoutKey, expandedId, measureTick])
// One card per category carries the batch control — see batchLeaders.
const batchLead = batchLeaders(ordered.map(({ suggestion }) => suggestion))
return (
<div className="petal-rail petal-no-print" aria-label="Suggestions">
{ordered.map(({ suggestion }) => (
@@ -101,7 +122,9 @@ export function SuggestionRail({
top={tops[suggestion.id] ?? 0}
active={activeId === suggestion.id}
expanded={expandedId === suggestion.id}
batchCount={batchLead.has(suggestion.id) ? (batchCounts[suggestion.type] ?? 0) : 0}
onAccept={onAccept}
onAcceptAll={onAcceptAll}
onDismiss={onDismiss}
onHover={onHover}
onActivate={onActivate}
@@ -117,7 +140,9 @@ interface CardProps {
top: number
active: boolean
expanded: boolean
batchCount: number
onAccept: (s: Suggestion) => void
onAcceptAll: (type: SuggestionType) => void
onDismiss: (s: Suggestion) => void
onHover: (id: string | null) => void
onActivate: (id: string) => void
@@ -125,17 +150,24 @@ interface CardProps {
}
const RailCard = forwardRef<HTMLDivElement, CardProps>(function RailCard(
{ suggestion, top, active, expanded, onAccept, onDismiss, onHover, onActivate, onToggleExpand },
{ suggestion, top, active, expanded, batchCount, onAccept, onAcceptAll, onDismiss, onHover, onActivate, onToggleExpand },
ref,
) {
const pack = usePack()
const meta = TYPE_META[suggestion.type]
const label = typeLabel(suggestion.type, pack)
const hasReplacement = suggestion.replacement.trim() !== ''
// Every other card truncates its diff to one line each: the original and the
// replacement differ by a word or two, and the explanation below is the part
// she reads. A translation is the reverse — the two lines are a whole sentence
// in each language, and they are the entire point of the card — so it wraps.
const clampDiff = suggestion.type !== 'translate'
return (
<div
ref={ref}
role="group"
aria-label={`${meta.label} suggestion`}
aria-label={`${label} suggestion`}
onMouseEnter={() => onHover(suggestion.id)}
onMouseLeave={() => onHover(null)}
className={`petal-rail-card${active ? ' petal-rail-card-active' : ''}`}
@@ -146,7 +178,7 @@ const RailCard = forwardRef<HTMLDivElement, CardProps>(function RailCard(
className="inline-flex items-center rounded-full px-2 py-0.5 text-[0.68rem] font-bold"
style={{ background: meta.color, color: 'var(--color-plum)' }}
>
{meta.label}
{label}
</span>
<button
type="button"
@@ -163,10 +195,16 @@ const RailCard = forwardRef<HTMLDivElement, CardProps>(function RailCard(
<button type="button" onClick={() => onActivate(suggestion.id)} className="mt-2 block w-full text-left">
{hasReplacement && (
<span className="flex flex-col gap-0.5" style={{ fontFamily: 'var(--font-body)' }}>
<span className="truncate text-[0.9rem] line-through" style={{ color: 'var(--color-muted)' }}>
<span
className={`text-[0.9rem] line-through${clampDiff ? ' truncate' : ''}`}
style={{ color: 'var(--color-muted)' }}
>
{suggestion.original}
</span>
<span className="truncate text-[0.9rem] font-medium" style={{ color: 'var(--color-plum)' }}>
<span
className={`text-[0.9rem] font-medium${clampDiff ? ' truncate' : ''}`}
style={{ color: 'var(--color-plum)' }}
>
{suggestion.replacement}
</span>
</span>
@@ -206,6 +244,17 @@ const RailCard = forwardRef<HTMLDivElement, CardProps>(function RailCard(
{expanded ? 'Hide Petal' : 'Ask Petal ✨'}
</button>
</div>
{hasReplacement && batchCount > 1 && (
<button
type="button"
onClick={() => onAcceptAll(suggestion.type)}
className="petal-accept-all mt-2 w-full rounded-full py-1 text-[0.7rem] font-bold"
style={{ color: 'var(--color-plum)', borderColor: meta.color }}
>
{batchLabel(suggestion.type, batchCount)}
</button>
)}
</div>
)
})
+6 -3
View File
@@ -1,6 +1,7 @@
import { useEffect, useRef, useState } from 'react'
import { usePack } from '../../i18n'
import { useAnchoredMenu } from './anchoredMenu'
// ToneSelect lets the writer set the document's target tone, which steers the
// grammar-checkpoint LLM toward the right register (an academic essay vs a casual
@@ -35,6 +36,7 @@ export function ToneSelect({ value, onChange }: Props) {
const pk = usePack()
const [open, setOpen] = useState(false)
const ref = useRef<HTMLDivElement>(null)
const { triggerRef, style: menuStyle } = useAnchoredMenu(open, 200)
const current = TONES.find((t) => t.value === value) ?? TONES[0]
// Click outside closes the menu.
@@ -48,8 +50,9 @@ export function ToneSelect({ value, onChange }: Props) {
}, [open])
return (
<div ref={ref} className="relative shrink-0">
<div ref={ref} className="shrink-0">
<button
ref={triggerRef}
type="button"
aria-label="Document tone"
aria-haspopup="listbox"
@@ -75,9 +78,9 @@ export function ToneSelect({ value, onChange }: Props) {
{open && (
<div
role="listbox"
className="petal-word-card absolute right-0 z-30 mt-1.5 p-1.5"
className="petal-word-card p-1.5"
style={{
width: 200,
...menuStyle,
background: 'var(--color-surface)',
border: '1px solid var(--color-border)',
borderRadius: 'var(--radius-card)',
+9 -3
View File
@@ -17,11 +17,16 @@ interface Props {
// heart toggles it; `onToggleSave` removes/re-adds it.
saved: boolean
onToggleSave: () => void
// A Chinese word's pinyin. Shown in place of the IPA line and *without* the
// slashes, because pinyin is not a phonetic transcription — it is how the word
// is spelled in letters, and the slashes would say something untrue about it
// in the one place a learner is looking for the truth about pronunciation.
pinyin?: string
style: React.CSSProperties
onReplace: (synonym: string) => void
}
export function WordCard({ word, info, loading, saved, onToggleSave, style, onReplace }: Props) {
export function WordCard({ word, info, loading, saved, onToggleSave, pinyin, style, onReplace }: Props) {
const t = usePack()
const definitions = info?.definitions ?? []
const synonyms = info?.synonyms ?? []
@@ -117,9 +122,10 @@ export function WordCard({ word, info, loading, saved, onToggleSave, style, onRe
when she has found a word she likes "can I use this?". Both are
quiet, muted lines: information she can take or leave, never a verdict
on her writing. */}
{(phonetic || band) && (
{(phonetic || pinyin || band) && (
<div className="mt-1.5 flex items-center gap-2 text-sm">
{phonetic && <span style={{ color: 'var(--color-muted)' }}>/{phonetic}/</span>}
{pinyin && <span style={{ color: 'var(--color-muted)' }}>{pinyin}</span>}
{!pinyin && phonetic && <span style={{ color: 'var(--color-muted)' }}>/{phonetic}/</span>}
{band && (
<span
className="rounded-full px-2 py-0.5 text-xs font-semibold"
@@ -0,0 +1,128 @@
import { describe, expect, it } from 'vitest'
import type { Suggestion, SuggestionType } from '../../api/client'
import { batchLeaders, planBatch } from './acceptBatch'
function sug(id: string, original: string, replacement: string, type: SuggestionType = 'grammar'): Suggestion {
return {
id,
doc_id: 'd1',
from_pos: 0,
to_pos: 0,
original,
replacement,
explanation: '',
type,
status: 'pending',
created_at: '2026-07-28T00:00:00Z',
}
}
// A stand-in for findRange: first occurrence in a flat string, or null.
function finder(text: string) {
return (needle: string) => {
const i = text.indexOf(needle)
return i < 0 ? null : { from: i, to: i + needle.length }
}
}
describe('planBatch', () => {
it('applies from the end backwards, so earlier spans keep their positions', () => {
const text = 'a apple and a orange and a egg'
const plan = planBatch(
[sug('1', 'a apple', 'an apple'), sug('2', 'a orange', 'an orange'), sug('3', 'a egg', 'an egg')],
finder(text),
)
expect(plan.steps.map((s) => s.suggestion.id)).toEqual(['3', '2', '1'])
expect(plan.missing).toEqual([])
expect(plan.skipped).toEqual([])
// Applying the plan in order against a mutable string must land every
// replacement where it belongs — this is the property the ordering exists
// for, and the one a single shared transaction depends on.
let out = text
for (const step of plan.steps) {
out = out.slice(0, step.from) + step.suggestion.replacement + out.slice(step.to)
}
expect(out).toBe('an apple and an orange and an egg')
})
it('orders by position in the document, not by the order the cards arrived', () => {
const plan = planBatch(
[sug('late', 'a egg', 'an egg'), sug('early', 'a apple', 'an apple')],
finder('a apple and a egg'),
)
expect(plan.steps.map((s) => s.suggestion.id)).toEqual(['late', 'early'])
})
it('settles a card whose span is gone — she already fixed it herself', () => {
const plan = planBatch(
[sug('1', 'a apple', 'an apple'), sug('2', 'a orange', 'an orange')],
finder('an apple and a orange'),
)
expect(plan.steps.map((s) => s.suggestion.id)).toEqual(['2'])
expect(plan.missing.map((s) => s.id)).toEqual(['1'])
})
it('leaves the second of two cards quoting the same words on screen', () => {
// findRange resolves both to the first occurrence, so applying both would
// overwrite the first edit with the second. The batch takes one and says so.
const plan = planBatch(
[sug('1', 'a apple', 'an apple'), sug('2', 'a apple', 'the apple')],
finder('a apple, a apple'),
)
expect(plan.steps.map((s) => s.suggestion.id)).toEqual(['1'])
expect(plan.skipped.map((s) => s.id)).toEqual(['2'])
expect(plan.missing).toEqual([])
})
it('keeps both of two spans that merely touch', () => {
// Adjacent is not overlapping: "to" ends exactly where "the" begins.
const plan = planBatch(
[sug('1', 'aa', 'AA'), sug('2', 'bb', 'BB')],
finder('aabb'),
)
expect(plan.steps.map((s) => s.suggestion.id)).toEqual(['2', '1'])
expect(plan.skipped).toEqual([])
})
it('skips an awareness-only card with nothing to insert', () => {
const plan = planBatch([sug('v', 'a apple', ' ', 'voice')], finder('a apple'))
expect(plan.steps).toEqual([])
expect(plan.skipped.map((s) => s.id)).toEqual(['v'])
})
it('returns an empty plan for an empty family', () => {
const plan = planBatch([], finder('anything'))
expect(plan).toEqual({ steps: [], missing: [], skipped: [] })
})
})
describe('batchLeaders', () => {
const card = (id: string, type: SuggestionType) => ({ id, type })
it('gives each category exactly one leader, the first of its kind', () => {
const leaders = batchLeaders([
card('g1', 'grammar'),
card('m1', 'mechanics'),
card('g2', 'grammar'),
card('m2', 'mechanics'),
card('g3', 'grammar'),
])
expect([...leaders]).toEqual(['g1', 'm1'])
})
it('follows the stacking order it is given, not the order types appear elsewhere', () => {
// The rail sorts by anchor, so "first" means highest on the page — the card
// she reads first, which is where the batch control belongs.
const leaders = batchLeaders([card('lower', 'grammar'), card('upper', 'grammar')])
expect([...leaders]).toEqual(['lower'])
})
it('leads a lone card of its type too — the count is what hides the button', () => {
expect([...batchLeaders([card('only', 'voice')])]).toEqual(['only'])
})
it('has no leaders for an empty stack', () => {
expect(batchLeaders([]).size).toBe(0)
})
})
+84
View File
@@ -0,0 +1,84 @@
import type { Suggestion, SuggestionType } from '../../api/client'
// batchLeaders picks the one card per category that carries the "Accept all
// Grammar (5)" control, given the cards in the order they are stacked. The rail
// can't carry a category header — cards are anchored to their own sentence, so a
// type's cards are scattered down the column — and the closest honest stand-in
// is the first card of its kind. Offering the same batch on all five would be
// five buttons saying one thing.
export function batchLeaders(ordered: { type: SuggestionType; id: string }[]): Set<string> {
const leaders = new Set<string>()
const claimed = new Set<SuggestionType>()
for (const { type, id } of ordered) {
if (claimed.has(type)) continue
claimed.add(type)
leaders.add(id)
}
return leaders
}
export interface BatchStep {
suggestion: Suggestion
from: number
to: number
}
export interface BatchPlan {
// The edits to apply, ordered LAST span first. Every position is resolved
// against the document as it stands *before* any of them are applied, and
// applying from the end backwards means an earlier span's position can't be
// shifted by a later one — so the whole batch can go into a single
// transaction, which is the point: one Ctrl+Z puts it all back.
steps: BatchStep[]
// Cards whose span is no longer in the document — she fixed it herself, or
// edited around it since the check ran. There is nothing to replace, but the
// card is stale and settling it is what a single Accept already does.
missing: Suggestion[]
// Cards deliberately left on screen: two suggestions quoting the same words
// (findRange resolves both to the first occurrence, so applying the second
// would overwrite the first), and anything with nothing to insert. A batch
// that silently dropped these would report edits it never made.
skipped: Suggestion[]
}
// planBatch decides what one "Accept all <category>" click actually does. It is
// pure and takes the span resolver as an argument so the ordering and overlap
// rules can be tested without a ProseMirror document — the part that goes wrong
// is the arithmetic, not the lookup.
export function planBatch(
list: Suggestion[],
resolve: (original: string) => { from: number; to: number } | null,
): BatchPlan {
const located: BatchStep[] = []
const missing: Suggestion[] = []
const skipped: Suggestion[] = []
for (const suggestion of list) {
if (suggestion.replacement.trim() === '') {
skipped.push(suggestion) // awareness-only (voice): nothing to accept
continue
}
const range = resolve(suggestion.original)
if (!range) {
missing.push(suggestion)
continue
}
located.push({ suggestion, from: range.from, to: range.to })
}
// Document order, then greedily keep the non-overlapping ones.
located.sort((a, b) => a.from - b.from || a.to - b.to)
const steps: BatchStep[] = []
let cursor = -Infinity
for (const step of located) {
if (step.from < cursor) {
skipped.push(step.suggestion)
continue
}
steps.push(step)
cursor = step.to
}
steps.reverse()
return { steps, missing, skipped }
}
+45
View File
@@ -0,0 +1,45 @@
import { useLayoutEffect, useRef, useState, type CSSProperties } from 'react'
// A dropdown that hangs off its own button in viewport coordinates.
//
// The tone and export menus used to be absolutely positioned inside their
// button's wrapper, which was fine until that wrapper became ChromeStrip — a
// horizontal scroller, and so a box that clips what overflows it. An absolute
// menu inside it is 36px tall and scrolls away with the pills. Positioning the
// menu against the viewport instead takes it out of the strip's hands entirely:
// nothing clips a fixed box unless an ancestor has a transform, and none of the
// editor's chrome does.
//
// The trade is that a fixed box doesn't follow its anchor, so anything that
// moves the button — the page scrolling under it, the strip scrolling, the
// window resizing — has to re-place the menu.
export function useAnchoredMenu(open: boolean, width: number) {
const triggerRef = useRef<HTMLButtonElement>(null)
// Nothing to place before the first measurement; keeping it off-screen rather
// than at 0,0 means no flash in the top-left corner on open.
const [style, setStyle] = useState<CSSProperties>({ position: 'fixed', top: -9999, left: -9999 })
useLayoutEffect(() => {
if (!open) return
const place = () => {
const el = triggerRef.current
if (!el) return
const r = el.getBoundingClientRect()
// Right-aligned under the button, as it always was — but pulled back
// inside the window if that would hang the menu off a narrow screen.
const left = Math.max(8, Math.min(r.right - width, window.innerWidth - width - 8))
setStyle({ position: 'fixed', top: r.bottom + 6, left, width, zIndex: 40 })
}
place()
window.addEventListener('resize', place)
// Capture, so this hears the editor pane and the pill strip scrolling —
// neither of which bubbles a scroll event to the window.
window.addEventListener('scroll', place, true)
return () => {
window.removeEventListener('resize', place)
window.removeEventListener('scroll', place, true)
}
}, [open, width])
return { triggerRef, style }
}
@@ -0,0 +1,74 @@
import { describe, expect, it } from 'vitest'
import { splitBilingual } from './bilingualReply'
// The contract these tests defend is "never hide an answer", not "parse the
// model". Every case that isn't cleanly two halves must still come back whole.
describe('splitBilingual', () => {
it('splits the pair language from the English at the blank line', () => {
const { native, en } = splitBilingual(
'“by foots” 不是固定说法,正确的是 “on foot”。\n\n"By foots" isnt a set phrase — the idiom is "on foot".',
)
expect(native).toBe('“by foots” 不是固定说法,正确的是 “on foot”。')
expect(en).toBe('"By foots" isnt a set phrase — the idiom is "on foot".')
})
it('works the same for a Latin pair, where both halves are Latin script', () => {
const { native, en } = splitBilingual(
'Dizemos "on foot", não "by foots".\n\nWe say "on foot", not "by foots".',
)
expect(native).toBe('Dizemos "on foot", não "by foots".')
expect(en).toBe('We say "on foot", not "by foots".')
})
it('renders a half-streamed reply as the pair language until the break arrives', () => {
// Mid-stream: the English half hasn't been written yet. The partial text is
// the whole bubble, not an empty one.
expect(splitBilingual('“by foots” 不是固定')).toEqual({
native: '“by foots” 不是固定',
en: '',
})
})
it('keeps a one-language reply whole', () => {
// A model that ignores the instruction costs styling, never content.
const single = 'We say "on foot" because the idiom is fixed.'
expect(splitBilingual(single)).toEqual({ native: single, en: '' })
})
it('treats extra blank lines as part of the English half', () => {
const { native, en } = splitBilingual('中文回答。\n\nFirst English point.\n\nSecond one.')
expect(native).toBe('中文回答。')
expect(en).toBe('First English point.\n\nSecond one.')
})
it('does not split on a blank line with nothing on one side', () => {
// A leading or trailing stray newline is not a separator; styling half of
// this as a translation of nothing would be worse than not splitting.
expect(splitBilingual('\n\nWe say "on foot".')).toEqual({
native: 'We say "on foot".',
en: '',
})
expect(splitBilingual('We say "on foot".\n\n')).toEqual({
native: 'We say "on foot".',
en: '',
})
})
it('accepts a separator line that carries whitespace', () => {
// Models emit "\n \n" often enough that requiring a bare "\n\n" would drop
// the split for a reply that followed the instruction.
const { native, en } = splitBilingual('中文回答。\n \nThe English answer.')
expect(native).toBe('中文回答。')
expect(en).toBe('The English answer.')
})
it('handles an empty reply', () => {
expect(splitBilingual('')).toEqual({ native: '', en: '' })
})
it('leaves single newlines inside a half alone', () => {
const { native, en } = splitBilingual('第一行\n第二行\n\nLine one\nLine two')
expect(native).toBe('第一行\n第二行')
expect(en).toBe('Line one\nLine two')
})
})
@@ -0,0 +1,43 @@
// Splitting Petal's chat reply into the two languages it was asked for.
//
// The Ask Petal prompt (internal/llm/prompts.go) asks for the pair language
// first, then the same answer in English, separated by one blank line. This is
// the reader of that contract — and it is deliberately forgiving, because the
// reply arrives from a small local model, token by token, and a rendering rule
// must never be able to hide an answer the writer could otherwise read.
//
// So there is exactly one failure mode and it is benign: anything that doesn't
// look like two halves is returned as `native` alone, which renders as one
// ordinary block. Nothing is dropped, ever.
export interface BilingualReply {
/** The pair language — or the whole reply, when there is only one half. */
native: string
/** The English half; '' when the reply hasn't reached the blank line yet. */
en: string
}
/**
* splitBilingual divides a reply at its first blank line.
*
* Streaming is the reason this splits at the *first* blank line rather than
* validating the shape: while tokens arrive the text is a native half with no
* separator yet, so it renders as the pair language and the English simply
* appears beneath it when the blank line lands. Any further blank lines stay
* inside the English half rather than starting a third section with nowhere to
* go.
*/
export function splitBilingual(content: string): BilingualReply {
const match = /\n[ \t]*\n/.exec(content)
if (!match) return { native: content, en: '' }
const native = content.slice(0, match.index).trim()
const en = content.slice(match.index + match[0].length).trim()
// A blank line with nothing on one side of it isn't two halves — it's a
// stray newline. Keep the reply whole rather than styling half of it as a
// translation of nothing.
if (native === '' || en === '') return { native: content.trim(), en: '' }
return { native, en }
}
+126
View File
@@ -0,0 +1,126 @@
import type { Node as PMNode } from '@tiptap/pm/model'
import { mapOffset } from './SuggestionHighlight'
import type { Segmenter } from '../../lib/segment'
import type { HanziInfo, WordInfo } from '../../api/client'
// The Chinese counterpart of `wordAt` (SpellCheck.ts): given a position in the
// document, which Chinese word is there.
//
// It lives in its own file rather than as a branch inside `wordAt` because the
// two answer the same question by genuinely different means — one runs a regex
// over the text, the other runs a shortest-path walk over a 188k-word list it
// had to fetch — and only one of them is about spelling at all. What they share
// is the part that matters for correctness: the offset→position mapping, which
// is `mapOffset`, the same function the suggestion, spell and search decoration
// layers all anchor through.
export interface HanziRange {
from: number
to: number
word: string
}
// blockAt finds the textblock containing pos, along with where that block starts
// — everything else here is arithmetic within one block.
//
// Segmentation is per-block for the same reason the spell tokenizer is: a word
// cannot span a paragraph break, and a block is the largest unit whose text is
// contiguous in the document.
function blockAt(doc: PMNode, pos: number): { node: PMNode; start: number } | null {
let found: { node: PMNode; start: number } | null = null
doc.descendants((node, nodePos) => {
if (found) return false
if (!node.isTextblock) return true
if (pos <= nodePos || pos >= nodePos + node.nodeSize) return false
found = { node, start: nodePos }
return false
})
return found
}
// offsetOf is the inverse of mapOffset: an absolute ProseMirror position to a
// character offset within the block's flattened text. Inline atoms (a hard
// break) occupy a position and contribute no text, so the two are not the same
// number and subtracting the block position would be wrong in any paragraph
// containing one.
function offsetOf(block: PMNode, blockStart: number, pos: number): number {
let textOffset = 0
let pmPos = blockStart + 1
let result = -1
block.forEach((child) => {
if (result >= 0) return
const len = child.isText ? (child.text?.length ?? 0) : 0
if (pos <= pmPos + child.nodeSize) {
result = textOffset + Math.max(0, Math.min(pos - pmPos, len))
return
}
textOffset += len
pmPos += child.nodeSize
})
return result >= 0 ? result : textOffset
}
// hanziWordAt resolves the Chinese word at a document position, or null when
// there is no Chinese there — which is the ordinary case in a mixed paragraph
// and is why the caller falls through to the Latin tokenizer.
export function hanziWordAt(doc: PMNode, pos: number, segmenter: Segmenter | null): HanziRange | null {
if (!segmenter) return null
const block = blockAt(doc, pos)
if (!block) return null
const text = block.node.textContent
if (!text) return null
const token = segmenter.wordAt(text, offsetOf(block.node, block.start, pos))
if (!token) return null
return {
from: mapOffset(block.node, block.start, token.from),
to: mapOffset(block.node, block.start, token.to),
word: token.word,
}
}
// hanziToWordInfo adapts a Chinese lookup into the shape the word card already
// renders.
//
// An adapter rather than a second card, because everything around the card is
// the same in both directions: it opens the same way, anchors the same way,
// captures into the same vocabulary garden, and reads aloud through the same
// voice — the zh pair already speaks Chinese, so 🔊 needs nothing new to say
// 公园 out loud. What differs is only which fields carry what.
//
// * `definitions` holds one entry per reading, labelled with its pinyin. The
// part-of-speech slot is where the card puts a short italic prefix, which
// is exactly the shape a reading label wants — and a reading *is* the thing
// that distinguishes these senses from each other (得 dé "to obtain" from 得
// de, the complement marker).
// * `gloss` stays empty. It means "translated into the writer's language",
// and for a writer learning Chinese that language is English, which is what
// the senses already are. Putting the English there too would print it
// twice.
// * The per-character fallback fills the same list, labelled by character, so
// a compound with no headword still says something true about itself.
export function hanziToWordInfo(info: HanziInfo): WordInfo {
const definitions =
info.readings.length > 0
? info.readings.map((r) => ({ part_of_speech: r.pinyin, definition: r.senses }))
: info.chars.map((c) => ({ part_of_speech: `${c.char} ${c.pinyin}`, definition: c.senses }))
return {
word: info.word,
gloss: '',
phonetic: '',
definitions,
synonyms: [],
frequency: 0,
difficulty: -1,
etymology: '',
}
}
// hanziPinyin is the word's own pronunciation, for the line under the headword.
// Empty when only the character fallback answered: the characters' readings are
// not the word's reading — 不 is bù alone and bú before a fourth tone — and
// printing them joined up would be inventing a pronunciation.
export function hanziPinyin(info: HanziInfo): string {
return info.readings[0]?.pinyin ?? ''
}
+42
View File
@@ -0,0 +1,42 @@
import { describe, expect, it } from 'vitest'
import { RAIL_MIN_MARGIN, railFitsBeside } from './railFit'
// The numbers below are not invented: they were measured in Chrome at the UX
// review's own 1517x810 viewport, with the 280px sidebar open and closed. They
// are here so the two layouts stay distinguishable if the constant is ever tuned.
describe('railFitsBeside', () => {
it('fits in distraction-free mode, where the pane spans the window', () => {
// 1517px window, sidebar collapsed: the 720px column centres at left 391,
// so its right edge is 1111 and 406px of margin remain.
expect(railFitsBeside(1517, 1111)).toBe(true)
})
it('does not fit with the document list open at the same window size', () => {
// Same window, 280px sidebar in flow: the column re-centres to right 1259 and
// the margin falls to 258 — the measurement item 5 reported. This is the case
// that must return false; rendering the rail here overhangs the viewport.
expect(railFitsBeside(1517, 1259)).toBe(false)
})
it('rejects the mid-animation width too, not just the settled one', () => {
// The sidebar animates over 280ms, so the recompute can land on an
// intermediate margin (266 was observed one frame in). Anything under the
// threshold has to read as "no rail", or the column flickers back in.
expect(railFitsBeside(1517, 1251)).toBe(false)
})
it('treats the threshold as inclusive', () => {
expect(railFitsBeside(1000, 1000 - RAIL_MIN_MARGIN)).toBe(true)
expect(railFitsBeside(1000, 1000 - RAIL_MIN_MARGIN + 1)).toBe(false)
})
it('has room to spare on a wide desktop', () => {
// 1920px maximised, sidebar open: margin 468.
expect(railFitsBeside(1920, 1452)).toBe(true)
})
it('never fits on a narrow window, whatever the column does', () => {
expect(railFitsBeside(900, 810)).toBe(false)
expect(railFitsBeside(768, 744)).toBe(false)
})
})
+19
View File
@@ -0,0 +1,19 @@
// Whether the margin rail has room to sit beside the editor.
//
// The rail is a 300px column with a 32px gutter (see `.petal-rail` in index.css);
// RAIL_MIN_MARGIN adds a little breathing space to the viewport edge. Below it the
// editor falls back to the inline card anchored under the word.
//
// This is a bare comparison, but it earns a name: the number decides which of two
// entirely different suggestion surfaces she gets, and the margin it measures moves
// for reasons that have nothing to do with the window size. The editor is a fixed
// 720px column centred in the pane, so collapsing the 280px sidebar (distraction-free
// mode) re-centres it and changes this margin by 140px without resizing anything.
// See the ResizeObserver in EditorCore for the other half of that story.
export const RAIL_MIN_MARGIN = 348
// `wrapperRight` and `innerWidth` are both viewport coordinates — i.e. exactly
// `wrapper.getBoundingClientRect().right` and `window.innerWidth`.
export function railFitsBeside(innerWidth: number, wrapperRight: number): boolean {
return innerWidth - wrapperRight >= RAIL_MIN_MARGIN
}
@@ -0,0 +1,54 @@
import { describe, expect, it } from 'vitest'
import type { SuggestionType } from '../../api/client'
import { fr } from '../../i18n/packs/fr'
import { ptPT } from '../../i18n/packs/pt-PT'
import { zh } from '../../i18n/packs/zh'
import { TYPE_META, typeLabel } from './suggestionMeta'
// Every type the API can send. Listed by hand rather than derived, so adding a
// suggestion type to the union without giving it a colour and a name fails here
// instead of rendering a card with `undefined` on its pill.
const ALL: SuggestionType[] = [
'grammar',
'phrasing',
'idiom',
'clarity',
'translate',
'voice',
'collocation',
'mechanics',
]
describe('TYPE_META', () => {
it('covers every suggestion type with a colour and a label', () => {
for (const type of ALL) {
expect(TYPE_META[type], type).toBeDefined()
expect(TYPE_META[type].color, `${type} colour`).toMatch(/^var\(--color-/)
expect(TYPE_META[type].label, `${type} label`).not.toBe('')
}
})
it('gives translate its own colour', () => {
const used = ALL.filter((t) => t !== 'translate').map((t) => TYPE_META[t].color)
expect(used).not.toContain(TYPE_META.translate.color)
})
})
describe('typeLabel', () => {
// The one bilingual pill, and it has to speak the writer's own pair — a
// hardcoded 翻译 would be simply wrong on the screen of a Portuguese writer.
it('names a translation in the writers own language', () => {
expect(typeLabel('translate', zh)).toBe('翻译 · Translate')
expect(typeLabel('translate', ptPT)).toBe('Tradução · Translate')
expect(typeLabel('translate', fr)).toBe('Traduction · Translate')
})
it('leaves every other type in English, whatever the pair', () => {
for (const p of [zh, ptPT, fr]) {
for (const type of ALL.filter((t) => t !== 'translate')) {
expect(typeLabel(type, p), `${type} on ${p.code}`).toBe(TYPE_META[type].label)
}
}
})
})
@@ -1,4 +1,5 @@
import type { SuggestionType } from '../../api/client'
import type { Pack } from '../../i18n'
// Per-type accent color + human label, mirroring the design tokens. Shared by the
// inline hover SuggestionCard and the margin SuggestionRail so a given suggestion
@@ -8,7 +9,32 @@ export const TYPE_META: Record<SuggestionType, { color: string; label: string }>
phrasing: { color: 'var(--color-peach)', label: 'Phrasing' },
idiom: { color: 'var(--color-lavender)', label: 'Idiom' },
clarity: { color: 'var(--color-sky)', label: 'Clarity' },
// The label here is only the fallback (and what a screen reader gets if the
// pack hasn't arrived yet) — see typeLabel.
translate: { color: 'var(--color-jade)', label: 'Translate' },
voice: { color: 'var(--color-honey)', label: 'Voice' },
collocation: { color: 'var(--color-blossom)', label: 'Word pairing' },
mechanics: { color: 'var(--color-sage)', label: 'Tidy-up' },
}
// typeLabel is what the pill says. It exists for exactly one type: a translation
// card names itself in the writer's own language first, because that language is
// its entire subject. Every other type keeps its English name — those are the
// terms she is learning, and she is learning them in English.
//
// Taking the pack rather than reading the module singleton keeps the label
// reactive: the pair language isn't known until /api/me answers, and a card
// rendered before then must relabel itself when it does.
export function typeLabel(type: SuggestionType, pack: Pack): string {
return type === 'translate' ? pack.editor.translateLabel : TYPE_META[type].label
}
// The label on the accept-a-whole-category button. It stays English even on a
// translation card, whose pill is bilingual: the pill names the kind of advice
// she is reading, while this names an action taken over the rest of the queue,
// and every other word in the action row — Accept, Dismiss, Ask Petal — is
// English too. A control that switched languages between cards would read as a
// different control.
export function batchLabel(type: SuggestionType, count: number): string {
return `Accept all ${TYPE_META[type].label} (${count})`
}
+6 -3
View File
@@ -1,6 +1,7 @@
import { useEffect, useRef, useState } from 'react'
import { api, type ExportFormat } from '../../api/client'
import { usePack } from '../../i18n'
import { useAnchoredMenu } from '../Editor/anchoredMenu'
// ExportMenu is the "get your writing out of Petal" dropdown. File formats are
// plain <a download> links to the server's export endpoint (which sets the
@@ -28,6 +29,7 @@ export function ExportMenu({ docId }: Props) {
const t = usePack()
const [open, setOpen] = useState(false)
const ref = useRef<HTMLDivElement>(null)
const { triggerRef, style: menuStyle } = useAnchoredMenu(open, 220)
useEffect(() => {
if (!open) return
@@ -39,8 +41,9 @@ export function ExportMenu({ docId }: Props) {
}, [open])
return (
<div ref={ref} className="relative shrink-0">
<div ref={ref} className="shrink-0">
<button
ref={triggerRef}
type="button"
aria-label="Export document"
aria-haspopup="menu"
@@ -63,9 +66,9 @@ export function ExportMenu({ docId }: Props) {
{open && (
<div
role="menu"
className="petal-word-card absolute right-0 z-30 mt-1.5 p-1.5"
className="petal-word-card p-1.5"
style={{
width: 220,
...menuStyle,
background: 'var(--color-surface)',
border: '1px solid var(--color-border)',
borderRadius: 'var(--radius-card)',
+30 -1
View File
@@ -19,6 +19,9 @@ interface Props {
// True when Petal can't reach its LLM helper — shows a gentle, reassuring note
// (the writing still saves locally, so this is awareness, not an error).
llmDown: boolean
// How many suggestions the rail is holding. Shown as a soft bilingual line
// beside the word count; hidden at zero.
suggestionCount: number
}
// Save-state labels. English except for the lapsed-session case, which is the
@@ -48,9 +51,23 @@ interface Indicator {
label: string
}
export function StatusBar({ wordCount, text, saveStatus, checking, voicing, collocating, llmDown }: Props) {
export function StatusBar({
wordCount,
text,
saveStatus,
checking,
voicing,
collocating,
llmDown,
suggestionCount,
}: Props) {
const t = usePack()
const label = saveLabel(saveStatus, t)
// The gentle counterpart of Grammarly's score: how much is waiting, never how
// well she wrote. Nothing waiting says itself — an empty rail — so a zero here
// would only be a verdict delivered after every check, which is the pressure
// this app's non-goals rule out.
const petals = suggestionCount > 0 ? t.status.petalsToPolish(suggestionCount) : null
const indicators: Indicator[] = [
{
@@ -109,6 +126,18 @@ export function StatusBar({ wordCount, text, saveStatus, checking, voicing, coll
</button>
{statsOpen && <StatsPanel text={text} wordCount={wordCount} />}
</div>
{petals && (
<>
<span aria-hidden>·</span>
{/* Native half first, as everywhere else in Petal. (The review wrote it
English-first; the pack's order is the one she reads all day.) */}
<span className="inline-flex items-center gap-1.5">
<span aria-hidden>🌸</span>
<span>{petals.native}</span>
<span style={{ opacity: 0.75 }}>· {petals.en}</span>
</span>
</>
)}
{indicators
.filter((i) => i.active)
.map((i) => (

Some files were not shown because too many files have changed in this diff Show More