drwily/petal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: drwily/petal:49e84278d579b7dabf5ae302c2e0f31fc812496e
Branches Tags
drwily/petal:main drwily/petal:feat/zh-learner-direction
..
compare: drwily/petal:9a0edd66796496580f43594d4914f8dbf90b6543
Branches Tags
drwily/petal:main drwily/petal:feat/zh-learner-direction
40 Commits
49e84278d5 .. 9a0edd6679
Author SHA1 Message Date
prosolis 9a0edd6679 Merge feat/deploy-phase15: containerized VPS deploy, auth, multi-language pairs 
Brings Phases 15-23 onto main: Docker/Traefik deploy at petal.parodia.dev,
OIDC sign-in replacing the basic-auth gate, per-user settings, a dictionary
that is no longer English+Chinese only, English+Portuguese and English+French
pairs with their Piper voices, and encrypted nightly backups on millenia.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 17:16:19 -07:00
prosolis be1ab5cef7 Her apostrophe was cutting French words in half 
Typography.ts rewrites every ' typed in the editor into a curly ’, but both
word regexes only counted the straight one. So "aujourd’hui" reached the
dictionary as "aujourd" + "hui", neither of them a French word, and one of the
commonest words in the language came back wearing two red underlines. Same for
quelqu’un, presqu’île, prud’homme. l’arbre only survived by accident, because
"l" happens to be a bare entry. withElision, written for exactly this, could
only ever fire on pasted text.

Both marks are word characters now, and combine() straightens on lookup — the
one place every lookup passes through — since the shipped word lists spell
theirs straight. Suggestions come back wearing whichever mark she actually
used, so accepting a pill never swaps her apostrophe.

œ was untokenizable too: U+0152/U+0153 sit outside the Latin-1 ranges, so
"cœur" split into "c" + "ur" and the orphan was long enough to underline. 586 œ
forms ship in fr.dic.gz and not one of them was reachable.

In the dictionary builder, the two cross-product paths added their forms
without the NEEDAFFIX check the single-affix paths apply, so a doubly-affixed
form that is still "not a word on its own" was accepted anyway — the exact
class of error the FLAG-aware rewrite exists to close. PFX and SFX are also
separate flag namespaces, and one shared `cross` dict let the second block
overwrite the first. Odd-length long-flag strings now stop the build instead of
dropping a character and expanding through the wrong paradigm.

The pt-PT and Québécois greps were case-sensitive against sentence-cased copy,
which let a leading "Actualmente…" through the guard added to catch it.

Note: this changes what the expander produces, but fr.dic.gz and pt-PT.dic.gz
are vendored and were built with the old behaviour. Both want regenerating on a
box that can fetch the upstream .deb, and BUILD_PLAN Phase 24's "pt-PT rebuild
is byte-identical" claim re-checked — if those bytes move, the NEEDAFFIX gap
was live in the Portuguese list too.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 17:07:30 -07:00
prosolis 071ea7b835 Petal learns French, and the pack that shipped was misspelling itself 
Phase 24, the fr half: langpack, Hunspell dictionary, Piper voice, and the
lexicon coverage that turned out to have been measured already (63.1%, better
than pt-PT's 62.1%). No migration; not deployed.

The plan recorded that build_ptpt_dictionary.py "generalizes" to French. It
did not. It handled single-character flags and plain PFX/SFX and stopped on
everything else, and fr.aff uses four of the things it stopped on. FLAG long
is the dangerous one: French flags are two characters, so the old reader's
set(flagstr) yields a bag of unrelated letters and expands every entry through
the wrong paradigm without ever erroring. Plus continuation flags (French
really does affix an affixed form), NEEDAFFIX on 68,075 of 84,140 stems, and
FULLSTRIP. Renamed build_hunspell_dictionary.py with a per-language profile,
asserting that CIRCUMFIX and FORBIDDENWORD are still unused rather than
assuming it — and it rebuilds pt-PT byte-identical to the shipped asset, which
is the only thing that makes "generalized" a claim rather than a hope.

Elision was decided by building both halves and measuring. Keeping l'arbre and
its thirty-three siblings: 3,159,832 forms, 8.25 MB gzipped. Dropping them:
473,326 and 1.19 MB. They are not new words, but the tokenizer keeps internal
apostrophes, so they genuinely would have been underlined — so they moved out
of the dictionary into withElision, which splits at a known clitic and still
requires the remainder to be a word (l'zzzz stays flagged). Real nspell: 369 ms
and 74 MB, against pt-PT's 842 ms and 139 MB, on the larger language.

Where the regional trap lives is the mirror image of Portuguese's: every fr_*
Piper voice is fr_FR and Debian's fr_FR/fr_CA/fr_BE dictionaries are one shared
word list, so nothing can be quietly wrong about the country and the whole
decision sits in the copy. What French has instead is the 1990 reform, packaged
three ways; comprehensive ships, because Petal never corrects her French and
coût and cout are both correct.

Then the interim review pass, at the user's suggestion and explicitly "for
now": four models read each Latin pack independently, and only findings at
least two of them reached on their own were applied — five per pack. It earned
its keep on the pack that was already live. pt-PT was carrying pre-Acordo
spellings (adjectivos, actualmente) in a file whose own header commits to
post-Acordo, plus Brazilian decepção, because the Phase 21 greps checked for
Brazilian vocabulary and never checked the pack against its own spelling
policy. That grep now exists and was confirmed to fail on the old text before
being kept. Where reviewers agreed a line was wrong but split on the fix, the
wording is mine and the reasoning is in BUILD_PLAN rather than averaged away.

Still owed, and both packs now say so precisely: a quorum of models agreeing is
agreement, not authority. No native speaker has read either pack, and none of
this has been seen in a browser.

go build/vet/test clean, tsc, vite build, vitest 190/190.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 16:19:26 -07:00
prosolis 9a2e909b85 Phases 22 and 23 are live: what the deploy actually found 
Two phases went up together because the tree could not separate them —
the switcher's copy keys sit in the same files as Phase 22's, and the user
chose to ship both rather than untangle them.

The migration backfill is the part worth recording: `source` came out
llm=100 / local=3, and the three are her existing mechanics rows claimed
by type, which is exactly what 0013 said it would do. 9 documents, 33
versions, 103 suggestions, integrity ok.

All three accounts are still on zh on purpose. Changing her pair from a
shell is the thing this phase was built to stop needing.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 15:09:49 -07:00
prosolis 1f4ca4775a Let her choose her own pair 
Raised by the user, not by the plan: there was no way to change language
in the mobile UI. There was no way anywhere. `users.pair_lang` has been
readable since Phase 19 and writable by nobody — /api/me was GET-only and
Upsert deliberately skips the column — which is also why "no pt-PT account
exists yet" has stood through two phases. Nothing could create one.

PATCH /api/me answers with the whole user rather than 204, so the client
re-reads the pair from the server instead of trusting its own request. One
write reaches everything: langpack, Hunspell dictionary, Piper voice,
lexicon provider and prompt language all read the column at use time.

The server refuses a pair it has no copy for, and auth.shippedPairs is
deliberately not internal/llm's list. That one names pairs the prompts can
talk about (fr and es, since Phase 19); this one names pairs Petal can
render itself in, which needs a langpack. Storing fr today would strand
her on Chinese with no way back except a lucky guess at a button she
cannot read.

The picker sits in the sidebar footer because the sidebar is the mobile
drawer — always one tap away. The status bar exists only while a document
is open, which is the wrong moment to find the app speaking a language you
can't read. Each language names itself, 中文 and Português: the one place
bilingual copy would get in the way.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 15:06:33 -07:00
prosolis 1bbc8fc8d3 Finish Phase 22: the half of Petal that works with the tunnel down 
Grammar lite, the false-friend list, the daily invitation and the offline
miscollocations — the four remaining §5–§6 items, all client-side and all
alive on a box that cannot reach the model.

The offline collocations forced a schema change. `type` had been doubling
as the answer to "which engine found this" — `mechanics` meant offline —
and that stops being true the moment an offline rule proposes a
collocation. Migration 0013 adds `source` (llm | local) and every pass now
scopes its DELETE by engine; without it the coach silently wiped every
offline chunk on the page. Existing rows backfill by type, so a pre-0013
collocation row is claimed as the coach's, which it was: the offline list
did not exist yet.

The rule pack is hand-curated rather than mined, and the entries left out
are the point — `married with` is wrong until "married with children",
`arrive to` wants at or in depending on the noun. A pack running on every
keystroke must not correct correct writing.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 15:05:55 -07:00
prosolis e9b8595456 Let the garden keep what she was given, not only what she sought 
Two halves of the same idea, both read out of work Petal already
records.

Planting: an accepted collocation is a learnable chunk, so it becomes a
phrase card. The scheduler didn't need to know — a three-word chunk
climbs the ladder exactly like a looked-up word. What needed care was
deciding what *isn't* a chunk (single words are word choice; a
six-word-plus "collocation" is a rewritten sentence, and sentences make
miserable flashcards), and that the example must be the *corrected*
sentence — the stored draft still holds the phrasing she just left
behind. Re-accepting the same chunk leaves the existing card alone
rather than resetting a schedule it has been climbing. The whole thing
is best-effort: accepting an edit must never fail because a flashcard
couldn't be made.

The growth journal: kept this month beside kept the month before, the
phrasing that stuck, the patterns that faded. The queries were the easy
part; the honesty is the feature. "Stuck" needs the phrase in a *second*
document, because one document is just the edit where she left it.
"Faded" says nothing at all unless she has been writing lately —
otherwise a month away from Petal comes back to her as progress, which
is the one way this could lie. And a suggestion had to start recording
when she *decided* it, not when the model proposed it, so 0012 adds
resolved_at and backfills the old rows to their created_at.

It lives as a second tab in the garden, and it feeds the kitten: after
an accept she now sometimes hears something true of her alone, once per
line, half the time, never waited for.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 14:16:59 -07:00
prosolis 7b845644be Phase 21 is live: what the pt-PT voice cost to deploy 
The plan's last open line was "Piper pt-PT voice instance on parodia",
and it hid two things.

A language was still a code change — read-aloud knew exactly two,
named in the Config struct. It discovers them from the environment now,
so fr and es cost a compose service and two .env lines.

And the voice repeated the phase's own lesson in a new place:
pt_PT-tugão-medium is the only European Portuguese voice Piper ships,
and its own downloader cannot fetch it — the ã goes into the HTTP
request line and http.client encodes ASCII.

Deploying is also what finally ran the reverse lookup against the real
550 MB dict.db, which the previous session left open because this laptop
has none: data → "date", comum → "common; usual", tarde → "evening",
with think, computer and garden correctly silent. think glossing to
pensar first confirms Phase 20's sense-agreement ordering on real data
rather than on a fixture.

Still owed, both said plainly in the phase: no pt-PT speaker has read the
pack, and no pt-PT account exists — both writers are on the zh pair, so
nothing she sees changed today.
 2026-07-27 13:32:15 -07:00
prosolis 3b714e297a Fetch the Portuguese voice Piper's own downloader can't 
piper.download_voices pastes the voice name straight into the HTTP request
line, and http.client encodes that as ASCII — so it dies with
UnicodeEncodeError on the ã in pt_PT-tugão-medium before a byte leaves the
container. Every pt_BR voice downloads fine. The failure lands precisely on
the one voice the pt-PT pair needs, and it is the *only* European voice in
the catalogue.

The entrypoint now falls back to fetching the model and its config itself
with the path percent-encoded, which is all the downloader was missing.
Same host, same files, same destination names, so the cached-voice check
and the server invocation are unchanged.
 2026-07-27 13:25:28 -07:00
prosolis 24c3533e18 Give read-aloud a Portuguese voice, and a slower one 
Phase 21's infra half. Two things the pt-PT pair needs from TTS, and one
thing every learner has wanted since Phase 11.

**A language is no longer a code change.** The handler knew exactly two
languages, named in the Config struct: English on TTS_ENDPOINT and Chinese
on TTS_ENDPOINT_ZH. Petal now discovers its Piper instances from the
environment — English keeps the unsuffixed pair it has always had, and
every other language is a TTS_ENDPOINT_<LANG>/TTS_VOICE_<LANG> pair — so
fr and es cost a compose service and two lines of .env. <LANG> is the base
tag, because an environment variable name cannot hold pt-PT's hyphen and
only one Portuguese model is loaded either way. A language configured by
halves is dropped rather than routed: half a configuration should reach
the client as "no voice here, use Web Speech", not as an instance that
errors on every tap. The startup line now names the voices it actually
resolved rather than the English endpoint it was handed — the same lesson
the dictionary line learned last week.

**pt_PT-tugão-medium is the only European voice Piper ships.** The other
five pt models in the catalogue are Brazilian, so the default anyone
reaches for is the wrong country — the same trap as `dictionary-pt`
packaging VERO, arriving through the catalogue rather than through the
model. Named explicitly in compose, with the query that checks it in the
deploy README.

**The slow replay** (SUGGESTIONS §5e) is `slow: true` on /api/tts, raising
Piper's length_scale to ~4/3. Piper stretches durations rather than
resampling, so it stays a voice instead of a groan. The pace is part of
the cache key — without it the slow replay of a word already heard at
normal speed would be served back at normal speed, which is the one
request where the difference is the whole point. 🐢 sits beside 🔊 on the
word card, the selection bubble and the garden flashcard; the Web Speech
fallback slows too, so the button means the same thing when Piper is down.

**And the other reading gets her own voice.** The `alsoIn` block — the
Portuguese sense of a word that is also English — now speaks in the pair's
locale, which the pack names (`locale`) rather than anything inferring it
from the letters. "comum" is spelled identically in both halves; a
detector would have to guess, and this is the same reason the gloss shows
both directions instead of picking one.

Tests: config discovery (both existing deployment shapes, half-configured
languages dropped, the pre-map voice defaults preserved), the slow scale
and its separate cache entry, pt routing on the base tag with pt-BR
landing on the European instance, and speech.ts's request body. The i18n
shape suite now asserts every pack names a speakable locale in its own
language — and that pt-PT's is not pt-BR.

Verified: go build/vet/test, tsc, vitest 125/125, vite build. Live smoke
against two fake Piper servers: en/pt × normal/slow all reached the right
instance at the right length_scale with four distinct cache entries, and
an unconfigured language still 404s.
 2026-07-27 13:21:45 -07:00
prosolis ccb43e5a4d Phase 21: Petal learns to be an English+Portuguese pair 
The plan said "Hunspell pt-PT vendored like en-US". Measuring that first is
what saved it: nspell expands affixes eagerly on construction, and European
Portuguese's 1,340 rules over 44,257 stems want over a gigabyte of browser
heap — ~340 MB for the first 12,000 entries, and no return at all after three
minutes on the whole file. So the expansion runs once at build time instead:
1,039,058 forms, 2.66 MB gzipped, read by the same nspell in 842 ms.

The obvious npm package would also have shipped the wrong language. Both
dictionary-pt and dictionary-pt-br carry VERO, the Brazilian word list, so
vendoring by name puts pt-BR spellings behind a pt-PT label — the drift
SUGGESTIONS §3 warns about, arriving through the packaging where no reviewer
can see it. The source is Projecto Natura's, and the build script now asserts
the fault lines (receção in, recepção out) before writing anything.

Spellcheck consults both dictionaries and flags only what both reject, which
is the no-detector answer to a pair with no script boundary. The word card
does the same in the other direction: "data" is a word in both languages, so
Petal shows both readings rather than guessing which she meant.

Writing the tests caught the one real bug — extendedAlphabet was a snapshot
while correct/suggest read live, and her dictionary arrives after English, so
every lookup would have resolved "cora" while the underlines were already
right.

Not done, and not claimed: the pack has not been read by a pt-PT speaker, and
the Piper voice is deferred with the deploy.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 12:43:02 -07:00
prosolis 4de83d0da5 Record the Spanish rebuild and how to do the next one 
dict.db now holds all five languages on the VPS. The deploy notes carry the
recipe, the two traps (SUBTLEX-US fails harmlessly; check the other languages'
counts are unchanged before shipping) and the measured coverage, so the next
rebuild is not archaeology.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 10:53:22 -07:00
prosolis 74bf600593 Make the dictionary startup line report rows, not capabilities 
It logged dictionary.Langs(), which is a compile-time constant of the languages
DreamDict *supports*. The database deployed until today supported Spanish and
contained none of it, so the line printed a confident "[en fr pt-PT es zh]"
over a file where every Spanish lookup came back empty — the exact failure the
line exists to catch, reported as success.

Contents() counts rows per language instead. For a file somebody has to copy
onto the box by hand, "what is in it" is the only question worth asking, and
the answer is now en=136615 es=102971 fr=56096 pt-PT=136300 zh=120883.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 10:51:09 -07:00
prosolis 86175f1559 Phases 19 and 20 are live: record what the deploy found 
Both are migration-free, so this was a rebuild: backup first, dict.db into the
LUKS volume and SHA-256 verified end to end, then pull and build. Startup logs
the languages it actually opened rather than the ones it hoped for, which is
the check worth having — the deployed binary found the deployed file with all
five in it.

Nothing she sees changed today. Both accounts are on the zh pair, the langpack
reads verbatim, and the Chinese gloss still comes from ECDICT. What shipped is
the capacity for the next pair.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 10:08:12 -07:00
prosolis 3640ce9324 Depend on dreamdict as a published module, not a local path 
The dev `replace ../dreamdict` was only ever a bridge for building against
uncommitted upstream work. It cannot survive the Docker build, whose context is
this repo alone, so it comes out now that the two dreamdict commits are pushed.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 10:02:17 -07:00
prosolis 97e9c269ec Phase 20: the dictionary stops being English and Chinese only 
Word lookups now come from DreamDict's dict.db for every pair but Chinese —
opened read-only beside petal.db, no service, nothing over the VPN, because a
hover gloss has to answer in milliseconds.

`Provider` is the two questions the popover and the tooltip already asked, so
the embedded *Lexicon satisfies it with no changes at all; Set.For(lang) is the
single place the choice between them is made. The prerequisite in the dreamdict
repo turned out to be two things, not one: the module path was unfetchable
*and* the query layer sat in internal/, which no other module may import
whatever the module is called. Both fixed upstream.

The plan's central assumption did not survive the data. It mapped
Gloss ← Translate(word, "en", L1) one-to-one; against the real 452 MB database
that table answers for 17% of the 2,000 commonest English words into pt-PT.
Wiktionary's translation sections are thin in that direction — "ephemeral",
"think" and "quickly" have no en→pt-PT row at all. Shared WordNet synsets
answer for 61%, so DreamDict gained Equivalents() and Petal glosses through it.
Ordering those was wrong in an instructive way too: sorting by frequency
glosses "think" as lembrar, "remember", because lembrar is the commoner
Portuguese word even though pensar shares six of think's synsets to lembrar's
one. Counting sense agreement first asks the right question.

The same measurement is why zh stays on ECDICT: DreamDict reaches a Chinese
gloss for 53% of those words, ECDICT for nearly all of them. The plan said
converge only if quality holds. It didn't, so nothing converged.

Two decisions about failure worth keeping. A missing dict.db is not an error —
a laptop checkout has never had one — but a present-and-never-imported one is,
because that is a half-finished deploy. And a pt-PT writer with no dictionary
falls back to the embedded datasets with the gloss suppressed, keeping
definitions, synonyms and phonetics rather than blanking the popover: an empty
field reads as "not found", the wrong language reads as broken.

The new fields surface as an etymology line and a three-band chip. Three, not
five: the difficulty score separates "everyday" from "you'll have to explain
this" but cannot rank obfuscate against serendipity, and a finer scale would be
a confident-looking lie. An unscored word gets no chip.

Writing the tests found two bugs first — trimEtymology sliced by byte, which
would have emitted invalid UTF-8 for exactly the Greek and Latin etymologies
the feature exists for, and its ellipsis path overran its own cap.

go build/vet/test, tsc, vite, vitest 96/96 clean; live smoke against the real
dict.db with one instance flipped from zh to pt-PT mid-run.

Not deployed: go.mod still replaces github.com/prosolis/dreamdict with
../dreamdict, so the Docker build needs the two upstream commits pushed and the
replace dropped. The deployed dict.db also predates DreamDict's Spanish data.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 09:38:50 -07:00
prosolis 336cae93e0 Phase 19: the copy stops being hardcoded Mandarin 
Every `中文 · English` string moves out of ~29 components into
web/src/i18n: one Pack type, a verbatim zh pack, and two ways to read
it — usePack() for components, pack() for the modules that build a line
when something happens rather than when something renders.

Anything with a value in it is a function on the pack rather than a
template at the call site, English pluralisation included: word order
isn't universal, and a pack author has to be able to move the number.
The roster constants (tones, rewrite styles, export formats, companions)
keep only value + emoji, so a label can't drift from its key.

On the server, internal/llm/lang.go replaces "Simplified Chinese" in the
three prompts that actually name her language. pt-PT is spelled
"European Portuguese (pt-PT, never Brazilian Portuguese)" in the prompt
itself, and each Lang carries her word for "why" so the tutor prompt
still recognises the question when she asks it her way.

pair_lang reaches the model through the row-scoped query each handler
already ran — the one that proves she owns the document — rather than a
second lookup that could disagree with it.

Also records Phase 18's deploy: migration 0011 rehearsed against a copy
of the live VPS database, then applied for real.
 2026-07-27 08:37:05 -07:00
prosolis 30d5e691c9 Phase 18: settings that belong to the writer, not the browser 
The mute toggle, the falling-petals toggle and the chosen companion lived in
localStorage, which is a property of the machine. Now that two people can sign
in to one Petal, sharing a laptop would have meant sharing a mascot and one
person's silence muting the other. Each key is namespaced by user id.

The awkward part is timing: sounds.ts and petals.ts read their value the moment
they are imported, long before /api/me can have answered. Rather than block
startup on the network for a mute flag, a read before the answer arrives sees
the old un-namespaced key -- on a single-writer browser, exactly the right
value -- and setPrefsScope then adopts it into that account's namespace and
tells every reader to look again. Adoption moves rather than copies, so the
first account inherits what was set before accounts existed and the second
starts from Petal's defaults.

The personal spelling dictionary moves further than that: onto the server. It
is built from her own writing, so it should not be readable by whoever sits
down at the same browser next -- but merely namespacing it would have split the
list she already has between her laptop and her tablet, which is worse than
where we started. A table keyed (user_id, lang, word) follows her instead. The
lang is the dictionary's, not hers: an English exception must not silence a
pt-PT flag once the second pair ships.

Adding a word takes effect in the editor immediately and persists in the
background, so the underline goes away the instant she asks. A browser still
holding the old list hands it over on first load, and only lets go once the
server has taken it.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 08:06:08 -07:00
prosolis ddc4164228 Note that the seeded 'local' row reappears after a migration 
Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 07:53:06 -07:00
prosolis 84ee6bfb9c Phase 17: run the migration, and fix the guard that locked out its own backup 
Claire's writing — 8 documents, 33 snapshots, 103 suggestions, 3 vocabulary
words and an image — now belongs to her account rather than to the pre-auth
'local' user, and the VPS is canonical. millenia was left running and
untouched as a frozen fallback; it diverges the moment either side is
written to, so it wants retiring rather than syncing.

The plan's stated prerequisite, that she log in once so her subject exists,
turned out to be false. Authentik's hashed_user_id sub is the user's uid,
derived from her id and the instance secret, so it can be read in advance —
which means the data moves first and she signs in to find her writing
already there, instead of to an empty Petal that fills in later.

The fix here is to the liveness guard, and it is the second attempt at it.
PRAGMA locking_mode = EXCLUSIVE goes on holding its lock after being set
back to NORMAL — SQLite only lets go on that connection's next database
access — so against a real WAL database the script locked itself out of its
own VACUUM INTO backup. It passed locally because the test database had
come out of VACUUM INTO and so was never in WAL mode: the fixture didn't
look like production, the same way the stub identity provider's slashless
issuer didn't. The probe now runs on its own connection and closes it.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 07:52:16 -07:00
prosolis 151df4565b Phase 17: a script to move the local user's writing onto a real account 
The destination is an OIDC subject id, which the app cannot know — it
belongs to the identity provider — so this runs deliberately, with Petal
stopped and a backup taken, rather than as a startup migration. documents,
tags, vocab_words and images carry user_id directly; versions, suggestions
and tag assignments hang off their parents and follow, which is why it has
to be one transaction with foreign keys off. Sessions for the old identity
are deleted rather than moved: a session is proof someone signed in, and
nobody ever signed in as 'local'.

Dry run by default, VACUUM INTO backup first, and it verifies every row it
expected to move actually moved — and that the source is left owning
nothing — before committing.

The 'is the app stopped?' guard took two attempts. BEGIN EXCLUSIVE, the
obvious check, sails past a running-but-idle Petal because in WAL mode it
only conflicts with another writer, which is precisely the case worth
catching. PRAGMA locking_mode = EXCLUSIVE conflicts with any connection at
all, since it locks the shared-memory index every WAL reader maps.

Sequencing this also turned up a crash waiting to happen: the image
backfill claims unowned files for 'local', which no longer exists after a
migration, and the resulting foreign-key error is fatal inside images.New.
Petal would have crash-looped the first time it started on a migrated
database. It now skips a missing owner, which costs nothing — the
migration moves the image rows itself.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 07:45:26 -07:00
prosolis 1b4a5f26df Phase 16 is live: record what deploying found 
Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 07:34:36 -07:00
prosolis e2f967c92b Deploy: drop the interim basic-auth gate now that Petal signs people in 
The gate existed because Petal authenticated nobody and a public hostname
was therefore a public, writable API. It no longer is: every /api route
answers 401 without a session, so the only thing an anonymous visitor
reaches is the app shell and its redirect to Authentik. The separate
unauthenticated /api/health router goes with it — it only existed to escape
the middleware. A second password in front of a real login is one more
thing to lose.

Also records the two problems this deployment actually hit, since both fail
before the login page appears and neither is obvious from the error: the
issuer's trailing slash is significant, and a provider created through the
API rather than the admin UI comes up with an empty grant_types.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 07:33:01 -07:00
prosolis 6d71276513 Auth: pass the OIDC issuer through untouched, trailing slash and all 
Deploying against the real Authentik turned this up immediately: its issuer
ends in a slash, OIDC requires the discovered issuer to match the configured
one byte-for-byte, and trimming it made discovery fail every time. The stub
in the tests happened to advertise a slashless issuer, so the whole suite
passed while the only provider Petal actually talks to could not be reached.
The stub now takes its issuer as a knob, and a regression test runs the flow
against one that ends in a slash.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 07:27:54 -07:00
prosolis 1cf207d73f Phase 16: Petal authenticates for itself 
Petal is now an OIDC client in its own right rather than trusting a header
from the proxy. The Phase-0 Resolver seam was the only integration point:
main.go picks the session store when Authentik is configured and the static
local user otherwise, and no handler or query moved for either.

internal/auth gains three pieces. session.go issues an opaque cookie token
and stores only its SHA-256, so a database copy yields nothing usable; the
30-day expiry slides on every request, throttled to one write an hour, and
logout deletes the row rather than just the cookie. oidc.go runs the
authorization-code flow with state, nonce and PKCE, and discovers the
provider lazily and on retry — an Authentik outage should block new logins
without stopping Petal booting or invalidating live sessions. users.go
provisions accounts from the token's claims and gates them on an allowlist
that matches emails as well as subject ids, since a subject is an opaque
uuid that doesn't exist until someone has already logged in once.

Migration 0010 lands sessions, images and users.pair_lang together. The
images table closes the capability-URL hole the Phase-0 audit flagged: a
hash was previously enough to fetch anyone's picture. Rows are keyed
(name, user_id) so one file can have several owners and deduplication
survives; a stranger gets 404 rather than 403, the cache header drops to
private, and files already on disk are claimed at startup or every image
already pasted into a document would 404.

On the frontend a single 401 interceptor feeds a warm bilingual sign-in
overlay, drawn over a still-visible editor because nothing has been taken
away. Behind it is the part that matters: a save that comes back 401
stashes its body to localStorage before anything else and stops the
auto-save loop, and reopening that document after signing in merges the
draft back and saves it. An expired session must not cost writing.

Writing the round-trip test against a stub identity provider turned up a
real bug: the one-shot state/nonce/PKCE cookies were cleared in a defer,
which runs after the redirect has written the response header, so the
clearing Set-Cookie was silently dropped and they lingered for their full
ten minutes.

Also swaps the emoji favicon for a drawn sakura, which renders as Petal's
own rose palette everywhere instead of whatever each platform's font
decides, and doubles as the app tile in Authentik.

Migration 0010 verified against a VACUUM INTO copy of the live millenia
database: counts intact, FTS still matching, the one existing image
claimed.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
 2026-07-27 07:21:32 -07:00
prosolis 42d857a878 millenia: nightly encrypted backup, supervision, and hardened Piper units 
The canonical instance -- the one with her actual writing -- turned out
to be the least protected thing in the estate:

- No scheduled backup at all; the newest snapshot was a month old. Now
  petal-backup.timer: VACUUM INTO, gzip, age-encrypt with the parodia
  public recipient, push to the VPS over headscale with a size check,
  prune both ends. Persistent=true because the box is not on 24/7.
  Neither machine can decrypt what it holds; the identity is offline.

- Petal ran as a bare ./petal with PPID 1, so a crash or reboot left it
  down until somebody noticed. Now petal.service, verified by kill -9.

- The Piper units retried forever without ever failing: RestartSec=3
  against systemd's default 10s window means the burst limit is never
  reached, which is how a dead service logged 26,800+ restarts over a
  day while read-aloud silently fell back to browser speech.
  StartLimitIntervalSec=300 makes a broken Piper show up in --failed.

backup-petal.sh now handles both deployment shapes (compose exec on the
VPS, local binary on millenia) and encrypts before anything leaves the
host. The VPS no longer uses it -- Petal rides parodia-backup there.
 2026-07-27 06:35:34 -07:00
prosolis 33e49ddb62 Encrypted data volume for the VPS, and two bugs the rehearsal caught 
Petal's data directory (petal.db, uploaded images, and the TTS cache --
which is synthesized audio of her sentences) now lives on a LUKS volume.
LUKS-on-a-file rather than gocryptfs because Petal is SQLite in WAL mode:
WAL needs a shared-memory index mapped consistently across processes, and
FUSE has a long history of subtle mmap/locking differences. A block
device with ext4 behaves exactly like a disk to SQLite.

The key sits on the same host, which is a deliberate availability
tradeoff and is documented as such: it stops a decommissioned disk or a
raw block-device read, not anyone holding the whole VM image.

Two bugs found by rehearsing a reboot rather than trusting the setup:

- Mounting over a directory HIDES its contents, it does not remove them.
  The first run left the original plaintext petal.db and WAL sitting on
  the unencrypted root filesystem, invisible under the mount -- exactly
  what the exercise was meant to eliminate. Now shredded before the
  mount, with a refusal if the mountpoint will not come up empty.

- systemd-cryptsetup was not installed on the host, so /etc/crypttab was
  being ignored entirely and the volume would never have unlocked at
  boot. The script now refuses to run without the generator present.
 2026-07-27 06:27:55 -07:00
prosolis 1d76ab1c82 Guard against Petal starting on an unmounted data volume 
The mountpoint directory exists whether or not the encrypted volume is
mounted, so a boot where the unlock failed would start Petal against an
empty unencrypted directory and serve a blank database -- the failure
mode that looks like data loss. .volume-ok lives on the encrypted
filesystem and is bind-mounted with create_host_path:false, so its
absence is a container start failure instead of a silent empty DB.
 2026-07-27 06:25:52 -07:00
prosolis 623bd02b9c Reach vLLM over headscale with a forwarder, and fix millenia's Piper 
Rebinding vllm-chat was the expensive option: Petal, Gogobee and Open
WebUI all point at 127.0.0.1:8000, and Open WebUI keeps its endpoint in
its own database rather than in env, so moving the bind address meant
editing three consumers and reloading a 35B AWQ model. The socat unit
adds a second listener on the headscale address instead -- local callers
untouched, no downtime, and the only new exposure is on the VPN. Bound
to 100.64.0.2 specifically, never 0.0.0.0: the far end is a public host.

Verified: a grammar checkpoint from petal.parodia.dev returns real
suggestions in ~3s over the VPN.

Also documents two things found on millenia that were invisible from
outside it:

- Piper had been dead since the Jul 26 reboot, 26,800+ failed restarts,
  with read-aloud silently falling back to browser Web Speech. An OS
  upgrade moved /usr/bin/python3 from 3.13 to 3.14, and venv/bin/python3
  is a symlink to the system interpreter, so lib/python3.13/site-packages
  went invisible -- sys.path had no site-packages at all. Recreating the
  venv lands Piper 1.6.0, which is what TTS_PATH exists for.
- The canonical instance has no automated backup (newest snapshot a
  month old) and runs unsupervised with PPID 1. Both written down; the
  backup one is pending the encryption-at-rest decision.

Backups on the VPS now ride parodia-backup (age-encrypted, offsite, S3),
using VACUUM INTO rather than that script's iterdump helper -- iterdump
does not reproduce an FTS5 virtual table, so a restore would have come
back with cross-document search silently missing.
 2026-07-27 06:15:00 -07:00
prosolis d01a0f1f0a Document the Phase 15 deploy and ungate the Spanish pair 
deploy/README.md becomes the real runbook: the VPS stack, Traefik, the
headscale LLM link, the interim edge gate, backups and restore. The
millenia Piper notes move to an appendix -- that instance still runs
them, and it is still canonical.

Two items are called out as outstanding rather than done, because both
need access to millenia: vLLM is not bound to its headscale interface,
so no AI pass works from the VPS yet, and parodia's ssh key is not
authorized there, so backups are VPS-local only -- which is not a backup
in the sense that matters. Each has its one-command fix written down.

Also folds in DreamDict gaining Spanish: es was explicitly gated on that
dataset existing, so it moves from "Later / not now" to a normal
follow-on pair after pt-PT, and the Phase 20 provider seam should cover
it from the start.
 2026-07-26 23:27:06 -07:00
prosolis 5b221cc7a3 Gate the VPS instance behind Traefik basic auth until Phase 16 
Petal authenticates nobody yet -- StaticResolver hands every request the
same local user -- so on a public host the whole API is open: anyone who
finds the hostname can read and write documents and fill the disk with
image uploads. Traefik holds the door until the OIDC flow exists.

/api/health keeps its own higher-priority router with no middleware, so
the acceptance criterion (public health endpoint, reachable by the
monitoring on this box) still holds. Both the middleware and that router
are deleted when Phase 16 lands.
 2026-07-26 23:21:43 -07:00
prosolis 2363ef2d37 Make Piper's synthesis path configurable (TTS_PATH) 
piper-tts 1.6.0 moved synthesis from POST / to POST /synthesize, with an
identical request body; the VPS sidecars run 1.6.0 and returned 405 to
every read-aloud request, while millenia's older server still expects /.
Rather than pinning both deployments to one Piper release, the path is
configuration -- default "/" keeps millenia working untouched, and the
compose stack sets /synthesize. The container healthcheck moves with it,
since it was probing the old route too.
 2026-07-26 23:15:01 -07:00
prosolis df6bc4989c Run the container as the host uid that owns the ./data bind mount 
The image's own petal user (uid 10001) has no claim on a bind-mounted
host directory, so SQLite came up with "unable to open database file
(14)" and the container restart-looped. Run as the stack directory's
owner instead of chowning ./data to 10001 -- the backup script gzips
snapshots in place from the host, so that account needs write access to
the same directory. Still non-root.
 2026-07-26 23:11:00 -07:00
prosolis 8410b6315b Phase 15: containerize Petal for the parodia.dev VPS 
Deploy plumbing so Petal can run on the public VPS behind the Traefik
already on that box, with vLLM reached over headscale.

- Dockerfile: node build -> go build -> alpine runtime. CGO stays off
  (modernc SQLite is pure Go), so the runtime layer exists only for
  ffmpeg (read-aloud transcodes Piper's WAV) and tzdata (the companion's
  bedtime nag and night mode read the local clock). Runs as uid 10001
  with /data as the single writable mount.

- docker-compose.yml: Traefik labels following this host's convention
  (external `traefik` network, `web-secure` entrypoint, `default` cert
  resolver). Petal publishes no host port. ./data is a bind mount, not a
  named volume, so the nightly backup and a restore are reachable from
  the host.

- Piper runs as two sibling containers rather than host systemd units.
  The plan assumed Piper was already installed on the VPS; it is not,
  the host has no lingering user session to keep user units alive, and
  containers keep the TTS ports on an internal network unreachable from
  anywhere but Petal. One image, voice chosen per service, model cached
  in a shared volume -- so the pt-PT voice is a new service, not a new
  image.

- db.Backup + a `-backup` flag: VACUUM INTO, not a file copy. Petal runs
  in WAL mode, so the newest committed pages may live in petal.db-wal;
  copying the three files separately can capture a torn mid-checkpoint
  state. VACUUM INTO reads one coherent snapshot without taking a write
  lock, and emits a single file with no -wal/-shm companions. Refuses an
  existing destination so a failed run can't destroy the last good
  backup.

- deploy/backup-petal.sh: nightly snapshot, compress, push to millenia
  over headscale with a post-transfer size check, prune both sides.

- deploy/petal.env.example: LLM_TIMEOUT raised 30s -> 90s for the
  WAN+VPN round trip, since the voice and collocation passes send a
  whole document and the timeout is a hard deadline on Complete.
 2026-07-26 23:08:22 -07:00
prosolis dae1213c68 Ratify the language-learning direction and expand it into Phases 15-22 
SUGGESTIONS.md (new): the product rationale. Every user gets one
(English + X) pair, X in {zh, pt-PT, fr, maybe es} - bilingual UI in the
pair, type in either language, direction inferred without a detector
(both-dictionaries spellcheck, show-both gloss on collision). Langpacks
keyed by X. LLM-minimalism as a standing principle: the LLM never gates
essential functionality; grammar-lite rules and an embedded
miscollocation list are planned as code-first layers beneath the LLM
families.

MULTIUSER_PLAN.md: all OPEN decisions settled - in-app OIDC (the
parodia.dev VPS hosting plan decides it), 30-day sliding sessions,
allowlist, migration-by-script, image-store ownership fixed alongside
auth, DreamDict imported as a package reading dict.db (module rename
prereq lives in the dreamdict repo), zh stays on ECDICT until compared.

BUILD_PLAN.md: the deferred bucket becomes checkboxed Phases 15-22
(deploy plumbing / OIDC / local-user migration / client-state
namespacing / langpack extraction / DreamDict provider / pt-PT pair /
learning loop + code-first layers) with standing rules: isolation tests
in the same commit as any user-scoped endpoint, LLM-minimalism, and
bilingual aesthetic as acceptance criteria.
 2026-07-26 22:24:04 -07:00
prosolis 316b6b305d Plan: DreamDict answers the Portuguese dictionary question 
OPEN #6 assumed the pt-PT gloss was gated on finding a dataset of
ECDICT's quality. It isn't — dreamdict already covers en, fr, pt-PT and
zh, and its API maps almost 1:1 onto lexicon.Result.

Replaces the data question with an integration one (OPEN #6a): HTTP
service, build-time extraction into Petal's embedded gz format, or
importing the dictionary package and opening dict.db read-only. Argues
for the third — same CGO-free sqlite driver Petal already depends on, no
runtime service, and it deletes ~11.6 MB of embedded data plus the
ECDICT build scripts.

Adds the migration caution that matters most: the zh path is in daily
use, so wire pt-PT/fr first (nothing to regress) and leave zh on ECDICT
until CC-CEDICT gloss quality has been compared on her real lookups.
 2026-07-26 21:48:51 -07:00
prosolis 023882a722 Add multi-user plan for review 
Phase 0 (identity plumbing) is done; this writes down what real
multi-user still needs and, more importantly, which decisions are
genuinely open. Six OPEN items are flagged for a reviewer to push back
on — chief among them whether Petal authenticates via Traefik
forward-auth or becomes an OIDC client itself, which everything else
depends on.
 2026-07-26 21:44:16 -07:00
prosolis 6901cdbbe4 Multi-user groundwork: request-scoped user identity 
Petal ran as a single hardcoded user, with db.LocalUserID named directly
at ~35 query sites. That made the caller's identity a compile-time
constant scattered across every package — nothing a real login could
replace without touching all of them.

New internal/auth moves it into the request context:

  - Middleware(Resolver) resolves the caller once per API request
  - handlers read auth.UserID(r.Context()) instead of naming a user
  - Resolver is the seam an Authentik session check drops into
  - StaticResolver(db.LocalUserID) keeps Petal single-user today

Behavior is unchanged. UserID returns "" rather than panicking when the
middleware is absent, so a mis-wired route fails closed: every query is
WHERE user_id = ?, which then matches nothing.

main.go splits /api into a public group (/health, /version) and an
authenticated group for everything else — a monitoring probe must not
need a session.

Two pre-existing access-control gaps fixed while threading, both
harmless with one user and not with two:

  - setStatus (accept/dismiss) updated a suggestion by bare id with no
    ownership check at all
  - listForDoc/fetchPending read a document's suggestions by doc_id
    alone; a suggestion quotes the sentence it corrects, so that leaked
    the source prose

Both now scope through documents.user_id.

Tests: internal/auth covers the context round-trip, the absent-context
case, and both 401 paths. Two-user isolation suites in docs and
suggestions mount the same routers twice behind two resolvers over one
database and assert a stranger gets 404 on every id-taking path, sees
nothing in list/search, and leaves the owner's data untouched.

Those suites earned their keep immediately: docs.fetch gained a userID
parameter but kept binding db.LocalUserID in the query. Unused
parameters are legal Go, so it compiled clean, vet was silent, and every
existing test passed while the lookup stayed unscoped.

Still global, out of scope and flagged in BUILD_PLAN.md: the image store
has no per-user association, and frontend localStorage keys are
per-browser rather than per-account.
 2026-07-26 21:42:37 -07:00
prosolis 61b3c6cd62 Disable thinking on the vLLM backend 
Qwen3-family models reason by default and prepend a plain-text preamble
ahead of the answer — not a <think> block, so it cannot be stripped
after the fact. Every Petal pass parses a JSON object out of the
completion, so an unsuppressed preamble fails the parse outright.

Send chat_template_kwargs {"enable_thinking": false} on every request,
matching the unconditional think:false the Ollama backend already sends.
 2026-07-26 21:25:56 -07:00
prosolis 78ed1dd281 Writing passport: evidence of process instead of an AI score 
She's submitting work that gets run through an AI detector and wants to
pre-check she won't be wrongly flagged. Petal should not answer that with
a detector of its own: they misfire badly on non-native English (Stanford
2023 found >50% of TOEFL essays flagged as AI vs. near-zero for native
writers), so a percentage aimed at an ESL writer is worse than nothing —
it either scares her off her own voice or gives false comfort.

So the artifact is provenance, not a verdict. Petal already snapshots
every ~3 minutes; this turns that history into a standalone printable
report: session breakdown, word-count growth, span, active time. No score
is emitted anywhere.

Two schema additions back it. preserve_history opts a document out of the
40-snapshot prune cap — right for recovery, wrong for provenance, where
you want the whole span including the oldest rows. content_hash/prev_hash
chain each snapshot to the one before it, so a history edited or thinned
after the fact fails verification. Pruning legitimately severs links, so a
link break reports as "gaps" unless preserve_history is on; only a hash
that fails against its own contents is unconditionally "broken".

The chart's x axis is snapshot order, not wall-clock, and that is the load
-bearing decision. On a linear time axis an essay written in three
sittings across three days renders as three vertical cliffs separated by
empty space — visually identical to text pasted in three chunks, i.e. the
report would have argued the opposite of the truth. Breaks are compressed
into explicitly labelled gutters instead. TestChartGivesWidthToWriting
pins it.

The report volunteers its largest single word-count jump and states its
own limits: it cannot show who was at the keyboard, or whether typed text
was composed or copied in. Overclaiming would be self-defeating — a reader
who catches it overstating discounts all of it.

HTML rather than server-rendered PDF, as with the other exports: a CJK-safe
PDF needs an embedded Unicode font or a headless browser. Print styles are
there so the browser's Save as PDF is the handoff path.

Claude-Session: https://claude.ai/code/session_016Yr6jELuRc7hyzYLccQKZd
 2026-07-19 11:45:08 -07:00
155 changed files with 16933 additions and 832 deletions
Expand all files Collapse all files
.dockerignore
+33
View File
@@ -0,0 +1,33 @@
# Keep the build context small and the image reproducible. Anything the build
# needs but that is gitignored (web/dist) is produced inside the image instead.
.git
.gitignore
.env
# Built by stage 1 — never copy a stale local build into the image.
web/dist
web/node_modules
# Local runtime state: the live database, images and TTS cache must never end
# up baked into an image layer.
data/
*.db
*.db-shm
*.db-wal
backups/
# Local build outputs
/petal
*.test
*.out
*.log
# Docs and tooling that don't affect the binary
*.md
!web/**/*.md
deploy/
scripts/
.vscode/
.idea/
.DS_Store
.env.example
+23 -5
View File
@@ -10,6 +10,12 @@ DATABASE_PATH=./data/petal.db
# On-disk store for images pasted/dropped/inserted in the editor
IMAGE_DIR=./data/images
# DreamDict's built dictionary (French, European Portuguese, Spanish, Mandarin),
# opened read-only beside petal.db. Optional: with no file here, word lookups use
# the embedded English/Chinese datasets, which is how a laptop checkout runs.
# Build one with `go run ./cmd/dictimport` in the dreamdict repo.
DICT_PATH=./data/dict.db
# LLM
LLM_BACKEND=vllm # vllm | ollama
LLM_ENDPOINT=http://localhost:8000 # vLLM :8000, Ollama :11434
@@ -27,17 +33,29 @@ TTS_ENDPOINT= # e.g. http://127.0.0.1:5005 — empty disable
TTS_ENDPOINT_ZH= # e.g. http://127.0.0.1:5006 — Chinese Piper instance
TTS_VOICE_EN=en_US-amy-medium # Piper voice id for English
TTS_VOICE_ZH=zh_CN-huayan-medium # Piper voice id for Chinese
TTS_PATH=/ # path Piper serves synthesis on: "/" up to piper-tts 1.5, "/synthesize" from 1.6.0
TTS_CACHE_DIR=./data/tts # on-disk store for synthesized clips (content-addressed)
TTS_TIMEOUT=15s
TTS_AUDIO_FORMAT=mp3 # mp3 | opus | wav — mp3/opus transcode Piper's WAV via ffmpeg
# --- Deferred (not wired in the local-dev build) ---
# Auth (Authentik OIDC) — deferred; single hardcoded local user for now
# SESSION_SECRET=change-me-to-random-64-char-string
# AUTHENTIK_URL=https://auth.parodia.dev
# --- Auth (Authentik OIDC) ---
#
# Login turns on only when the issuer, client id and secret are all set. Leave
# them commented out for local development and Petal runs as the single
# hardcoded `local` user, exactly as it did before auth landed.
#
# AUTHENTIK_URL is the issuer of the Petal provider in Authentik (the value of
# its "OpenID Configuration Issuer" field). The redirect URI to register there
# is BASE_URL + /auth/callback.
# AUTHENTIK_URL=https://auth.parodia.dev/application/o/petal/
# AUTHENTIK_CLIENT_ID=petal
# AUTHENTIK_CLIENT_SECRET=
#
# Who may sign in: comma-separated OIDC subject ids and/or email addresses.
# Empty = anyone Authentik authenticates.
# PETAL_ALLOWED_SUBS=her@example.com,me@example.com
# --- Deferred (not wired in the local-dev build) ---
# Copyleaks (Tier-2 plagiarism) — deferred; needs a public webhook
# COPYLEAKS_ENABLED=false
.gitignore
+4
View File
@@ -10,6 +10,10 @@ web/dist/*
!web/dist/.gitkeep
*.log
# Python
__pycache__/
*.pyc
# Local env & data
.env
*.db
BUILD_PLAN.md
+217 -3
View File
@@ -135,9 +135,209 @@ Multi-session build. **Source of truth for what's done and what's next.** Update
- [x] Verified: tsc clean, vite build OK, companion vitest 45/45. **Real-browser screenshots** (local Playwright + Chromium, clock mocked to 23:30): day = warm cream + pink sakura petals; night = dark plum-indigo + twinkling stars + glowing sleepy kitten. Both pretty (acceptance criterion).
### Deferred (post-v1-local)
- [ ] Authentik OIDC auth + session middleware ← **on hold: user doing foundational work first**
- [ ] Copyleaks Tier-2 + webhook HMAC
- [ ] Dockerfile, docker-compose, Traefik, deploy to write.parodia.dev
- [x] **Multi-user groundwork** (2026-07-26) — request-scoped identity. New `internal/auth`: `Middleware(Resolver)` resolves the caller once per API request and stores the id in the context; handlers read it via `auth.UserID(r.Context())` instead of naming `db.LocalUserID`. `StaticResolver(db.LocalUserID)` keeps Petal single-user today. **Auth itself is still deferred** — but every query is now scoped to whoever the resolver says is calling, so landing Authentik is a one-line change in `main.go` plus a `Resolver` implementation.
- [ ] Copyleaks Tier-2 + webhook HMAC (still parked — needs a public webhook; revisit after Phase 15)
- Authentik OIDC + deploy: **no longer deferred — expanded into Phases 15–17 below** (decisions ratified 2026-07-26; see `MULTIUSER_PLAN.md`). Deploy landed 2026-07-26 (Phase 15); Authentik itself already runs on the same VPS, so Phase 16 has its IdP waiting.
## Execution phases 15–22 (added 2026-07-26)
Decisions behind these are ratified in `MULTIUSER_PLAN.md` (all OPENs settled) and `SUGGESTIONS.md` (the *why*; Q1–Q3 settled). **Standing rules for every phase below:**
- **Isolation tests in the same commit** as any new user-scoped endpoint (the `docs/isolation_test.go` suites are the template — this discipline caught a real unscoped-query bug once already).
- **LLM-minimalism** (SUGGESTIONS §6): the LLM never gates essential functionality; new essential features are code+data first.
- **Aesthetic + bilingual-in-the-pair copy remain acceptance criteria** on every user-visible change.
- Verify per project convention: go build/vet/test, tsc, vite build, vitest, live smoke on a throwaway DB/port.
### Phase 15 — Deploy plumbing (parodia.dev + headscale) ✅ (2026-07-26/27)
Petal hosted on the parodia.dev VPS; vLLM stays on millenia over headscale. Auth (Phase 16) needs the stable `BASE_URL`/redirect URI this phase creates. **Hostname: `petal.parodia.dev`** (DNS already pointed at the VPS). Runbook: `deploy/README.md`.
- [x] Dockerfile (multi-stage: `npm run build` → `go build` → alpine runtime) + docker-compose. CGO stays off (modernc SQLite is pure Go), so the runtime layer exists only for **ffmpeg** (read-aloud transcode) and **tzdata** (the bedtime nag + night mode read the local clock). Non-root; `/data` is the single writable mount. **`.dockerignore`** keeps the live DB and a stale local `web/dist` out of the image.
- [x] Traefik route + HTTPS on `petal.parodia.dev` — labels follow the host's existing convention (external `traefik` network, `web-secure` entrypoint, `default` cert resolver, `compression@file`) plus Petal's own header middleware. No host port is published; Traefik is the only way in. `BASE_URL=https://petal.parodia.dev` set for Phase 16's redirect URI.
- [x] `LLM_ENDPOINT` → millenia's headscale address (`100.64.0.2:8000`); `LLM_TIMEOUT` **30s → 90s** for the WAN+VPN round trip (the voice/collocation passes send a whole document and the timeout is a hard deadline on `Complete`). **Exposed with a forwarder, not a rebind** (`deploy/vllm-headscale-proxy.service`, socat): `vllm-chat.service` is shared — Petal, **Gogobee** and Open WebUI all point at `127.0.0.1:8000`, and Open WebUI keeps its endpoint in its own database rather than in env, so rebinding meant editing three consumers and reloading a 35B AWQ model. The forwarder adds a second listener on `100.64.0.2` only (never `0.0.0.0` — the far end is a public host), zero downtime, zero consumer changes. Model is `qwen3.6-35b`. **Verified end to end: a grammar checkpoint from `petal.parodia.dev` returns real suggestions in ~3s over the VPN.**
- [x] TTS — **deviation from the plan, deliberate**: Piper was *not* actually installed on parodia, and the `reala` account has no lingering session to keep user systemd units alive. Runs as **two sibling containers** (`piper-en`, `piper-zh`) off one image, models cached in a shared volume, on an internal network with no published ports. pt-PT in Phase 21 is a fourth service, not a new image. **Found + fixed while wiring**: piper-tts 1.6.0 moved synthesis from `POST /` to `POST /synthesize` (identical body); rather than pin both deployments to one release, the path is now config (`TTS_PATH`, default `/` so millenia is untouched).
- [x] Backups — `db.Backup` uses **`VACUUM INTO`**, not a file copy: in WAL mode the newest committed pages may live in `petal.db-wal`, and copying the three files separately can capture a torn mid-checkpoint state. `VACUUM INTO` reads one coherent snapshot including the WAL, takes no write lock (safe against the live app), and emits a single file with no companions; it refuses an existing destination so a failed run can't destroy the last good backup. Driven by a `-backup` flag on the binary.
- **On the VPS: folded into the host's existing `parodia-backup`** (age-encrypted, offsite to S3, 14-day retention, dead-man snitch) rather than a parallel cron — the user pointed out that layer already existed. **Found a real bug while doing it:** that script's `sqlite_dump` helper uses Python `iterdump`, which **does not reproduce an FTS5 virtual table** — it emits `documents_fts` as a raw `sqlite_master` row plus shadow tables, and replaying the result dies with `no such table`. Cross-document search would have been silently missing after any restore. Added a `sqlite_file_dump` helper using `VACUUM INTO` instead; round-trip verified (counts + a live FTS `MATCH`).
- **On millenia: `petal-backup.timer`** — the canonical instance had **no scheduled backup at all** (newest snapshot a month old), which mattered far more than the staging one. Nightly 03:20, `Persistent=true` (the box isn't on 24/7), snapshot → gzip → **age-encrypt with the parodia public recipient** → push to the VPS over headscale with a size check → prune both ends. Verified the pushed archive is real age ciphertext and that neither box can decrypt it.
- [x] Migration decision: **millenia stays canonical** (user's call). The VPS runs an empty staging DB so she moves accounts exactly once, when Phase 16/17 land.
- [x] **Encryption at rest** (not in the original plan — the user raised it mid-session, correctly). VPS data dir is now a **LUKS2 volume** (`deploy/setup-encrypted-data.sh`), covering `petal.db`, `images/` **and the TTS cache** (synthesized audio of her sentences). LUKS-on-a-file rather than gocryptfs because SQLite in WAL mode needs a shared-memory index mapped consistently across processes and FUSE has a long history of mmap/locking differences. Key on the same box — a deliberate availability tradeoff, documented honestly: it stops a decommissioned disk or a raw block-device read, **not** anyone holding the whole VM image. Canary-verified: a marker written through the app is absent from the raw image and present through the mount. **Two bugs caught by rehearsing a reboot rather than trusting the clean run** — (1) mounting over a directory *hides* its contents rather than removing them, so the first pass left the original plaintext `petal.db` and WAL on the unencrypted root filesystem, invisible under the mount (now shredded pre-mount, with a refusal if the mountpoint won't come up empty); (2) **`systemd-cryptsetup` wasn't installed**, so `/etc/crypttab` was ignored entirely and the volume would never have unlocked at boot. Added a **mount-liveness guard** (`.volume-ok` bind-mounted with `create_host_path: false`) so an unmounted volume is a loud container start failure instead of Petal quietly serving a blank database. ⚠️ A true reboot is untested — the VPS also runs matrix/lemmy/akkoma/gitea/authentik, so that's the user's call. millenia remains unencrypted at rest (LVM, no LUKS).
- [x] **Supervision** (also not in the original plan). millenia's Petal had been running as a bare `./petal` with **PPID 1** — no unit, no screen session — so a crash or reboot left it silently down; now `petal.service`, verified by `kill -9`. **Piper's silent-failure mode fixed**: with `RestartSec=3` against systemd's default 10s window the burst limit was never reached, so a dead service looped **26,800+ times over a day without entering `failed`**; both units now set `StartLimitIntervalSec=300`/`StartLimitBurst=5`. Still missing: an external probe (uptime-kuma monitors on `/api/health` and `/api/tts` — needs the UI, written up in `deploy/README.md` §7).
- [x] **Interim edge gate** (not in the original plan; added once the instance was live). Petal authenticates nobody yet — `StaticResolver` hands every request the same `local` user — so on a public host the whole API was open to read/write and image upload. Traefik basic auth holds the door until Phase 16, with `/api/health` exempt on its own higher-priority router. Deleted when OIDC lands.
- [x] Acceptance — verified over public HTTPS **with the LLM link down** (it genuinely is): Hunspell dictionaries 200, gloss + word lookup (incl. phonetic) 200, doc create/save, FTS search on 春天, md + docx export, vocab capture/list, read-aloud EN + zh (real mp3 via ffmpeg, cache hit on repeat, 404 for an unconfigured language so the client falls back). `POST /check` → the warm 502 that renders as 小助手在休息. `/api/health` public; HTTP 301 → HTTPS with a valid cert.
### Phase 16 — Auth (in-app OIDC) + image-store ownership ✅ (2026-07-27) — live on petal.parodia.dev
Option B ratified. `go-oidc` + `x/oauth2`; config fields already existed. The `Resolver` seam from Phase 0 was the only integration point — no handler or query moved.
- [x] OIDC login flow (`internal/auth/oidc.go`): `/auth/login` → Authentik → `/auth/callback` → provision → session. **state** (cookie vs param, constant-time) + **nonce** (ID-token claim vs cookie, so a token minted for another attempt is refused) + **PKCE S256**. Discovery is **lazy and retried**: an Authentik outage blocks new logins but leaves every existing session working, since those need only Petal's own DB — the app must not fail to boot because the IdP is briefly down.
- [x] `sessions` table (migration `0010`); opaque token in `petal_session` (`HttpOnly`, `SameSite=Lax`, `Secure` only when `BASE_URL` is https — flagging it on a plain-http dev server makes the browser silently drop it). **The table stores only the token's SHA-256**, so a DB copy yields no usable session. **30-day sliding expiry**, all time math in SQLite `datetime()` (canonical UTC), the extension throttled to one write per hour per session. `/auth/logout` deletes the row, not just the cookie; `RevokeAll` signs one writer out everywhere; expired rows pruned at startup.
- [x] Allowlist: `PETAL_ALLOWED_SUBS`, comma-separated. **Matches a subject id *or* an email**, case-insensitively — a deliberate widening of the plan: a subject is an opaque uuid that doesn't exist until first login, so a subject-only list means letting someone in, reading a log line, and editing config. Empty = anyone Authentik authenticates. A rejected valid login gets the warm bilingual "这个 Petal 不是给你写的 · This Petal isn't yours to write in" page and **no provisioned account**.
- [x] `main.go` picks the resolver from config: `SessionStore` (which is itself the `Resolver`) when `AUTHENTIK_URL`/id/secret are all set, `StaticResolver(local)` otherwise — so local dev and every pre-auth deployment behave exactly as before. `/auth/*` mounts on the root router, outside `/api`.
- [x] Frontend: one 401 interceptor in `api/client.ts` (`UnauthorizedError` + an `onUnauthorized` hook covering `req`, the image upload and the SSE chat stream) → `useSession` → `SignInOverlay` (warm bilingual, editor still visible behind it — nothing has been taken away). **Draft rescue** (`lib/drafts.ts`): a save that 401s stashes its body in `localStorage` keyed by doc id *before* anything else, auto-save then stops (further attempts would only 401 and re-stash), and opening that doc after re-login merges it back and schedules a save. StatusBar says 已保存在本机 · Kept on this device — where the writing is, not what failed. Sidebar footer gains the account + 退出 · Sign out (hidden when the id is still `local`).
- [x] **Image store ownership** (OPEN #5): `images` table (`PRIMARY KEY (name, user_id)`) — **one row per owner, not one owner per file**, so the same picture uploaded by two people is still stored once and dedup survives; the file is deleted only with its last row. Fetch joins on the caller and answers **404, not 403** (whether a hash exists is itself information). `Cache-Control` went `public` → `private` — a shared cache must never hand one writer's image to another. Files already on disk are claimed for the local user at startup (idempotent), because a row is now what makes an image fetchable and every picture already pasted into a document would otherwise 404.
- [x] `users.pair_lang` (default `'zh'`) added in the same migration; login refreshes email/display name but never touches it — it's Petal's setting, not the IdP's.
- [x] Tests: `session_test.go` (lifecycle, expiry + prune, sliding renewal, hash-not-token storage, cross-user non-interchangeability, `RevokeAll`, FK cascade, middleware wiring, user upsert, allowlist matrix) and `oidc_test.go` — **the whole round trip against a stub IdP** (RSA-signed ID tokens, real discovery + JWKS): PKCE challenge present, verifier reaches the token endpoint, state mismatch → 400, **replayed nonce from another attempt → 400**, allowlist refusal → the bilingual 403 with no account created, provider error → 403, already-signed-in login short-circuits home. `images/handler_test.go` gained two-user isolation, cross-user dedup + last-owner file deletion, and backfill idempotency. `web/src/lib/drafts.test.ts` covers the rescue (round-trip, per-doc, take-consumes, expiry, corrupted entry, storage that throws).
- **A real bug the round-trip test caught:** the one-shot state/nonce/PKCE cookies were cleared with `defer o.clearTemp(w)` — which runs *after* the redirect has written the response header, so the `Set-Cookie` was silently dropped and they lingered in the browser for their full 10 minutes. Now cleared up front.
- Verified: go build/vet/test, tsc, vite build, vitest 76/76 all clean. Migration `0010` applied to **a copy of the live millenia DB** (`VACUUM INTO` snapshot): 10 migrations apply, documents/vocab/versions counts unchanged, FTS search still returns hits, the one existing image claimed, `pair_lang` defaulted. Live smoke against the binary on a throwaway DB: auth-off → `/api/me` is `local` and everything 200s; auth-on → `/api/docs` and `/api/me` 401, `/api/health` still public, `/auth/login` with an unreachable IdP renders the warm 503 page, `/auth/logout` redirects home; a hand-inserted session row → 200 with the cookie, 401 without it, with a bad one, and once expired.
- [x] **Deployed** (user: "do it! register it!"). Provider + application registered in Authentik (slug `petal`, confidential, strict redirect `https://petal.parodia.dev/auth/callback`, openid/profile/email, implicit-consent authorization flow), created through `ak shell` since the available API token is a limited invite-minter bot. `.env` filled in on the VPS, image rebuilt, container recreated. **The Traefik basic-auth gate is gone** along with the separate unauthenticated `/api/health` router that existed only to escape it — Petal 401s every `/api` route without a session, so an anonymous visitor gets the app shell and a redirect, and a second password in front of a real login is one more thing to lose. Verified over public HTTPS: `/api/health` 200, `/api/docs` **401 with no basic-auth challenge**, `/auth/login` → Authentik with state+nonce+PKCE in the URL, following it lands on the real sign-in page, `/petal.svg` served as the favicon. The final step — typing her password — is hers.
- **Two bugs deploying caught that the whole test suite could not**, both fatal before the login page ever renders: (1) the code trimmed the issuer's **trailing slash**, and Authentik's issuer has one — OIDC requires a byte-for-byte match, so discovery failed every time while the stub IdP (which advertised a slashless issuer) kept passing. Fixed, and the stub's issuer is now a knob with a regression test that ends in a slash. (2) A provider created through `ak shell` rather than the admin UI comes up with **`grant_types = []`**, which authentik reads as "no grant type is permitted here" and answers with `invalid_request` / *The request is otherwise malformed*. Both are written up in `deploy/README.md` §4.
- **Allowlist is currently `prosolis@proton.me` only.** That Authentik instance fronts ~40 accounts across several applications, so an empty list was not an option, and guessing which account is hers would either lock her out or let a stranger in. Adding her is one line in `.env` plus a restart. Note that an Authentik account with **no email set** (e.g. `akadmin`) can't match an email-based entry — use its subject id.
### Phase 17 — Migrate the `local` user ✅ (2026-07-27) — her writing now lives on her account
Script, app stopped, backup first (OPEN #4). **The "she logs in once first" dependency turned out not to exist**: authentik's default `hashed_user_id` sub mode makes the subject `User.uid`, which is derived from her user id and the instance secret — stable, and readable before she has ever signed in (`ak shell -c "…User.objects.get(username='claire').uid"`). So the data can move *first*, and she signs in to find her writing already there rather than to an empty Petal that fills in later.
- [x] `scripts/migrate_local_user.*`: single transaction, `PRAGMA foreign_keys=OFF`, re-point `documents`/`tags`/`vocab_words` **and `images`** (versions/suggestions follow parents; `images` is new in Phase 16 and carries `user_id` directly — miss it and every pasted picture 404s), delete the empty provisioned row, verify row counts before commit; refuses to run if the app is up or the target has data
- [x] `scripts/migrate_local_user.py` — dry-run by default, `VACUUM INTO` backup before touching anything, one transaction with `PRAGMA foreign_keys=OFF`, re-points `documents`/`tags`/`vocab_words`/`images`, deletes the old user row, and **verifies every expected row actually moved (and that the source is left owning nothing) before it commits**, rolling back otherwise. Refuses to merge into an account that already owns writing. Runbook in the script header.
- **The "is the app stopped?" guard needed a second attempt.** `BEGIN EXCLUSIVE` — the obvious check — passes straight through against a *running but idle* Petal, because in WAL mode it only conflicts with another writer. That is exactly the case the guard exists to catch, and it would have failed silently. `PRAGMA locking_mode = EXCLUSIVE` conflicts with any connection at all, since it locks the shared-memory index every WAL reader maps; verified against a live server.
- [x] **Run for real.** Her 8 documents, 33 snapshots, 103 suggestions, 3 vocabulary words and 1 image moved from `local` onto `5f47d955…` (Claire, `clairew8@pm.me`). Sequence: `-backup` snapshot of millenia's live DB (taken while it kept running — `VACUUM INTO` needs no write lock), shipped to the VPS, installed over the throwaway staging database (kept as `petal.db.staging-*`), **started once so migration `0010` applied**, stopped, migrated, started. Verified over public HTTPS with a short-lived probe session, then removed: `/api/me` is her, 8 documents listed, her image 200s, vocabulary garden and version history intact, search returns hits — and 401 without the cookie.
- **millenia is a frozen fallback, not a mirror** (user's call: "both, VPS first"). It was left running and completely untouched, still serving the same writing under the pre-auth `local` user. The two diverge the moment anything is written on either, so it wants retiring rather than syncing.
- **A second bug in the guard, found by running it against production rather than a test file.** `PRAGMA locking_mode = EXCLUSIVE` keeps holding the lock after being set back to `NORMAL` — SQLite only releases it on that connection's next database access — so on a **WAL** database the script locked itself out of its own `VACUUM INTO` backup. It passed locally because the test database had come out of `VACUUM INTO` and so wasn't in WAL mode at all — the same shape of miss as the trailing-slash issuer: the fixture didn't look like production. The probe now runs on its own connection and closes it, and the fix was re-verified against a database that had genuinely been served in WAL mode.
- **Startup crash averted while sequencing this**: the image backfill claims unowned files for `local`, which stops existing after the migration — a foreign-key error inside `images.New`, which `main.go` treats as fatal. Petal would have entered a crash loop the first time it started on a migrated database. The backfill now skips a missing owner (there is nothing to claim in that case anyway; the migration moves the image rows itself).
### Phase 18 — Per-user, per-language client state ✅ (2026-07-27)
- [x] **Preferences namespaced by account** (`web/src/lib/prefs.ts`) — `petal.sound`, `petal.petals` and `petal.companion` now read/write `<base>.u.<userId>`. The wrinkle is timing: `sounds.ts` and `petals.ts` read their value at *import* time, long before `/api/me` answers, so rather than block startup on the network for a mute flag, a read before the answer sees the **legacy un-namespaced key** (on a single-writer browser, exactly the right value) and `setPrefsScope` — called from `useSession` the moment `/api/me` resolves — adopts it and fires `onPrefsScopeChange` so each module re-reads. `PetalCompanion` re-reads too, unless she's already swapped mascots in the meantime.
- [x] **Legacy adoption is a move, not a copy** (user's call): the first account to sign in on a browser inherits whatever was set back when Petal had no accounts, and the key is then deleted so the *second* account starts from Petal's defaults rather than from a stranger's choices. An existing scoped value is never overwritten by the legacy one.
- [x] **Personal spell dictionary promoted to a server table** (the plan's nice-to-have; user chose it) — new `internal/spell` package + migration `0011_personal_dictionary`: `personal_words (user_id, lang, word, created_at)`, `PRIMARY KEY (user_id, lang, word)`, cascading with the account. `GET/POST/DELETE /api/spell/words`; adds are idempotent, a delete of a word that was never there is a success (the caller's intent already holds), and **every response carries the full resulting list** so the client never has to merge two views of one set. Namespacing it in `localStorage` instead would have *fragmented* the list she already has across her laptop and tablet — strictly worse than before; a table means it follows her.
- [x] `lang` is the **dictionary's** language, not the writer's — an en-US personal word must not silence a pt-PT flag once the second pair ships. Normalised (`trim`/lowercase, default `en`) so `EN`/`en`/absent can't split one list into three.
- [x] `useSpellChecker` is server-backed: nspell loads, then the word list arrives from her account and is replayed in. A browser still holding the Phase-7 `petal.spell.personal` key hands it over on first load — but **only lets go of it once the server has accepted it**, so a failed request costs nothing. `addWord` takes effect in the editor immediately and persists in the background: the underline goes away the instant she asks, whatever the network is doing. A word list that fails to load costs correct words being flagged, never writing.
- [x] Tests: `internal/spell/handlers_test.go` — lifecycle (idempotent add, bulk add, repeat delete, `[]` not `null`), languages-don't-merge incl. the case-normalisation case, junk rejection, and the standing-rule **two-user isolation** (mount twice behind two resolvers over one DB: Bob sees none of Alice's words, his identical word is his own row, his delete doesn't reach hers, deleting the account takes the dictionary with it). `web/src/lib/prefs.test.ts` — legacy adoption, move-not-copy, two accounts on one browser, never-overwrite, listener fires once per real change, storage-throws safety.
- Verified: go build/vet/test, tsc, vite build, vitest 82/82 all clean; live smoke on a throwaway DB (:8073) — add/bulk-add/list/delete, pt-PT list independent of en, CJK word accepted, 400 on empty.
- [x] **Rehearsed, then deployed** (2026-07-27). The rehearsal the previous session couldn't run: `VACUUM INTO` snapshot of the live VPS database, pulled down, migrated by the Phase-18 binary — 11 migrations apply, every count unchanged (2 users, 8 documents, 33 versions, 103 suggestions, 3 vocab words, 1 image), FTS still matching, `integrity_check` and `foreign_key_check` both clean, `personal_words` present and empty. Then the deploy: off-box encrypted backup first, `git pull` + `docker compose up -d --build`, all three containers healthy, `0011` applied to the live DB with her writing untouched. Verified over public HTTPS: `/api/health` 200, `/api/docs` and the new `/api/spell/words` **401 without a session**.
- ⚠️ The authenticated live probe of `/api/spell/words` (hand-inserted session row, as Phases 16/17 used) was **blocked by this session's permission classifier** — minting a session token reads as credential fabrication. Not worked around. The endpoint's full lifecycle is covered by `internal/spell/handlers_test.go` and was smoke-tested end to end on a throwaway DB when it was built; what remains unproven in production is only that it answers 200 for a real cookie, which the shared middleware already governs for every other route.
### Phase 19 — Langpack extraction (the copy chore) ✅ (2026-07-27)
Pure refactor, zero visible change; prerequisite for every new pair (SUGGESTIONS §2, Q2 settled).
- [x] Every `中文 · English` string from the ~29 frontend files (plus `tips.ts`, `prose.ts`, `companions.ts`, `stats.ts`) now lives in `web/src/i18n`: `types.ts` (the `Pack` shape), `packs/zh.ts` (today's copy, **verbatim** — sentinel assertions in `i18n.test.ts` guard against a quiet rewording), `index.ts` (the accessor).
- [x] Two access paths, matching where copy is built: `usePack()` for components (a `useSyncExternalStore` subscription, so a pack arriving after first paint re-renders), and `pack()` for the modules that compose a line when something *happens* rather than when something renders — the companion and the prose checker read it at call time, never at import time.
- [x] **Anything with a value in it is a function on the pack**, not a template assembled at the call site (`reviewDue(n)`, `daysAgo(n)`, `duplicateTitle(title)`, every prose rule). Word order isn't universal; a pack author must be able to move the number. English pluralisation moved into the pack with it.
- [x] `Line` renamed its Mandarin half `zh` → `native` throughout (companion bubbles, tone/style pills, history badges, stat rows). `gradeBand` now returns a band *name* rather than a label, and the roster constants (`TONES`, `REWRITE_STYLES`, export formats, companions) keep only value + emoji — the label is a pack lookup keyed by the same value, with a test asserting no roster entry is unlabelled.
- [x] `internal/llm/lang.go`: the three prompts that *name* the writer's language — the collocation gloss, Ask Petal's "answer in her language", the explanation translator — take a `Lang` instead of saying "Simplified Chinese" outright. pt-PT is spelled **"European Portuguese (pt-PT, never Brazilian Portuguese)"** in the prompt itself, since a model that has read far more pt-BR needs telling. `Why` carries her word for "why" (为什么 / porquê / …) so the tutor prompt still recognises the question. An unknown code falls back rather than erroring — a prompt is the wrong place to discover a config problem.
- [x] Wired to `users.pair_lang` on both sides: `useSession` calls `setPackLang` the moment `/api/me` answers, and each LLM handler reads the column **in the row-scoped query it already ran** (the one that proves she owns the document) rather than in a second lookup that could disagree with it.
- Tests: `internal/llm/lang_test.go` (fallback matrix; each prompt names the writer's language and *not* Chinese; the zh pair reads exactly as before), `internal/suggestions/pairlang_test.go` (the column reaches the model for collocation + translate, zh unchanged — **verified to fail when the join is removed**), `web/src/i18n/i18n.test.ts` (default before `/api/me`, fallback for an unshipped pair, no spurious notifications, verbatim sentinels, interpolation incl. plurals, no empty string anywhere in the pack, every companion/tone/style labelled).
- Verified: go build/vet/test, tsc, vite build, vitest 90/90 clean; live smoke on a throwaway DB (:8074) — doc create/save, search, md export, spell add, warm 502 from the collocation pass with the LLM down; the built bundle still carries the zh strings.
- [x] **Deployed 2026-07-27**, together with Phase 20. Pure refactor with no migration, so it was a rebuild; the langpack is live and reads exactly as before, which is the whole point of a verbatim `zh` pack.
### Phase 20 — DreamDict as a lexicon provider ✅ (2026-07-27)
Option 3 ratified (import package, read-only `dict.db`).
- [x] **The prerequisite was bigger than the plan thought.** Renaming the module was necessary but not sufficient: DreamDict's query layer lived in `internal/dictionary`, which no other module may import whatever the module is called. Both fixed upstream in one commit — `module github.com/prosolis/dreamdict`, `internal/dictionary` → `dictionary`, with a package comment saying why reading a built database is public API while building one stays internal. `internal/loader` is untouched, and DreamDict's own tests pass unchanged.
- [x] **Provider seam** (`internal/lexicon/provider.go`): a `Provider` is the two questions the popover and the tooltip have always asked (`Lookup`, `Gloss`), which the embedded `*Lexicon` already satisfied unmodified. `Set.For(lang)` is the single place the choice is made. `OpenDreamDict` opens `dict.db` read-only beside `petal.db`; **a missing file returns `(nil, nil)`, not an error** — a laptop checkout has never had one — while a file that is *present but unimported* does error, because that one is somebody's half-finished deploy.
- [x] **The absent-dictionary case degrades better than "no data".** A pt-PT writer with no `dict.db` falls back to the embedded datasets **with the gloss suppressed** (`glossless`), so she keeps English definitions, synonyms and phonetics — all compiled into the binary and all correct for her — and loses only the translation. Handing her the Chinese gloss would be worse than handing her nothing: empty reads as "not found", wrong-language reads as Petal being broken.
- [x] pt-PT + fr + **es** wired to DreamDict; **zh stays on ECDICT**, and the routing test is the guard on that decision. The comparison the plan asked for was run against the real 452 MB database: DreamDict reaches a Chinese gloss for **53%** of the 2,000 commonest English words, against ECDICT's essentially total coverage of them. Quality did not hold, so nothing converged. es routes to DreamDict from day one and simply finds no rows in the April build — which is the same code path as any unglossed word.
- [x] **The plan's central assumption was wrong, and measuring it is what found that.** `Gloss ← Translate(word, "en", L1)` was mapped 1:1 in `MULTIUSER_PLAN.md`; against real data that table answers for **17%** of common English words into pt-PT (16% into fr). Wiktionary's translation sections are thin in the en→X direction. Going through shared Princeton WordNet synset ids instead answers for **61%**, and it is where the words a learner wants live — "ephemeral", "think" and "quickly" have no en→pt-PT translation row at all. New `dictionary.Equivalents(word, from, to)` upstream does that, falling back to the translations table, for **62%** combined. A gloss absent five times in six is not a gloss.
- [x] **Ranking, argued from a wrong answer.** Ordering equivalents by target-word frequency glosses "think" as *lembrar* — "remember" — because lembrar is the commoner Portuguese word even though pensar shares six of think's synsets to lembrar's one. Counting shared senses first, frequency second, asks which candidate means the same thing *most often*: think → pensar; achar; lembrar, write → escrever, garden → jardim, house → casa before firma.
- [x] The de-inflection walk (`candidates`) is shared with the embedded path, because `dict.db` stores headwords — "running" has no row. The first candidate that *has definitions* becomes the headword every other field is read from, so one popover never mixes "running"'s frequency with "run"'s senses. The gloss walks separately, since a word can have an equivalent and no definition.
- [x] **Surfaced where cheap**: a band chip beside the phonetic (`wordband.ts`) and an etymology line at the foot of the card. Following Phase 19, `wordBand` returns a band *name* and the langpack owns the wording. **Three bands, not five** — the difficulty score is a heuristic over length and corpus counts, good enough to separate "everyday" from "you will need to explain this" and not good enough to rank *obfuscate* against *serendipity*; a finer scale would be a confident-looking lie. Thresholds come from the real distribution (136k headwords bunch between 0.45 and 0.60; the words a writer reaches for sit under 0.40). An unscored word renders **no chip at all**.
- [x] The pair language is read **per request** in `providerFor` — a word lookup has no row-scoped query to piggyback on, unlike Phase 19's handlers — and a failed read falls back to today's embedded behaviour rather than failing the lookup. `Cache-Control` dropped from `public` to `private`: the same URL now answers in a different language per writer.
- Tests: `internal/lexicon/dreamdict_test.go` — fixture is a **real dict.db on disk**, so open/stat/seeded is the production path; missing vs. unseeded vs. unreadable, every field filled, gloss follows the writer not the word (pt-PT/fr/es/de), the synset path, de-inflection carrying *all* fields to one headword, miss-is-not-an-error, the NULL-difficulty sentinel, IPA chosen over CMU, and the handler tests (two writers/one URL/two languages, unknown caller, private caching). Upstream: `Equivalents` ordering, synset-over-translation, fallback. Frontend: `wordband.test.ts` (bands pinned to real scores; difficulty 0.0 is a score, not a missing value) and an i18n assertion that no band can be unlabelled.
- **Two bugs the tests found before the browser did**: `trimEtymology` sliced by byte, which would put invalid UTF-8 in the JSON for exactly the etymologies that matter (ἐφήμερος, ephemerus), and its ellipsis path overran its own cap.
- Verified: go build/vet/test, tsc, vite build, vitest 96/96 clean, both repos. Live smoke on a throwaway DB (:8075) against the real 452 MB `dict.db` — startup logs the languages it actually got, zh unchanged, then the same instance flipped to pt-PT and re-queried.
- [x] Deploy documented (`deploy/README.md` §4b, `DICT_PATH` through Dockerfile/compose/.env.example): `dict.db` ships into the data dir, stays out of the backups because they name `petal.db` explicitly, and is rebuildable from public data.
- [x] **Deployed 2026-07-27**, with Phase 19. The two dreamdict commits went to GitHub (`prosolis/dreamdict` main), the `replace` came out for a real pseudo-version, and Petal shipped as a rebuild — no migration in either phase, so `schema_migrations` is still 11. Order: off-box encrypted backup first (`✓ petal.db.age`), then `dict.db` copied into the LUKS volume and **SHA-256 verified end to end**, then pull + `docker compose up -d --build`. All three containers healthy; startup logs `dictionary: DreamDict open at /data/dict.db ([en fr pt-PT es zh])`, so the deployed binary opened the deployed file and found every language. Over public HTTPS: `/api/health` 200, `/api/docs`, `/api/word/…` and `/api/gloss/…` all **401 without a session**. Her writing untouched — 2 users, 8 documents, 33 versions, 103 suggestions, 3 vocabulary words, 1 image, FTS still matching, `integrity_check` ok. Both accounts are on the zh pair, so **nothing about her experience changed today**; what shipped is the capacity for the next pair.
- [x] **`dict.db` rebuilt with Spanish and deployed** (2026-07-27, user: "if we need to redeploy DreamDict to add Spanish support, then do so"). Millenia's dreamdict checkout turned out to carry ~490 lines of **uncommitted** changes; checking rather than pulling over them showed an *earlier draft* of the regional-variant work since committed upstream (main has the reviewed `wordListQuery` refactor, millenia the pre-refactor `Words`) — nothing unique at risk, but not mine to discard, so that checkout was left untouched and the build ran from a clean clone. Import: 6m15s, `es` 102,971 words / 71,680 definitions, **every other language byte-identical to April** — which is what says a language was added rather than the rest quietly shifted. Coverage of the 2,000 commonest English words: **es 68.6%** (best of the four), fr 63.1%, pt-PT 62.1% (both unchanged), **zh 53.2% — re-measured, still under ECDICT, so Chinese stays put**. Shipped millenia→parodia direct over headscale, SHA-256 verified both ends, April file kept as `dict.db.april-backup`.
- **The startup log was lying, and the rebuild is what exposed it.** It printed `dictionary.Langs()` — a compile-time constant of the languages DreamDict *supports* — so it had been reporting a confident `[en fr pt-PT es zh]` over the April file that contained no Spanish at all: exactly the failure the line existed to catch, rendered as success. It now counts rows (`en=136615 es=102971 fr=56096 pt-PT=136300 zh=120883`), with a test asserting a fixture holding only two languages cannot name five. For a file somebody copies onto the box by hand, "what is in it" is the only question worth asking.
- A second thing worth recording from the rebuild: the SUBTLEX-US download now fails (source moved behind a manual export), which looked like it would silently cost English frequency data. It doesn't — the loader falls back to `SUBTLEX-US.txt`. Chasing it down showed English "frequency" is mostly **SCOWL's commonness bucket** (1000/800/600/…/50), refined by SUBTLEX for ~1,600 words — which is the quantised distribution measured earlier, and independent confirmation that the band chip was right to read `difficulty` rather than `frequency`.
- ⚠️ As in Phase 18, the authenticated live probe — seeing `/api/word` answer 200 for a real cookie — was **not performed**: minting a session row reads as credential fabrication to this session's classifier. The lookup path was smoke-tested end to end against this exact `dict.db` (same SHA-256) on a throwaway instance, including the zh→pt-PT flip, and the shared middleware governs that last step for every other route.
### Phase 21 — The pt-PT pair (first Latin pair, proves the model)
SUGGESTIONS §1/§3/§3a. French and **Spanish** follow the same groove afterwards — es is no longer gated now that DreamDict has Spanish data (2026-07-26). pt-PT still goes first: it's the pair with a real user behind it, and it's the one that proves the langpack + both-dictionaries model.
Phase 20 left this ready: `dict.db` on the VPS now holds all five languages, and pt-PT gloss coverage of common English words is 62%.
**Code half built 2026-07-27** (user: "let's continue the build plan"; scope confirmed: code only, no VPS work; the pack written but flagged unreviewed). Not deployed — no migration, so it is a rebuild whenever the user wants it.
- [x] **pt-PT spelling dictionary — and it could not be "vendored like en-US".** nspell expands affixes *eagerly on construction*: English's ~50k stems and small rule set are fine, European Portuguese's **1,340 affix rules over 44,257 stems** are not. Measured here: ~340 MB of heap for the first 12,000 entries and no return at all after three minutes on the whole file, i.e. comfortably over a gigabyte for a browser to load a spellchecker. So `scripts/build_ptpt_dictionary.py` runs Hunspell's expansion **once at build time** — 1,039,058 surface forms, 15 MB of text, **2.66 MB gzipped**, which nspell then reads with no affix machinery at all in **842 ms / ~120 MB**. The runtime path is byte-for-byte the English one, which is the real prize. The shipped `.aff` keeps only upstream's TRY/KEY/REP/MAP, which shape *corrections* rather than membership — so "telemovel" still corrects to "telemóvel" and "cao" still knows about "ção".
- [x] **npm's `dictionary-pt` is not European Portuguese.** Both it and `dictionary-pt-br` package VERO (*Verificador Ortográfico Livre*, Brasil) — the obvious vendoring step would have shipped Brazilian spellings under a pt-PT label, which is the §3 drift risk arriving through the packaging rather than through the model. The real source is Projecto Natura's (Universidade do Minho), which LibreOffice ships and Debian packages as `hunspell-pt-pt`; its aff declares `LANG pt_PT`. The build script **asserts the fault lines before it writes anything**: accepts `receção`, `húmido`, `telemóvel`, `autocarro`, `comboio`, `ótimo`, `pensámos`, `escrevêssemos`; rejects `recepção`, `úmido`, `ônibus`, `óptimo`. A source that fails those is not the dictionary it is for.
- [x] **Both-dictionaries spellcheck** (`useSpellChecker`): English always loads; her language loads when a pair ships one; a token is flagged only when **every** loaded dictionary rejects it. Correction pills **interleave** the two rather than concatenating — otherwise English fills all five and a misspelt Portuguese word gets no Portuguese suggestion, which is the one case the second dictionary was loaded for. No dictionary at all accepts everything: a failed fetch must not underline the whole document.
- [x] **The tokenizer had to become a property of the checker, not a constant.** `[A-Za-z]` cuts "coração" into "cora" and "o", both short enough that `isCheckable` discards them — so the word was silently never checked *and* a right-click would have offered a definition of "cora". The wide alphabet is `[A-Za-zÀ-ÖØ-öø-ÿ]`, deliberately skipping × and ÷, which hide inside that Latin-1 range. It stays **off** for a writer with no Latin second language: widening it there can only find new words to underline (the *café* and *naïve* she borrows) and no mistake she actually made. `wordAt` takes the same flag, so the underline and every lookup agree.
- [x] **Gloss/WordCard both directions** — new `lexicon.Reverse` on the lookup and a `reverse` line on the hover tip. A Latin pair has no script boundary: *data*, *sale*, *comum*, *tarde* and *ali* are real words on both sides, and there is no honest way to look at one in a mixed document and know which was meant. Petal asks both directions and shows whatever answers — no detector, so it cannot be wrong about her writing, and for a learner the collision is the interesting part. The English de-inflection walk is deliberately **not** applied in reverse: `candidates` knows -s/-ed/-ing, and running it over Portuguese would be right by accident and wrong by rule.
- [x] Prompts pinned to **European Portuguese, never pt-BR** — already done in Phase 19 (`internal/llm/lang.go` spells it out inside the prompt, with *porquê* carried alongside so the tutor recognises her question).
- [x] pt-PT langpack written (`web/src/i18n/packs/pt-PT.ts`), and pt-PT is now a real switch rather than a fallback. Post-Acordo spellings with the European lexicon (*ficheiro*, *ecrã*, *guardar*, *sinónimo*, *académico*, *Iniciar sessão*), *estás a escrever* rather than the gerund, and second-person *tu* — a companion in a private notebook, not a form. A test greps the built pack for Brazilian forms, because that is exactly the error nobody reviewing the diff can see.
- [ ] ⚠️ **The pack is NOT reviewed by a pt-PT speaker** — SUGGESTIONS §3's own bar, and the one item here I cannot meet. Flagged at the top of the file and left unchecked deliberately; expect a speaker to change the register before the vocabulary.
- [x] Companion tips/cheers/bedtime lines in the pt-PT pack. Not a translation of the zh pack: the bedtime proverbs are Portuguese ones and there is a false-friends tip the Mandarin pair had no use for. The English wit in the bedtime lines is the user's own and is kept word for word across packs.
- [x] **Piper pt-PT voice on parodia** ✅ (2026-07-27) — `piper-pt` sidecar, fourth service off the one image. Two things had to change first. (1) **A language stopped being a code change**: the handler knew exactly two, named in the Config struct, so Petal now *discovers* its instances from the environment — English on the unsuffixed `TTS_ENDPOINT`/`TTS_VOICE_EN`, everything else on a `TTS_ENDPOINT_<LANG>`/`TTS_VOICE_<LANG>` pair, base tag only (an env var name can't hold pt-PT's hyphen, and one Portuguese model is loaded either way). Half a configuration is dropped rather than routed, so it reads to the client as "use Web Speech" rather than erroring on every tap. fr and es are now a compose service and two `.env` lines. (2) The startup line names the voices it *resolved* (`en=… pt=… zh=…`), the same lesson as the dictionary line.
- [x] **`pt_PT-tugão-medium` is the only European voice Piper ships** — the other five `pt_*` models are Brazilian, so the default anyone reaches for is the wrong country: `dictionary-pt`'s trap again, arriving through the catalogue instead of the model. And it does not download: `piper.download_voices` pastes the voice name into the request line, `http.client` encodes that ASCII, and it dies on the *ã* before a byte leaves the container — precisely and only on the voice the pt-PT pair needs. The entrypoint now falls back to fetching the model and its config itself with the path percent-encoded, which is all the downloader was missing.
- [x] **Slow replay** (SUGGESTIONS §5e) — `slow: true` on `/api/tts` raises `length_scale` to ~4/3 (≈0.75× pace); Piper stretches durations rather than resampling, so it stays a voice. The pace is **part of the cache key**: without it the slow replay of a word already heard at normal speed is served back at normal speed, which is the one request where the difference is the whole point. 🐢 beside 🔊 on the word card, the selection bubble and the garden flashcard; the Web Speech fallback slows too, so the button means the same thing when Piper is down.
- [x] **L1 voice** — the `alsoIn` block speaks in the pair's locale, which the **pack names** (`locale`) rather than anything inferring it from the letters. "comum" is spelled identically in both halves; the component that knows it is rendering her language says so, exactly as the both-directions gloss avoids a detector.
- [x] Acceptance ✅ (2026-07-27), with one part that cannot be met from here. **Verified on the box against the real 550 MB `dict.db`** (throwaway DB on :8091, auth off, real Piper sidecars): the pt-PT gloss path (*think* → **pensar**; achar; lembrar — Phase 20's sense-agreement ordering holding on real data, not just the fixture), and **the first real collision lookups** — *data* → "date / Indicação da época…", *comum* → "common; usual", *tarde* → "evening; afternoon", *ali* → "there", while *think*, *computer* and *garden* correctly carry **no** reverse block. Read-aloud: pt-PT/en-US × normal/slow all 200 with the slow clips ~27% longer and five distinct cache entries; zh unchanged; an unconfigured language (fr) still 404s. **zh-pair user sees zero change**: flipped back, the popover is byte-for-byte ECDICT again (gloss, phonetic, no reverse). Her live data untouched throughout — 8 documents, 33 versions, 103 suggestions, FTS matching, integrity ok, `schema_migrations` still at 11.
- [ ] ⚠️ **No pt-PT account exists yet.** Both accounts are on the zh pair, so nothing she sees changed today; what shipped is the capacity. The browser half of the pt-PT experience (the 2.66 MB dictionary inflating in a real tab, the wide alphabet, the pills interleaving) is covered by unit tests and by the assets being served — 577 B aff, 2,661,813 B gz over public HTTPS — but not by a human in a browser signed into a pt-PT account. That and the native-speaker review are what Phase 21 still owes.
- Tests: `internal/lexicon/dreamdict_test.go` gains a real collision in the fixture (*data*: English facts, Portuguese date) — both readings on a collision, **no** reverse block for an English-only word, the tooltip carrying only the reverse gloss, and the embedded/glossless providers staying silent (a Chinese reading of an English word is worse than none). Frontend: `spellchecker.test.ts` (either-accepts, flag-only-if-both-reject, a Portuguese word never flagged for being unknown to English, no-dictionary-accepts-everything, a dictionary arriving *after* the checker was built, interleaved pills) and `SpellCheck.test.ts` (the narrow alphabet still cutting "coração", the wide one not, CJK never tokenized under either, × and ÷ excluded). The i18n suite now runs its shape assertions over *every* pack — a shape only the first author's pack satisfies is a coincidence, not a shape.
- **A bug the test found, not the code review**: `extendedAlphabet` was a value computed when the checker was built while `correct`/`suggest` read live. Her dictionary arrives *after* English, so the underlines would have been right while every lookup was still resolving "cora". It is a getter now.
- Verified: go build/vet/test, tsc, vite build, vitest 116/116 clean. The shipped asset loaded in a real nspell (842 ms, 139 MB, pt-PT variants correct both ways). Live smoke on a throwaway DB (:8091): both dictionary files served (577 B aff, 2,661,813 B gz), the gz inflating to 1,039,058 forms with `receção` present, and the zh word lookup unchanged. **Not verified against real data**: this laptop has no `dict.db`, so the reverse-lookup path is exercised by the fixture only — the first real pt-PT collision lookup happens on the VPS.
### Phase 22 — Learning loop + code-first layers ✅ (2026-07-27) — the last phase of the plan
Each item independent and small; order within is free (SUGGESTIONS §5–§6).
**First two built 2026-07-27** (user: "continue the build plan"; code only, no VPS work — not deployed, and there is no migration to undo, so it is a rebuild whenever the user wants it).
**Remaining four built 2026-07-27** (user: "let's finish the last phase of the build plan"). With them the left-hand column of the SUGGESTIONS §6 table is complete: **spell, define, gloss, pronounce, catch the common mistakes, review vocabulary, prove authorship — every daily-writing need now works on a box with the tunnel down.** The model adds depth and conversation when it is reachable and holds nothing hostage when it isn't. Carries one migration (`0013_suggestion_source`), so unlike the earlier code-only sessions this is a deploy rather than a rebuild.
- [x] **Growth journal** (Q3 settled) ✅ (2026-07-27) — `GET /api/suggestions/growth`, a read-side view of a table Petal already keeps: no new capture, no model call, nothing leaves the box. Three signals, and the work was in deciding which ones are *honest* rather than in computing them.
- **Kept** — edits she took on board in the last 30 days, with the 30 before it offered flat beside it. That second number is the whole of the self-comparison rule: there is no target, no average and no other account anywhere in these queries.
- **Stuck** — accepted phrasing that now appears in **two or more** of her own documents. One document is not evidence: it is the edit itself, still sitting where it was applied. The second is her reaching for the phrase on her own, which is the only thing the line actually claims. Candidate phrases are filtered through `vocab.PhraseKey`, the *same* definition of "a learnable chunk" the garden plants, so the journal and the garden can never disagree about what counts.
- **Faded** — a pattern corrected ≥2× in the earlier window and not since. **Guarded by "has she written lately?"**: without that check, a month away from Petal is reported back to her as progress, which is the one way this feature could lie. Test named for the guard, not the query.
- **The dates had to come from her decision, not the model's proposal** — migration `0012_suggestion_resolved_at`. `created_at` is when a checkpoint *offered* an edit; a suggestion offered in April and accepted in June is June's growth. Existing rows backfill to `created_at`, which is exactly the approximation the journal would otherwise have had to make (and is very nearly right — edits are settled minutes after a checkpoint); pending rows keep NULL, because nothing has been decided. Tested against a database rewound to before the column, since that is the only shape the live box will ever present.
- **Surface**: a second tab *inside* the garden (🌷 Garden / 🌱 Growth) rather than new chrome — same idea seen twice, the garden as objects and the journal as change over time. A review session hides the tabs: mid-flashcard is no moment to be offered a different page.
- **Feeds the companion**, which was the point: on an accept the kitten prefers a line that is true of *her* ("you're using 'make a decision' on your own now! 🌱") over one that would fit anybody — half the time, so it stays a surprise, once per line per session, so personal praise never becomes wallpaper. The journal is fetched on the first accept and **never awaited**: the cheer goes out now, personal or not.
- Copy is bound by the same two rules as the SQL, and a test greps both packs for *error/mistake/wrong/streak/average/erro/errada/错误* — the framing is the feature, and it's the part a future edit would quietly undo.
- [x] **Plant accepted collocations** in the vocabulary garden as phrase cards ✅ (2026-07-27) — scheduler untouched, as predicted: `vocab.Plant` writes the same row `capture` does, so a three-word chunk climbs the SM-2-lite ladder exactly like a looked-up word, blossoms with `reps`, and cloze-blanks in review. The garden now holds both halves of learning — what she sought out, and what she was gently given.
- **Only collocations.** The other families fix *this* sentence (a comma, "their"→"there"); a collocation is the one that hands over something reusable, and reusable is the only thing worth reviewing in a week.
- **What isn't a chunk**: `PhraseKey` rejects single words (that's word choice, and lookup already gardens it), anything over 6 words or 60 runes (a rewritten sentence wearing a collocation's label makes a miserable flashcard), and digit/symbol-only text. The cap counts **runes** — a byte cap would drop Portuguese chunks for being accented.
- **The example is the *corrected* sentence.** The stored `content_text` is still the pre-accept draft (the client applies the replacement in the editor), so the sentence around `original` is extracted and swapped server-side. Otherwise the flashcard would quiz her on the phrasing she had just left behind.
- **ON CONFLICT DO NOTHING**, unlike capture's refresh-the-context upsert. Accepting the same collocation again months later is evidence the chunk is still being learned; the worst possible response is to overwrite its first context and reset a schedule it has been climbing. Test asserts the card keeps `interval_days = 7`.
- **Best-effort, always.** Planting runs after the status write and swallows its own errors: accepting an edit is what she asked for, and it must not fail — or feel slower — because a flashcard couldn't be made. A rewrite too long to plant still returns 204.
- Verified live on a throwaway DB (:8099, no dictionary, no LLM): accept → card `make a decision` with example *"I had to make a decision about the job."* bounded to its own sentence, then the journal reporting `kept:1`, `stuck:[{make a decision, docs:2}]` once the phrase appeared in a second document, and a seeded two-month-old pattern surfacing under `faded`.
- Tests: `internal/vocab/plant_test.go` (PhraseKey table incl. rune-vs-byte, plant-once, unplantable is a silent no-op), `internal/suggestions/plant_test.go` (corrected-sentence example, only-collocations, idempotent-and-never-resets, sentence-rewrite skipped without failing the accept), `internal/suggestions/growth_test.go` (both windows, stuck needs a second document, the wrote-recently guard, still-happening excluded, and a per-writer isolation test seeding bob), `internal/db/db_test.go` (the backfill). Frontend: `journalCheers.test.ts` (silent before the fetch lands, once per line, one fetch however often warmed, silent on failure, pack resolved at call time) plus journal assertions in `i18n.test.ts`.
- Verified: go build/vet, `go test ./internal/...` clean, tsc, vite build, vitest 131/131.
- ✅ **Deployed 2026-07-27** with the rest of Phase 22 (migrations 0012+0013). ⚠️ Still not seen in a browser, and the pt-PT journal copy is part of the pack a native speaker has not reviewed.
- [x] **Daily writing invitation** from the companion ✅ (2026-07-27) — offered to a *blank page* about a minute into a session, at most once a day. Petal always has a document open, so "a session that starts with no doc open" became "the page in front of her is still empty", which is the state the invitation was actually for.
- **The stored value is a date, and that is the entire mechanism.** No count, no run of days, nothing that degrades with absence: coming back after a month reads exactly like coming back tomorrow. That is the one property this feature could lose silently, so the rule lives in its own file (`invitation.ts`) rather than inside the heartbeat, and the test names it — *treats a month away the same as a day away*.
- **Both answers spend the day's invitation.** Being asked again after "not today" would make no into a negotiation. Declining costs a sleepy `好吧，我继续睡 😴` and nothing else; letting the bubble time out is a third way of saying no.
- **Accepting titles the blank page with the prompt**, so the question she agreed to answer is still in front of her once the bubble has gone.
- Copy is bound the way the journal's is: a test greps both packs for *streak / in a row / every day / missed / 连续 / 打卡 / todos os dias* — the framing is the feature.
- [x] **False-friend list** per pair ✅ (2026-07-27) — ~19 curated en↔pt entries in the pt-PT pack; **zh has none, and that is the honest answer**, not an unwritten one: the trap needs a shared script to spring.
- **Never a correction.** Two surfaces, both heads-up only: a lavender block at the top of the WordCard (above the definition — it is the thing she would not think to check), and at most one companion note per pass. No `fix`, so it never becomes a card. *Actually* may well be the word she meant; the flag says what the English one means and stops. A test greps the entries for *wrong / mistake / errado* — this is the mistake that makes a learner feel foolish, and the tone is the whole point.
- [x] **Embedded miscollocation list** ✅ (2026-07-27) — the do/make, say/tell, heavy-rain families as ten curated patterns, and **they file as `collocation`, not as a new family**. Same rail, same warm phrasing, and — the reason it matters — an accepted chunk plants in the vocabulary garden exactly as the coach's would. The writer never learns which engine spoke.
- **That forced a schema change**: `type` had been doubling as the answer to "which engine found this" (`mechanics` meant offline). The moment an offline rule proposes a collocation that breaks — so migration `0013_suggestion_source` adds `source` (llm | local) and every pass now scopes its DELETE by engine. Without it the coach silently wiped every offline chunk on the page, and the offline pass left the coach's rows to accumulate. Both directions are tested; existing rows backfill by type, and a pre-0013 collocation row is correctly claimed as the coach's, since the offline list did not exist yet.
- **The span tiebreak moved with it**: an exact offline card beats an overlapping LLM one by *source*, not by type — an offline miscollocation is as exact as an offline comma.
- Replacements agree with the tense she wrote in (`did a mistake` → `made a mistake`), and a rule never proposes a phrase identical to what she already wrote.
- [x] **Grammar lite** rule-pack ✅ (2026-07-27) — the deterministic `mechanics` family already *was* the fourth family (Phase 8), so this was the rule pack it had been waiting for rather than new plumbing: preposition pairs, doubled comparatives, `people is`, and per-pair L1 interference. All client-side, instant, no debounce, no rate limit, alive on a VPN-down box.
- **Sourcing decision (SUGGESTIONS Q6): hand-curated, not mined.** LanguageTool's corpus is broad because it aims at recall; this pack aims at the opposite. Every entry here is a pairing that is wrong in essentially *all* contexts, and the ones that are only usually wrong were left out on purpose — `married with` is a mistake until "married with children", `arrive to` wants at or in depending on the noun, `different than` is ordinary American English. Each rule is tested in both directions, and the guard cases are the correct English sitting next to the mistake.
- **L1 rules are gated by pair, and the gating is what lets them be confident**: a near-certainty for a Portuguese speaker is only a guess for anybody else. pt/fr/es get *ter 30 anos* → "I am 30 years old" (subject and tense carried into the correction), "I am agree", "since three years" → "for three years". zh gets 很喜欢 → "very like", 开灯 → "open the light", and 虽然…但是 → "although … but".
- **The zh rules the plan named and this pack does not implement**: dropped articles and he/she slips. Neither is detectable from text alone — "She said he was late" is a perfect sentence whichever pronoun was meant — and flagging them would mean correcting correct writing, which is the one thing a rule pack running on every keystroke must not do. Said in a comment where the rules are, not only here.
- Verified live on a throwaway DB (:8099, no dictionary, **no LLM configured at all**): an offline `did a mistake` → card → accept → garden card *made a mistake* with the example bounded to its own corrected sentence, and the journal reporting `kept:1`.
- Tests: `grammarLite.test.ts` (30, every rule in both directions), `invitation.test.ts` (7), `offline_test.go` (the six engine-split cases), `db_test.go` (the 0013 backfill), plus false-friend shape/tone guards in `i18n.test.ts`.
- ✅ **Deployed 2026-07-27.** ⚠️ Still not seen in a browser; the pt-PT copy added here is part of the pack a native speaker has not reviewed.
### Phase 23 — Choosing her own pair (2026-07-27)
Raised by the user, not by the plan: *"I see no way to change my language in the mobile UI."* She was right, and the gap was total — `users.pair_lang` had been readable since Phase 19 and writable by nobody. `/api/me` was GET-only, `Upsert` deliberately skips the column, and no screen anywhere offered the choice. Phases 19–21 built the machinery for a second pair and then left the switch off the wall, which is why ⚠️ *"no pt-PT account exists yet"* stood unresolved through two phases: **nothing could create one.**
- [x] **`PATCH /api/me`** (`auth.UpdateMeHandler`) — answers with the whole updated user rather than 204, so the client re-reads the pair from the server instead of assuming its own request took. One write reaches everything: the langpack, the Hunspell dictionary, the Piper voice, the lexicon provider and the prompt language all read `users.pair_lang` at use time.
- [x] **The server refuses a pair it has no copy for.** `auth.shippedPairs` is deliberately *not* `internal/llm`'s language list — that one names every pair the **prompts** can talk about (cheap to add; fr and es have been in it since Phase 19), this one names every pair Petal can **render itself in**, which needs a langpack. Storing `fr` today would strand her on Chinese copy with no way back except a lucky guess at a button she cannot read.
- [x] **The picker lives in the sidebar footer**, beside her name and the way out — because the sidebar *is* the mobile drawer, and it is the only chrome that is always one tap away on a phone. The status bar was the other candidate and is wrong: it exists only while a document is open, which is exactly the wrong moment to discover the app is speaking a language you can't read.
- [x] **Each language names itself** — 中文, Português, and nothing else. The one place in Petal where bilingual copy would actively get in the way: a writer who has landed on the wrong pair cannot read "Portuguese" written in Chinese. The `aria-label` carries the English for a screen reader, which has no such problem.
- [x] **No reload.** The pack was already a subscription (Phase 19), and `useSpellChecker` already reloads on `pack.code` while read-aloud already reads `pack().locale` — so the 2.66 MB pt-PT dictionary inflates, the wide alphabet turns on and the voice changes on the tap. Nothing here needed new plumbing; the switch is the only part that was missing.
- [x] Tests: `internal/auth/pairlang_test.go` (round-trip and back again — a writer who tries a pair must be able to return; every unshipped code refused with the column unmoved; 400 vs 401 split so a lapsed session still becomes the sign-in overlay). `i18n.test.ts` asserts `shippedPacks()` offers exactly the pairs that have copy, and that every code it offers actually resolves.
- Verified: go build/vet, `go test ./...` clean, tsc, vite build, vitest 173/173. ⚠️ **Not seen in a browser** — no Chrome extension on this laptop, and the picker's appearance in a real mobile drawer is exactly the part unit tests cannot cover.
**Deployed 2026-07-27** (`1f4ca47`), and it carried Phase 22 with it — the two could not be separated in the tree, so the user chose to ship both. Pre-deploy snapshot `data/backups/petal-pre-phase22-20260727T220410Z.db` (VACUUM INTO against the live app). On the box: migrations 0012 and 0013 applied, `source` backfilled **llm=100 / local=3** — the three are her pre-existing `mechanics` rows, claimed by type exactly as the migration intended. Her writing came through untouched: 9 documents, 33 versions, 103 suggestions, `integrity_check` ok. All four containers healthy, read-aloud still resolving `en/pt/zh`, dictionary still open on all five languages. `PATCH /api/me` answers **401 without a session** rather than 405, which is the only half of it a signed-out probe can prove — the route is mounted and behind auth.
- **All three accounts are still on `zh`, deliberately.** Flipping her pair is hers to do now that the button exists, and doing it for her from a shell is precisely the change this phase was built to stop needing.
### Phase 24 — the fr pair ✅ (2026-07-27, code half) — and what the plan got wrong about it
Scope agreed with the user 2026-07-27: *"switcher for Chinese and Portuguese now, plan support for others in a later session or two."* Then, this session: **French end to end, code only, deploy its own step** — es follows as a repeat of a proven groove rather than two half-finished pairs. Phase 21 was supposed to be the groove and mostly was; the exception is item 3, which the plan had recorded as solved and was not.
1. [x] **The langpack** (`web/src/i18n/packs/fr.ts`, 470 lines). Metropolitan French, tutoiement, *se connecter* rather than *login* — and the regional decision lives **entirely here**, unlike pt: Debian's `fr_FR`, `fr_CA`, `fr_BE`, `fr_CH`, `fr_LU` and `fr_MC` are all symlinks to one word list, so there is no dictionary to get wrong and nothing but the copy to get right. A vitest greps the built pack for *courriel*, *clavarder*, *magasiner* and *fin de semaine*, exactly as the pt-PT one greps for Brazilian forms — the error nobody reviewing the diff can see. The pack punctuates the way French does (« guillemets », a space before ! ? : ;), which is *also* the habit `prose.spaceBeforePunct` warns her about in her English: the copy demonstrates the rule its own prose note tells her not to carry across. An ordinary space, not U+202F — a narrow no-break space is invisible in a diff and the next pack author would strip it by accident.
2. [~] **Reviewed by a quorum of models, not by a native speaker** (2026-07-27, user: "perhaps for now, we could leverage multiple LLMs to act as reviewers… accept the responses that have the most alignment amongst them" — explicitly an interim measure). Four models read each pack independently as native speakers, blind to one another, returning verbatim substrings so agreement could be counted mechanically rather than judged. **Threshold: a finding is applied only if ≥2 of 4 reached it on their own.** Eight reviews, 32 findings, 12 above threshold, 10 applied.
- **fr, applied**: `Fatiguée, on écrit mal` (**4/4**), `Clique droit sur un mot anglais` → *Fais un clic droit* (3/4), `je me suis emmêlée` (3/4), `touche pour changer` → *appuie* (2/4), `laisse une espace` → *un espace* (2/4).
- **pt-PT, applied**: `adjectivos` → *adjetivos* (3/4), `actualmente` → *atualmente* (3/4), `decepção` → *deceção* (2/4), `Cão abanão` → *Cão abana-rabo* (2/4), `Ouves? Pois não…` → *Pois não ouves…* (2/4).
- **Where the reviewers agreed a line was wrong but not on the fix**, the wording is mine and the reasoning is written down rather than averaged: `Fatiguée, on écrit mal` split 2–2 between keeping *on* and switching to *tu*, and *both* camps' stated objection (feminine agreement with impersonal *on*) survives the *on* wording — so **Quand tu es fatiguée, tu écris mal** is the only candidate that answers every reviewer, and it matches the pack's own tutoiement. `je me suis emmêlée` drew three different fixes; *emmêlé les pinceaux* is the actual idiom and makes the participle invariable, which also settles the fourth reviewer's point that the companion is a *chat* and therefore masculine.
- **The finding that justifies the exercise**: pt-PT was carrying **pre-Acordo spellings** — *adjectivos*, *actualmente* — in direct contradiction of its own header, plus Brazilian *decepção*. Phase 21's vitest greps the pack for Brazilian *vocabulary* and never checked the pack against its own stated *spelling policy*, so this had been shipped and reviewed and was still invisible. The grep now covers nine pre-Acordo forms, and was confirmed to fail on the old text before being kept.
- **Below threshold, deliberately not applied** (1/4 each): *very* also modifies adverbs, so `veryBeforeVerb` is incomplete rather than false — and the rule that renders it only runs for the zh pair anyway; `aide à lire`; `et toi aussi tu devrais`; `Cansada não se escreve bem`; `Já vais em`; `está toda a gente a dormir`; the `longSentence` infinitive; `breaks[1]`.
- ⚠️ **Still not a native speaker.** A quorum of models agreeing is agreement, not authority: it caught a *clique droit* that is not French and an adjective disagreeing with *on*; it cannot catch a line that is correct and lifeless. The ⚠️ at the top of both packs now says which review happened rather than none.
3. [x] **Hunspell dictionary — and "the pt-PT script generalizes" was wrong.** It handled single-character flags and plain PFX/SFX and *stopped* on anything else, which was the right call and not a generalization: `fr.aff` uses four of the things it stopped on, and every one changes which words are accepted. **`FLAG long`** — French flags are two characters (`S.`, `L'`, `Um`), so `set(flagstr)` yields a bag of unrelated letters and expands every entry through the wrong paradigm; this is the one that fails silently. **Continuation flags** — French really does affix an affixed form (`PFX Um 0 0/S.`), which pt-PT dropped after asserting it was safe to. **`NEEDAFFIX`** on 68,075 of 84,140 stems, the bare form arriving through a zero-append rule. **`FULLSTRIP`**. `CIRCUMFIX` and `FORBIDDENWORD` are declared-but-unused and the script now *asserts* that rather than assuming it. Renamed `scripts/build_hunspell_dictionary.py` with a per-language profile; **the pt-PT rebuild is byte-identical to the shipped asset**, which is what says the generalization did not change the pair that already worked.
- **Which of three, not which of six.** fr is packaged by how it treats the 1990 reform: `-classical`, `-revised`, `-comprehensive`. Petal ships **comprehensive**, because Petal never corrects her French — the only thing this dictionary can do is underline something, and *coût* and *cout* are both correct French taught in different decades. The MUST_ACCEPT list *proves* which package was used: classical rejects `cout`, revised rejects `coût`, only comprehensive accepts both.
- **Elision is the size decision, and it moved out of the dictionary.** Both halves were built and measured: keeping the elided forms is **3,159,832 forms / 8.25 MB gzipped**; dropping them is **473,326 / 1.19 MB**. They are not new words — thirteen clitics glued to words already in the list — but the tokenizer keeps internal apostrophes, so `l'arbre` really does arrive whole and really would have been underlined. `withElision` splits at a *known* clitic and checks the remainder: `l'arbre` costs one extra hash probe instead of seven megabytes, `zzz'arbre` is still flagged because zzz is not a word French elides, and `l'zzzz` is still flagged because the remainder must itself be a word. Stems carrying their own apostrophe (`aujourd'hui`, `quelqu'un`, `presqu'île`) are kept verbatim and match directly; `entr'aide` is absent for the same reason Dicollecte omits it.
- Loaded in a real nspell: **369 ms, 74 MB** for 473,326 forms — cheaper than pt-PT's 842 ms / 139 MB, on a bigger language. Suggestions do the thing an ESL writer needs most: `ecrire` → *écrire*, `francais` → *français*.
4. [x] **Piper voice** — `piper-fr`, a fourth sidecar off the same image, plus `TTS_ENDPOINT_FR` and `TTS_VOICE_FR`. No Go at all, which is Phase 21's discovery holding: a language is configuration now. The exact opposite of pt's trap — every `fr_*` voice in the catalogue is `fr_FR`, so there is no wrong country to default to, and `fr_FR-siwis-medium` is chosen to match the register of the other three rather than to avoid anything. ASCII, so the percent-encoded download fallback `tugão` needed never fires.
5. [x] **Lexicon coverage** — already measured, and better than the pair that shipped: the Phase 20 rebuild put fr at **63.1%** of the 2,000 commonest English words against pt-PT's 62.1%. Both directions answer with no code change; `lexicon.Set.For` routes every non-Chinese pair to DreamDict already.
- Free, because Phase 19 and 22 did them: `internal/llm/lang.go` already carries fr, `grammarLite`'s L1 rules already gate *ter 30 anos* / "I am agree" / "since three years" to pt+fr+es, and the sidebar picker derives itself from the shipped packs — so the switch offering **Français** is not a line of new UI.
- Tests: `i18n.test.ts` (the Québécois grep; French spacing and guillemets kept in the copy; agreement in the interpolated lines — *1 fleur* / *3 fleurs*, *1 chose retenue* / *5 choses retenues*; the fr false friends *attend* and *pass*, which pt-PT has no use for). `spellchecker.test.ts` gains seven elision cases including both directions of the flag-it/don't rule. `pairlang_test.go` now round-trips **every** shipped pair rather than the first one — the Go allowlist and the frontend's PACKS are two copies of one fact — and its unshipped examples moved to `es`/`fr-CA`. `config_test.go` discovers a fourth voice.
- Verified: go build/vet, `go test ./...` clean, tsc, vite build, vitest 190/190 (33 in the i18n suite after the review pass). **Not seen in a browser** — no Chrome extension on this laptop; the 1.19 MB dictionary inflating in a real tab and the picker's third entry in a real mobile drawer are what unit tests cannot cover.
- **Not deployed.** No migration, so it is a rebuild whenever the user wants it; the Piper sidecar wants `docker compose up -d piper-fr` and a voice download on the box.
### Phase 25 (planned) — the es pair
Everything above, minus the surprises: item 3's expander now handles what Spanish's `es_ES.aff` is likely to need (single-char flags, no compounding), so the work is the langpack, a native review, `hunspell-es` (packaged per country — check what `es_ES` actually is before vendoring), a `piper-es` service with `es_ES-davefx-medium`, and nothing at all for the lexicon: es is the *best*-covered pair in `dict.db` at 68.6%.
### Later / explicitly not now
- Learner-facing Chinese writing (the zh pair's second direction) — own phase with its own spec (SUGGESTIONS §4); only after Phases 19–21 prove the pair model
- ~~Spanish pair — gated on DreamDict growing an es dataset~~ **ungated 2026-07-26** (DreamDict added Spanish). Now a normal follow-on pair after pt-PT and fr — see Phase 25.
- Reactive-animation puppy companion — wishlist, low priority; `companions.ts` roster + mood engine is the drop-in point
- Copyleaks Tier-2 — revisit once Phase 15 provides a public webhook endpoint
### Next-up (post-v1 product, agreed with user 2026-06-26)
- [x] **Phase 9 — ESL superpowers**: inline Chinese gloss on hover/select; "say it more naturally" / tone-rewrite. ✅ (see Phase 9 above)
@@ -147,6 +347,20 @@ Multi-session build. **Source of truth for what's done and what's next.** Update
- [x] **Phase 14 — companion warmth + bedtime nag + night mode**: more encouraging phrases, a gentle "go to bed" nudge after 11pm, and a calm dark theme + falling stars at night. ✅ (see Phase 14 above)
## Session log
- 2026-07-27: **Phase 24 — the fr pair, and a "generalizes" that did not** (user: "resume the build plan"; scope chosen with the user: French end to end, code only, deploy its own step). The plan's five items were meant to be mechanical, and four of them were — the Piper voice is a compose service and two env lines because Phase 21 made a language configuration; the lexicon needed nothing at all, fr having been measured at 63.1% during Phase 20's rebuild, better than the pair that already shipped; the sidebar picker grew a third entry without a line of UI because it derives itself from the shipped packs. **Item 3 was the one that had been recorded as done and wasn't.** `build_ptpt_dictionary.py` was said to generalize; it handled single-character flags and plain PFX/SFX and stopped on everything else, and `fr.aff` uses four of the things it stopped on. `FLAG long` is the dangerous one: French flags are two characters, so the pt-PT reader's `set(flagstr)` yields a bag of unrelated letters and expands every entry through the wrong paradigm without erroring. Plus continuation flags (French really does affix an affixed form), NEEDAFFIX on 68,075 of 84,140 stems, and FULLSTRIP. The rewritten `build_hunspell_dictionary.py` carries a per-language profile and asserts that CIRCUMFIX and FORBIDDENWORD are still unused — and **rebuilds pt-PT byte-identical to the shipped asset**, which is the only thing that makes "generalized" a claim rather than a hope. **The second decision was elision, and it was made by measuring both halves**: keeping `l'arbre` and its thirty-three siblings costs 3,159,832 forms and 8.25 MB gzipped; dropping them costs 473,326 and 1.19 MB. They are not new words, but the tokenizer keeps internal apostrophes, so they really would have been underlined — so they moved out of the dictionary and into `withElision`, which splits at a known clitic and still requires the remainder to be a word (`l'zzzz` stays flagged). Real nspell: 369 ms and 74 MB for the larger language, against pt-PT's 842 ms and 139 MB. **Where the regional trap lives is the mirror image of Portuguese's**: every `fr_*` Piper voice is fr_FR and every Debian fr dictionary is one shared word list, so nothing can be quietly wrong about the country — the whole decision is in the copy, which is why the pack is greped for *courriel* and *magasiner* the way pt-PT is greped for *arquivo*. What French does have instead is the 1990 reform, packaged three ways; Petal ships comprehensive, because Petal never corrects her French and *coût* and *cout* are both correct. go build/vet/test, tsc, vite, vitest 190/190. **Two things owed and both said plainly**: no native speaker has read the pack (SUGGESTIONS §3's bar, unmet for pt-PT too), and nothing here has been seen in a browser. **Then, same session, an interim answer to the first of those** (user: "perhaps for now, we could leverage multiple LLMs to act as reviewers?"): four models reviewed each Latin pack independently, and only findings ≥2 of them reached on their own were applied — five per pack. It earned its keep on the pack that was *already shipped*: pt-PT had **pre-Acordo spellings in a file whose own header commits to post-Acordo**, because the Phase 21 greps checked for Brazilian vocabulary and never checked the pack against its own spelling policy. That grep now exists and was confirmed to fail on the old text. Where reviewers agreed a line was wrong but split on the fix, the wording is mine and the reasoning is in the phase entry rather than averaged away. Still not a native speaker, and both packs now say so precisely.
- 2026-07-27: **Phase 22 finished — the build plan's last four items, and the LLM stops holding anything hostage** (user: "let's finish the last phase of the build plan"; code only, no VPS work). The four remaining items shared one theme, and it only became visible while building them: **§6's left-hand column is now complete.** Spell, define, gloss, pronounce, catch the common mistakes, review vocabulary, prove authorship — every daily-writing need works with the tunnel down. **The plan asked for "grammar lite as a fourth suggestion family", and the fourth family already existed**: Phase 8's deterministic `mechanics` pass was the plumbing, so this was the rule pack it had been waiting for rather than new machinery — preposition pairs, doubled comparatives, `people is`, plus per-pair L1 interference. **Q6 answered by hand-curating rather than mining LanguageTool**: that corpus is broad because it aims at recall, and this pack aims at the exact opposite, so every entry is a pairing wrong in essentially *all* contexts and the ones only *usually* wrong were left out on purpose — `married with` is a mistake until "married with children", `arrive to` wants at or in depending on the noun, `different than` is ordinary American English. Each rule is pinned in both directions, the guard case being the correct English next to the mistake. **The L1 rules are gated by pair, and the gating is what earns them their confidence** — *ter 30 anos* → "I am 30 years old" is a near-certainty for a Portuguese writer and only a guess for anyone else. The two zh rules the plan itself named are the ones this pack **refuses** to implement: dropped articles and he/she slips are not detectable from text alone ("She said he was late" is perfect whichever pronoun was meant), and flagging them would mean correcting correct writing. **The miscollocation list forced the session's one real design change.** It had to file as `collocation` rather than as its own family — same rail, same phrasing, and an accepted chunk plants in the garden exactly as the coach's would — but `type` had been quietly doubling as the answer to *which engine found this*, and that breaks the instant an offline rule proposes a collocation. Migration `0013_suggestion_source` splits the two apart: each pass now scopes its DELETE by engine, and the span tiebreak moved with it (an exact offline card beats an overlapping LLM one by source, not by type — an offline miscollocation is as exact as an offline comma). Without it the coach silently wiped every offline chunk on the page and the offline pass left the coach's rows to pile up; both directions are now tested, and a pre-0013 collocation row correctly backfills to the coach, since the offline list did not exist yet. **The daily invitation's whole substance is one stored date** — no count, no run of days, nothing that gets worse for being away, so a month away reads exactly like a day away; it lives in its own file because that is the property this feature would lose silently, and the test is named for it rather than for the query. Both answers spend the day's invitation, because being asked again after "not today" would make no a negotiation. **False friends are the one thing here that never becomes a card**: ~19 curated en↔pt entries, shown as a lavender block above the WordCard's definition and as at most one companion note per pass, with no `fix` anywhere — *actually* may well be the word she meant, and this is the mistake that makes a learner feel foolish rather than merely corrected. zh has none, which is the honest answer and not an unwritten one: the trap needs a shared script. Copy for the invitation and the false friends is greped by tests the same way the journal's is (*streak / in a row / 连续 / todos os dias*; *wrong / mistake / errado*) — the framing is the feature, and it is the part a future edit would undo while meaning well. Verified: go build/vet, `go test ./internal/...` clean, tsc, vite build, vitest 172/172 (30 new rule cases, 7 invitation, plus false-friend shape/tone guards), and a live throwaway DB on :8099 with **no LLM configured at all** — offline `did a mistake` → card → accept → garden card *made a mistake*, example bounded to its own corrected sentence, journal `kept:1`. ⚠️ **Not deployed and not seen in a browser**, and this one carries a migration, so it is a deploy rather than a rebuild. The pt-PT copy added here joins the pack a native speaker still has not reviewed.
- 2026-07-27: **Phase 21 deployed — the pt-PT pair has a voice** (user: "continue the build plan"; scope chosen: deploy Phase 21 to the VPS rather than start Phase 22). The plan's remaining line was "Piper pt-PT voice instance on parodia", and it hid two things. **A language was still a code change**: read-aloud knew exactly two, named in the Config struct as `TTSEndpointZH`/`TTSVoiceZH`, so adding Portuguese meant editing Go to add Portuguese. Petal now discovers its Piper instances from the environment — English keeps the unsuffixed pair, everything else is `TTS_ENDPOINT_<LANG>`/`TTS_VOICE_<LANG>`, base tag only because an env var name cannot hold pt-PT's hyphen — and a language configured by halves is dropped rather than routed, so it reaches the client as "no voice, use Web Speech" instead of erroring on every tap. fr and es now cost a compose service and two `.env` lines. **And the voice itself repeated Phase 21's own lesson in a new place**: `pt_PT-tugão-medium` is the *only* European Portuguese voice in Piper's catalogue — the other five are Brazilian — so, exactly as with `dictionary-pt` packaging VERO, the default anyone reaches for ships the wrong country. Then it wouldn't download at all: `piper.download_voices` pastes the voice name into the HTTP request line and `http.client` encodes that as ASCII, so it dies with `UnicodeEncodeError` on the *ã* before a byte leaves the container — a failure that lands on precisely the one voice this pair needs and on no other. The entrypoint falls back to fetching the model and its config itself with the path percent-encoded, which is all the downloader was missing. **The slow replay** (§5e) went in while there: `slow: true` raises `length_scale` to ~4/3, and the pace is part of the **cache key** — without that, asking to hear slowly a word already heard at speed serves the fast clip back, which is the one request where the difference is the entire point. **The L1 voice asks the pack, not the letters**: a new `locale` field, because "comum" is spelled the same in both halves and a detector would have to guess — the same reason the gloss shows both directions. **Deploying is what finally ran the reverse lookup against real data**, the item the previous session left open because this laptop has no `dict.db`: *data* → "date", *comum* → "common; usual", *tarde* → "evening; afternoon", *ali* → "there", with *think*, *computer* and *garden* correctly silent; and *think* glossing to **pensar** first confirms Phase 20's sense-agreement ordering on the real 550 MB database rather than on a fixture. zh flipped back is byte-for-byte ECDICT again. go build/vet/test, tsc, vitest 125/125, vite; laptop smoke against two fake Pipers, then the real thing on the box. Her data untouched: 8 documents, 33 versions, 103 suggestions, FTS matching, integrity ok, `schema_migrations` still at 11 (no migration in this phase). **Two things Phase 21 still owes, both said plainly**: the pack has not been read by a pt-PT speaker, and no pt-PT account exists — both writers are on the zh pair, so nothing she sees changed today and the browser half of the Portuguese experience has never had a human in front of it.
- 2026-07-27: **Phase 21 (code half) — the pt-PT pair, and the plan's one-line assumption about the dictionary** (user: "let's continue the build plan"; scope confirmed: code only, the Piper voice and the deploy deferred, the pack written but flagged unreviewed). The plan said "Hunspell pt-PT vendored like en-US", and that turned out to be the load-bearing sentence. **nspell expands affixes eagerly on construction** — it materialises every surface form the moment you build it. English survives that; European Portuguese's 1,340 affix rules over 44,257 stems do not. Measured before deciding anything: ~340 MB of heap for the first 12,000 entries, and no return at all after three minutes on the whole file — over a gigabyte, in a browser, on a tablet. So the expansion moved to build time: `scripts/build_ptpt_dictionary.py` writes 1,039,058 forms, 2.66 MB gzipped, which the *same* nspell then reads in 842 ms using ~120 MB, and the runtime path stays byte-for-byte the English one. The `.aff` keeps only TRY/KEY/REP/MAP, which shape corrections rather than membership, so "telemovel" still corrects to "telemóvel". **A second thing the obvious route would have got wrong quietly**: npm's `dictionary-pt` is not European Portuguese — both it and `dictionary-pt-br` package VERO (Brasil), so vendoring the obvious package name ships Brazilian spellings under a pt-PT label. That is §3's pt-BR drift arriving through the *packaging* rather than through the model, and nobody reviewing the diff would see it. The real source is Projecto Natura's, packaged as `hunspell-pt-pt`; the build script now asserts the fault lines (`receção`/`húmido`/`pensámos` in, `recepção`/`úmido`/`ônibus`/`óptimo` out) before it writes a byte, and a vitest greps the built langpack for *sinônimo*, *arquivo*, *tela*, *você*. **Both-dictionaries spellcheck** landed as §3a specifies — flag only what every loaded dictionary rejects, interleave the correction pills so English can't fill all five — and dragged a smaller thing with it: the tokenizer had to become a property of the checker rather than a constant, because `[A-Za-z]` cuts "coração" into "cora", which is both silently unchecked *and* what a right-click would have looked up. The wide alphabet stays off for a writer with no Latin second language, where it could only earn her new squiggles. **Gloss both directions**: a Latin pair has no script boundary, so *data*, *sale* and *comum* are words on both sides and there is no honest way to know which she meant — Petal asks both and shows what answers, which needs no detector and therefore cannot be wrong about her writing. The reverse direction deliberately skips the English de-inflection walk, which over Portuguese would be right by accident and wrong by rule. **Writing the tests found the bug**: `extendedAlphabet` was a snapshot taken when the checker was built while `correct`/`suggest` read live — and her dictionary arrives *after* English, so the underlines would have been right while every lookup still resolved "cora". go build/vet/test, tsc, vite, vitest 116/116 clean; the shipped asset loaded in a real nspell; live smoke on a throwaway DB served both files and left the zh lookup untouched. **Two things outstanding and both said plainly**: the pack has *not* been read by a pt-PT speaker (SUGGESTIONS §3's own bar, and not one I can meet), and this laptop has no `dict.db`, so the reverse-lookup path is covered by a fixture rather than by a real collision — the first of those happens on the VPS.
- 2026-07-27: **dict.db rebuilt with Spanish, and a log line caught lying** (user: "if we need to redeploy DreamDict to add Spanish support, then do so"). Millenia's dreamdict checkout held ~490 lines of uncommitted work; rather than pull over it, comparing file contents showed an earlier draft of the regional-variant work already committed upstream — nothing unique, but not mine to discard, so it was left alone and the rebuild ran from a clean clone pushed over from the laptop (millenia has no GitHub SSH). Import took 6m15s and added **es: 102,971 words**, leaving en/fr/pt-PT/zh byte-identical — the check that distinguishes "added a language" from "quietly changed everything". Coverage measured before shipping: **es 68.6%**, the best of the four; **zh re-measured at 53.2%**, so the ECDICT decision stands on fresh evidence rather than on the earlier number. Shipped direct millenia→parodia over headscale, hashed both ends, kept the April file for rollback. **The rebuild's real find was in Petal, not DreamDict**: the startup line reported `dictionary.Langs()`, a compile-time constant of *supported* languages, so it had been printing a cheerful `[en fr pt-PT es zh]` over a database with no Spanish in it — the exact failure it existed to catch, reported as success, and something I had already claimed as proof the deploy was good. It now counts rows. Chasing a failed SUBTLEX-US download (benign — the loader falls back to `.txt`) also confirmed English "frequency" is mostly SCOWL's commonness bucket, which independently vindicates the band chip reading `difficulty` instead.
- 2026-07-27: **Phase 20 — DreamDict becomes the dictionary for every pair but Chinese** (user: "let's continue the build plan"; scope confirmed: build the seam against the existing April `dict.db`, rebuild it later, code + local verification only). The prerequisite was bigger than the plan recorded: renaming DreamDict's module path was necessary but useless on its own, because the query layer lived in `internal/dictionary` and no module may import another's `internal`. Both fixed upstream — the package is now `dictionary`, with a comment saying why *reading* a built database is public API while the loaders that build one stay internal. In Petal, `Provider` is the two questions the popover already asked, so the embedded `*Lexicon` satisfied it with no changes at all, and `Set.For(lang)` is the one place the choice is made. **The measurement is the story of the phase.** `MULTIUSER_PLAN.md` mapped `Gloss ← Translate(word, "en", L1)` 1:1; against the real 452 MB database that table answers for **17%** of the 2,000 commonest English words into pt-PT. Wiktionary's translation sections are thin in that direction — "ephemeral", "think" and "quickly" have no en→pt-PT row at all. The shared-synset path answers for **61%**, so a new upstream `Equivalents` queries that and falls back to translations for 62% combined. Then the *ordering* was wrong in an instructive way: sorting by target frequency glosses "think" as *lembrar* — "remember" — because lembrar is commoner in Portuguese, even though pensar shares six of think's synsets to lembrar's one. Counting sense agreement first fixes it (think → pensar; write → escrever; garden → jardim). The same measurement is what kept **zh on ECDICT**: DreamDict reaches a Chinese gloss for 53% of those words where ECDICT reaches nearly all — the plan said converge only if quality holds, and it didn't. Two other decisions worth keeping: a missing `dict.db` is **not an error** (a laptop has never had one) but a present-and-unimported one is; and a pt-PT writer without a dictionary falls back to the embedded datasets **with the gloss suppressed**, keeping the English half rather than blanking the popover — an empty field reads as "not found", the wrong language reads as broken. The new fields surface as **three** bands, not five, because the difficulty score can separate "everyday" from "you'll have to explain this" but cannot rank *obfuscate* against *serendipity*, and a finer scale would be a confident-looking lie. Writing the tests found two bugs first: `trimEtymology` sliced by byte, which would have emitted invalid UTF-8 for precisely the Greek and Latin etymologies the feature exists for, and its ellipsis path overran its own cap. go build/vet/test, tsc, vite, vitest 96/96 clean in both repos; live smoke on a throwaway DB against the real dictionary, one instance flipped from zh to pt-PT mid-run. **Then deployed, with Phase 19** (user: "do it"): dreamdict pushed to GitHub, the `replace` swapped for a real pseudo-version, encrypted off-box backup first, `dict.db` copied into the LUKS volume and SHA-256-verified, then a rebuild — no migration in either phase, so `schema_migrations` stayed at 11 and her writing came through untouched (8 documents, 33 versions, 103 suggestions, FTS matching, integrity ok). Both accounts are on the zh pair, so **nothing she sees changed today**; what shipped is the capacity for the next pair. Outstanding: the deployed `dict.db` predates DreamDict's Spanish data and needs rebuilding before the es pair ships.
- 2026-07-27: **Phase 18 deployed, and Phase 19 — the copy stops being hardcoded Mandarin** (user: "let's continue the build plan"; sequencing confirmed: rehearse + deploy 18, then start 19). The rehearsal the previous session was blocked from running went first: a `VACUUM INTO` snapshot of the live VPS database, migrated locally by the Phase-18 binary, every count unchanged and FTS/integrity/foreign keys clean, `personal_words` created empty — then the deploy itself (off-box encrypted backup, rebuild, all three containers healthy, `0011` applied to the live DB with her writing untouched, `/api/spell/words` 401 without a session over public HTTPS). **One check was refused and not worked around**: minting a probe session row to see the endpoint answer 200 for a real cookie reads as credential fabrication to this session's classifier; the endpoint's lifecycle is covered by tests and the shared middleware governs that last step for every other route. **Phase 19** then lifted every `中文 · English` literal out of ~29 files into `web/src/i18n` — one `Pack` type, a verbatim `zh` pack, and two access paths chosen by *when* copy is built: `usePack()` for components, `pack()` for the companion and prose checker, which compose a line when something happens rather than when something renders. The interesting decisions were about what a pack must be allowed to control: **every string with a value in it is a function** (`reviewDue(n)`, `daysAgo(n)`, even English pluralisation) because word order isn't universal; the roster constants keep only value + emoji so a label can never drift from its key; and `gradeBand` returns a band *name* rather than a label. On the server, `internal/llm/lang.go` replaces "Simplified Chinese" in the three prompts that name her language — with pt-PT spelled **"European Portuguese (pt-PT, never Brazilian Portuguese)"** in the prompt itself, and her word for "why" carried alongside so the tutor still recognises the question. `pair_lang` is read **in the row-scoped query each handler already ran**, not a second lookup that could disagree with it — and the test for that was checked by breaking the join and watching it fail. go build/vet/test, tsc, vite, vitest 90/90 clean; live smoke on a throwaway DB. **Phase 19 is not deployed** — no migration, so it's a rebuild whenever the user wants it.
- 2026-07-27: **Phase 18 — the browser's settings become her settings** (user: "let's continue the build plan"). Two scope calls taken with the user: the personal spell dictionary goes **server-side** rather than being namespaced in place, and the pre-account `localStorage` keys are **adopted then rescoped** by the first writer to sign in. New `web/src/lib/prefs.ts` namespaces `petal.sound`/`petal.petals`/`petal.companion` by user id; the interesting part is timing — those modules read their value at *import* time, before `/api/me` can possibly have answered, so a pre-scope read deliberately sees the legacy key (the right value on a single-writer browser) and `setPrefsScope`, called from `useSession`, adopts it and notifies every reader. Adoption **moves** rather than copies, so account two starts from Petal's defaults instead of inheriting a stranger's mascot. New `internal/spell` package + migration `0011`: `personal_words` keyed `(user_id, lang, word)`, where `lang` is the **dictionary's** language, not the writer's — an English exception must not silence a pt-PT flag when the second pair ships. `useSpellChecker` now replays her list from her account, hands over any browser-held Phase-7 list on first load (releasing it only once the server has taken it), and persists an added word in the background so the underline vanishes the instant she asks. **The reason for the server table over cheaper namespacing**: keying the existing list by user in `localStorage` would have *fragmented* the words she already has across her laptop and tablet — the "cheap" fix was the one that made things worse. Tests: full lifecycle + languages-don't-merge + junk + the standing-rule two-user isolation suite in Go, and legacy-adoption/move-not-copy/two-accounts/storage-throws in vitest. go build/vet/test, tsc, vite, vitest 82/82 clean; live smoke on a throwaway DB. **Not deployed** — and the customary rehearsal of `0011` against a copy of the live VPS database was blocked by the session's permission classifier, so that check is outstanding (it is a plain `CREATE TABLE`, so lower-risk than `0005`/`0010`, but the convention exists for a reason).
- 2026-07-27: **Phase 17 — Claire's writing moved onto her real account** (user: "claire is local user today in Petal. let's make sure to migrate existing data to her account"). `scripts/migrate_local_user.py`: dry-run by default, own `VACUUM INTO` backup, one transaction with foreign keys off, re-points `documents`/`tags`/`vocab_words`/`images`, verifies every expected row moved before committing. **The plan's stated prerequisite — "she logs in once so her sub exists" — turned out to be false**: authentik's `hashed_user_id` sub is `User.uid`, derived from her id and the instance secret, so it is readable in advance and the data could move *first*; she signs in to find her writing already there instead of to an empty Petal. Her 8 documents, 33 snapshots, 103 suggestions, 3 vocabulary words and 1 image now belong to `5f47d955…`, verified end to end over public HTTPS. Per the user's call the VPS is now canonical and millenia was left running and untouched as a frozen fallback (it diverges the moment either is written to — retire it rather than sync it). **Three bugs, each found by a different kind of contact with reality**: (1) the image backfill claims files for `local`, which stops existing after a migration — a foreign-key error inside `images.New`, which `main.go` treats as fatal, so Petal would have crash-looped on first start against a migrated database; (2) `BEGIN EXCLUSIVE` was the wrong liveness check, since in WAL mode it only conflicts with another *writer* and sails past a running-but-idle Petal — exactly the case the guard exists for; (3) the replacement, `PRAGMA locking_mode = EXCLUSIVE`, holds its lock past being reset to `NORMAL`, so on a real WAL database the script locked itself out of its own backup — invisible locally because the test file had come from `VACUUM INTO` and wasn't in WAL mode. Same shape as Phase 16's trailing-slash issuer: the fixture didn't look like production.
- 2026-07-27: **Phase 16 built — Petal authenticates for itself** (user: "let's continue the build plan"; box access granted mid-session). New `internal/auth` surface on top of the Phase-0 `Resolver` seam: `session.go` (opaque cookie, **SHA-256-at-rest**, 30-day sliding expiry throttled to one write an hour, revoke/revoke-all/prune), `oidc.go` (login/callback/logout with state + nonce + PKCE, **lazy retried discovery** so an IdP outage can't stop Petal booting or invalidate live sessions), `users.go` (provisioning upsert keyed on `sub`, `/api/me`, allowlist). Migration `0010` lands `sessions`, `images` and `users.pair_lang` together. `main.go` picks the resolver from config, so a laptop build is unchanged. **Image ownership** closes the capability-URL hole flagged in the Phase-0 audit — one row per owner keeps dedup, a stranger gets 404 not 403, `Cache-Control` dropped to `private`, and pre-existing files are claimed at startup or they'd all 404. Frontend: a single 401 interceptor, a warm bilingual sign-in overlay over a still-visible editor, and a **draft rescue** to localStorage so an expired session can't cost writing — the auto-save stashes the body it couldn't send and reclaims it after re-login. **Three deliberate deviations from the plan**, all noted above: the allowlist matches emails as well as subject ids (a subject doesn't exist until first login, so a subject-only list is unusable in advance); `SESSION_SECRET` was dropped from config rather than left unused (nothing signs anything — sessions are opaque and server-side); and image rows are keyed `(name, user_id)` rather than owned singly, which is what preserves deduplication. **A real bug caught by writing the round-trip test rather than by reading the code**: the one-shot state/nonce/PKCE cookies were cleared in a `defer`, i.e. after the redirect had already written the header, so the clearing `Set-Cookie` was silently dropped. Verified: full go/tsc/vite/vitest suites, migration `0010` against a `VACUUM INTO` copy of the live millenia DB (counts intact, FTS still matching, image claimed), and a live smoke against the binary in both auth-off and auth-on modes including a hand-inserted session (valid → 200; absent/forged/expired → 401). **Then deployed** (user: "do it! register it!"): provider + application registered in Authentik via `ak shell`, `.env` filled in, image rebuilt, and the **Traefik basic-auth gate removed** — Petal holds its own door now. Deploying immediately found two things no test could: the issuer's **trailing slash is significant** (Authentik's has one, OIDC compares byte-for-byte, and my normalising it away broke discovery while the slashless stub kept passing — now a knob with a regression test), and a provider created through the shell rather than the admin UI comes up with **empty `grant_types`**, which authentik answers with `invalid_request` before the login page renders. Verified over public HTTPS: health 200, `/api/docs` 401 with no basic-auth challenge, `/auth/login` → Authentik with state+nonce+PKCE, following it lands on the real sign-in page. Also swapped the emoji favicon for a **drawn sakura** (`web/public/petal.svg`) that renders in Petal's own rose palette everywhere instead of at each platform's discretion, and doubles as the Authentik app tile (inlined as a data URI, since this authentik doesn't serve `/media`). **The allowlist is `prosolis@proton.me` only** — that IdP fronts ~40 accounts, so empty was not an option and guessing her account would either lock her out or let a stranger in; adding her is one `.env` line and a restart.
- 2026-07-27: **Phase 15 finished off on the two boxes** (user granted millenia access mid-session: `ssh reala@192.168.1.212`, and parodia is `ssh reala@100.64.0.1` over headscale). **LLM link**: rather than rebinding vLLM as planned, `deploy/vllm-headscale-proxy.service` (socat) adds a listener on `100.64.0.2` only — `vllm-chat.service` is shared with **Gogobee** and **Open WebUI** (whose endpoint lives in its own DB, not env), so a rebind meant three consumer edits and a 35B reload; the forwarder cost nothing and no downtime. Grammar checkpoint from the VPS now returns real suggestions in ~3s. **Backups**: the user pointed out the VPS already has daily provider VM backups *and* an age-encrypted offsite `parodia-backup` job, so Petal was folded into the latter instead of running a parallel cron — and doing so **exposed a real bug in that job's `sqlite_dump` helper**: Python's `iterdump` does not reproduce an FTS5 virtual table, so any restore would have come back with cross-document search silently missing (fixed with a `VACUUM INTO`-based helper, round-trip verified). The **bigger** find: millenia, which holds her actual writing, had **no scheduled backup at all** — now `petal-backup.timer`, age-encrypted with the parodia public recipient and pushed off-box, neither machine able to decrypt it. **Encryption at rest** (user raised it; correctly): VPS data dir is now LUKS2 covering the DB, images *and* the TTS cache; key on-box as a deliberate availability tradeoff, documented for what it does and doesn't stop. Rehearsing a reboot caught two bugs a clean run would have hidden — the plaintext originals were still on the unencrypted root fs *under* the mount, and `systemd-cryptsetup` wasn't installed so crypttab was being ignored entirely and the volume would never have unlocked at boot. Added a `.volume-ok` guard so an unmounted volume fails loudly instead of serving a blank DB. **Millenia hygiene**: Piper had been dead since the Jul 26 reboot — **26,800+ failed restarts**, read-aloud silently degrading to browser Web Speech — because an OS upgrade moved `/usr/bin/python3` 3.13→3.14 and the venv's `site-packages` went invisible; venv recreated (lands Piper 1.6.0, which is what `TTS_PATH` exists for), both voices verified through Petal. Petal itself was running unsupervised at PPID 1 and is now `petal.service` (verified by `kill -9`); the Piper units got `StartLimitIntervalSec`/`Burst` so a broken service enters `failed` instead of looping forever unnoticed. Remaining: an external uptime-kuma probe (needs the UI), a true VPS reboot test (shared public host, user's call), and millenia is still unencrypted at rest.
- 2026-07-26: **Phase 15 complete — Petal is deployed at https://petal.parodia.dev** (user: "let's start this build plan"; scope confirmed as artifacts **plus** the actual deploy, millenia stays canonical, hostname `petal.parodia.dev`). Stack: `Dockerfile` (node → go → alpine; CGO off, so the runtime layer carries only ffmpeg + tzdata), `docker-compose.yml` behind the host's existing Traefik, and **two Piper sidecars** instead of the planned host systemd units — Piper turned out never to have been installed on the VPS and the account has no lingering session, so containers on an internal network with no published ports are both simpler and tighter. **Three real problems found by deploying rather than by planning:** (1) the image's `petal` user (uid 10001) has no claim on a bind-mounted host directory → SQLite `unable to open database file (14)` and a restart loop; the container now runs as the stack directory's owner (still non-root, and the host account keeps write access the backup script needs); (2) piper-tts **1.6.0 moved synthesis from `POST /` to `POST /synthesize`** with an identical body → every read-aloud 405'd; rather than pin both deployments to one Piper release the path became config (`TTS_PATH`, default `/`, so millenia is untouched); (3) once the instance was live it was **a public, writable, unauthenticated API** — Petal authenticates nobody yet, so Traefik basic auth now holds the door until Phase 16, with `/api/health` exempt on its own higher-priority router. Backups: `db.Backup` via **`VACUUM INTO`** (WAL-coherent, no write lock, single file, refuses to overwrite) behind a `-backup` flag so the nightly job snapshots the running container; `deploy/backup-petal.sh` compresses, pushes to millenia with a size check, prunes both sides; cron at 03:15; restore documented and verified by round-tripping an archive through the binary. Tests: `internal/db/backup_test.go` (WAL capture, seeded user survives, no `-wal`/`-shm` companions, refuses an existing destination, missing source), `internal/tts` path-normalisation + configured-path. go build/vet/test clean. **Acceptance verified over public HTTPS with the LLM link genuinely down**: dictionaries, gloss, word lookup + phonetic, doc create/save, CJK FTS search, md/docx export, vocab capture, read-aloud EN + zh (real mp3, cache hit, 404-fallback for an unconfigured language) — all fine; `/check` → the warm 502 that renders as 小助手在休息; health public, HTTP→HTTPS with a valid cert. **Two items outstanding, both needing millenia access I don't have**: vLLM isn't bound to its headscale interface (so no AI pass works yet), and parodia's ssh key isn't authorized on millenia (so backups are VPS-local only — not yet a real off-box backup). Both have one-command fixes in `deploy/README.md` §3 and §5. Also this session: **DreamDict gained Spanish**, so the es pair is no longer gated — folded into Phases 20/21 and the "Later" bucket. Next: **Phase 16 (auth)** — Authentik already runs on the same VPS.
- 2026-07-26: **Product direction + execution plan ratified** (user: "make it so, number one"). New `SUGGESTIONS.md` (product rationale for the language-learning direction): the **pair model** — every user gets one (English + X) pair, X ∈ {zh, pt-PT, fr, maybe es}, bilingual UI in the pair, type in either language, direction inferred (no detector: both-dictionaries spellcheck, show-both gloss on collision); **langpacks** keyed by X; **LLM-minimalism** as a standing principle (LLM is garnish, never a gatekeeper — grammar-lite rule pack + embedded miscollocation list planned as code-first layers). Deployment settled: Petal on the **parodia.dev VPS**, vLLM on millenia over **headscale** (the only cross-VPN dependency; Piper is VPS-local). All `MULTIUSER_PLAN.md` OPENs ratified: in-app OIDC (B), 30-day sliding sessions, allowlist, migration script, image-store fix with auth, DreamDict via package import (Option 3, module rename prereq in the dreamdict repo), zh stays on ECDICT until compared. Everything expanded into **Phases 15–22** above with standing rules (isolation tests same-commit, LLM-minimalism, bilingual aesthetic). Ready for implementation handoff starting at Phase 15.
- 2026-07-26: **Multi-user groundwork** (user: "let's start preparing Petal for multi-user support"; scope agreed as plumbing-only, aimed at Authentik). New **`internal/auth`** package — context-carried identity (`WithUser`/`UserID`), a `Resolver` seam (`Resolve(*http.Request) (string, error)`), `StaticResolver` for today's single user, and `Middleware` that 401s anything unresolved. `main.go` splits `/api` into a **public group** (`/health`, `/version` — a monitoring probe must not need a session) and an **authenticated group** carrying everything else. All ~35 `db.LocalUserID` call sites across `docs`/`suggestions`/`vocab` now read the caller from the request; helpers that had no request in scope (`fetch`, `ownsDoc`, `ownsTag`, `tagsByDoc`, `fetchVersion`, `passportVersions`, `fetchPending`, vocab `fetch`) take an explicit `userID` param. `UserID` returns `""` rather than panicking when middleware is absent, so a mis-wired route **fails closed** (every query is `WHERE user_id = ?` → matches nothing). **Two real access-control gaps found and fixed while threading**: `setStatus` (accept/dismiss) updated a suggestion by bare id with **no ownership check at all**, and `fetchPending`/`listForDoc` read a document's suggestions by `doc_id` alone — a leak of the quoted source sentences. Both now scope through `documents.user_id`. **A third bug was caught by the new tests, not by the compiler**: `docs.fetch` gained a `userID` parameter but kept binding `db.LocalUserID` in the query — legal Go (unused params compile), silently unscoped, and it would have shipped. New tests: `internal/auth/auth_test.go` (round-trip, absent-context, both 401 paths) and **two-user isolation suites** (`docs/isolation_test.go`, `suggestions/isolation_test.go`) that mount the same routers twice behind two resolvers over one DB and assert a stranger gets 404 on get/update/delete/export/passport/snapshot/version-preview/restore/tag-assign/tag-rename/tag-delete/suggestion-accept/dismiss, sees nothing in list/search/version-list, and leaves the owner's data untouched. go build/vet/test all clean. **Still global, deliberately out of scope** (flagged for the auth phase): the image store is content-addressed with no per-user association or DB row — any authenticated user holding a hash can fetch any image (capability-URL security, needs a table + migration to fix); `export-all` is correctly scoped; frontend `localStorage` keys (`petal.spell.personal`, `petal.companion`, sound/petals prefs) are per-browser, not per-account, so they'd bleed across users sharing a device.
- 2026-06-26: **Phases 12 + 13 complete** (collocation coach + vocabulary garden — "finish the rest of the build plan except Authentik/Traefik"). **Phase 12**: collocation drops in as a third suggestion family reusing the whole `runPass`/`pendingScope` machinery — `llm/collocation.go` (`RunCollocation`, 25s floor, reuses `ParseCheckpoint`), `collocationSystemPrompt`/`CollocationMessages` (warm "Natives usually say…" + Mandarin gloss, defers grammar elsewhere), migration `0005` **rebuilds** the suggestions table to extend the `type` CHECK (SQLite can't ALTER a CHECK), `collocationScope` + `CollocationLimit` + `POST /{id}/collocation`. **Caught a latent bug**: `grammarScope` was `type != 'voice'` → would wipe collocation flags; fixed to `type NOT IN ('voice','collocation')`. Frontend: `--color-blossom` pink, "Make it sound natural 🌸" toolbar pill, `collocating`/`runCollocation` in `useCheckpoint`, StatusBar dot — all into the existing rail/card. **Phase 13**: new `internal/vocab` package — migration `0006_vocab_garden` (`vocab_words`, SM-2-lite columns, doc_id `ON DELETE SET NULL`, `UNIQUE(user_id,word)`), `scheduler.go` (Leitner ladder 1/3/7/16/35 → geometric; gentle "again", no streak-shaming), `handlers.go` (capture-upsert/list/due/review/delete, all owner-scoped, time math via SQLite `datetime()` so stored values stay canonical-UTC). Auto-capture wired into `EditorCore.openWordLookup` (only dictionary-known words, captures the surrounding sentence + doc_id) + a 🤍/💚 toggle on `WordCard`. `GardenPanel` slide-over: blossom grid (bloom stage by reps), flashcard review (sentence blanked, flip, again/good/easy, direction alternates recognition↔production), sleepy-kitten footer; opened from a global 🌷 header button. Tests: `TestCollocationPassCoexists`, vocab `scheduler_test.go` + `handlers_test.go`, db CHECK test extended. go build/vet/test + tsc + vite + vitest (51/51) all clean; migration verified against a copy of the live DB; live backend smoke (throwaway DB) walked the full vocab lifecycle + the warm-502 collocation path. **Remaining: only the deferred infra bucket** — Authentik auth, Copyleaks Tier-2 (needs a public webhook), Docker/Traefik/deploy — all on hold per the user's "except Authentik/Traefik".
- 2026-06-26: **Phase 14 complete** (companion warmth + bedtime nag + night mode). `tips.ts`: `ENCOURAGEMENTS` 5→10 lines; new `BEDTIME` array (4 lines, user-supplied English wit + gentle Mandarin leads). `useCompanion.ts`: bedtime branch in the 10s heartbeat (after idle-return + break, before the generic tip); only nudges while actively writing; own `lastBedtime` ref + 30min `BEDTIME_GAP`, respects `PROACTIVE_GAP`; new `'bedtime'` `BubbleTone` lingers ~4s longer. **Night mode** (added same session, user request): `lib/night.ts` centralizes `isBedtime()` + window (now shared by the nag too); `hooks/useNightMode.ts` toggles `petal-night` on `<html>` (60s re-check); `index.css` `html.petal-night` re-points only the palette tokens → whole UI flips via `var()` (no component edits), 600ms dusk fade, print stays white; `PetalFall` gains a `night` prop → chunky cartoon power stars (`makeCartoonStar`, Mario/Kirby-style, 5 candy colors) mixed ~70/30 with small twinkle sparkles, gentle spin + shallow shimmer, effect re-inits on flip; App: `useNightMode()` → `<PetalFall night={night}/>`. tsc + vite clean, companion vitest 45/45; verified with real-browser Playwright screenshots (clock mocked to 23:30) — day petals/cream vs night stars/dark-plum, both pretty. Bedtime window is `BEDTIME_FROM`/`BEDTIME_TO` (local clock) for easy retune.
- 2026-06-26: **Phase 11 complete** (writer power-ups, batch requested as "do it all"). Seven features: (1) in-doc **Find & Replace** — `SearchHighlight` decoration extension + `FindReplace` bar (Ctrl/Cmd+F, match-case, replace-all back-to-front, DOM scroll that doesn't trigger the selection bubble); (2) **read-aloud** Web Speech util + 🔊 in WordCard & selection bubble; (3) **keyboard/touch access** — Ctrl/Cmd+D caret lookup, Ctrl/Cmd+J rewrite, touch long-press (refactored `handleContextMenu` → shared `openWordLookup(pos)`); (4) **export-all** backup zip (`GET /api/docs/export-all`, `TestExportAll`, sidebar download links); (5) **smart typography** input-rules extension (curly quotes/em-dash/ellipsis, ASCII-only so CJK untouched); (6) **duplicate doc + sidebar sort + outline popover**; (7) **English phonetic** (pivoted from pinyin — IPA is what an English learner needs; pinyin annotates Chinese she already reads) via `scripts/build_phonetic.py` + embedded `phonetic.json.gz` + `Result.Phonetic` + WordCard `/ˈrɪvər/` line — **full 46,579-word dataset built from ECDICT** (the csv re-download worked; `--seed` mode kept as a csv-free fallback). Also folded in this session: the **selection-bubble vs copy/paste fix** (bubble deferred to pointer-up + container `pointer-events:none` so it never sits where you click). go build/vet/test + tsc + vite all clean; live smoke verified word-phonetic (incl. de-inflection) + export-all zip (de-duped CJK names, route priority). Next: deferred bucket (auth/Copyleaks/deploy), still on hold per user.
Dockerfile
+70
View File
@@ -0,0 +1,70 @@
# Petal — multi-stage build producing the single self-contained binary.
#
# Stage 1 builds the frontend; stage 2 compiles the Go server with web/dist
# embedded (go:embed), so the runtime image carries one executable and no
# assets. modernc's SQLite is pure Go, so CGO stays off and the binary is
# static — the runtime layer exists only for ffmpeg (read-aloud transcodes
# Piper's WAV to mp3) and CA certificates.
# ---------- stage 1: frontend ----------
FROM node:22-alpine AS web
WORKDIR /src/web
# Install deps against the lockfile alone so this layer caches across source
# edits. The Hunspell dictionaries come from a devDependency, so a plain
# `npm ci` (not --omit=dev) is required for the spell checker to ship.
COPY web/package.json web/package-lock.json ./
RUN npm ci
COPY web/ ./
RUN npm run build
# ---------- stage 2: server ----------
FROM golang:1.25-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# The build context's web/dist is gitignored and excluded by .dockerignore;
# take the freshly built one from stage 1 so go:embed picks it up.
COPY --from=web /src/web/dist ./web/dist
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/petal ./cmd/server
# ---------- stage 3: runtime ----------
FROM alpine:3.21
# ffmpeg: read-aloud pipes Piper's WAV through it to mp3/opus. tzdata: the
# companion's bedtime nag and night mode read the local clock, so the container
# needs a real timezone rather than bare UTC.
RUN apk add --no-cache ca-certificates ffmpeg tzdata \
&& adduser -D -u 10001 petal
WORKDIR /app
COPY --from=build /out/petal /app/petal
# Mount point for petal.db (+ -wal/-shm), the image store, the TTS cache and
# DreamDict's read-only dict.db. dict.db is deployed alongside rather than baked
# in: it is ~450 MB, changes a few times a year, and is shared with other
# services on the host — putting it in the image would multiply it by every tag.
RUN mkdir -p /data && chown -R petal:petal /data
VOLUME ["/data"]
USER petal
EXPOSE 8080
ENV PORT=8080 \
DATABASE_PATH=/data/petal.db \
IMAGE_DIR=/data/images \
TTS_CACHE_DIR=/data/tts \
DICT_PATH=/data/dict.db
# Same endpoint Traefik and the uptime probe use; needs no session by design.
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD wget -qO- http://127.0.0.1:8080/api/health || exit 1
ENTRYPOINT ["/app/petal"]
MULTIUSER_PLAN.md
+390
View File
@@ -0,0 +1,390 @@
# Petal multi-user plan
**Status:** Phase 0 (identity plumbing) landed 2026-07-26 in `6901cdb`.
**Phase A (authentication) + Phase C's image store built 2026-07-27** — in-app
OIDC, server-side sessions, allowlist, provisioning, the frontend 401 path and
per-owner images. Not yet configured against the live Authentik; see
`BUILD_PLAN.md` Phase 16 and `deploy/README.md` §4. Phase B (migrating the
`local` user) still waits on her first real login.
**All OPEN decisions ratified by the user 2026-07-26** (recommendations
accepted as written) — see each OPEN for its settled answer. Execution phases
live in `BUILD_PLAN.md` (Phase 15 onward); product rationale for the language
work is in `SUGGESTIONS.md`.
Settled context that postdates the original draft: Petal will be hosted on the
**parodia.dev VPS**, reaching vLLM on millenia over **headscale VPN**; Piper
TTS is already installed on parodia (VPS-local, no VPN hop). The LLM is the
only cross-VPN dependency, and per the LLM-minimalism principle
(SUGGESTIONS.md §6) it must never gate essential functionality.
---
## 1. Where Petal is today
Single user, by construction. `db.Open` seeds one row (`users.id = 'local'`) and,
until this week, every query named that constant directly.
What was already right: **the schema has been multi-user-shaped from day one.**
`documents`, `tags`, and `vocab_words` all carry `user_id`; `document_versions`
and `suggestions` scope through their parent document. No migration is needed to
support a second user — only a way to know which user is asking.
### What Phase 0 changed
- New `internal/auth`: `Middleware(Resolver)` resolves the caller once per API
request and stores the id in the request context. Handlers read
`auth.UserID(r.Context())`.
- `Resolver` is a one-method interface — `Resolve(*http.Request) (string, error)`
— and is the only thing a real identity provider has to implement.
- `StaticResolver(db.LocalUserID)` supplies today's single user, so behavior is
unchanged.
- `/api` is split into a public group (`/health`, `/version`) and an
authenticated group (everything else).
- Two pre-existing access-control gaps fixed: `setStatus` (accept/dismiss) had
**no ownership check at all**, and `fetchPending` read suggestions by `doc_id`
alone — which leaked the quoted source sentences.
- Two-user isolation test suites (`docs`, `suggestions`) mount the same routers
twice behind two resolvers over one database.
**The remaining work is not "make Petal multi-user."** It is "authenticate
someone, provision them, and clean up the three places where data is still
global."
---
## 2. Goals and non-goals
**Goals**
- Two or more people use one Petal instance without seeing each other's writing.
- The existing local user's data survives, attached to a real account.
- Adding a user is an operator action, not a code change.
**Non-goals (explicitly out, unless a reviewer argues otherwise)**
- Sharing, collaboration, or multi-author documents. Petal is a private writing
space; every feature to date assumes one reader. Sharing would change the
passport's meaning (authorship evidence) and is a product decision, not an
auth one.
- Roles, permissions, or an admin UI.
- Public signup. Accounts are provisioned deliberately.
---
## 3. Phase A — authentication
### OPEN #1: forward-auth vs. in-app OIDC — **SETTLED: Option B (in-app OIDC)**
Ratified 2026-07-26. The deciding fact arrived with the deployment plan: Petal
will live on the public parodia.dev VPS, which is exactly the environment where
Option A's "must never be reachable except through Traefik" invariant is a
footgun. Original analysis kept below for the record.
**Option A — Traefik forward-auth to an Authentik outpost.**
Traefik is already in the deferred deploy bucket. Authentik's outpost terminates
the login, and Petal receives a trusted header (`X-authentik-uid`, plus email and
name). The `Resolver` becomes ~20 lines: read the header, map to a user id.
- *For:* no OIDC library, no session store, no cookie handling, no redirect
plumbing, no token refresh, no logout endpoint. Petal keeps zero auth code.
Login/MFA/password reset are entirely Authentik's problem.
- *Against:* Petal is only secure if it is **never reachable except through
Traefik**. Anyone who can hit the container directly can forge the header and
become any user. That's a deployment invariant enforced by network config, not
by code — and it is exactly the kind of invariant that quietly breaks. It also
makes local development awkward (no proxy → no identity), though
`StaticResolver` covers that.
**Option B — Petal is an OIDC client itself.**
`github.com/coreos/go-oidc` + `golang.org/x/oauth2`, a `/auth/callback` route,
and a signed session cookie. The config fields already exist
(`AUTHENTIK_URL`, `AUTHENTIK_CLIENT_ID`, `AUTHENTIK_CLIENT_SECRET`,
`SESSION_SECRET`).
- *For:* self-contained and safe to expose directly. No trust-the-proxy
invariant. Works the same in dev and prod.
- *Against:* meaningfully more code — a session table or signed-cookie scheme,
CSRF on the callback, token expiry, logout. Two new dependencies in a project
that currently has exactly two.
**My recommendation: Option B**, but not confidently. The deciding factor for me
is that "must never be reachable directly" is a footgun that survives long after
whoever set it up has forgotten, and Petal already holds someone's private
journals. But if the deployment is definitively a single Traefik-fronted box on a
LAN and will stay that way, Option A is *much* less code and I'd not object.
A reviewer should weigh: how likely is this instance ever exposed beyond the LAN?
Does the operator want Petal to be independently deployable?
### Session handling (assumes Option B)
- Server-side sessions in a `sessions` table (id, user_id, expires_at,
created_at, user_agent), cookie holds an opaque random id.
Preferred over signed stateless cookies because it makes logout and
revocation actually work — worth the one table.
- Cookie: `HttpOnly`, `SameSite=Lax`, `Secure` when `BASE_URL` is https.
- **OPEN #2 — SETTLED: 30-day sliding expiry** (ratified 2026-07-26). An
editor that logs you out mid-draft is hostile, and auto-save makes a
surprise 401 genuinely costly. Sliding: each authenticated request extends
the session.
### The 401 problem
Every frontend fetch currently assumes success. Once a session can expire,
**any** call can return 401 mid-session — including the 1.5s auto-save, which is
the one that must not fail silently.
Proposal: a single interceptor in `web/src/api/client.ts` that, on 401, halts
auto-save, surfaces a warm bilingual "请重新登录 / Please sign in again" state
rather than a raw error, and preserves unsaved editor content across the
re-login (localStorage draft keyed by doc id). This is small but easy to forget,
and getting it wrong means lost writing.
---
## 4. Phase B — user provisioning and migration
### Provisioning
On first successful login, upsert a `users` row from the OIDC claims
(`sub` → `users.id`, plus email and name). No signup flow; whoever Authentik lets
in gets an account.
**OPEN #3 — SETTLED: yes, allowlist** (ratified 2026-07-26). Authentik may
host other applications with a broader user set than Petal should have. Gate
via a `PETAL_ALLOWED_SUBS` env var (or an Authentik group claim check —
implementer's choice, env var is simpler); a valid login not on the list gets
a warm bilingual "this Petal isn't yours to write in" page, not a 500.
### Migrating the existing `local` user
The live database on millenia holds real writing under `user_id = 'local'`. That
data must end up owned by the wife's real account.
Recommended: a migration that **renames** rather than copies — update the
`users.id` and let `ON UPDATE CASCADE`… except SQLite FKs here are declared
without `ON UPDATE`, so this needs either a deliberate multi-table update inside
one transaction (`documents`, `tags`, `vocab_words` — versions and suggestions
follow their parents) with `PRAGMA foreign_keys=OFF` around it, or an explicit
one-off admin command.
I'd rather do this as a **documented one-off script run with the app stopped and
a copy of the DB taken first** than as an automatic startup migration, because it
depends on knowing the new OIDC subject id, which isn't available until that
person logs in once. Sequence: deploy auth → she logs in → new empty account is
created → stop app, back up, run script to move `local`'s rows onto her real id,
delete the empty row → restart.
**OPEN #4 — SETTLED: documented one-off script** (ratified 2026-07-26), run
with the app stopped and a DB backup taken first, per the existing `scripts/`
convention. No admin endpoint.
---
## 5. Phase C — the data that is still global
Found during the Phase 0 audit. None of these break with two users; all of them
leak or bleed.
### Image store — the real one
`internal/images` is a flat content-addressed directory. There is no per-user
association and no database row at all. Any authenticated user who knows a
sha256 can fetch any other user's image.
That is capability-URL security. Hashes aren't guessable, so this is not an
emergency — but "unguessable filename" is not access control, and images pasted
into a private journal are exactly the content that shouldn't rely on it.
Proposal: an `images` table (hash, user_id, content_type, created_at, size) with
the fetch handler joining on the caller. Content addressing is kept — the same
image uploaded by two users is stored once on disk and simply has two rows, so
deduplication survives. Deleting the last row referencing a hash removes the
file.
**OPEN #5 — SETTLED: fix it in the same phase as auth** (ratified
2026-07-26). The moment a second account exists the exposure is real, and the
fix requires a migration either way.
### Frontend `localStorage`
`petal.spell.personal` (personal dictionary), `petal.companion` (chosen mascot),
plus sound and petal-effect preferences are all per-browser. Two users on one
device share them — and the personal dictionary is the one that matters, since
it's built from someone's own writing.
Cheapest fix: namespace every key by user id once the client knows who it is.
The honest fix for the dictionary is to move it server-side into a table, which
also means it follows a user between devices — arguably a feature.
### Not affected
`export-all` is correctly scoped. The TTS cache is content-addressed audio of
text the requester supplied, no cross-user inference. The lexicon is a static
dataset identical for everyone.
---
## 6. Phase D — per-user language (and DreamDict)
Tracked here because it lands on the same `users` row and shouldn't be designed
twice.
**English is always the target language.** What varies is the user's *native*
language — the one glosses and explanations are written in. Mandarin ships today;
European Portuguese (pt-PT, explicitly not pt-BR) is wanted; French is possible.
### OPEN #6 is answered: DreamDict
The original worry here was data sourcing — Petal's gloss comes from ECDICT
(English↔Chinese), and a pt-PT equivalent of comparable quality and license
looked like the blocker.
`~/git/dreamdict` already solves it, and more completely than expected. It
supports **en, fr, pt-PT, and zh** (~136k/56k/136k/121k words), and its shape maps
almost 1:1 onto `lexicon.Result`:
| Petal field | DreamDict |
|---|---|
| `Gloss` | `Translate(word, "en", L1)` |
| `Phonetic` | pronunciation (CMU + IPA for en, Wiktionary IPA elsewhere) |
| `Definitions` | `Define(word, lang)` — curated sources ranked above Wiktionary |
| `Synonyms` | `Synonyms(word, lang)` |
It also carries data Petal has no equivalent for and could use: `Antonyms`,
`Frequency`, `Difficulty`, and `Etymology`.
> **Correction (2026-07-27, Phase 20).** The `Gloss` row of that table is wrong.
> `Translate(word, "en", L1)` reads Wiktionary's translation sections, which are
> thin in the en→X direction: measured on the real `dict.db`, it answers for
> **17%** of the 2,000 commonest English words into pt-PT and 16% into fr.
> Meaning has to come through shared WordNet synset ids instead (**61%**), which
> is what DreamDict's new `Equivalents(word, from, to)` does — falling back to
> the translations table, for 62% combined. The 1:1 mapping was assumed from the
> API surface and never checked against the data; it did not survive contact
> with it. Same measurement on zh reads 53% against ECDICT's near-total coverage
> of those words, which is why the zh pair did **not** converge.
So Phase D stops being gated on data and becomes an integration decision.
### OPEN #6a (new): how to integrate — **SETTLED: Option 3** (ratified 2026-07-26)
Import the package, open `dict.db` read-only. Prerequisite stands: DreamDict's
module path must be renamed (or `replace`-directed) first — **that change
lives in the dreamdict repo, not this one.** The migration caution below also
stands: pt-PT/fr wire to DreamDict first; zh stays on ECDICT until compared on
real lookups. Options kept below for the record.
**Option 1 — HTTP client.** Petal calls DreamDict on localhost:7777, exactly the
pattern already used for Piper TTS (including graceful degradation when it's
down).
- *For:* zero coupling, DreamDict updates independently, all endpoints available.
- *Against:* a second service Petal now depends on at runtime, and the gloss is a
350ms hover tooltip where "the dictionary service is down" is a visible
regression from today's always-there embedded data.
**Option 2 — build-time extraction.** A script (sibling to the existing
`scripts/build_gloss.py`) generates Petal's embedded `.json.gz` datasets per
language from DreamDict's `dict.db`.
- *For:* preserves the embedded/offline property exactly; no runtime dependency;
no architectural change at all.
- *Against:* every language multiplies the binary (the four current gz files are
already ~11.6 MB); updating the dictionary means rebuilding and redeploying
Petal; the richer fields are lost unless separately extracted.
**Option 3 — import the package, open `dict.db` read-only.** DreamDict's
`internal/dictionary` is a plain library with `NewReadOnly(dbPath)`, and its only
dependency is `modernc.org/sqlite` — the same CGO-free driver Petal already uses.
Petal opens `dict.db` as a second read-only handle beside `petal.db`.
- *For:* no service, no HTTP, no new dependency, lookups stay local-file fast,
all four languages at once, and it deletes ~11.6 MB of embedded gz plus the
ECDICT build scripts. One dictionary, maintained once, shared with GogoBee.
- *Against:* Petal stops being a self-contained binary in the "just run it" sense
— `dict.db` has to be deployed alongside. In practice Petal already ships a
data directory (`petal.db`, images, TTS cache), so this is a smaller loss than
it first sounds.
**My recommendation: Option 3.** It is the only one that gets all four languages,
keeps lookups offline and instant, and *removes* code rather than adding a
subsystem. Option 1's runtime dependency buys flexibility Petal doesn't need for
a dictionary that changes a few times a year.
**Prerequisite:** DreamDict's module path is currently `module dreamdict`, which
isn't fetchable. Importing it needs the module renamed to something like
`gitea.parodia.dev/drwily/dreamdict` (or a local `replace` directive for
development). Small, but it must happen first.
### Migration caution
Whichever option wins, the zh path is **currently working and in daily use**. The
gloss quality difference between ECDICT and CC-CEDICT is unknown and matters more
than the architecture.
Proposal: introduce DreamDict behind Petal's existing lexicon interface as a
*provider*, wire pt-PT and fr to it first (nothing to regress — they don't exist
yet), and keep zh on ECDICT until the two have been compared on real lookups from
her actual documents. Converge only if quality holds. This also de-risks the whole
change: if DreamDict turns out to be a poor fit, only the unshipped languages are
affected.
### Still per-user regardless
Native language becomes a `users` column, and these become per-user lookups:
LLM prompt copy (`internal/llm/prompts.go`, currently Mandarin-first), companion
tips (`tips.ts`), the L1 Piper voice (Piper has pt-PT voices), and the CJK font
stacks (not needed for Latin-script L1). English-side machinery — nspell en-US,
the phonetic dataset, the EN voice — is unaffected and stays shared.
## 7. Suggested sequence
1. ~~Settle OPEN #1~~ **Settled: in-app OIDC.**
2. Deploy plumbing: Dockerfile, Traefik, real hostname on parodia.dev, HTTPS,
headscale route to vLLM on millenia (bound to the headscale interface
only), VPS-local Piper, off-VPS DB backup. Auth needs a stable `BASE_URL`
and a redirect URI.
3. Auth itself: OIDC `Resolver`, sessions table (30-day sliding), allowlist,
frontend 401 handling with draft preservation.
4. Image store table + migration (same phase, per OPEN #5).
5. Provision the second real account; migrate `local`'s data (script, app
stopped, backup first).
6. `localStorage` namespacing (key by user **and** language — see
SUGGESTIONS.md §8).
7. Per-user language pair. **No longer gated on data** — DreamDict covers all
four languages. Sequence within it: rename DreamDict's module path → wire
it in as a lexicon provider → pt-PT/fr first → compare zh quality →
converge if it holds. The pair model and langpack shape are specified in
SUGGESTIONS.md §1–§3.
These are expanded into checkboxed execution phases in `BUILD_PLAN.md`
(Phase 15 onward) — that file remains the source of truth for progress.
---
## 8. Risks
- **Silent unscoping.** Phase 0 hit this exactly once: `docs.fetch` took a
`userID` parameter and kept binding `db.LocalUserID` in the query. Unused
parameters are legal Go — it compiled, `vet` was silent, and every existing
test passed while the lookup stayed unscoped. Only the two-user isolation test
caught it. **Every new user-scoped endpoint should get an isolation case in the
same commit**; the existing suites are the template.
- **Migrating live data.** The wife's real writing is the thing being moved.
Back up first, run with the app stopped, verify counts before deleting
anything.
- **A 401 mid-draft losing work.** See Phase A.
- **Scope creep into sharing.** Multi-user and collaboration are different
products. Adding accounts should not quietly become adding sharing.
---
## 9. Questions for the reviewer — all answered 2026-07-26
1. ~~Forward-auth or in-app OIDC?~~ **In-app OIDC** (OPEN #1).
2. ~~Session lifetime?~~ **30-day sliding** (OPEN #2).
3. ~~Allowlist?~~ **Yes**, `PETAL_ALLOWED_SUBS` or group claim (OPEN #3).
4. ~~Script vs. admin endpoint?~~ **Script**, app stopped, backup first (OPEN #4).
5. ~~Image store timing?~~ **Same phase as auth** (OPEN #5).
6. ~~pt-PT dictionary data?~~ **DreamDict**, integrated per **Option 3**
(import package, read-only `dict.db`; module rename is the prerequisite)
(OPEN #6a).
7. ~~zh gloss regression risk?~~ **zh stays on ECDICT** until compared against
DreamDict on real lookups from her actual documents; converge only if
quality holds.
README.md
+12 -2
View File
@@ -32,6 +32,16 @@ cd .. && go build -o petal ./cmd/server
Configuration is via environment variables — copy `.env.example` to `.env`.
## Deployment
```bash
docker compose up -d --build # petal + the two Piper read-aloud sidecars
```
Behind Traefik on the parodia.dev VPS; vLLM stays on millenia over headscale.
Full runbook — first deploy, the LLM link, backups and restore — in
[`deploy/README.md`](./deploy/README.md).
## Status
Early build, multi-session. Auth (Authentik), Copyleaks plagiarism, and Docker/Traefik
deployment are deferred — see `BUILD_PLAN.md`.
Early build, multi-session. Auth (Authentik OIDC) is next; Copyleaks plagiarism is
still parked — see `BUILD_PLAN.md`.
SUGGESTIONS.md
+364
View File
@@ -0,0 +1,364 @@
# Petal — product suggestions: becoming essential for language learners
**Status:** written 2026-07-26 against `feat/writing-passport`; **ratified by
the user 2026-07-26** (recommendations accepted — reviewer Q1–Q3 settled
below; Q4–Q6 remain genuinely open and don't block execution). This document
is the *why*; the checkboxed execution phases live in `BUILD_PLAN.md`
(Phase 15 onward).
**The brief:** make Petal essential for two audiences — ESL writers (native
Mandarin / pt-PT / French → English), and English natives learning Mandarin,
European Portuguese, French, maybe Spanish. Preserve privacy and warmth.
**The language model (settled by the user, 2026-07-26):** every user has
exactly one language **pair, with English always one half** — (en + X),
X ∈ {zh, pt-PT, fr, maybe es}. This is a deliberate scope decision: never an
X↔Y pair without English, never more than one pair per user. The UI is
bilingual in the pair everywhere (tips, pet responses, cards), and the user
may **type in either language of the pair**; Petal infers direction from the
text rather than asking.
---
## 1. What the pair model implies
The wife's zh setup is already exactly this — she writes Mandarin and English
mixed, the UI is zh+en bilingual, and Petal adapts per span (CJK is never
spellchecked, English words gloss to Chinese). So the pair model isn't a new
design; it's a *promotion of today's behavior to the spec*. Three consequences:
- **Schema:** one column, `users.pair_lang` (the X half; default `'zh'`).
No per-document language, no target/native split. Add it in whatever
migration Phase B's provisioning touches — one column now vs. a real
migration later, the same logic that put `user_id` in the schema on day one.
- **Direction is inferred, not declared.** The zh pair gets inference for free
(script boundaries separate the languages). Latin pairs don't — see §3a,
which is the one genuinely new problem the pair model creates.
- **Every bilingual surface stays two-language**, just parameterized: the
`中文 · English` pattern becomes `X · English`. Nothing about the UI's shape
changes, which is why the copy extraction in §2 is safe to do early.
## 2. Languages as data, not code ("langpacks")
Adding pt-PT today means editing code in many places. A quick census: **29
frontend files** carry hardcoded zh-first bilingual strings (`tips.ts`,
`GardenPanel`, `StatusBar`, `WordCard`, every popover…), plus the
Mandarin-first prompt copy in `internal/llm/prompts.go`. Adding each new
language by hunting through those files doesn't scale to four pairs and would
slowly erode the bilingual-copy quality that makes Petal feel cared-for.
Because English is always one half, a **langpack is keyed by X alone** — one
pack per pair, holding everything that varies:
- UI copy pairs — extract the existing `中文 · English` strings into a copy
module; the current strings become the `zh` pack verbatim, so nothing
visible changes. This is the biggest single chore in the whole effort;
better done once than per-language.
- Companion tip/cheer/bedtime lines (`tips.ts` is already data-shaped —
closest to done).
- LLM prompt copy: bilingual explanation phrasing, "natives usually say…"
example pairs, and the both-directions framing (the text may be English, X,
or mixed — respond appropriately).
- Hunspell dictionary for X where one exists (`pt-PT`, `fr`, `es` upstream;
zh has none — see §4). The en-US dictionary is shared by every pair.
- DreamDict wiring: gloss both directions (`en→X` and `X→en`), phonetics.
- Piper voices for X (both for reading X text aloud and as the L1 voice);
font stack (CJK stacks only for zh).
Shared across all pairs, untouched: nspell en-US, the English IPA dataset, the
EN Piper voice, and all of the editor machinery.
This is refactoring, not product, so it's tempting to skip. Don't: it's the
difference between "Spanish is a data drop" and "Spanish is a month."
## 3. Sequencing: Latin-script targets first, and in this order
**pt-PT → fr → es.** Everything needed for these exists already: Hunspell
dictionaries, Piper voices, DreamDict data (en/fr/pt-PT/zh), and — critically —
the entire decoration/anchoring machinery (`wordAt`, spell tokenizer, suggestion
re-anchoring) already works, because these languages are space-delimited and
Latin-script like English.
Caveats worth writing down now:
- **DreamDict has no Spanish.** "Maybe Spanish" is gated on adding es to
DreamDict first, or a separate dataset. Cheap to note, expensive to discover
later.
- **pt-BR drift is the main quality risk.** Qwen will default to Brazilian
Portuguese in both explanations and "natives say…" examples. Prompts must pin
European Portuguese explicitly, and the pt-PT pack should be reviewed by a
pt-PT speaker before it's trusted — same standard the zh copy got by being
written for a real reader. The multi-user plan's ECDICT-vs-DreamDict
compare-on-real-lookups discipline applies here too.
### 3a. The Latin+Latin wrinkle: inferring direction without a script boundary
The zh pair gets "which language is this word?" for free — the script answers
it, and all of today's behavior (CJK never spellchecked, English words gloss
to Chinese) hangs off that. In an en+fr or en+pt pair, both halves are Latin
script, so the two per-word decisions need a real answer:
- **Spellcheck:** load both Hunspell dictionaries and pass a token if *either*
accepts it; flag only words wrong in both. This never falsely squiggles
correct writing in either language — the failure mode is missing a French
word that happens to be a valid English word, which is the gentle direction
to fail in. Correction pills can offer both dictionaries' suggestions.
- **Gloss/WordCard:** look the word up in both directions via DreamDict; if it
exists in only one language, done. For collisions (*chat*, *pain*, *sale*
are all real words in both English and French), show both compactly — a
two-line card ("🇫🇷 chat → cat · 🇬🇧 chat → bavarder") is honest, needs no
detector, and is arguably *delightful* for a learner. A sentence-level
language guess can order the lines, but shouldn't hide either.
No trained language detector, no heuristics that can be wrong about someone's
writing — both-dictionaries membership plus show-both-on-collision covers it.
The LLM passes need nothing: the prompt already sees the mixed text whole.
- The Hunspell tokenizer's current rule "CJK is never tokenized" stays correct
for the zh pair unchanged.
## 4. The zh pair's *other* direction is a separate epic — say so explicitly
The en+zh pair already exists, but only one direction of it is built: today
Petal deliberately ignores typed hanzi (never tokenized, never flagged, never
glossed) — exactly right for a zh-native writer practicing English, and
exactly insufficient for an English native *learning* Chinese, for whom the
hanzi side is the whole point. Supporting that direction breaks assumptions
that are load-bearing everywhere:
- No spaces → `wordAt`, the spell tokenizer, and word-boundary lookups need
real word segmentation (a jieba-style segmenter, client- or server-side).
- Hunspell has no concept of Chinese; "spellcheck" becomes wrong-character
(错别字) detection — a different problem, probably LLM-assisted.
- Smart-typography input rules and the IME interact; input rules are currently
ASCII-gated, which is correct, but selection/caret behavior mid-IME
composition needs testing.
- The learning aids that matter are different: pinyin annotation (useful here,
unlike for the current user who reads hanzi), tone-mark help, HSK-level word
difficulty, hanzi stroke/handwriting practice.
None of this is unbuildable, but it is its **own phase with its own spec**, not
part of the langpack drop. Recommendation: ship the pt-PT/fr pairs first to
prove the pair model, and treat learner-facing Chinese writing as Petal's next
big product bet after that — it's also the most differentiated one (very few
warm, private tools exist for writing practice in Chinese).
## 5. Deepening the learning loop (all local, all gentle)
Petal's suggestion pipeline currently *corrects and forgets*. The vocabulary
garden proved that capturing what the user already does (lookups) creates a
learning surface for free. The same move is available twice more:
### 5a. Growth journal (patterns from accepted suggestions)
Accepted grammar/collocation suggestions are a record of what the writer is
learning. Aggregate them **locally** into gentle patterns: "this month you've
mostly stopped mixing 在/at" / "make a decision has stuck — you've used it
right 4 times since." Two framing rules that keep it warm: it reports *growth*,
never an error tally, and it only ever compares the writer to her own past
self. Feeds the companion's cheer pool with genuinely personal material
("上次你还问过这个词，这次自己用对了! 🌱"). Data is already in the
`suggestions` table (status + type + original/replacement); this is a read-side
feature, no new capture needed.
### 5b. Plant accepted collocations in the garden
An accepted collocation ("do a decision" → "make a decision") is a learnable
chunk, exactly like a looked-up word. Auto-capture it into the vocabulary
garden as a phrase card (the SM-2-lite scheduler doesn't care that it's two
words). The garden then reflects *both* halves of learning: words she sought
out, and phrasing she was gently given.
### 5c. Companion as tutor-lite: a daily invitation to write
The companion nudges about breaks and bedtime but never *invites writing*. A
once-a-day bilingual prompt ("写 50 个字：今天让你微笑的一件小事 · Write 50
words: one small thing that made you smile today"), offered when a session
starts with no doc open. Explicitly **no streaks, no guilt** — the existing
no-streak-shaming ethos in the SR scheduler is the right precedent; a declined
prompt just gets a sleepy "好吧，我继续睡 😴". Prompt lists live in the
native-language pack.
### 5d. Use DreamDict's richer fields
The multi-user plan notes DreamDict carries `Frequency`, `Difficulty`,
`Antonyms`, `Etymology` with "no equivalent" in Petal. Three cheap, high-value
surfaces:
- A **frequency/difficulty chip** in the WordCard ("common word" / "advanced")
— helps a learner decide whether a word is worth gardening.
- **Etymology for the en-native audience**: Romance-language learners live on
cognates; a one-line "from Latin *decidere*, like English *decide*" is the
single best memory hook for pt/fr/es vocabulary.
- **False friends**: a small curated list per pair (en↔pt: *embarrassed* ≠
*embaraçada*-adjacent traps, *actually*/*atualmente*; en↔fr likewise),
surfaced as a warm heads-up in the WordCard and as a collocation-style
gentle flag when one is used suspiciously. Tiny data, disproportionate
trust-building — this is the mistake that makes learners feel foolish, and
catching it kindly is very Petal.
### 5e. Read-aloud, slower
Piper voices exist per target language; wire the target-language voice into the
existing 🔊 surfaces, and add a **slow toggle** (Piper's `length_scale`) —
learners replaying a sentence at 0.75× is one of the oldest, most-loved
listening aids, and it's a query parameter away.
## 6. LLM-minimalism: essential help in plain code, the model as garnish
**Stated by the user (2026-07-26):** with the LLM on the far side of a VPN,
preserve as much essential functionality as possible in ordinary code inside
Petal, and rely on the LLM as little as possible. This deserves to be a
standing design principle, not just a deployment reaction — it's also what
keeps Petal instant (no 38-second checkpoints for things a lookup can answer)
and private by construction.
Where Petal stands today, by dependency:
| Already pure code (survives VPN-down) | LLM-only today |
|---|---|
| Spellcheck (Hunspell), gloss/definitions/synonyms/phonetics (embedded lexicon → DreamDict), thesaurus, vocabulary garden + SR review, search, tags, versions + writing passport, export, find/replace, typography, TTS (Piper, VPS-local) | Grammar checkpoint, collocation coach, voice pass, Ask Petal, tone rewrite |
Everything in §5 lands in the left column by design (growth journal, garden
planting, daily prompts, DreamDict fields, false friends — all lookups and
local aggregation). The right column splits into two groups:
**Worth a code-first layer (the essential two):**
- **Grammar lite** — a rule-pack of high-precision, data-driven checks for
the classic ESL patterns: a/an before vowel sounds, uncountables
("informations", "advices", "furnitures"), subject–verb agreement in simple
clauses, doubled comparatives, common preposition pairs ("depend of" →
"depend on"), per-pair L1-interference rules (zh: dropped articles, he/she
slips; pt/fr: "have X years" for age). These run instantly on every edit —
no debounce, no 30s rate limit — as a fourth suggestion family through the
existing rail. The bar is **precision over recall**: an offline rule must be
near-certain before it flags, because a wrong correction is colder than a
missed one. LanguageTool's open rule corpus is a mineable source for
vetted patterns (extract data, not the Java).
- **Collocation data** — the same curated-list move as false friends: the
do/make, say/tell, strong-tea/heavy-rain families that fill every ESL
collocation workbook are a few hundred entries of data, not a model. A
small embedded miscollocation list catches the top offenders offline; the
LLM pass, when reachable, adds the long tail. Same family, same rail, same
warm phrasing — the writer never needs to know which engine spoke.
**Inherently LLM (degrade warmly, don't imitate):** Ask Petal, tone rewrite,
and the voice pass are open-ended language generation — a code fake would be
worse than the existing honest "小助手在休息" state. Leave them as the
garnish they are.
The framing that falls out: **the LLM never holds essential functionality
hostage.** Every daily-writing need — spell, define, gloss, pronounce, catch
the common mistakes, review vocabulary, prove authorship — works on a
disconnected VPS. The model adds depth and conversation when the tunnel is up.
## 7. The writing passport is an ESL flagship — treat it as one
The passport exists because AI detectors misfire on non-native English (the
commit message cites the Stanford TOEFL finding). That's not a side feature —
for the ESL audience it may be *the* reason to adopt Petal over any other
editor: **the tool that protects you from being wrongly accused, instead of
scoring you.** No product change needed beyond making sure it works identically
for any target language (it should — it's language-agnostic snapshot history).
Worth a prominent place in the README/landing copy when Petal gets one.
## 8. Ties into MULTIUSER_PLAN.md
For the open questions there, this document's brief implies:
- **OPEN #1 (auth):** Option B (in-app OIDC), and the planned deployment
settles it. The user's stated topology (2026-07-26) is: **Petal hosted on
the parodia.dev VPS, reaching vLLM on millenia over headscale VPN.** A
public-internet app is exactly the case where "must never be reachable
except through Traefik" is a footgun — one proxy misconfiguration on a VPS
and forged identity headers reach the app. In-app OIDC is safe to expose
directly.
- **Deployment topology consequences** worth writing into the plan's Phase 2
(deploy plumbing):
- The LLM becomes the only cross-VPN runtime dependency (Piper is already
installed on parodia.dev, so TTS stays VPS-local). The warm
"小助手在休息" degradation path was built for a flaky co-tenant Ollama; a
VPN link-down hits the same path, so the architecture already fails
gently — but checkpoint latency now includes a WAN+VPN round trip, worth
a look at the 60s LLM timeout.
- Everything offline-by-design (DreamDict lookups, spellcheck, gloss,
garden, search, the whole editor) keeps working when the VPN is down —
another argument for MULTIUSER_PLAN Option 3 over an HTTP dictionary
service, which would otherwise add a second cross-machine dependency.
- vLLM and Piper on millenia should bind to the headscale interface only,
never 0.0.0.0 on the LAN-facing side.
- The writing moves onto rented VPS disk. "The writing never leaves the
box" (§8) becomes "the box is a VPS" — at-rest encryption and an
off-VPS backup of `petal.db` (e.g. nightly to millenia over the same
VPN) deserve a line in the deploy phase.
- **OPEN #6a (DreamDict):** Option 3 (import, read-only `dict.db`), agreed —
it's the only option where four languages stay offline and instant, which
§6 and §9 treat as non-negotiable.
- **Phase B provisioning** should set `users.pair_lang` from the operator's
provisioning step or a first-run picker — add the column in the same
migration. (The plan's Phase D "native language becomes a `users` column"
becomes this: one column, the X half of the pair.)
- **localStorage namespacing** matters slightly more than the plan says once a
household mixes pairs: the personal spell dictionary is per-*language* as
well as per-user (a user's en words and pt-PT words must not merge into one
Hunspell overlay). Key by `user + lang`.
## 9. Privacy & warmth guardrails (the checklist for every item above)
Everything suggested here passes these; future ideas should too.
1. **Offline-first, always — and code-first (§6).** Every essential surface
must work with the LLM unreachable and the network unplugged
(DreamDict-as-local-file preserves this; an HTTP dictionary service would
not). The LLM only ever adds depth to something that already works.
Cloud APIs are off the table even when they'd be easier.
2. **The writing never leaves the box.** No telemetry, no "anonymous usage
stats," ever. The growth journal (§5a) is computed locally from local rows.
3. **No scores, no percentages, no red.** Petal already refuses AI-detection
scores and classic red squiggles; the growth journal and false-friend flags
must hold the same line — evidence and gentle phrasing, never grades.
4. **No streaks, no guilt.** The SR scheduler set the precedent (gentle
"again", no wipe). Daily prompts (§5c) are invitations, not obligations.
5. **Both languages of the pair, always visible** — every explanation, tip,
and pet response renders bilingual in (en + X). That's the warmth: being
helped in the language you think in, next to the one you're learning.
6. **The kitten stays asleep.** Every new companion behavior routes through
the existing mood/cooldown engine; 瞌睡猫 keeps mumbling helpful things
without waking up. (A reactive-animation puppy is on the wishlist — low
priority per the user; the `companions.ts` roster + mood engine is already
the drop-in point, richer per-mood Lottie segments are the only new work.)
## 10. Suggested sequence (interleaved with the multi-user plan's)
1. Add the `users.pair_lang` column (with the Phase B migration or sooner).
2. Extract the bilingual UI copy into the langpack (zh pack = today's strings
verbatim; pure refactor, no visible change).
3. DreamDict integration per MULTIUSER_PLAN Option 3 (module rename → lexicon
provider → pt-PT/fr wired first, zh compared before converging).
4. pt-PT as the first full second pair: Hunspell pt-PT, Piper pt-PT voices,
pinned-pt-PT prompts, native-speaker copy review, and the both-dictionaries
spellcheck + show-both-gloss behavior from §3a. French follows the same
groove; Spanish gated on DreamDict es data.
5. Learning-loop features (§5) — each is small and independent; growth journal
and garden-planting of collocations first, since they're read-side over
existing data.
6. Code-first layers (§6): the embedded miscollocation list first (same shape
as false friends, drops into the existing collocation family), then
grammar lite as its own suggestion family. Both are per-pair data, so
they slot naturally into the langpacks from step 2.
7. Learner-facing Chinese writing (the zh pair's second direction): spec it as
its own phase (§4) only after the pair model is proven on pt-PT/fr.
## 11. Questions for the reviewer (1–3 settled 2026-07-26)
1. ~~§3a's no-detector stance~~ **Settled: yes** — pass if either dictionary
accepts it, show both glosses on collision, no language detector.
2. ~~UI-copy extraction first?~~ **Settled: yes** — the extraction (§2) is a
prerequisite chore, done before pt-PT is wired.
3. ~~Growth journal framing~~ **Settled: build it** with the two framing rules
as hard constraints (growth only, self-comparison only).
4. When (not whether) to build the learner-facing hanzi direction of the zh
pair — after pt-PT/fr, or is it wanted sooner?
5. Spanish: worth asking DreamDict to grow an es dataset now, or park it?
6. Grammar lite (§6): hand-curate the rule pack from ESL teaching materials
(small, fully understood), or mine LanguageTool's open rule corpus for
vetted patterns (bigger head start, needs licensing + quality triage)?
cmd/server/main.go
+127 -15
View File
@@ -1,9 +1,11 @@
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"flag"
"io/fs"
"log"
"net/http"
@@ -12,12 +14,14 @@ import (
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/config"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/docs"
"gitea.parodia.dev/drwily/petal/internal/images"
"gitea.parodia.dev/drwily/petal/internal/lexicon"
"gitea.parodia.dev/drwily/petal/internal/llm"
"gitea.parodia.dev/drwily/petal/internal/spell"
"gitea.parodia.dev/drwily/petal/internal/suggestions"
"gitea.parodia.dev/drwily/petal/internal/tts"
"gitea.parodia.dev/drwily/petal/internal/vocab"
@@ -25,8 +29,25 @@ import (
)
func main() {
backupTo := flag.String("backup", "",
"write a consistent copy of the database to this path and exit (no server)")
flag.Parse()
cfg := config.Load()
// Backup mode short-circuits before anything else starts: no migrations, no
// seed, no listener. It runs against the live database safely (VACUUM INTO
// takes only a read transaction), so the nightly job is
// docker compose exec petal /app/petal -backup /data/backups/<name>.db
// against the running container rather than a copy of three WAL files.
if *backupTo != "" {
if err := db.Backup(cfg.DatabasePath, *backupTo); err != nil {
log.Fatalf("backup: %v", err)
}
log.Printf("backup written to %s", *backupTo)
return
}
database, err := db.Open(cfg.DatabasePath)
if err != nil {
log.Fatalf("database: %v", err)
@@ -34,6 +55,52 @@ func main() {
defer database.Close()
log.Printf("database ready at %s", cfg.DatabasePath)
// Identity. With Authentik configured, Petal is an OIDC client in its own
// right: /auth/login starts a real login and the session cookie it issues is
// what every API request is resolved from. Without it — local development,
// and every deployment before auth landed — StaticResolver hands out the
// single hardcoded local user, so nothing about running Petal on a laptop
// changes.
sessions := auth.NewSessionStore(database.DB)
users := auth.NewUserStore(database.DB)
var resolver auth.Resolver = auth.StaticResolver(db.LocalUserID)
var oidcClient *auth.OIDC
if cfg.AuthEnabled() {
oidcClient = auth.NewOIDC(context.Background(), auth.Options{
IssuerURL: cfg.AuthentikURL,
ClientID: cfg.AuthentikClientID,
ClientSecret: cfg.AuthentikClientSecret,
BaseURL: cfg.BaseURL,
Allowed: auth.ParseAllowlist(cfg.AllowedSubs),
}, sessions, users)
resolver = sessions
if n, err := sessions.Prune(); err == nil && n > 0 {
log.Printf("auth: pruned %d expired session(s)", n)
}
log.Printf("auth: OIDC enabled (issuer=%s, redirect=%s)", cfg.AuthentikURL, oidcClient.RedirectURI())
} else {
log.Printf("auth: OIDC not configured — running as the single %q user", db.LocalUserID)
}
// The dictionary behind word lookups. dict.db is DreamDict's built database
// — French, European Portuguese, Spanish and Mandarin in one read-only file
// beside petal.db. It is optional on purpose: a laptop checkout has never
// had one, and the Chinese pair doesn't need one, so its absence downgrades
// lookups rather than stopping Petal. A file that is present but broken is
// a different matter and gets said out loud.
dict, err := lexicon.OpenDreamDict(cfg.DictPath)
if err != nil {
log.Printf("dictionary: %s unusable (%v) — falling back to the embedded datasets", cfg.DictPath, err)
}
defer dict.Close()
lexSet := lexicon.NewSet(dict)
if lexSet.HasDreamDict() {
log.Printf("dictionary: DreamDict open at %s (%s)", cfg.DictPath, lexSet.Contents())
} else {
log.Printf("dictionary: no dict.db at %s — English/Chinese only", cfg.DictPath)
}
r := chi.NewRouter()
r.Use(middleware.RequestID)
r.Use(middleware.RealIP)
@@ -65,6 +132,29 @@ func main() {
_, _ = w.Write([]byte(`{"version":"` + version + `"}`))
})
// Everything below serves or mutates a particular user's data, so it sits
// behind the auth middleware. /health and /version deliberately stay
// outside it: they carry no user data, and a monitoring probe (or the
// client's update poll) must not need a session to reach them.
//
// The middleware resolves the caller once and hands handlers the answer via
// auth.UserID(r.Context()). Which resolver it runs is the only thing that
// changed when auth landed: the session store in a deployment with
// Authentik configured, the static local user otherwise. No handler or
// query moved for either.
api.Group(func(pr chi.Router) {
pr.Use(auth.Middleware(resolver))
// Who am I? The frontend namespaces its per-account browser state by
// this id and shows the signed-in writer.
pr.Get("/me", users.MeHandler())
// …and the one thing about herself she can change: which language
// Petal is her pair in. It lives here rather than under a /settings
// tree because there is exactly one setting and it is a property of
// the user row — the same row /me reads back.
pr.Patch("/me", users.UpdateMeHandler())
llmClient := llm.NewLLMClient(cfg)
sug := suggestions.New(database, llmClient)
@@ -73,41 +163,63 @@ func main() {
docsHandler := docs.New(database)
docsRouter := docsHandler.Routes()
sug.RegisterDocRoutes(docsRouter)
api.Mount("/docs", docsRouter)
pr.Mount("/docs", docsRouter)
// Tag management (the roster) and cross-document full-text search.
api.Mount("/tags", docsHandler.TagRoutes())
api.Mount("/search", docsHandler.SearchRoutes())
pr.Mount("/tags", docsHandler.TagRoutes())
pr.Mount("/search", docsHandler.SearchRoutes())
// Per-suggestion actions (accept/dismiss) under /api/suggestions.
api.Mount("/suggestions", sug.Routes())
pr.Mount("/suggestions", sug.Routes())
// Offline lexicon: full word lookups (gloss + definition + synonyms) for
// the right-click popover, and the lightweight Chinese-only gloss for the
// inline hover/select tooltip. One handler so the datasets load once.
lex := lexicon.NewHandler()
api.Mount("/word", lex.Routes())
api.Mount("/gloss", lex.GlossRoutes())
// the right-click popover, and the lightweight gloss-only lookup for the
// inline hover/select tooltip. One handler over one provider Set, so the
// embedded datasets and dict.db are each opened once. Which of them
// answers depends on the caller's language pair — so unlike before, the
// response is no longer identical for everyone, and it stays behind auth
// for that reason as much as for the API surface.
lex := lexicon.NewHandler(database.DB, lexSet)
pr.Mount("/word", lex.Routes())
pr.Mount("/gloss", lex.GlossRoutes())
// Vocabulary garden: words the writer looks up are captured here and
// surfaced for gentle spaced-repetition review.
api.Mount("/vocab", vocab.New(database).Routes())
pr.Mount("/vocab", vocab.New(database).Routes())
// Editor image uploads, stored on disk and served back by content hash.
imgHandler, err := images.New(cfg.ImageDir)
// The personal spelling dictionary — the words she's told Petal to stop
// flagging. Kept server-side (rather than in the browser) so it belongs
// to her account and follows her between devices.
pr.Mount("/spell", spell.New(database).Routes())
// Editor image uploads, stored on disk and served back by content hash
// to whoever owns them. Files already on disk from before ownership
// existed are claimed for the local user at startup.
imgHandler, err := images.New(cfg.ImageDir, database.DB, db.LocalUserID)
if err != nil {
log.Fatalf("image store: %v", err)
}
api.Mount("/images", imgHandler.Routes())
pr.Mount("/images", imgHandler.Routes())
// Read-aloud: proxy short passages to a local Piper TTS server. Only
// mounted when TTS_ENDPOINT is configured; otherwise the frontend falls
// back to the browser's Web Speech API on its own.
if ttsHandler, ok := tts.New(cfg); ok {
api.Mount("/tts", ttsHandler.Routes())
log.Printf("read-aloud enabled (TTS endpoint=%s)", cfg.TTSEndpoint)
pr.Mount("/tts", ttsHandler.Routes())
// Name the languages, not just the English endpoint: which
// voices a deployment actually reached is the thing worth
// seeing at boot, and a missing sidecar is silent otherwise
// (a 404 the client answers by quietly using Web Speech).
log.Printf("read-aloud enabled (voices: %s)", strings.Join(ttsHandler.Languages(), ", "))
}
})
})
// Login lives outside /api: these are browser navigations, and they must be
// reachable without a session — that is their entire job.
if oidcClient != nil {
r.Mount("/auth", oidcClient.Routes())
}
// Everything else: serve the embedded SPA (with index.html fallback for client routing).
r.NotFound(spaHandler())
deploy/README.md
+648 -48
View File
@@ -1,57 +1,573 @@
# Deploying read-aloud (Piper TTS) to millenia
# Deploying Petal
Petal's read-aloud generates audio with a local **Piper** neural-TTS server and
transcodes it to mp3 with **ffmpeg**. Both run on millenia (192.168.1.212);
nothing leaves the box. If Piper is down or `TTS_ENDPOINT` is unset, the frontend
falls back to the browser's Web Speech API automatically.
Two deployments exist right now:
## 1. Piper as a systemd service (one-time)
| | host | shape | status |
|---|---|---|---|
| **parodia** | `petal.parodia.dev` / `100.64.0.1` | docker compose behind the host's Traefik | **canonical** since 2026-07-27 — she signs in here |
| **millenia** | `192.168.1.212` / `100.64.0.2` | `petal.service`, bare binary on `:8088`, Piper as user systemd units | frozen fallback: a copy of her writing as it stood at the move, still owned by the pre-auth `local` user |
Her writing moved to the VPS when sign-in landed (Phase 16/17): it is the
instance that actually authenticates, its data directory is LUKS-encrypted, and
it is reachable from anywhere. millenia was left running and untouched as a
fallback — but the two diverge the moment anything is written on either, so it
should be retired rather than kept in step. Everything below is the VPS side;
the millenia Piper notes are kept in the appendix because that instance still
runs them.
---
## 1. The VPS stack
`docker-compose.yml` at the repo root brings up three containers:
- **petal** — the single Go binary with the frontend embedded. Publishes no host
port; Traefik is the only way in.
- **piper-en** / **piper-zh** / **piper-pt** / **piper-fr** — read-aloud. Each
Piper HTTP server loads exactly one voice, so every language is its own
container off one image, with the models cached in a shared volume. They sit
on an internal network with no published ports, so only Petal can reach them.
Adding pt-PT in Phase 21 was a third service and fr in Phase 24 a fourth —
never a new image, and since Phase 21 never any Go either (the languages are
discovered from `TTS_ENDPOINT_<LANG>`/`TTS_VOICE_<LANG>`).
They run as containers rather than the host systemd units millenia uses because
Piper was never actually installed on the VPS, and the `reala` account has no
lingering session to keep user units alive across logout.
### Prerequisites on the host
- Docker with the compose plugin, and the existing external `traefik` network
- A DNS A record for the hostname pointing at the VPS (`petal.parodia.dev` is
already in place)
### First deploy
```bash
ssh reala@100.64.0.1
git clone https://gitea.parodia.dev/drwily/petal.git ~/petal
cd ~/petal
cp deploy/petal.env.example .env
```
Then edit `.env`:
- `PETAL_UID` / `PETAL_GID` — `id -u` / `id -g` for this account. `./data` is a
bind mount, so the image's own `petal` user has no claim on it; a mismatch
shows up as `unable to open database file (14)` and a restart loop.
- `AUTHENTIK_URL` / `AUTHENTIK_CLIENT_ID` / `AUTHENTIK_CLIENT_SECRET` /
`PETAL_ALLOWED_SUBS` — sign-in, see §4. Without them Petal runs as the single
`local` user and must not be exposed.
- `LLM_MODEL` / `LLM_CHAT_MODEL` — see §3.
```bash
mkdir -p data/backups
docker compose up -d --build
docker compose ps # all three healthy
```
### Updating
```bash
cd ~/petal && git pull && docker compose up -d --build
```
The frontend is embedded in the binary, so a rebuild is the whole deploy. The
client polls `/api/version` (a hash of the built `index.html`) and offers a
refresh when it changes.
---
## 2. What Traefik does
Labels follow the convention the other services on this box use: the external
`traefik` network, the `web-secure` entrypoint, the `default` cert resolver and
`compression@file`. Petal adds its own response-header middleware
(`frame-ancestors 'self'`, HSTS, nosniff, `Referrer-Policy: same-origin`).
There is no auth middleware at the edge: Petal does its own (§4). `/api/health`
and `/api/version` sit outside Petal's own auth for the same reason they always
did — a monitoring probe must not need a session, and neither carries user
data.
---
## 3. The LLM link over headscale
The vLLM backend stays on millenia and is reached over headscale
(`100.64.0.2`). **This is the only cross-VPN dependency**, and by the
LLM-minimalism principle it never gates essential functionality — spell check,
gloss, vocabulary garden, search, export and read-aloud all keep working with
the link down, and the status bar shows the warm
`🌙 小助手在休息 · Petal's helper is resting · 文字已保存`.
`LLM_TIMEOUT` is raised from the local-network default of 30s to **90s**: the
voice and collocation passes send a whole document, the timeout is a hard
deadline on the completion call, and a WAN+VPN round trip eats the margin.
### How the link is exposed — a forwarder, not a rebind
vLLM stays bound to `127.0.0.1:8000`. `vllm-headscale-proxy.service` (a socat
unit, in this directory) adds a second listener on `100.64.0.2:8000` that
forwards to it.
The plan originally said to rebind vLLM itself. That turned out to be the
expensive option: `vllm-chat.service` is **shared** — Petal, Gogobee and Open
WebUI all point at `127.0.0.1:8000`, and Open WebUI stores its endpoint in its
own database rather than in env — so moving the bind address would mean editing
three consumers and reloading a 35B AWQ model, minutes of downtime for all of
them. The forwarder adds a door instead of moving one: local callers are
untouched, and the only new exposure is on the VPN interface.
It binds `100.64.0.2` specifically, **never** `0.0.0.0`: the far end of this
link is a public host, and the LAN has no business seeing an unauthenticated
inference endpoint.
```bash
sudo install -m 0644 deploy/vllm-headscale-proxy.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now vllm-headscale-proxy
ss -lntp | grep 8000 # expect BOTH 127.0.0.1:8000 and 100.64.0.2:8000
```
The model id (`qwen3.6-35b`) goes into `LLM_MODEL` / `LLM_CHAT_MODEL` on the
VPS. Verified end to end: a grammar checkpoint from `petal.parodia.dev` returns
real suggestions in ~3s over the VPN.
---
## 4. Sign-in (Authentik OIDC)
Petal is an OIDC client in its own right: it runs the login itself rather than
trusting a header from the proxy. Nothing about the container has to be
unreachable for that to be safe.
Login turns on only when `AUTHENTIK_URL`, `AUTHENTIK_CLIENT_ID` and
`AUTHENTIK_CLIENT_SECRET` are all set. With any of them missing Petal falls back
to the single hardcoded `local` user — which is what local development wants,
and what every deployment did before this landed. A host serving the public
must have them set.
### Register Petal in Authentik
In the Authentik admin UI (**Applications → Providers → Create → OAuth2/OpenID
Provider**):
| Field | Value |
| --- | --- |
| Client type | Confidential |
| Redirect URI | `https://petal.parodia.dev/auth/callback` (strict) |
| Scopes | `openid`, `profile`, `email` |
| Signing key | any (Petal fetches the JWKS from discovery) |
Then create an **Application** bound to that provider, and copy the client id,
the client secret, and the provider's **OpenID Configuration Issuer** (it looks
like `https://auth.parodia.dev/application/o/petal/` — the issuer, not the
`.well-known` URL; Petal appends that itself).
Put them in `.env`:
```
AUTHENTIK_URL=https://auth.parodia.dev/application/o/petal/
AUTHENTIK_CLIENT_ID=…
AUTHENTIK_CLIENT_SECRET=…
PETAL_ALLOWED_SUBS=her@example.com,me@example.com
```
`PETAL_ALLOWED_SUBS` is the guest list: comma-separated OIDC subject ids and/or
email addresses. Authentik fronts several applications on this host, and being a
valid user there does not mean being a user here. Leaving it empty lets in
everyone Authentik authenticates. Emails are accepted alongside subject ids
precisely so the list can be written *before* anyone has logged in — a subject
is an opaque uuid that doesn't exist until first sign-in.
A valid login that isn't on the list gets a warm bilingual "this Petal isn't
yours to write in" page, and no account is provisioned.
### Checking it
```bash
curl -si https://petal.parodia.dev/api/docs | head -1 # 401 without a session
curl -si https://petal.parodia.dev/auth/login | grep -i location # → Authentik
docker compose logs petal | grep '^.*auth:' # issuer + redirect at boot
```
The startup log prints the redirect URI it will use; if Authentik rejects the
login with a redirect-uri mismatch, compare that line against what's registered.
Two things bit this deployment, both worth checking first if a login dies early:
- **The issuer's trailing slash is significant.** Authentik's is
`…/application/o/petal/`, OIDC requires the discovered issuer to match the
configured one byte-for-byte, and normalising the slash away makes discovery
fail with `did not match the issuer URL returned by provider`.
- **A provider created through the API or `ak shell` has an empty
`grant_types`**, which authentik reads as "no grant type is permitted here"
and answers with `invalid_request` / *The request is otherwise malformed*
before the login page ever appears. The admin UI fills the list in for you;
scripted creation must set it (`authorization_code`, `refresh_token`).
Discovery is lazy and retried, so an Authentik outage blocks *new* logins but
leaves existing sessions working — those only need Petal's own database.
### Sessions
Opaque token in a `petal_session` cookie (`HttpOnly`, `SameSite=Lax`, `Secure`
on https); the `sessions` table stores only its SHA-256, so a database copy
yields nothing usable. Thirty-day sliding expiry — every request pushes it out,
throttled to one write an hour. `/auth/logout` deletes the row, not just the
cookie. Expired rows are pruned at startup.
To sign someone out everywhere immediately:
```bash
docker compose exec petal sh -c \
"sqlite3 /data/petal.db \"DELETE FROM sessions WHERE user_id = '<sub>'\""
```
### The edge gate is gone
Until Phase 16 there was a Traefik basic-auth middleware in front of everything,
because Petal authenticated nobody and a public hostname was a public API. It
was removed when OIDC went live on 2026-07-27, together with the separate
unauthenticated `/api/health` router that existed only to escape it: every `/api`
route now answers 401 without a session, and the only thing an anonymous visitor
gets is the app shell and a redirect to sign in.
If you ever run this stack *without* `AUTHENTIK_*` configured — Petal then falls
back to the single `local` user — put the gate back before pointing DNS at it:
```yaml
traefik.http.routers.petal.middlewares: compression@file,petal-headers,petal-auth
traefik.http.middlewares.petal-auth.basicauth.users: ${PETAL_BASIC_AUTH:?}
```
with `htpasswd -nbB petal 'your-password'` in `.env` as `PETAL_BASIC_AUTH`.
---
## 4a. Moving an account (`scripts/migrate_local_user.py`)
Petal ran as one hardcoded user (`users.id = 'local'`) before sign-in existed.
Moving that writing onto a real account is a deliberate, one-off operation:
```bash
docker compose stop petal
python3 scripts/migrate_local_user.py data/petal.db --to <oidc-sub> # dry run
python3 scripts/migrate_local_user.py data/petal.db --to <oidc-sub> \
--email her@example.com --name "Her Name" --apply
docker compose up -d petal
```
The subject id is **knowable before she has ever logged in**. With authentik's
default `hashed_user_id` sub mode it is the user's `uid`:
```bash
docker exec authentik-server-1 ak shell -c \
"from authentik.core.models import User; print(User.objects.get(username='claire').uid)"
```
so the data can move first and she signs in to find it already there.
**Start Petal once on the incoming database before migrating.** A database
carried over from another instance may be a schema behind, and the app applies
migrations at startup; the script moves rows and does not touch the schema.
The script is dry-run by default, takes its own `VACUUM INTO` backup, runs as
one transaction with foreign keys off, and verifies the row counts before it
commits. It refuses to run while anything else has the database open, and
refuses to merge into an account that already owns writing.
**`local` comes back, and that's expected.** `db.Open` seeds that row on every
startup, so it reappears the moment Petal restarts after a migration. It owns
nothing — the writing is on the real account — and it is only ever resolved to
by `StaticResolver`, which a deployment with `AUTHENTIK_*` set never uses. Check
`SELECT COUNT(*) FROM documents WHERE user_id = 'local'` if you want to be sure
a migration took; the presence of the row itself says nothing.
---
## 4b. The dictionary (`dict.db`)
Word lookups for the French, European Portuguese and Spanish pairs come from
[DreamDict](https://github.com/prosolis/dreamdict)'s built database, which Petal
opens **read-only** beside `petal.db`. Petal imports DreamDict's `dictionary`
package directly — there is no DreamDict service to run and nothing to reach
over the VPN, which matters because a hover gloss must answer in milliseconds.
`dict.db` is **optional**. With no file at `DICT_PATH` Petal logs
```
dictionary: no dict.db at /data/dict.db — English/Chinese only
```
and serves lookups from the datasets compiled into the binary. The Chinese pair
is unaffected either way — it stays on ECDICT (see below) — and a non-Chinese
writer still gets English definitions, synonyms and pronunciation, losing only
the translation. **A dictionary that failed to deploy costs the gloss, not the
popover.** A file that is present but was never imported is a different matter
and is logged as an error.
### Installing it
The database is built by DreamDict's own import CLI from ~6 GB of source data;
it is not built on the VPS. Copy the built file into the data volume:
```bash
# on the machine holding a built dict.db (millenia: ~/dreamdict/data/dict.db)
scp ~/dreamdict/data/dict.db reala@100.64.0.1:/home/reala/petal/data/dict.db
# on parodia
chown "$(id -u):$(id -g)" /home/reala/petal/data/dict.db
docker compose restart petal # the handle is opened once, at startup
```
Expect ~450 MB. It sits inside the LUKS volume with everything else (§6). The
backups name `petal.db` explicitly rather than sweeping the data directory
(§5), so `dict.db` stays out of them — which is the right outcome and worth
keeping: it is rebuildable from public data and would otherwise dominate every
nightly snapshot. Petal never writes to it.
### Why Chinese doesn't use it
The zh pair stays on the embedded ECDICT gloss, deliberately. Measured on the
deployed database, DreamDict reaches a Chinese gloss for 53% of the 2,000
commonest English words; ECDICT covers essentially all of them and is in daily
use by a real writer. `lexicon.Set.For` is where that decision lives — one
`switch`, changed the day a comparison on her actual lookups says otherwise.
For pt-PT and French the same measurement reads 62% and 63%, which is why they
use DreamDict: there is no alternative source for them at all.
### Rebuilding it
Rebuilt 2026-07-27 to add Spanish (the previous file predated DreamDict's
Spanish support). The recipe, since it will be needed again:
```bash
# on millenia, from a clean checkout of dreamdict main
./scripts/download-dict-data.sh ~/dreamdict/data # idempotent; skips what's there
go run ./cmd/dictimport --data ~/dreamdict/data --db ./dict.db --clean
```
~6 minutes on 32 cores; the data directory is ~7 GB and mostly already
downloaded. **Build to a new path, never over a file in use** — then verify by
hash on both ends before swapping.
Two things worth knowing before trusting a rebuild:
- The SUBTLEX-US download fails (the source moved behind a manual export). It
does not matter: the loader falls back to `SUBTLEX-US.txt`, which is present,
and English "frequency" is mostly SCOWL's commonness bucket anyway —
1000/800/600/…/50, refined by SUBTLEX for only ~1,600 words. That is why the
word-difficulty chip reads `difficulty`, not `frequency`.
- Check the *other* languages' counts are unchanged before shipping. The 2026-07
rebuild came out byte-identical for en/fr/pt-PT/zh, which is what says it
added a language rather than quietly shifting the rest.
Gloss coverage of the 2,000 commonest English words, after the rebuild:
**es 68.6%**, fr 63.1%, pt-PT 62.1%, zh 53.2%. The startup line reports actual
per-language row counts, so a database missing a language says so.
---
## 5. Backups
### On the VPS — folded into `parodia-backup`
Petal rides the host's existing offsite job (`/usr/local/bin/parodia-backup`,
`parodia-backup.timer`, nightly ~03:40): age-encrypted to S3, 14-day retention,
dead-man snitch. The host holds only the age *public* recipient, so it writes
backups it cannot itself decrypt.
```
push petal.db.age sqlite_file_dump /home/reala/petal/data/petal.db
```
`/home/reala/petal/.env` is in the same job's secrets tarball — it carries the
interim basic-auth hash and, from Phase 16, the OIDC client secret.
**Why `sqlite_file_dump` and not the script's existing `sqlite_dump`:** that
helper uses Python's `iterdump`, which **does not reproduce an FTS5 virtual
table**. It emits `documents_fts` as a raw `sqlite_master` row plus its shadow
tables, and replaying the result dies with `no such table: documents_fts` —
verified by round-tripping a real dump on 2026-07-27. Petal's cross-document
search would have been silently missing after any restore. `sqlite_file_dump`
runs `VACUUM INTO` instead: a genuine database file, virtual tables intact, WAL
folded in, no write lock. Restore is a copy rather than a replay.
> If `apply.db` ever gains a virtual table, it needs the same treatment.
### Restore (VPS)
```bash
age -d -i <offline-identity> petal.db.age > /tmp/petal.db # from S3
cd ~/petal
docker compose stop petal # stop writers first
mv data/petal.db data/petal.db.before-restore # keep the current state
rm -f data/petal.db-wal data/petal.db-shm # a stale WAL against a new file
cp /tmp/petal.db data/petal.db
docker compose start petal
docker compose logs petal --tail 5 # expect "database ready"
```
To sanity-check an archive before committing to it, have Petal open it in a
scratch directory — a clean exit means it reads end to end:
```bash
mkdir -p /tmp/restore-check && cp /tmp/petal.db /tmp/restore-check/petal.db
docker run --rm -v /tmp/restore-check:/data --user "$(id -u):$(id -g)" \
--entrypoint sh petal:local -c '/app/petal -backup /data/verify.db'
```
### On millenia — `petal-backup.timer`
Until 2026-07-27 her actual writing had **no scheduled backup at all**; the
newest snapshot was a month old. It now runs nightly at 03:20
(`Persistent=true`, because the box isn't on 24/7 and a missed window would
otherwise be skipped silently):
```bash
sudo install -m 0644 deploy/petal-backup.service deploy/petal-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now petal-backup.timer
sudo systemctl start petal-backup.service # prove it before trusting it
```
`deploy/backup-petal.sh` snapshots via `petal -backup`, gzips, **age-encrypts
with the parodia public recipient**, pushes to the VPS over headscale with a
post-transfer size check, and prunes both ends. The private identity is offline,
so neither millenia nor the VPS can decrypt what it is holding — verified.
A manual snapshot any time, no tooling required:
```bash
cd ~/petal && ./petal -backup ~/petal/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).db
```
Restore is a copy — stop Petal, drop the file in as `data/petal.db`, remove any
stale `-wal`/`-shm`, start.
---
## 6. Encryption at rest
### VPS — `/home/reala/petal/data` is a LUKS volume
`deploy/setup-encrypted-data.sh` puts the data directory on LUKS2 over a sparse
file at `/var/lib/petal-crypt.img`. That covers `petal.db`, uploaded `images/`,
**and the TTS cache** — which is synthesized audio of her sentences and is easy
to forget.
LUKS-on-a-file rather than gocryptfs because Petal is SQLite in WAL mode: WAL
needs a shared-memory index (`-shm`) mapped consistently across processes, and
FUSE has a long history of subtle mmap/locking differences. A block device with
ext4 behaves exactly like a disk to SQLite, which is the only guarantee worth
having under a database.
**What it protects, honestly.** The key lives at `/etc/petal/dataset.key` on the
same host so the volume auto-unlocks at boot. That is a deliberate availability
tradeoff:
| | |
|---|---|
| protects against | a decommissioned or resold disk; reading the raw block device; casual browsing of a filesystem snapshot that excludes `/etc` |
| does **not** protect against | anyone holding the whole VM image — they get the keyfile with the ciphertext; or anything at all while the host is running and mounted |
Real protection from a provider-side snapshot needs the key off-box (fetched
over the VPN at boot). Considered, not chosen.
Two things this setup got wrong the first time, both caught by rehearsing a
reboot rather than trusting a clean run — worth knowing if you rebuild it:
- **Mounting over a directory hides its contents, it does not remove them.** The
first pass left the original plaintext `petal.db` and WAL sitting on the
unencrypted root filesystem, invisible under the mount. The script now shreds
the originals before mounting and refuses to continue if the mountpoint will
not come up empty.
- **`systemd-cryptsetup` was not installed**, so `/etc/crypttab` was ignored
entirely and the volume would never have unlocked at boot. The script now
refuses to run without the generator present.
Check it any time:
```bash
sudo ./deploy/setup-encrypted-data.sh --status
```
### The mount-liveness guard
The mountpoint directory exists whether or not the volume is mounted, so a boot
where the unlock failed would start Petal against an empty unencrypted
directory and quietly serve a blank database — the failure that looks like data
loss. `data/.volume-ok` lives on the encrypted filesystem and is bind-mounted
with `create_host_path: false`, turning that into a loud container start
failure:
```
Error response from daemon: invalid mount config for type "bind":
bind source path does not exist: /home/reala/petal/data/.volume-ok
```
Verified by unmounting and attempting a start.
**A true reboot has not been tested** — the VPS also runs matrix, lemmy, akkoma,
gitea and authentik, so rebooting it is your call. The boot path was rehearsed
through `local-fs.target`, which pulls the mount, which pulls the unlock.
### millenia is not encrypted at rest
LVM, no LUKS. Her canonical writing sits in plaintext on the home box. Backups
leaving it are age-encrypted; the disk itself is not.
---
## 7. Supervision and monitoring
Petal on millenia ran for months as a bare `./petal` with PPID 1 — no unit, no
screen session — so a crash or reboot left it down until someone noticed. It is
now `petal.service`:
```bash
sudo install -m 0644 deploy/petal.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now petal.service
```
Verified by `kill -9`-ing it and watching systemd bring it back.
**Piper's silent-failure mode is fixed.** Both units now set
`StartLimitIntervalSec=300` / `StartLimitBurst=5`. With `RestartSec=3` and
systemd's default 10-second window, only ~3 restarts ever landed inside it, so
the burst limit was never reached and a dead service looped **26,800+ times over
a day without ever entering `failed`**. A genuinely broken Piper now shows up in
`systemctl --user --failed`.
### Still to do — an external probe
Nothing yet watches millenia from outside. uptime-kuma already runs on the VPS
and can reach millenia over headscale, so the missing piece is two monitors
(they need the uptime-kuma UI, hence not scripted here):
- `http://100.64.0.2:8088/api/health` — Petal itself
- a POST to `http://100.64.0.2:8088/api/tts` — catches a dead Piper, which
`/api/health` will not, because read-aloud degrades silently to browser
speech
---
## Appendix — Piper on millenia (user systemd units)
millenia still runs Piper as user services; these are the original notes.
```bash
# from this repo, on your workstation:
scp deploy/piper.service deploy/setup-piper.sh 192.168.1.212:/tmp/
ssh 192.168.1.212 'cd /tmp && sudo ./setup-piper.sh'
```
`setup-piper.sh` creates `~/piper/venv`, installs `piper-tts[http]`, downloads the
`en_US-amy-medium` voice into `~/piper/voices`, installs+enables `piper.service`
(loopback :5005), and smoke-tests it. Idempotent.
`setup-piper.sh` creates `~/piper/venv`, installs `piper-tts[http]`, downloads
`en_US-amy-medium` into `~/piper/voices`, installs and enables `piper.service`
(loopback `:5005`), and smoke-tests it. Idempotent. Check it with
`systemctl status piper` / `journalctl -u piper -f`.
Check it any time: `systemctl status piper`, `journalctl -u piper -f`.
## 2. Point petal at Piper + redeploy the binary
Add to petal's environment (its `.env` or launch env):
```
TTS_ENDPOINT=http://127.0.0.1:5005
TTS_VOICE_EN=en_US-amy-medium
TTS_AUDIO_FORMAT=mp3
```
Then ship the rebuilt binary (`go build -o petal ./cmd/server` already done) and
restart the petal `:8088` session. Confirm the log line:
`read-aloud enabled (TTS endpoint=http://127.0.0.1:5005)`.
## 3. Verify
```bash
# on millenia — end-to-end through petal, including ffmpeg transcode:
curl -sf -X POST localhost:8088/api/tts \
-H 'Content-Type: application/json' \
-d '{"text":"hello there","lang":"en-US"}' -o /tmp/petal-tts.mp3 \
&& file /tmp/petal-tts.mp3 # expect: Audio file ... MPEG ... layer III
```
- Second identical call is served from the cache (`~/petal/.../data/tts/*.mp3`).
- A language with no configured instance returns 404 → client uses Web Speech.
- Browser check via the uitest harness (`~/petal/uitest`): tap a word in the
WordCard / select a sentence and hit speak — expect the natural Piper voice.
## Chinese voice (live)
Each Piper HTTP server loads ONE model, so Chinese runs as a **second instance**:
`piper-zh.service` on :5006 with `zh_CN-huayan-medium`. Deployed via:
Chinese runs as a second instance (`piper-zh.service`, `:5006`,
`zh_CN-huayan-medium`):
```bash
scp deploy/piper-zh.service 192.168.1.212:~/.config/systemd/user/
@@ -60,6 +576,90 @@ ssh 192.168.1.212 'export XDG_RUNTIME_DIR=/run/user/$(id -u)
systemctl --user daemon-reload && systemctl --user enable --now piper-zh.service'
```
Then in petal's `start.sh`: `TTS_ENDPOINT_ZH=http://127.0.0.1:5006` and
`TTS_VOICE_ZH=zh_CN-huayan-medium`. The handler maps language → instance from config,
so adding more languages is just another instance + env pair (no code change).
Petal's env then carries `TTS_ENDPOINT=http://127.0.0.1:5005`,
`TTS_ENDPOINT_ZH=http://127.0.0.1:5006` and the matching voice ids. The handler
maps language → instance from config, so another language is another instance
plus an env pair, no code change.
**Adding a language (Phase 21 made this literal).** Petal discovers its Piper
instances from the environment: English is the unsuffixed
`TTS_ENDPOINT`/`TTS_VOICE_EN`, and every other language is a
`TTS_ENDPOINT_<LANG>`/`TTS_VOICE_<LANG>` pair. `<LANG>` is the *base* tag —
`PT`, not `PT_PT`, because an environment variable name cannot hold a hyphen and
only one Portuguese model is loaded regardless. Both halves must be set: an
endpoint with no voice is dropped, so a half-finished language reads to the
browser as "no voice here, use Web Speech" instead of erroring on every tap. The
startup line names what it actually resolved:
```
read-aloud enabled (voices: en=en_US-amy-medium, pt=pt_PT-tugão-medium, zh=zh_CN-huayan-medium)
```
**Portuguese: `pt_PT-tugão-medium` is the only European voice Piper ships.** The
other five `pt_*` models in the catalogue are all Brazilian, so the voice has to
be named explicitly for the same reason the Hunspell dictionary did (Phase 21):
the obvious default is the wrong country. Check what exists before assuming:
```bash
docker exec petal-piper-en python -c "import urllib.request,json; \
d=json.load(urllib.request.urlopen('https://huggingface.co/rhasspy/piper-voices/resolve/main/voices.json')); \
print([k for k in d if k.startswith('pt')])"
```
**French: the opposite situation, and worth knowing it is.** Every `fr_*` voice
in the catalogue is `fr_FR`, so there is no wrong country to land on by default
and no Québec voice to choose instead; `fr_FR-siwis-medium` is picked to match
the register of the other three rather than to avoid anything. The name is also
plain ASCII, so the entrypoint's percent-encoded download fallback — which
exists only because `tugão` broke `piper.download_voices` — never fires here.
**Slow replay.** `POST /api/tts` takes `slow: true`, which raises Piper's
`length_scale` to about 4/3 (≈0.75× pace). It is a separate cache entry, not a
playback-rate trick, so the slow clip is synthesized once and then instant.
**Piper version note:** piper-tts moved synthesis from `POST /` to
`POST /synthesize` in 1.6.0, with an identical request body. `TTS_PATH` selects
which — it defaults to `/`, and both the VPS compose and millenia's `start.sh`
now set `/synthesize`. If read-aloud starts returning 502 after a Piper upgrade,
that flag is the fix.
**The venv is fragile across Python upgrades.** On 2026-07-27 millenia's Piper
was found dead with **26,800+ failed restarts**, silently since the Jul 26
reboot — read-aloud had been falling back to browser Web Speech the whole time.
Root cause: an OS upgrade moved `/usr/bin/python3` from 3.13 to 3.14, and
`venv/bin/python3` is a *symlink to the system interpreter*, so the venv's
`lib/python3.13/site-packages` became invisible — `sys.path` contained no
site-packages at all. The failure surfaced as the misleading
`No module named piper.http_server` even though `http_server.py` was sitting
right there on disk.
Fix (what was done — recreating the venv, not repairing it):
```bash
systemctl --user stop piper.service piper-zh.service
mv ~/piper/venv ~/piper/venv.broken-py313
python3 -m venv ~/piper/venv
~/piper/venv/bin/pip install "piper-tts[http]"
~/piper/venv/bin/python -c 'import piper.http_server' # must not raise
systemctl --user start piper.service piper-zh.service
```
That reinstall lands 1.6.0, so it must be paired with `TTS_PATH=/synthesize` in
`start.sh` and a binary new enough to read that variable. Voices in
`~/piper/voices` survive and do not need re-downloading.
Worth knowing: `Restart=on-failure` will retry forever without ever alerting.
Neither service reports its health anywhere, which is why this went unnoticed
for a day. A `/api/tts` probe in uptime-kuma would have caught it.
Verify end to end (through Petal, including the ffmpeg transcode):
```bash
curl -sf -X POST localhost:8088/api/tts \
-H 'Content-Type: application/json' \
-d '{"text":"hello there","lang":"en-US"}' -o /tmp/petal-tts.mp3 \
&& file /tmp/petal-tts.mp3 # expect: MPEG ADTS, layer III
```
A second identical call is served from the cache; a language with no configured
instance returns 404 so the client falls back to Web Speech.
deploy/backup-petal.sh
Executable
+113
View File
@@ -0,0 +1,113 @@
#!/usr/bin/env bash
# Nightly off-box backup of Petal's database.
#
# ./backup-petal.sh # snapshot, compress, encrypt, push, prune
# ./backup-petal.sh --local-only # snapshot + prune, skip the remote push
#
# Used on millenia, driven by petal-backup.timer (see deploy/README.md). The
# VPS does not use this script -- Petal rides parodia-backup there.
#
# The snapshot goes through `petal -backup`, which uses SQLite's VACUUM INTO:
# one coherent file including anything still in the WAL, taken without a write
# lock, so it is safe against the live running app. That is why this script
# never touches petal.db / -wal / -shm directly — copying those three
# separately can capture a torn mid-checkpoint state.
#
# Everything below is overridable from the environment.
set -euo pipefail
# Stack directory (holds docker-compose.yml and ./data).
STACK_DIR="${STACK_DIR:-$HOME/petal}"
# Where snapshots land on the VPS before being pushed off-box. Inside ./data so
# the container can write it through the existing bind mount.
LOCAL_DIR="${LOCAL_DIR:-$STACK_DIR/data/backups}"
# Off-VPS destination: millenia over headscale. Empty disables the push.
REMOTE_HOST="${REMOTE_HOST:-100.64.0.2}"
REMOTE_USER="${REMOTE_USER:-}"
REMOTE_DIR="${REMOTE_DIR:-petal-backups}"
# Retention, in days, on each side.
KEEP_LOCAL_DAYS="${KEEP_LOCAL_DAYS:-7}"
KEEP_REMOTE_DAYS="${KEEP_REMOTE_DAYS:-30}"
# age public recipient. Set it and every archive is encrypted before it leaves
# (and at rest locally too); leave it empty and the script says so loudly.
AGE_RECIPIENT="${AGE_RECIPIENT:-}"
local_only=0
[ "${1:-}" = "--local-only" ] && local_only=1
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
name="petal-${stamp}.db"
cd "$STACK_DIR"
mkdir -p "$LOCAL_DIR"
snapshot="${LOCAL_DIR}/${name}"
# Two deployment shapes: the VPS runs the compose stack, millenia runs a bare
# binary. Either way the snapshot goes through `petal -backup` (VACUUM INTO),
# which is safe against the live process, so neither has to stop writing.
if [ -f "$STACK_DIR/docker-compose.yml" ] && docker compose ps --status running 2>/dev/null | grep -q petal; then
echo ">> snapshotting via the running container -> data/backups/${name}"
# ./data/backups on the host is the container's /data/backups.
docker compose exec -T petal /app/petal -backup "/data/backups/${name}"
elif [ -x "$STACK_DIR/petal" ]; then
echo ">> snapshotting via the local binary -> ${snapshot}"
# DATABASE_PATH must match the running instance; start.sh is the source of
# truth for it, so read it from there rather than guessing.
DB_PATH="$(sed -n 's/^export DATABASE_PATH=//p' "$STACK_DIR/start.sh" 2>/dev/null | tail -1)"
DATABASE_PATH="${DB_PATH:-$STACK_DIR/data/petal.db}" "$STACK_DIR/petal" -backup "$snapshot"
else
echo "no way to snapshot: neither a running petal container nor $STACK_DIR/petal" >&2
exit 1
fi
[ -s "$snapshot" ] || { echo "snapshot missing or empty: $snapshot" >&2; exit 1; }
echo ">> compressing"
gzip -9 "$snapshot"
archive="${snapshot}.gz"
# Encrypt with age when a recipient is configured. The recipient is a PUBLIC
# key -- this host can write backups it cannot itself decrypt, and the private
# identity stays offline. Same custody model as parodia-backup. Without this,
# an off-box copy is just her writing sitting in plaintext on another machine.
if [ -n "$AGE_RECIPIENT" ]; then
age -r "$AGE_RECIPIENT" -o "${archive}.age" "$archive"
shred -uz "$archive" 2>/dev/null || rm -f "$archive"
archive="${archive}.age"
else
echo " (AGE_RECIPIENT unset: this backup is NOT encrypted)" >&2
fi
echo " $(du -h "$archive" | cut -f1) ${archive}"
if [ "$local_only" -eq 0 ] && [ -n "$REMOTE_HOST" ]; then
target="${REMOTE_HOST}"
[ -n "$REMOTE_USER" ] && target="${REMOTE_USER}@${REMOTE_HOST}"
echo ">> pushing to ${target}:${REMOTE_DIR}/"
ssh -o BatchMode=yes "$target" "mkdir -p '${REMOTE_DIR}'"
scp -q -o BatchMode=yes "$archive" "${target}:${REMOTE_DIR}/"
# Verify by size rather than trusting scp's exit code alone — a truncated
# transfer that still exits 0 would leave a backup that only looks fine.
local_size="$(stat -c%s "$archive")"
remote_size="$(ssh -o BatchMode=yes "$target" "stat -c%s '${REMOTE_DIR}/$(basename "$archive")'")"
if [ "$local_size" != "$remote_size" ]; then
echo "size mismatch after transfer: local ${local_size}, remote ${remote_size}" >&2
exit 1
fi
echo " verified ${remote_size} bytes"
echo ">> pruning remote copies older than ${KEEP_REMOTE_DAYS} days"
ssh -o BatchMode=yes "$target" \
"find '${REMOTE_DIR}' \\( -name 'petal-*.db.gz' -o -name 'petal-*.db.gz.age' \\) -type f -mtime +${KEEP_REMOTE_DAYS} -delete"
elif [ "$local_only" -eq 1 ]; then
echo ">> --local-only: skipping the remote push"
else
echo ">> REMOTE_HOST is empty: skipping the remote push" >&2
fi
echo ">> pruning local copies older than ${KEEP_LOCAL_DAYS} days"
find "$LOCAL_DIR" \( -name 'petal-*.db.gz' -o -name 'petal-*.db.gz.age' \) -type f -mtime "+${KEEP_LOCAL_DAYS}" -delete
echo ">> done"
deploy/petal-backup.service
+23
View File
@@ -0,0 +1,23 @@
[Unit]
Description=Nightly backup of Petal's database (millenia)
Documentation=file:///home/reala/petal/deploy/README.md
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=reala
WorkingDirectory=/home/reala/petal
# Encrypted with the parodia age recipient before it leaves the box, then
# pushed to the VPS over headscale. The recipient is a public key and the
# private identity is offline, so neither millenia nor the VPS can decrypt what
# they are holding. Replaces nothing -- before this there was no scheduled
# backup of her writing at all; the newest snapshot on 2026-07-27 was a month
# old.
Environment=AGE_RECIPIENT=age19n4k55m9d50xew5vj2ehmcsf3wuj7fhgmfpckadpvcya4032q9dqrt4yjw
Environment=REMOTE_USER=reala
Environment=REMOTE_HOST=100.64.0.1
Environment=REMOTE_DIR=petal-backups-millenia
ExecStart=/home/reala/petal/deploy/backup-petal.sh
Nice=10
IOSchedulingClass=idle
deploy/petal-backup.timer
+12
View File
@@ -0,0 +1,12 @@
[Unit]
Description=Nightly Petal database backup (millenia)
[Timer]
OnCalendar=*-*-* 03:20:00
# The box is not on 24/7; without this a missed window would just be skipped
# and the backup would silently never run.
Persistent=true
RandomizedDelaySec=300
[Install]
WantedBy=timers.target
deploy/petal.env.example
+82
View File
@@ -0,0 +1,82 @@
# Petal — production environment for the parodia.dev VPS.
# Copy to the stack directory as `.env` (docker-compose.yml reads it via
# env_file) and fill in the model names. Values the image already fixes
# (PORT, DATABASE_PATH, IMAGE_DIR, TTS_CACHE_DIR, TTS endpoints) are set in
# docker-compose.yml, not here.
# --- Routing -----------------------------------------------------------------
# Must match the DNS A record and the Traefik Host() rule.
PETAL_HOST=petal.parodia.dev
# Absolute origin the app knows itself by. Phase 16's OIDC redirect URI is
# built from this, so it has to be the real public HTTPS origin.
BASE_URL=https://petal.parodia.dev
# The companion's bedtime nag and the night theme read the container clock.
TZ=Europe/Lisbon
# The container runs as this uid/gid so it can write the ./data bind mount.
# Set both to the output of `id -u` / `id -g` for the account owning the stack
# directory. Wrong values show up as "unable to open database file (14)".
PETAL_UID=1001
PETAL_GID=1001
# --- Interim edge gate (delete when Phase 16 auth lands) ---------------------
# Petal has no authentication of its own yet — StaticResolver hands every
# request the same local user — so Traefik holds the door with basic auth until
# the OIDC flow exists. user:bcrypt-hash, as produced by:
# htpasswd -nbB petal 'your-password'
# /api/health is deliberately exempt (its own router) so monitoring still works.
PETAL_BASIC_AUTH=
# --- LLM (millenia, over headscale) ------------------------------------------
# The only cross-VPN dependency. Petal degrades warmly when it's unreachable:
# spell check, gloss, garden, search, export and read-aloud all keep working and
# the status bar shows 小助手在休息 · Petal's helper is resting.
#
# 100.64.0.2 is millenia on the headscale network. vLLM must be bound to that
# interface (NOT 0.0.0.0 — this host is public); see deploy/README.md.
LLM_BACKEND=vllm
LLM_ENDPOINT=http://100.64.0.2:8000
LLM_MODEL=
LLM_CHAT_MODEL=
# 30s is the local-network default. Over WAN + VPN, with the voice and
# collocation passes sending a whole document, that truncates real work — the
# request is a hard deadline on Complete, and a timeout surfaces as the same
# warm 502 as an unreachable model. 90s leaves headroom without letting a
# genuinely wedged backend hang the pass forever.
LLM_TIMEOUT=90s
# --- Read-aloud (Piper sidecars) ---------------------------------------------
# Endpoints are wired in docker-compose.yml; these pick the voice each sidecar
# loads. Changing one means recreating that container so it downloads the model.
#
# A language is routable only when both halves are set — a TTS_ENDPOINT_XX with
# no TTS_VOICE_XX reads as "no voice for this language" and the browser's own
# synthesizer takes over, rather than as an instance that errors on every
# request. Adding es is a compose service plus a pair of lines here.
#
# pt_PT-tugão-medium is the only European Portuguese voice Piper ships; every
# other pt model in the catalogue is Brazilian. French has the opposite
# property — every fr voice in the catalogue is fr_FR — so there is no wrong
# country to land on and no non-ASCII name to trip the downloader.
TTS_VOICE_EN=en_US-amy-medium
TTS_VOICE_ZH=zh_CN-huayan-medium
TTS_VOICE_PT=pt_PT-tugão-medium
TTS_VOICE_FR=fr_FR-siwis-medium
TTS_AUDIO_FORMAT=mp3
TTS_TIMEOUT=15s
# --- Auth (Authentik OIDC) ---------------------------------------------------
# Authentik already runs on this host. Set all three and Petal authenticates
# for itself; leave any unset and it falls back to the single `local` user
# (which on a public host means the Traefik basic-auth gate must stay).
#
# AUTHENTIK_URL is the provider's issuer, and the redirect URI to register in
# Authentik is https://petal.parodia.dev/auth/callback.
# AUTHENTIK_URL=https://auth.parodia.dev/application/o/petal/
# AUTHENTIK_CLIENT_ID=petal
# AUTHENTIK_CLIENT_SECRET=
#
# Who may sign in: comma-separated subject ids and/or emails. Empty = anyone
# Authentik authenticates, which is wider than this instance wants.
# PETAL_ALLOWED_SUBS=
deploy/petal.service
+24
View File
@@ -0,0 +1,24 @@
[Unit]
Description=Petal writing editor (millenia)
# Petal ran unsupervised for a long time -- a bare ./petal with PPID 1, no unit
# and no screen session -- so a crash or a reboot left it silently down until
# somebody noticed. It also wants vLLM up first, though it degrades warmly if
# the model is unreachable, so this is Wants and not Requires.
After=network-online.target vllm-chat.service
Wants=network-online.target vllm-chat.service
[Service]
Type=simple
User=reala
WorkingDirectory=/home/reala/petal
# start.sh carries the environment (ports, LLM endpoint, Piper endpoints and
# TTS_PATH) and execs the binary, so the service supervises Petal itself rather
# than a shell wrapper.
ExecStart=/home/reala/petal/start.sh
Restart=on-failure
RestartSec=5
StandardOutput=append:/home/reala/petal/petal.log
StandardError=append:/home/reala/petal/petal.log
[Install]
WantedBy=multi-user.target
deploy/piper-zh.service
+8
View File
@@ -2,6 +2,14 @@
Description=Piper TTS HTTP server — Chinese voice (read-aloud backend for petal)
After=network-online.target
Wants=network-online.target
# Give up loudly instead of retrying forever. This service once failed 26,800+
# times over a day without anyone noticing: RestartSec=3 means only ~3 restarts
# land inside systemd's default 10s StartLimitIntervalSec, so the default burst
# of 5 was never reached and the unit never entered `failed`. Widening the
# window to 5 minutes makes a genuinely broken Piper show up in
# `systemctl --user --failed` while still riding out transient blips.
StartLimitIntervalSec=300
StartLimitBurst=5
[Service]
Type=simple
deploy/piper.service
+8
View File
@@ -2,6 +2,14 @@
Description=Piper TTS HTTP server (read-aloud backend for petal)
After=network-online.target
Wants=network-online.target
# Give up loudly instead of retrying forever. This service once failed 26,800+
# times over a day without anyone noticing: RestartSec=3 means only ~3 restarts
# land inside systemd's default 10s StartLimitIntervalSec, so the default burst
# of 5 was never reached and the unit never entered `failed`. Widening the
# window to 5 minutes makes a genuinely broken Piper show up in
# `systemctl --user --failed` while still riding out transient blips.
StartLimitIntervalSec=300
StartLimitBurst=5
[Service]
Type=simple
deploy/piper/Dockerfile
+37
View File
@@ -0,0 +1,37 @@
# Piper neural-TTS HTTP server — the read-aloud backend Petal proxies to.
#
# One image, any voice: the model is named by PIPER_VOICE at runtime and
# downloaded into the shared /voices volume on first start. Each Piper server
# loads exactly one voice, so a new language is a new service in
# docker-compose.yml, not a new image (English and Chinese today; pt-PT lands
# with the Portuguese pair).
#
# python:3.12 rather than 3.13 — piper-tts pulls onnxruntime, whose wheel
# coverage for 3.13 still lags.
FROM python:3.12-slim
RUN pip install --no-cache-dir "piper-tts[http]" \
&& useradd -m -u 10002 piper
ENV PIPER_VOICE=en_US-amy-medium \
PIPER_DATA_DIR=/voices \
PIPER_PORT=5000
RUN mkdir -p /voices && chown piper:piper /voices
VOLUME ["/voices"]
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
USER piper
EXPOSE 5000
# The server has no dedicated health route, so synthesizing a single word is
# the honest check: it proves the model loaded, not just that a port is open.
HEALTHCHECK --interval=60s --timeout=20s --start-period=180s --retries=3 \
CMD python -c "import os,urllib.request,json; \
urllib.request.urlopen(urllib.request.Request('http://127.0.0.1:'+os.environ['PIPER_PORT']+'/synthesize', \
data=json.dumps({'text':'ok','voice':os.environ['PIPER_VOICE']}).encode(), \
headers={'Content-Type':'application/json'}), timeout=15).read(1)"
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
deploy/piper/entrypoint.sh
Executable
+57
View File
@@ -0,0 +1,57 @@
#!/usr/bin/env bash
# Fetch the configured voice if the shared volume doesn't have it yet, then
# serve it. The download is the only step that needs the internet, and it runs
# once per voice for the life of the volume — Petal itself stays offline-first.
set -euo pipefail
voice="${PIPER_VOICE:?PIPER_VOICE must be set}"
data_dir="${PIPER_DATA_DIR:-/voices}"
port="${PIPER_PORT:-5000}"
if [ ! -f "${data_dir}/${voice}.onnx" ]; then
echo ">> downloading voice ${voice} into ${data_dir}"
# piper.download_voices cannot fetch a voice whose name isn't ASCII, and the
# only European Portuguese voice in the catalogue is pt_PT-tugão-medium:
# the downloader pastes the name straight into the request line, and
# http.client encodes that as ASCII, so it dies with UnicodeEncodeError on
# the ã before a byte leaves the container. Every pt_BR voice downloads
# fine — the failure lands precisely on the voice the pt-PT pair needs.
#
# So: try the supported path, and fall back to fetching the two files
# ourselves with the URL percent-encoded, which is all the downloader was
# missing. Same host, same files, same destination names.
python -m piper.download_voices "${voice}" --data-dir "${data_dir}" || {
echo ">> download_voices failed for ${voice}; fetching directly (non-ASCII voice name)"
python - "${voice}" "${data_dir}" <<'PY'
import json, sys, urllib.parse, urllib.request
voice, data_dir = sys.argv[1], sys.argv[2]
BASE = "https://huggingface.co/rhasspy/piper-voices/resolve/main/"
catalogue = json.load(urllib.request.urlopen(BASE + "voices.json", timeout=120))
entry = catalogue.get(voice)
if entry is None:
sys.exit(f"no voice named {voice!r} in the catalogue")
# The catalogue keys the files by repo path; only the model and its config are
# needed to serve (MODEL_CARD is licence text).
for path in entry["files"]:
if not path.endswith((".onnx", ".onnx.json")):
continue
url = BASE + urllib.parse.quote(path)
dest = f"{data_dir}/{path.rsplit('/', 1)[-1]}"
print(f">> {url} -> {dest}", flush=True)
with urllib.request.urlopen(url, timeout=600) as r, open(dest, "wb") as out:
while chunk := r.read(1 << 20):
out.write(chunk)
PY
}
fi
echo ">> serving ${voice} on :${port}"
# 0.0.0.0 is safe here: the container sits on Petal's internal compose network
# with no published ports, so only Petal can reach it.
exec python -m piper.http_server \
-m "${voice}" \
--data-dir "${data_dir}" \
--host 0.0.0.0 --port "${port}"
deploy/setup-encrypted-data.sh
Executable
+149
View File
@@ -0,0 +1,149 @@
#!/usr/bin/env bash
# One-time setup: put Petal's data directory on an encrypted volume.
#
# sudo ./setup-encrypted-data.sh # create + migrate + persist
# sudo ./setup-encrypted-data.sh --status # report, change nothing
#
# WHAT THIS DOES AND DOES NOT PROTECT
# -----------------------------------
# The volume auto-unlocks from a keyfile stored on the same host. That is a
# deliberate choice (availability over paranoia), and it means:
#
# protects against : a decommissioned or resold disk, someone reading the
# raw block device, casual browsing of a filesystem-level
# snapshot that does not include /etc
# does NOT protect : anyone who takes the whole VM image -- they get
# against /etc/petal/dataset.key along with the ciphertext; and
# anything at all once the host is running and mounted
#
# For real protection against a provider-side snapshot the key has to live off
# the box (fetched over the VPN at boot). That was considered and not chosen.
#
# WHY LUKS-ON-A-FILE RATHER THAN gocryptfs
# ----------------------------------------
# Petal is SQLite in WAL mode. WAL needs a shared-memory index (-shm) mapped
# consistently across processes, and FUSE filesystems have a long history of
# subtle mmap/locking differences. A LUKS block device with ext4 on top behaves
# exactly like a normal disk to SQLite, which is the only guarantee worth having
# under a database.
set -euo pipefail
IMG="${IMG:-/var/lib/petal-crypt.img}"
SIZE="${SIZE:-8G}"
MAPPER_NAME="${MAPPER_NAME:-petal-data}"
KEYFILE="${KEYFILE:-/etc/petal/dataset.key}"
MOUNTPOINT="${MOUNTPOINT:-/home/reala/petal/data}"
STACK_DIR="${STACK_DIR:-/home/reala/petal}"
OWNER_UID="${OWNER_UID:-1001}"
OWNER_GID="${OWNER_GID:-1001}"
[ "$(id -u)" -eq 0 ] || { echo "must run as root" >&2; exit 1; }
# systemd-cryptsetup ships the generator that turns /etc/crypttab into units.
# On a minimal Debian it is NOT installed, and without it crypttab is silently
# ignored -- the volume simply never unlocks at boot. Found the hard way.
if [ ! -x /usr/lib/systemd/system-generators/systemd-cryptsetup-generator ]; then
echo "!! systemd-cryptsetup-generator is missing: /etc/crypttab would be ignored at boot."
echo " install it first: apt-get install systemd-cryptsetup"
exit 1
fi
status() {
echo "image : $IMG $( [ -f "$IMG" ] && echo "($(du -h --apparent-size "$IMG" | cut -f1) apparent, $(du -h "$IMG" | cut -f1) on disk)" || echo "(absent)")"
echo "mapper : /dev/mapper/$MAPPER_NAME $( [ -e "/dev/mapper/$MAPPER_NAME" ] && echo "(open)" || echo "(closed)")"
echo "keyfile : $KEYFILE $( [ -f "$KEYFILE" ] && echo "(present, mode $(stat -c%a "$KEYFILE"))" || echo "(absent)")"
echo "mountpoint : $MOUNTPOINT $(mountpoint -q "$MOUNTPOINT" && echo "(mounted)" || echo "(NOT mounted)")"
grep -q "^$MAPPER_NAME " /etc/crypttab 2>/dev/null && echo "crypttab : present" || echo "crypttab : MISSING"
grep -q " $MOUNTPOINT " /etc/fstab 2>/dev/null && echo "fstab : present" || echo "fstab : MISSING"
}
if [ "${1:-}" = "--status" ]; then status; exit 0; fi
if [ -f "$IMG" ]; then
echo "$IMG already exists — refusing to re-create. Use --status." >&2
exit 1
fi
echo ">> stopping the stack so nothing is writing to $MOUNTPOINT"
if [ -f "$STACK_DIR/docker-compose.yml" ]; then
( cd "$STACK_DIR" && docker compose down )
fi
echo ">> generating keyfile $KEYFILE (root-only)"
install -d -m 0700 "$(dirname "$KEYFILE")"
if [ ! -f "$KEYFILE" ]; then
dd if=/dev/urandom of="$KEYFILE" bs=512 count=1 status=none
chmod 0400 "$KEYFILE"
fi
echo ">> creating $SIZE sparse image at $IMG"
truncate -s "$SIZE" "$IMG"
chmod 0600 "$IMG"
echo ">> LUKS format + open"
cryptsetup luksFormat --type luks2 --batch-mode --key-file "$KEYFILE" "$IMG"
cryptsetup luksOpen --key-file "$KEYFILE" "$IMG" "$MAPPER_NAME"
echo ">> mkfs + mount"
mkfs.ext4 -q -L petal-data "/dev/mapper/$MAPPER_NAME"
# Preserve whatever is already in the plaintext directory, then swap it in.
STAGING=""
if [ -d "$MOUNTPOINT" ] && [ -n "$(ls -A "$MOUNTPOINT" 2>/dev/null)" ]; then
STAGING="$(mktemp -d)"
echo ">> preserving existing plaintext data -> $STAGING"
cp -a "$MOUNTPOINT/." "$STAGING/"
# Critical, and easy to miss: mounting over a directory HIDES its contents,
# it does not remove them. Skip this and the original plaintext petal.db sits
# on the unencrypted root filesystem forever, invisible under the mount,
# defeating the entire exercise. Clear the mountpoint before mounting.
echo ">> shredding the plaintext originals under the mountpoint"
find "$MOUNTPOINT" -mindepth 1 -type f -exec shred -uz {} + 2>/dev/null || true
find "$MOUNTPOINT" -mindepth 1 -depth -type d -exec rmdir {} + 2>/dev/null || true
[ -z "$(ls -A "$MOUNTPOINT" 2>/dev/null)" ] || {
echo "!! $MOUNTPOINT is not empty after cleanup; refusing to mount over live data" >&2
echo " (data is preserved at $STAGING)" >&2
exit 1
}
fi
mkdir -p "$MOUNTPOINT"
mount "/dev/mapper/$MAPPER_NAME" "$MOUNTPOINT"
if [ -n "$STAGING" ]; then
echo ">> restoring data onto the encrypted volume"
cp -a "$STAGING/." "$MOUNTPOINT/"
find "$STAGING" -type f -exec shred -uz {} + 2>/dev/null || true
rm -rf "$STAGING"
fi
# Mount-liveness sentinel: docker-compose bind-mounts this file with
# create_host_path:false, so an unmounted volume becomes a loud container start
# failure rather than Petal quietly serving an empty database.
touch "$MOUNTPOINT/.volume-ok"
chown -R "$OWNER_UID:$OWNER_GID" "$MOUNTPOINT"
echo ">> persisting across reboots"
# systemd-cryptsetup loop-mounts a regular file source on its own.
if ! grep -q "^$MAPPER_NAME " /etc/crypttab 2>/dev/null; then
echo "$MAPPER_NAME $IMG $KEYFILE luks,nofail" >> /etc/crypttab
fi
# nofail: a problem here must never wedge the boot of a host running half a
# dozen other services.
# x-systemd.before=docker.service is the important one: without it Docker can
# start first, find $MOUNTPOINT empty, and bring Petal up against a blank
# unencrypted directory that the real volume then hides.
if ! grep -q " $MOUNTPOINT " /etc/fstab 2>/dev/null; then
echo "/dev/mapper/$MAPPER_NAME $MOUNTPOINT ext4 defaults,nofail,x-systemd.requires=/dev/mapper/$MAPPER_NAME,x-systemd.before=docker.service 0 2" >> /etc/fstab
fi
systemctl daemon-reload
echo ">> restarting the stack"
if [ -f "$STACK_DIR/docker-compose.yml" ]; then
( cd "$STACK_DIR" && docker compose up -d )
fi
echo
status
deploy/vllm-headscale-proxy.service
+40
View File
@@ -0,0 +1,40 @@
[Unit]
Description=Expose millenia's vLLM chat server on the headscale interface only
# Why a forwarder instead of just rebinding vLLM: vllm-chat.service is shared.
# Petal, Gogobee and Open WebUI all talk to 127.0.0.1:8000, and Open WebUI keeps
# its endpoint in its own database rather than in env, so moving vLLM's bind
# address would mean editing three consumers and reloading a 35B AWQ model
# (minutes of downtime for all of them). This adds a second door instead: local
# callers keep loopback untouched, and only the headscale address gains a
# listener. Nothing about vllm-chat changes.
#
# Deliberately NOT 0.0.0.0 — this reaches a public VPS over the VPN, and the
# LAN has no business seeing an unauthenticated inference endpoint.
After=network-online.target tailscaled.service vllm-chat.service
Wants=network-online.target
BindsTo=vllm-chat.service
[Service]
Type=simple
# fork: one child per connection, so a single client can't block the others.
# reuseaddr: survive a restart while sockets are still in TIME_WAIT.
# The bind address is millenia's headscale IP; if tailscaled hasn't brought the
# interface up yet the bind fails and Restart retries until it has.
ExecStart=/usr/bin/socat -d TCP-LISTEN:8000,bind=100.64.0.2,fork,reuseaddr TCP:127.0.0.1:8000
Restart=always
RestartSec=5
# Long generations hold a connection open; don't let systemd reap a healthy one.
TimeoutStopSec=10
# The process only shuttles bytes between two sockets — give it nothing else.
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6
DynamicUser=true
[Install]
WantedBy=multi-user.target
docker-compose.yml
+180
View File
@@ -0,0 +1,180 @@
# Petal on the parodia.dev VPS.
#
# docker compose up -d --build
#
# Fronted by the host's existing Traefik (external `traefik` network, the
# `web-secure` entrypoint and the `default` cert resolver — same convention the
# other services on this box use). Petal itself never binds a host port; the
# only way in is through Traefik over HTTPS.
#
# Read-aloud runs as two sibling containers rather than host systemd services:
# each Piper HTTP server loads exactly one voice, the host has no lingering
# user session to keep systemd units alive, and keeping them on the internal
# network means the TTS ports are unreachable from anywhere but Petal.
#
# Copy deploy/petal.env.example to .env before the first `up`.
name: petal
services:
petal:
build:
context: .
dockerfile: Dockerfile
image: petal:local
container_name: petal
restart: unless-stopped
# ./data is a bind mount, so the image's own `petal` user (uid 10001) has no
# claim on it — the host's ownership wins and the container can't open
# petal.db. Run as whoever owns the stack directory instead. Keeping it the
# host user (rather than chowning ./data to 10001) is deliberate: the backup
# script gzips snapshots in place from the host, so the host account needs
# write access to the same directory. Still never root.
user: "${PETAL_UID:-1001}:${PETAL_GID:-1001}"
env_file: .env
environment:
# Fixed by the image layout; kept here so they're visible at a glance.
PORT: "8080"
DATABASE_PATH: /data/petal.db
IMAGE_DIR: /data/images
TTS_CACHE_DIR: /data/tts
# DreamDict's built dictionary, read-only, deployed into the data volume
# (see deploy/README.md). Absent it, word lookups fall back to the
# embedded English/Chinese datasets rather than failing.
DICT_PATH: /data/dict.db
# Piper sidecars. Each server loads one voice, so English and Chinese are
# separate containers; the handler maps language → instance from config.
TTS_ENDPOINT: http://piper-en:5000
TTS_ENDPOINT_ZH: http://piper-zh:5000
# A language is discovered from the TTS_ENDPOINT_<LANG>/TTS_VOICE_<LANG>
# pair, so fr and es cost a service and two lines rather than a code
# change. <LANG> is the base tag — an env var name can't hold pt-PT's
# hyphen, and there is one Portuguese voice loaded either way.
TTS_ENDPOINT_PT: http://piper-pt:5000
TTS_ENDPOINT_FR: http://piper-fr:5000
# The sidecars run piper-tts 1.6.0, which serves synthesis on
# /synthesize; millenia's older server keeps the default "/".
TTS_PATH: /synthesize
# The companion's bedtime nag and night mode read the local clock.
TZ: ${TZ:-Europe/Lisbon}
volumes:
# A bind mount, not a named volume: petal.db must be trivially reachable
# from the host for the nightly backup and for a restore.
- ./data:/data
# Mount-liveness guard. On the VPS ./data is an encrypted LUKS volume, and
# the mountpoint directory still exists when that volume is NOT mounted —
# so without this, a boot where the unlock failed would start Petal
# against an empty unencrypted directory and quietly serve a blank
# database. .volume-ok lives on the encrypted filesystem, and
# create_host_path: false turns its absence into a container start
# failure instead. Harmless elsewhere: create the file once and it is a
# no-op. See deploy/README.md §6.
- type: bind
source: ./data/.volume-ok
target: /data/.volume-ok
read_only: true
bind:
create_host_path: false
networks:
- traefik
- internal
depends_on:
- piper-en
- piper-zh
- piper-pt
labels:
traefik.enable: "true"
traefik.docker.network: traefik
traefik.http.routers.petal.rule: Host(`${PETAL_HOST:-petal.parodia.dev}`)
traefik.http.routers.petal.entrypoints: web-secure
traefik.http.routers.petal.tls: "true"
traefik.http.routers.petal.tls.certResolver: default
traefik.http.routers.petal.service: petal
# No edge gate: Petal authenticates for itself now (Authentik OIDC), so
# every /api route answers 401 without a session and the only thing served
# to an anonymous visitor is the app shell and its sign-in redirect. The
# basic-auth middleware that stood here until Phase 16 — plus the separate
# unauthenticated router /api/health needed to escape it — is gone; a
# second password in front of a real login is just one more thing to lose.
traefik.http.routers.petal.middlewares: compression@file,petal-headers
traefik.http.services.petal.loadbalancer.server.port: "8080"
# Petal is a private writing space: no framing, no sniffing, HSTS on.
traefik.http.middlewares.petal-headers.headers.customresponseheaders.Content-Security-Policy: frame-ancestors 'self'
traefik.http.middlewares.petal-headers.headers.customresponseheaders.Strict-Transport-Security: max-age=31536000; includeSubDomains
traefik.http.middlewares.petal-headers.headers.customresponseheaders.X-Content-Type-Options: nosniff
traefik.http.middlewares.petal-headers.headers.customresponseheaders.Referrer-Policy: same-origin
piper-en:
build:
context: deploy/piper
image: petal-piper:local
container_name: petal-piper-en
restart: unless-stopped
environment:
PIPER_VOICE: ${TTS_VOICE_EN:-en_US-amy-medium}
volumes:
- piper-voices:/voices
networks:
- internal
piper-zh:
build:
context: deploy/piper
image: petal-piper:local
container_name: petal-piper-zh
restart: unless-stopped
environment:
PIPER_VOICE: ${TTS_VOICE_ZH:-zh_CN-huayan-medium}
volumes:
- piper-voices:/voices
networks:
- internal
# European Portuguese, for the pt-PT pair. pt_PT-tugão-medium is the *only*
# European voice in Piper's catalogue — the other five Portuguese models are
# all pt_BR — so the default anyone reaches for is the Brazilian one, exactly
# as it was with the Hunspell dictionary in Phase 21. Named here rather than
# left to the image default for that reason.
piper-pt:
build:
context: deploy/piper
image: petal-piper:local
container_name: petal-piper-pt
restart: unless-stopped
environment:
PIPER_VOICE: ${TTS_VOICE_PT:-pt_PT-tugão-medium}
volumes:
- piper-voices:/voices
networks:
- internal
# French, for the fr pair. The opposite situation to Portuguese: every French
# voice Piper ships is fr_FR, so there is no wrong country to land on by
# default, and the name is plain ASCII so the entrypoint's percent-encoded
# fallback (added for tugão) never has to fire. siwis-medium to match the
# register of the other three.
piper-fr:
build:
context: deploy/piper
image: petal-piper:local
container_name: petal-piper-fr
restart: unless-stopped
environment:
PIPER_VOICE: ${TTS_VOICE_FR:-fr_FR-siwis-medium}
volumes:
- piper-voices:/voices
networks:
- internal
networks:
# Created and owned by the host's Traefik stack.
traefik:
external: true
# Petal ↔ Piper only. Not reachable from the internet or the other stacks.
internal:
driver: bridge
volumes:
# Downloaded voice models, shared read-mostly by both Piper instances so the
# same model is never fetched twice.
piper-voices:
go.mod
+4
View File
@@ -3,7 +3,11 @@ module gitea.parodia.dev/drwily/petal
go 1.25.0
require (
github.com/coreos/go-oidc/v3 v3.20.0
github.com/go-chi/chi/v5 v5.3.0
github.com/go-jose/go-jose/v4 v4.1.4
github.com/prosolis/dreamdict v0.0.0-20260727163219-302a39d5c768
golang.org/x/oauth2 v0.36.0
modernc.org/sqlite v1.53.0
)
go.sum
+8
View File
@@ -1,7 +1,11 @@
github.com/coreos/go-oidc/v3 v3.20.0 h1:EtE0WIBHk03N+DqGkY4+UONzzZHk7amKt6IyNd7OsZE=
github.com/coreos/go-oidc/v3 v3.20.0/go.mod h1:DYCf24+ncYi+XkIH97GY1+dqoRlbaSI26KVTCI9SrY4=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/go-chi/chi/v5 v5.3.0 h1:halUjDxhshgXHMrao5bB8eNBXo/rnzwr8m5m36glehM=
github.com/go-chi/chi/v5 v5.3.0/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
@@ -12,10 +16,14 @@ github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWE
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/prosolis/dreamdict v0.0.0-20260727163219-302a39d5c768 h1:+7NP78QlAVcXMviA23cMBtxwvYwDhADnABa3JBhbApI=
github.com/prosolis/dreamdict v0.0.0-20260727163219-302a39d5c768/go.mod h1:s4D+Q++6Qjq8X7/ZXEMStGXA6tScm4GadFo0+HF4BHY=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4=
golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
internal/auth/auth.go
+77
View File
@@ -0,0 +1,77 @@
// Package auth answers one question for every API request: who is asking?
//
// Until now Petal ran as a single hardcoded user and every query passed
// db.LocalUserID directly. That made the identity of the caller a compile-time
// constant scattered across ~35 call sites — nothing a real login could ever
// replace without touching all of them. This package moves that identity into
// the request context, resolved once by [Middleware], so handlers read the
// current user instead of naming one.
//
// The identity itself still comes from [StaticResolver] today, which returns
// the same local user for everyone. Swapping in Authentik later means writing
// one Resolver (validate the session cookie → user id) and changing the single
// line in main.go that constructs it. No handler changes.
package auth
import (
"context"
"net/http"
"gitea.parodia.dev/drwily/petal/internal/httputil"
)
// ctxKey is unexported so no other package can plant a user id in the context
// without going through [WithUser].
type ctxKey struct{}
// WithUser returns a copy of ctx carrying userID as the authenticated caller.
// Handlers never call this; [Middleware] does, and tests use it to build a
// request that looks authenticated.
func WithUser(ctx context.Context, userID string) context.Context {
return context.WithValue(ctx, ctxKey{}, userID)
}
// UserID returns the authenticated user id carried by ctx, or "" if the request
// never passed through [Middleware].
//
// Returning "" rather than panicking keeps an unauthenticated request failing
// *closed*: every query in Petal is scoped `WHERE user_id = ?`, so an empty id
// matches no rows — a missing middleware leaks nothing, it just returns empty
// results. Handlers may therefore use the value directly without checking it.
func UserID(ctx context.Context) string {
id, _ := ctx.Value(ctxKey{}).(string)
return id
}
// Resolver maps an inbound request to the id of the user making it. Returning
// an error, or an empty id, rejects the request with a 401.
//
// This is the seam a real identity provider drops into: an Authentik resolver
// validates the session cookie and returns the user id it maps to.
type Resolver interface {
Resolve(r *http.Request) (string, error)
}
// StaticResolver resolves every request to the same user id, ignoring the
// request entirely. It is how Petal runs today — a single-user app whose one
// user now arrives through the same path a logged-in user eventually will.
type StaticResolver string
// Resolve implements [Resolver].
func (s StaticResolver) Resolve(*http.Request) (string, error) { return string(s), nil }
// Middleware resolves the caller with res and stores the result in the request
// context for [UserID]. Requests the resolver rejects — or resolves to an empty
// id — never reach the handler; they get a 401 instead.
func Middleware(res Resolver) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
userID, err := res.Resolve(r)
if err != nil || userID == "" {
httputil.ErrorJSON(w, http.StatusUnauthorized, "not signed in")
return
}
next.ServeHTTP(w, r.WithContext(WithUser(r.Context(), userID)))
})
}
}
internal/auth/auth_test.go
+75
View File
@@ -0,0 +1,75 @@
package auth
import (
"context"
"net/http"
"net/http/httptest"
"testing"
)
// errResolver rejects every request, standing in for a real resolver that finds
// no valid session.
type errResolver struct{ err error }
func (e errResolver) Resolve(*http.Request) (string, error) { return "", e.err }
func TestUserIDRoundTrip(t *testing.T) {
ctx := WithUser(context.Background(), "alice")
if got := UserID(ctx); got != "alice" {
t.Fatalf("UserID = %q, want alice", got)
}
}
// A request that never passed through the middleware must report no user rather
// than panicking — every query is `WHERE user_id = ?`, so an empty id fails
// closed (matches nothing) instead of falling back to some default account.
func TestUserIDAbsentIsEmpty(t *testing.T) {
if got := UserID(context.Background()); got != "" {
t.Fatalf("UserID on bare context = %q, want empty", got)
}
}
func TestMiddlewareInjectsResolvedUser(t *testing.T) {
var seen string
h := Middleware(StaticResolver("local"))(http.HandlerFunc(
func(_ http.ResponseWriter, r *http.Request) { seen = UserID(r.Context()) },
))
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/", nil))
if seen != "local" {
t.Fatalf("handler saw user %q, want local", seen)
}
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200", rec.Code)
}
}
// Both rejection paths — an explicit error and a silent empty id — must 401
// without ever entering the handler. The empty case matters most: a resolver
// that returns ("", nil) by mistake would otherwise hand handlers an empty user
// id, and while that fails closed at the SQL layer, it should never get there.
func TestMiddlewareRejectsUnresolved(t *testing.T) {
for name, res := range map[string]Resolver{
"resolver error": errResolver{err: http.ErrNoCookie},
"empty user id": StaticResolver(""),
} {
t.Run(name, func(t *testing.T) {
called := false
h := Middleware(res)(http.HandlerFunc(
func(http.ResponseWriter, *http.Request) { called = true },
))
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/", nil))
if called {
t.Fatal("handler ran for an unauthenticated request")
}
if rec.Code != http.StatusUnauthorized {
t.Fatalf("status = %d, want 401", rec.Code)
}
})
}
}
internal/auth/oidc.go
+390
View File
@@ -0,0 +1,390 @@
package auth
import (
"context"
"crypto/rand"
"crypto/subtle"
"encoding/base64"
"errors"
"fmt"
"log"
"net/http"
"strings"
"sync"
"github.com/coreos/go-oidc/v3/oidc"
"github.com/go-chi/chi/v5"
"golang.org/x/oauth2"
)
// Temporary cookies that carry one login attempt from /auth/login to
// /auth/callback. They live for ten minutes and are cleared the moment the
// callback runs.
const (
stateCookie = "petal_oidc_state"
nonceCookie = "petal_oidc_nonce"
pkceCookie = "petal_oidc_pkce"
loginAttemptTTL = 600 // seconds
)
// Options configures the OIDC client.
type Options struct {
IssuerURL string // Authentik's issuer, e.g. https://auth.example.com/application/o/petal/
ClientID string
ClientSecret string
BaseURL string // Petal's public base URL; the redirect URI is derived from it
Allowed Allowlist
}
// OIDC implements Petal's half of an authorization-code login against
// Authentik: /auth/login starts it, /auth/callback finishes it by provisioning
// the account and issuing a session, /auth/logout ends it.
//
// Petal is the OIDC client itself rather than trusting a proxy-injected header.
// The header approach is far less code, but it is only safe while the container
// is unreachable except through that proxy — an invariant enforced by network
// configuration, not by anything in the repository, on a public host that also
// runs half a dozen other services. Petal holds someone's private journals; it
// should be safe to expose directly.
type OIDC struct {
opts Options
sessions *SessionStore
users *UserStore
secure bool
// The provider is discovered over the network, which means it can fail at
// startup for reasons that have nothing to do with Petal. Discovery is
// therefore lazy and retried: an Authentik outage blocks new logins but
// leaves every existing session working, since those only need the database.
mu sync.Mutex
provider *oidc.Provider
oauth *oauth2.Config
verifier *oidc.IDTokenVerifier
}
// NewOIDC builds the login flow. It attempts discovery once so a misconfigured
// issuer shows up in the startup log rather than on the writer's first login,
// but a failure here is not fatal.
func NewOIDC(ctx context.Context, opts Options, sessions *SessionStore, users *UserStore) *OIDC {
o := &OIDC{
opts: opts,
sessions: sessions,
users: users,
secure: strings.HasPrefix(strings.ToLower(opts.BaseURL), "https://"),
}
if err := o.discover(ctx); err != nil {
log.Printf("auth: OIDC discovery failed (%v) — login will retry on demand", err)
}
return o
}
// RedirectURI is the callback Authentik must have registered for this client.
func (o *OIDC) RedirectURI() string {
return strings.TrimSuffix(o.opts.BaseURL, "/") + "/auth/callback"
}
// discover resolves the provider metadata and builds the oauth2 config.
func (o *OIDC) discover(ctx context.Context) error {
o.mu.Lock()
defer o.mu.Unlock()
if o.provider != nil {
return nil
}
// The issuer is passed through exactly as configured, trailing slash and
// all: OIDC requires the discovered issuer to match the requested one
// byte-for-byte, and Authentik's ends in a slash. (go-oidc trims it itself
// when building the .well-known URL, so a slash here costs nothing.)
provider, err := oidc.NewProvider(ctx, o.opts.IssuerURL)
if err != nil {
return err
}
o.provider = provider
o.verifier = provider.Verifier(&oidc.Config{ClientID: o.opts.ClientID})
o.oauth = &oauth2.Config{
ClientID: o.opts.ClientID,
ClientSecret: o.opts.ClientSecret,
Endpoint: provider.Endpoint(),
RedirectURL: o.RedirectURI(),
Scopes: []string{oidc.ScopeOpenID, "profile", "email"},
}
return nil
}
// ready returns the discovered client, discovering it first if an earlier
// attempt failed.
func (o *OIDC) ready(ctx context.Context) (*oauth2.Config, *oidc.IDTokenVerifier, error) {
if err := o.discover(ctx); err != nil {
return nil, nil, err
}
o.mu.Lock()
defer o.mu.Unlock()
return o.oauth, o.verifier, nil
}
// Routes mounts the login endpoints. Mount at "/auth", outside /api: these are
// browser navigations, not API calls, and they must be reachable without a
// session — that is their whole purpose.
func (o *OIDC) Routes() chi.Router {
r := chi.NewRouter()
r.Get("/login", o.login)
r.Get("/callback", o.callback)
r.Get("/logout", o.logout)
r.Post("/logout", o.logout)
return r
}
// login starts an authorization-code flow with PKCE.
func (o *OIDC) login(w http.ResponseWriter, r *http.Request) {
// Already signed in? Don't bounce a valid session through the IdP.
if _, err := o.sessions.Resolve(r); err == nil {
http.Redirect(w, r, "/", http.StatusFound)
return
}
conf, _, err := o.ready(r.Context())
if err != nil {
o.page(w, http.StatusServiceUnavailable,
"登录暂时不可用", "Sign-in is unavailable right now",
"Petal 联系不上登录服务。请稍后再试。",
"Petal can't reach the sign-in service. Please try again in a moment.")
return
}
state, err := randomToken()
if err != nil {
o.page(w, http.StatusInternalServerError, "出了点问题", "Something went wrong", "请再试一次。", "Please try again.")
return
}
nonce, err := randomToken()
if err != nil {
o.page(w, http.StatusInternalServerError, "出了点问题", "Something went wrong", "请再试一次。", "Please try again.")
return
}
pkce := oauth2.GenerateVerifier()
// state defends the callback against CSRF (a forged callback can't know the
// cookie); nonce ties the returned ID token to this attempt; PKCE binds the
// code to this client even if it leaks in transit.
o.setTemp(w, stateCookie, state)
o.setTemp(w, nonceCookie, nonce)
o.setTemp(w, pkceCookie, pkce)
http.Redirect(w, r, conf.AuthCodeURL(state,
oidc.Nonce(nonce),
oauth2.S256ChallengeOption(pkce),
), http.StatusFound)
}
// callback completes the flow: verify, allowlist, provision, issue a session.
func (o *OIDC) callback(w http.ResponseWriter, r *http.Request) {
// Expire the one-shot login cookies up front, not on the way out: every exit
// from here writes a response, and a Set-Cookie added after the header is
// written is silently dropped. They're read from the request below, so
// clearing them on the response now costs nothing.
o.clearTemp(w)
if errParam := r.URL.Query().Get("error"); errParam != "" {
o.page(w, http.StatusForbidden,
"登录未完成", "Sign-in didn't finish",
"登录服务拒绝了这次请求。你可以再试一次。",
"The sign-in service turned that request down. You can try again.")
return
}
state, err := r.Cookie(stateCookie)
if err != nil || state.Value == "" ||
subtle.ConstantTimeCompare([]byte(state.Value), []byte(r.URL.Query().Get("state"))) != 1 {
o.page(w, http.StatusBadRequest,
"这个登录链接过期了", "That sign-in link expired",
"请回到 Petal 重新登录。",
"Head back to Petal and sign in again.")
return
}
conf, verifier, err := o.ready(r.Context())
if err != nil {
o.page(w, http.StatusServiceUnavailable,
"登录暂时不可用", "Sign-in is unavailable right now",
"Petal 联系不上登录服务。请稍后再试。",
"Petal can't reach the sign-in service. Please try again in a moment.")
return
}
pkce, err := r.Cookie(pkceCookie)
if err != nil {
o.page(w, http.StatusBadRequest, "这个登录链接过期了", "That sign-in link expired",
"请回到 Petal 重新登录。", "Head back to Petal and sign in again.")
return
}
token, err := conf.Exchange(r.Context(), r.URL.Query().Get("code"), oauth2.VerifierOption(pkce.Value))
if err != nil {
log.Printf("auth: code exchange failed: %v", err)
o.page(w, http.StatusBadGateway, "登录没有成功", "Sign-in didn't go through",
"请再试一次。", "Please try again.")
return
}
claims, err := o.claims(r.Context(), verifier, token)
if err != nil {
log.Printf("auth: id token rejected: %v", err)
o.page(w, http.StatusBadGateway, "登录没有成功", "Sign-in didn't go through",
"请再试一次。", "Please try again.")
return
}
// Nonce check: this ID token must belong to the attempt that started here.
nonce, err := r.Cookie(nonceCookie)
if err != nil || subtle.ConstantTimeCompare([]byte(nonce.Value), []byte(claims.nonce)) != 1 {
o.page(w, http.StatusBadRequest, "这个登录链接过期了", "That sign-in link expired",
"请回到 Petal 重新登录。", "Head back to Petal and sign in again.")
return
}
if !o.opts.Allowed.Permits(claims.Subject, claims.Email) {
log.Printf("auth: rejected sign-in for sub=%s email=%s (not on the allowlist)", claims.Subject, claims.Email)
o.page(w, http.StatusForbidden,
"这个 Petal 不是给你写的", "This Petal isn't yours to write in",
"你的账号是有效的，但还没有被邀请到这个 Petal。如果这是个误会，找管理员说一声就好。",
"Your account is valid, but it hasn't been invited to this Petal. If that's a mistake, a word with whoever runs it will sort it out.")
return
}
if err := o.users.Upsert(claims.Subject, claims.Email, claims.displayName()); err != nil {
log.Printf("auth: provisioning failed: %v", err)
o.page(w, http.StatusInternalServerError, "出了点问题", "Something went wrong",
"请再试一次。", "Please try again.")
return
}
session, err := o.sessions.Create(claims.Subject, r.UserAgent())
if err != nil {
log.Printf("auth: session creation failed: %v", err)
o.page(w, http.StatusInternalServerError, "出了点问题", "Something went wrong",
"请再试一次。", "Please try again.")
return
}
SetSessionCookie(w, session, o.secure)
log.Printf("auth: signed in %s (%s)", claims.Email, claims.Subject)
http.Redirect(w, r, "/", http.StatusFound)
}
// logout revokes the session server-side and clears the cookie. Doing both
// matters: clearing only the cookie leaves a token that still works if it was
// ever captured.
func (o *OIDC) logout(w http.ResponseWriter, r *http.Request) {
if c, err := r.Cookie(SessionCookie); err == nil && c.Value != "" {
if err := o.sessions.Revoke(c.Value); err != nil {
log.Printf("auth: revoke failed: %v", err)
}
}
ClearSessionCookie(w, o.secure)
http.Redirect(w, r, "/", http.StatusFound)
}
// idClaims is the subset of the ID token Petal cares about.
type idClaims struct {
Subject string `json:"sub"`
Email string `json:"email"`
Name string `json:"name"`
PreferredUsername string `json:"preferred_username"`
nonce string
}
func (c idClaims) displayName() string {
if c.Name != "" {
return c.Name
}
if c.PreferredUsername != "" {
return c.PreferredUsername
}
return c.Email
}
// claims verifies the ID token in a token response and extracts its claims.
func (o *OIDC) claims(ctx context.Context, verifier *oidc.IDTokenVerifier, token *oauth2.Token) (idClaims, error) {
raw, ok := token.Extra("id_token").(string)
if !ok || raw == "" {
return idClaims{}, errors.New("no id_token in the token response")
}
idToken, err := verifier.Verify(ctx, raw)
if err != nil {
return idClaims{}, err
}
var claims idClaims
if err := idToken.Claims(&claims); err != nil {
return idClaims{}, err
}
if claims.Subject == "" {
claims.Subject = idToken.Subject
}
claims.nonce = idToken.Nonce
return claims, nil
}
func (o *OIDC) setTemp(w http.ResponseWriter, name, value string) {
http.SetCookie(w, &http.Cookie{
Name: name,
Value: value,
Path: "/auth",
HttpOnly: true,
Secure: o.secure,
SameSite: http.SameSiteLaxMode,
MaxAge: loginAttemptTTL,
})
}
func (o *OIDC) clearTemp(w http.ResponseWriter) {
for _, name := range []string{stateCookie, nonceCookie, pkceCookie} {
http.SetCookie(w, &http.Cookie{
Name: name, Value: "", Path: "/auth",
HttpOnly: true, Secure: o.secure, SameSite: http.SameSiteLaxMode, MaxAge: -1,
})
}
}
func randomToken() (string, error) {
b := make([]byte, 24)
if _, err := rand.Read(b); err != nil {
return "", err
}
return base64.RawURLEncoding.EncodeToString(b), nil
}
// page renders one of the flow's dead ends. Every one of them is a full stop in
// front of someone who was just trying to write, so they read as warm bilingual
// sentences rather than as a status code — the same standard as the rest of the
// app, and the reason these aren't plain http.Error calls.
func (o *OIDC) page(w http.ResponseWriter, status int, titleZH, titleEN, bodyZH, bodyEN string) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.WriteHeader(status)
fmt.Fprintf(w, `<!doctype html>
<html lang="zh"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>%s · Petal</title>
<style>
:root { color-scheme: light dark; }
body { margin:0; min-height:100vh; display:flex; align-items:center; justify-content:center;
background:#fdf8f5; color:#5b4b52;
font-family:'Nunito','PingFang SC','Microsoft YaHei','Noto Sans CJK SC',system-ui,sans-serif; }
main { max-width:30rem; padding:2.5rem; text-align:center; }
.mark { font-size:2.5rem; }
h1 { font-size:1.5rem; margin:.75rem 0 .25rem; font-weight:700; }
h2 { font-size:1rem; margin:0 0 1.25rem; font-weight:600; opacity:.65; }
p { line-height:1.7; margin:.4rem 0; }
p.en { opacity:.7; font-size:.95rem; }
a { display:inline-block; margin-top:1.75rem; padding:.6rem 1.4rem; border-radius:999px;
background:#f3c7d3; color:#5b4b52; text-decoration:none; font-weight:700; }
@media (prefers-color-scheme: dark) { body { background:#231b28; color:#e9dfe6; } a { background:#7c5f78; color:#fdf8f5; } }
</style></head>
<body><main>
<div class="mark">🌸</div>
<h1>%s</h1><h2>%s</h2>
<p>%s</p><p class="en">%s</p>
<a href="/">回到 Petal · Back to Petal</a>
</main></body></html>
`, titleEN, titleZH, titleEN, bodyZH, bodyEN)
}
internal/auth/oidc_test.go
+402
View File
@@ -0,0 +1,402 @@
package auth
import (
"context"
"crypto/rand"
"crypto/rsa"
"encoding/json"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
jose "github.com/go-jose/go-jose/v4"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// These tests run the whole login round-trip against a stub identity provider:
// discovery, the redirect out, the callback back, and the session that comes out
// the other end. The flow is the one place in Petal where getting a detail wrong
// (an unchecked state, a nonce nobody compares) is both easy and invisible —
// everything still "works" from the browser's point of view.
// stubIdP is a minimal OpenID provider: discovery, a JWKS, and a token endpoint
// that mints a signed ID token for whoever the test says just logged in.
type stubIdP struct {
*httptest.Server
key *rsa.PrivateKey
clientID string
// issuer as advertised by discovery and asserted in tokens. Defaults to the
// server's URL; a test can give it a trailing slash, which is what Authentik
// does and which OIDC requires to match byte-for-byte.
issuer string
// Claims the next token exchange will assert.
sub, email, name string
// nonce echoed into the token; set from the login attempt's cookie.
nonce string
// lastForm records what Petal sent to /token, so the test can assert PKCE.
lastForm url.Values
}
func newStubIdP(t *testing.T, clientID string) *stubIdP {
t.Helper()
key, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
t.Fatal(err)
}
idp := &stubIdP{key: key, clientID: clientID}
mux := http.NewServeMux()
idp.Server = httptest.NewServer(mux)
idp.issuer = idp.URL
t.Cleanup(idp.Close)
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{
"issuer": idp.issuer,
"authorization_endpoint": idp.URL + "/authorize",
"token_endpoint": idp.URL + "/token",
"jwks_uri": idp.URL + "/jwks",
"id_token_signing_alg_values_supported": []string{"RS256"},
})
})
mux.HandleFunc("/jwks", func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(jose.JSONWebKeySet{
Keys: []jose.JSONWebKey{{Key: key.Public(), Algorithm: "RS256", Use: "sig"}},
})
})
mux.HandleFunc("/token", func(w http.ResponseWriter, r *http.Request) {
_ = r.ParseForm()
idp.lastForm = r.PostForm
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{
"access_token": "stub-access-token",
"token_type": "Bearer",
"id_token": idp.idToken(t),
})
})
return idp
}
// idToken mints a signed ID token asserting the currently configured claims.
func (idp *stubIdP) idToken(t *testing.T) string {
t.Helper()
signer, err := jose.NewSigner(
jose.SigningKey{Algorithm: jose.RS256, Key: idp.key},
(&jose.SignerOptions{}).WithType("JWT"),
)
if err != nil {
t.Fatal(err)
}
payload, _ := json.Marshal(map[string]any{
"iss": idp.issuer,
"aud": idp.clientID,
"sub": idp.sub,
"email": idp.email,
"name": idp.name,
"nonce": idp.nonce,
"exp": time.Now().Add(time.Hour).Unix(),
"iat": time.Now().Unix(),
})
signed, err := signer.Sign(payload)
if err != nil {
t.Fatal(err)
}
raw, err := signed.CompactSerialize()
if err != nil {
t.Fatal(err)
}
return raw
}
// newFlow wires Petal's login routes to a stub provider.
func newFlow(t *testing.T, allowed Allowlist) (*stubIdP, http.Handler, *SessionStore, *UserStore) {
t.Helper()
sessions, users, _ := newStores(t)
idp := newStubIdP(t, "petal")
o := NewOIDC(context.Background(), Options{
IssuerURL: idp.URL,
ClientID: "petal",
ClientSecret: "shh",
BaseURL: "http://petal.test",
Allowed: allowed,
}, sessions, users)
return idp, o.Routes(), sessions, users
}
// cookieJar collects Set-Cookie headers across the redirect chain, standing in
// for the browser that would normally carry them.
type cookieJar map[string]string
func (j cookieJar) absorb(rec *httptest.ResponseRecorder) {
for _, c := range rec.Result().Cookies() {
if c.MaxAge < 0 || c.Value == "" {
delete(j, c.Name)
continue
}
j[c.Name] = c.Value
}
}
func (j cookieJar) attach(r *http.Request) *http.Request {
for name, value := range j {
r.AddCookie(&http.Cookie{Name: name, Value: value})
}
return r
}
// start runs /auth/login and returns the redirect target plus the cookies it set.
func start(t *testing.T, flow http.Handler) (*url.URL, cookieJar) {
t.Helper()
rec := httptest.NewRecorder()
flow.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/login", nil))
if rec.Code != http.StatusFound {
t.Fatalf("login status=%d body=%s", rec.Code, rec.Body)
}
target, err := url.Parse(rec.Header().Get("Location"))
if err != nil {
t.Fatal(err)
}
jar := cookieJar{}
jar.absorb(rec)
return target, jar
}
func TestLoginRoundTrip(t *testing.T) {
idp, flow, sessions, users := newFlow(t, nil)
idp.sub, idp.email, idp.name = "sub-her", "her@example.com", "Her Name"
target, jar := start(t, flow)
// The redirect must carry everything the flow depends on later.
q := target.Query()
if q.Get("state") == "" || q.Get("nonce") == "" {
t.Fatalf("login redirect missing state/nonce: %s", target)
}
if q.Get("code_challenge") == "" || q.Get("code_challenge_method") != "S256" {
t.Fatalf("login redirect missing PKCE challenge: %s", target)
}
if q.Get("redirect_uri") != "http://petal.test/auth/callback" {
t.Fatalf("redirect_uri = %q", q.Get("redirect_uri"))
}
if jar[stateCookie] != q.Get("state") {
t.Fatal("the state cookie does not match the state sent to the provider")
}
idp.nonce = jar[nonceCookie]
// Come back as the provider would.
rec := httptest.NewRecorder()
flow.ServeHTTP(rec, jar.attach(
httptest.NewRequest(http.MethodGet, "/callback?code=abc&state="+url.QueryEscape(q.Get("state")), nil)))
if rec.Code != http.StatusFound || rec.Header().Get("Location") != "/" {
t.Fatalf("callback status=%d location=%q body=%s", rec.Code, rec.Header().Get("Location"), rec.Body)
}
// PKCE: the code verifier must reach the token endpoint.
if v := idp.lastForm.Get("code_verifier"); v == "" {
t.Fatal("token exchange sent no code_verifier")
}
// The account was provisioned from the token's claims...
user, err := users.Get("sub-her")
if err != nil {
t.Fatalf("user was not provisioned: %v", err)
}
if user.Email != "her@example.com" || user.DisplayName != "Her Name" {
t.Fatalf("unexpected provisioned user %+v", user)
}
// ...and the response carries a session that resolves to them.
jar.absorb(rec)
token := jar[SessionCookie]
if token == "" {
t.Fatal("callback issued no session cookie")
}
got, err := sessions.Resolve(withCookie(token))
if err != nil || got != "sub-her" {
t.Fatalf("session resolved to %q (err=%v), want sub-her", got, err)
}
// The one-shot login cookies must not linger.
for _, name := range []string{stateCookie, nonceCookie, pkceCookie} {
if jar[name] != "" {
t.Fatalf("%s survived the callback", name)
}
}
// Signing out revokes server-side, not just in the browser.
out := httptest.NewRecorder()
flow.ServeHTTP(out, jar.attach(httptest.NewRequest(http.MethodGet, "/logout", nil)))
if out.Code != http.StatusFound {
t.Fatalf("logout status=%d", out.Code)
}
if _, err := sessions.Resolve(withCookie(token)); err == nil {
t.Fatal("the session survived signing out")
}
}
// A callback whose state doesn't match the cookie is a forged one.
func TestCallbackRejectsBadState(t *testing.T) {
idp, flow, sessions, _ := newFlow(t, nil)
idp.sub, idp.email = "sub-her", "her@example.com"
_, jar := start(t, flow)
idp.nonce = jar[nonceCookie]
rec := httptest.NewRecorder()
flow.ServeHTTP(rec, jar.attach(
httptest.NewRequest(http.MethodGet, "/callback?code=abc&state=some-other-state", nil)))
if rec.Code != http.StatusBadRequest {
t.Fatalf("status=%d, want 400", rec.Code)
}
assertNoSession(t, sessions, rec)
// And so is one with no state cookie at all.
bare := httptest.NewRecorder()
flow.ServeHTTP(bare, httptest.NewRequest(http.MethodGet, "/callback?code=abc&state=x", nil))
if bare.Code != http.StatusBadRequest {
t.Fatalf("status=%d for a cookieless callback, want 400", bare.Code)
}
}
// An ID token minted for a different login attempt must not be accepted, even
// though it is perfectly valid and correctly signed.
func TestCallbackRejectsReplayedNonce(t *testing.T) {
idp, flow, sessions, _ := newFlow(t, nil)
idp.sub, idp.email = "sub-her", "her@example.com"
_, jarA := start(t, flow)
_, jarB := start(t, flow)
idp.nonce = jarB[nonceCookie] // a token belonging to the *other* attempt
rec := httptest.NewRecorder()
flow.ServeHTTP(rec, jarA.attach(
httptest.NewRequest(http.MethodGet, "/callback?code=abc&state="+url.QueryEscape(jarA[stateCookie]), nil)))
if rec.Code != http.StatusBadRequest {
t.Fatalf("status=%d, want 400", rec.Code)
}
assertNoSession(t, sessions, rec)
}
// Being a valid user at the identity provider is not the same as being a user
// here, and the refusal has to read like Petal rather than like a stack trace.
func TestCallbackHonoursAllowlist(t *testing.T) {
idp, flow, sessions, users := newFlow(t, ParseAllowlist("her@example.com"))
idp.sub, idp.email, idp.name = "sub-stranger", "stranger@example.com", "A Stranger"
_, jar := start(t, flow)
idp.nonce = jar[nonceCookie]
rec := httptest.NewRecorder()
flow.ServeHTTP(rec, jar.attach(
httptest.NewRequest(http.MethodGet, "/callback?code=abc&state="+url.QueryEscape(jar[stateCookie]), nil)))
if rec.Code != http.StatusForbidden {
t.Fatalf("status=%d, want 403", rec.Code)
}
if body := rec.Body.String(); !strings.Contains(body, "这个 Petal 不是给你写的") ||
!strings.Contains(body, "isn't yours to write in") {
t.Fatalf("refusal page is not the warm bilingual one: %s", body)
}
assertNoSession(t, sessions, rec)
if _, err := users.Get("sub-stranger"); err == nil {
t.Fatal("a rejected login still provisioned an account")
}
// The person on the list gets in through the same door.
idp.sub, idp.email, idp.name = "sub-her", "her@example.com", "Her Name"
_, jar2 := start(t, flow)
idp.nonce = jar2[nonceCookie]
ok := httptest.NewRecorder()
flow.ServeHTTP(ok, jar2.attach(
httptest.NewRequest(http.MethodGet, "/callback?code=abc&state="+url.QueryEscape(jar2[stateCookie]), nil)))
if ok.Code != http.StatusFound {
t.Fatalf("an allowed writer was turned away: status=%d body=%s", ok.Code, ok.Body)
}
}
// The provider refusing the login (a cancelled consent, a locked account) is a
// dead end, not a session.
func TestCallbackHandlesProviderError(t *testing.T) {
_, flow, sessions, _ := newFlow(t, nil)
rec := httptest.NewRecorder()
flow.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/callback?error=access_denied", nil))
if rec.Code != http.StatusForbidden {
t.Fatalf("status=%d, want 403", rec.Code)
}
assertNoSession(t, sessions, rec)
}
// Authentik's issuer ends in a slash, and OIDC requires the discovered issuer to
// match the configured one byte-for-byte. Normalising it away made discovery
// fail against the real provider while every stub test still passed.
func TestDiscoveryKeepsTrailingSlashIssuer(t *testing.T) {
sessions, users, _ := newStores(t)
idp := newStubIdP(t, "petal")
idp.issuer = idp.URL + "/"
idp.sub, idp.email = "sub-her", "her@example.com"
o := NewOIDC(context.Background(), Options{
IssuerURL: idp.issuer,
ClientID: "petal",
ClientSecret: "shh",
BaseURL: "http://petal.test",
}, sessions, users)
flow := o.Routes()
// A failed discovery renders the 503 "sign-in is unavailable" page instead
// of redirecting, so reaching the provider at all is the assertion.
target, jar := start(t, flow)
if !strings.HasPrefix(target.String(), idp.URL+"/authorize") {
t.Fatalf("login went to %q, want the provider's authorize endpoint", target)
}
// And the ID token it issues, whose `iss` carries the same slash, verifies.
idp.nonce = jar[nonceCookie]
rec := httptest.NewRecorder()
flow.ServeHTTP(rec, jar.attach(
httptest.NewRequest(http.MethodGet, "/callback?code=abc&state="+url.QueryEscape(jar[stateCookie]), nil)))
if rec.Code != http.StatusFound {
t.Fatalf("callback status=%d body=%s", rec.Code, rec.Body)
}
}
// Signing in when already signed in shouldn't bounce a good session through the
// identity provider.
func TestLoginSkipsWhenAlreadySignedIn(t *testing.T) {
_, flow, sessions, _ := newFlow(t, nil)
token, err := sessions.Create(db.LocalUserID, "")
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
flow.ServeHTTP(rec, withCookie(token))
// withCookie builds a GET "/" request; point it at the login route.
rec = httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/login", nil)
req.AddCookie(&http.Cookie{Name: SessionCookie, Value: token})
flow.ServeHTTP(rec, req)
if rec.Code != http.StatusFound || rec.Header().Get("Location") != "/" {
t.Fatalf("status=%d location=%q, want a redirect home", rec.Code, rec.Header().Get("Location"))
}
}
// assertNoSession fails if a response handed out a usable session cookie.
func assertNoSession(t *testing.T, sessions *SessionStore, rec *httptest.ResponseRecorder) {
t.Helper()
for _, c := range rec.Result().Cookies() {
if c.Name == SessionCookie && c.Value != "" {
if _, err := sessions.Resolve(withCookie(c.Value)); err == nil {
t.Fatal("a rejected login was given a working session")
}
}
}
}
internal/auth/pairlang_test.go
+118
View File
@@ -0,0 +1,118 @@
package auth
import (
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// patchMe drives UpdateMeHandler as the given user would reach it: behind the
// middleware, which is the only thing that puts an id in the context.
func patchMe(t *testing.T, users *UserStore, id, body string) *httptest.ResponseRecorder {
t.Helper()
r := httptest.NewRequest(http.MethodPatch, "/me", strings.NewReader(body))
r = r.WithContext(WithUser(r.Context(), id))
w := httptest.NewRecorder()
users.UpdateMeHandler()(w, r)
return w
}
func TestSetPairLang(t *testing.T) {
_, users, _ := newStores(t)
if err := users.SetPairLang("bob", "pt-PT"); err != nil {
t.Fatalf("set pt-PT: %v", err)
}
if u, _ := users.Get("bob"); u.PairLang != "pt-PT" {
t.Fatalf("pair_lang = %q, want pt-PT", u.PairLang)
}
// Every pair with a langpack, not just the first one: this list and the
// frontend's PACKS are two copies of the same fact, and the day they
// disagree is the day she can pick a pair the app cannot render.
if err := users.SetPairLang("bob", "fr"); err != nil {
t.Fatalf("set fr: %v", err)
}
if u, _ := users.Get("bob"); u.PairLang != "fr" {
t.Fatalf("pair_lang = %q, want fr", u.PairLang)
}
// And back — a writer who tries a pair and doesn't like it must be able to
// return, which is the whole reason the picker exists.
if err := users.SetPairLang("bob", "zh"); err != nil {
t.Fatalf("set zh: %v", err)
}
if u, _ := users.Get("bob"); u.PairLang != "zh" {
t.Fatalf("pair_lang = %q, want zh", u.PairLang)
}
}
// A pair the frontend has no langpack for must not be storable. Accepting it
// would leave her looking at Chinese copy with no way back except a lucky guess.
func TestSetPairLangRejectsUnshippedPairs(t *testing.T) {
_, users, _ := newStores(t)
// "es" is the real case here — the pair whose pack has not been written yet.
// "pt-BR" is the near-miss that matters most: a Brazilian code must not be
// quietly served European copy and a European voice.
for _, lang := range []string{"es", "pt-BR", "fr-CA", "klingon", "", " "} {
if err := users.SetPairLang("bob", lang); err == nil {
t.Fatalf("stored unshipped pair %q", lang)
}
}
if u, _ := users.Get("bob"); u.PairLang != "zh" {
t.Fatalf("a refused write still moved pair_lang to %q", u.PairLang)
}
}
func TestSetPairLangUnknownUser(t *testing.T) {
_, users, _ := newStores(t)
if err := users.SetPairLang("nobody", "pt-PT"); err == nil {
t.Fatal("set a pair language on an account that does not exist")
}
}
func TestUpdateMeHandler(t *testing.T) {
_, users, _ := newStores(t)
w := patchMe(t, users, "bob", `{"pair_lang":"pt-PT"}`)
if w.Code != http.StatusOK {
t.Fatalf("status = %d, want 200 (%s)", w.Code, w.Body.String())
}
// The whole user comes back, so the client can re-read the pair from the
// server instead of assuming its request took.
var got db.User
if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil {
t.Fatalf("decode: %v", err)
}
if got.ID != "bob" || got.PairLang != "pt-PT" {
t.Fatalf("response = %+v, want bob on pt-PT", got)
}
}
func TestUpdateMeHandlerRejects(t *testing.T) {
_, users, _ := newStores(t)
for name, body := range map[string]string{
"unshipped pair": `{"pair_lang":"es"}`,
"missing field": `{}`,
"not json": `pt-PT`,
} {
if w := patchMe(t, users, "bob", body); w.Code != http.StatusBadRequest {
t.Fatalf("%s: status = %d, want 400", name, w.Code)
}
}
if u, _ := users.Get("bob"); u.PairLang != "zh" {
t.Fatalf("a rejected request still moved pair_lang to %q", u.PairLang)
}
// A caller the middleware never resolved (or whose row is gone) is a lapsed
// session, not a bad request — the client turns 401 into the sign-in overlay.
if w := patchMe(t, users, "nobody", `{"pair_lang":"pt-PT"}`); w.Code != http.StatusUnauthorized {
t.Fatalf("unknown user: status = %d, want 401", w.Code)
}
}
internal/auth/session.go
+171
View File
@@ -0,0 +1,171 @@
package auth
import (
"crypto/rand"
"crypto/sha256"
"database/sql"
"encoding/base64"
"encoding/hex"
"errors"
"net/http"
"time"
)
// SessionCookie is the cookie carrying the opaque session token.
const SessionCookie = "petal_session"
const (
// sessionTTL is how long a session lives without use. Thirty days, sliding:
// every authenticated request pushes the expiry back out. An editor that
// logs you out mid-draft is hostile, and Petal auto-saves every 1.5s, so a
// surprise 401 costs real writing.
sessionTTL = 30 * 24 * time.Hour
// sessionTTLModifier is the same span as a SQLite datetime() modifier. All
// expiry math happens inside SQLite so stored values stay canonical UTC and
// never depend on the server's local clock or on Go/SQLite parsing agreeing.
sessionTTLModifier = "+30 days"
// sessionRenewAfter throttles the sliding extension: a session is only
// pushed forward once its expiry has drifted this far from the maximum. It
// turns "a write on every request" into "a write at most once an hour per
// session" while leaving the sliding window indistinguishable to the user.
sessionRenewAfter = "-1 hour"
)
// ErrNoSession means the request carried no session cookie, or one that is
// unknown or expired. It is not an internal failure: the caller is simply not
// signed in.
var ErrNoSession = errors.New("no valid session")
// SessionStore issues, validates and revokes login sessions, and is itself the
// [Resolver] the API middleware runs on.
//
// The cookie holds a random token; the table stores only its SHA-256. A dump of
// the database therefore hands an attacker no usable session — the same reason
// passwords are never stored as given. Server-side rows (rather than a signed
// stateless cookie) are what make logout and revocation actually revoke.
type SessionStore struct {
db *sql.DB
}
// NewSessionStore returns a store backed by the given database.
func NewSessionStore(db *sql.DB) *SessionStore { return &SessionStore{db: db} }
// Create issues a new session for userID and returns the token to put in the
// cookie. The token is never stored; only its hash is.
func (s *SessionStore) Create(userID, userAgent string) (string, error) {
raw := make([]byte, 32)
if _, err := rand.Read(raw); err != nil {
return "", err
}
token := base64.RawURLEncoding.EncodeToString(raw)
if len(userAgent) > 256 {
userAgent = userAgent[:256]
}
_, err := s.db.Exec(
`INSERT INTO sessions (id, user_id, expires_at, user_agent)
VALUES (?, ?, datetime('now', ?), ?)`,
hashToken(token), userID, sessionTTLModifier, userAgent,
)
if err != nil {
return "", err
}
return token, nil
}
// Resolve implements [Resolver]: it reads the session cookie, validates it, and
// returns the user it belongs to — extending the session's life while it does.
func (s *SessionStore) Resolve(r *http.Request) (string, error) {
c, err := r.Cookie(SessionCookie)
if err != nil || c.Value == "" {
return "", ErrNoSession
}
return s.userFor(c.Value)
}
// userFor validates a raw token and slides its expiry forward.
func (s *SessionStore) userFor(token string) (string, error) {
id := hashToken(token)
var userID string
err := s.db.QueryRow(
`SELECT user_id FROM sessions WHERE id = ? AND expires_at > datetime('now')`, id,
).Scan(&userID)
if errors.Is(err, sql.ErrNoRows) {
return "", ErrNoSession
}
if err != nil {
return "", err
}
// Slide the window. Throttled, and deliberately not fatal: a failed
// extension shortens one session's life, which is no reason to reject a
// request that is otherwise perfectly authenticated.
_, _ = s.db.Exec(
`UPDATE sessions SET expires_at = datetime('now', ?)
WHERE id = ? AND expires_at < datetime('now', ?, ?)`,
sessionTTLModifier, id, sessionTTLModifier, sessionRenewAfter,
)
return userID, nil
}
// Revoke deletes the session behind a token. Unknown tokens are not an error —
// signing out of a session that is already gone is a success, not a failure.
func (s *SessionStore) Revoke(token string) error {
_, err := s.db.Exec(`DELETE FROM sessions WHERE id = ?`, hashToken(token))
return err
}
// RevokeAll deletes every session for a user, signing them out everywhere.
func (s *SessionStore) RevokeAll(userID string) error {
_, err := s.db.Exec(`DELETE FROM sessions WHERE user_id = ?`, userID)
return err
}
// Prune removes expired rows and returns how many it deleted. Nothing depends
// on it for correctness — expired sessions are already rejected on lookup — it
// just keeps the table from accumulating dead rows forever.
func (s *SessionStore) Prune() (int64, error) {
res, err := s.db.Exec(`DELETE FROM sessions WHERE expires_at <= datetime('now')`)
if err != nil {
return 0, err
}
return res.RowsAffected()
}
// hashToken maps a raw session token to the id stored in the table.
func hashToken(token string) string {
sum := sha256.Sum256([]byte(token))
return hex.EncodeToString(sum[:])
}
// SetSessionCookie writes the session cookie. Secure is set only when Petal is
// served over https — flagging it on a plain-http dev server would make the
// browser drop the cookie and silently break local login.
func SetSessionCookie(w http.ResponseWriter, token string, secure bool) {
http.SetCookie(w, &http.Cookie{
Name: SessionCookie,
Value: token,
Path: "/",
HttpOnly: true,
Secure: secure,
SameSite: http.SameSiteLaxMode,
MaxAge: int(sessionTTL / time.Second),
})
}
// ClearSessionCookie expires the session cookie in the browser. The matching
// server-side row must be revoked separately — that's the half that counts.
func ClearSessionCookie(w http.ResponseWriter, secure bool) {
http.SetCookie(w, &http.Cookie{
Name: SessionCookie,
Value: "",
Path: "/",
HttpOnly: true,
Secure: secure,
SameSite: http.SameSiteLaxMode,
MaxAge: -1,
})
}
internal/auth/session_test.go
+311
View File
@@ -0,0 +1,311 @@
package auth
import (
"errors"
"net/http"
"net/http/httptest"
"path/filepath"
"strings"
"testing"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// newStores opens a database holding two users and returns the session and user
// stores over it.
func newStores(t *testing.T) (*SessionStore, *UserStore, *db.DB) {
t.Helper()
database, err := db.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatalf("open db: %v", err)
}
t.Cleanup(func() { database.Close() })
if _, err := database.Exec(
`INSERT INTO users (id, email, display_name) VALUES (?, ?, ?)`,
"bob", "bob@petal.local", "Bob",
); err != nil {
t.Fatalf("seed second user: %v", err)
}
return NewSessionStore(database.DB), NewUserStore(database.DB), database
}
// withCookie builds a request carrying a session token.
func withCookie(token string) *http.Request {
r := httptest.NewRequest(http.MethodGet, "/", nil)
r.AddCookie(&http.Cookie{Name: SessionCookie, Value: token})
return r
}
func TestSessionLifecycle(t *testing.T) {
sessions, _, _ := newStores(t)
token, err := sessions.Create(db.LocalUserID, "test-agent")
if err != nil {
t.Fatalf("create: %v", err)
}
got, err := sessions.Resolve(withCookie(token))
if err != nil {
t.Fatalf("resolve: %v", err)
}
if got != db.LocalUserID {
t.Fatalf("resolved %q, want %q", got, db.LocalUserID)
}
// Signing out must invalidate the token server-side, not just in the browser:
// clearing only the cookie leaves a token that still works if it ever leaked.
if err := sessions.Revoke(token); err != nil {
t.Fatalf("revoke: %v", err)
}
if _, err := sessions.Resolve(withCookie(token)); !errors.Is(err, ErrNoSession) {
t.Fatalf("revoked token still resolves (err=%v)", err)
}
}
// A request with no cookie, or a token nobody issued, is simply not signed in.
func TestSessionRejectsUnknown(t *testing.T) {
sessions, _, _ := newStores(t)
if _, err := sessions.Resolve(httptest.NewRequest(http.MethodGet, "/", nil)); !errors.Is(err, ErrNoSession) {
t.Fatalf("bare request err=%v, want ErrNoSession", err)
}
if _, err := sessions.Resolve(withCookie("not-a-real-token")); !errors.Is(err, ErrNoSession) {
t.Fatalf("forged token err=%v, want ErrNoSession", err)
}
}
// The table stores a hash, so a database dump yields no usable session.
func TestSessionTokenIsNotStored(t *testing.T) {
sessions, _, database := newStores(t)
token, err := sessions.Create(db.LocalUserID, "")
if err != nil {
t.Fatalf("create: %v", err)
}
var stored string
if err := database.QueryRow(`SELECT id FROM sessions`).Scan(&stored); err != nil {
t.Fatal(err)
}
if stored == token || strings.Contains(stored, token) {
t.Fatal("the raw session token is stored in the database")
}
if stored != hashToken(token) {
t.Fatal("stored id is not the token's hash")
}
}
func TestSessionExpiry(t *testing.T) {
sessions, _, database := newStores(t)
token, err := sessions.Create(db.LocalUserID, "")
if err != nil {
t.Fatalf("create: %v", err)
}
if _, err := database.Exec(
`UPDATE sessions SET expires_at = datetime('now','-1 minute')`,
); err != nil {
t.Fatal(err)
}
if _, err := sessions.Resolve(withCookie(token)); !errors.Is(err, ErrNoSession) {
t.Fatalf("expired session still resolves (err=%v)", err)
}
n, err := sessions.Prune()
if err != nil {
t.Fatalf("prune: %v", err)
}
if n != 1 {
t.Fatalf("pruned %d rows, want 1", n)
}
}
// The window slides: using a session pushes its expiry back out, so someone who
// writes in Petal every few days is never signed out mid-draft.
func TestSessionSlidesForward(t *testing.T) {
sessions, _, database := newStores(t)
token, err := sessions.Create(db.LocalUserID, "")
if err != nil {
t.Fatalf("create: %v", err)
}
// Pretend the session has been idle for a fortnight.
if _, err := database.Exec(
`UPDATE sessions SET expires_at = datetime('now','+16 days')`,
); err != nil {
t.Fatal(err)
}
if _, err := sessions.Resolve(withCookie(token)); err != nil {
t.Fatalf("resolve: %v", err)
}
var extended bool
if err := database.QueryRow(
`SELECT expires_at > datetime('now','+29 days') FROM sessions`,
).Scan(&extended); err != nil {
t.Fatal(err)
}
if !extended {
t.Fatal("using a session did not extend it")
}
}
// Two live sessions must each resolve to their own writer — the whole point.
func TestSessionsAreNotInterchangeable(t *testing.T) {
sessions, _, _ := newStores(t)
aliceToken, err := sessions.Create(db.LocalUserID, "")
if err != nil {
t.Fatal(err)
}
bobToken, err := sessions.Create("bob", "")
if err != nil {
t.Fatal(err)
}
for token, want := range map[string]string{aliceToken: db.LocalUserID, bobToken: "bob"} {
got, err := sessions.Resolve(withCookie(token))
if err != nil {
t.Fatalf("resolve: %v", err)
}
if got != want {
t.Fatalf("token resolved to %q, want %q", got, want)
}
}
// Revoking one session leaves the other alone.
if err := sessions.Revoke(aliceToken); err != nil {
t.Fatal(err)
}
if _, err := sessions.Resolve(withCookie(bobToken)); err != nil {
t.Fatalf("bob was signed out by alice's logout: %v", err)
}
// RevokeAll signs one writer out everywhere and nobody else.
second, _ := sessions.Create("bob", "phone")
if err := sessions.RevokeAll("bob"); err != nil {
t.Fatal(err)
}
for _, token := range []string{bobToken, second} {
if _, err := sessions.Resolve(withCookie(token)); !errors.Is(err, ErrNoSession) {
t.Fatalf("RevokeAll left a session alive (err=%v)", err)
}
}
}
// The session store is itself the Resolver the API middleware runs on, so a
// valid cookie must carry all the way through to the handler.
func TestSessionStoreDrivesMiddleware(t *testing.T) {
sessions, _, _ := newStores(t)
token, err := sessions.Create("bob", "")
if err != nil {
t.Fatal(err)
}
var seen string
h := Middleware(sessions)(http.HandlerFunc(
func(_ http.ResponseWriter, r *http.Request) { seen = UserID(r.Context()) },
))
rec := httptest.NewRecorder()
h.ServeHTTP(rec, withCookie(token))
if rec.Code != http.StatusOK || seen != "bob" {
t.Fatalf("status=%d user=%q, want 200/bob", rec.Code, seen)
}
rec2 := httptest.NewRecorder()
h.ServeHTTP(rec2, httptest.NewRequest(http.MethodGet, "/", nil))
if rec2.Code != http.StatusUnauthorized {
t.Fatalf("status=%d for a cookieless request, want 401", rec2.Code)
}
}
func TestUserUpsert(t *testing.T) {
_, users, database := newStores(t)
if err := users.Upsert("sub-123", "her@example.com", "Her Name"); err != nil {
t.Fatalf("upsert: %v", err)
}
user, err := users.Get("sub-123")
if err != nil {
t.Fatalf("get: %v", err)
}
if user.Email != "her@example.com" || user.DisplayName != "Her Name" {
t.Fatalf("unexpected user %+v", user)
}
if user.PairLang != "zh" {
t.Fatalf("pair_lang = %q, want the zh default", user.PairLang)
}
// A rename upstream is reflected here; Petal's own settings are not touched.
if _, err := database.Exec(`UPDATE users SET pair_lang = 'pt-PT' WHERE id = 'sub-123'`); err != nil {
t.Fatal(err)
}
if err := users.Upsert("sub-123", "new@example.com", "New Name"); err != nil {
t.Fatalf("second upsert: %v", err)
}
user, _ = users.Get("sub-123")
if user.Email != "new@example.com" || user.DisplayName != "New Name" {
t.Fatalf("login did not refresh the profile: %+v", user)
}
if user.PairLang != "pt-PT" {
t.Fatalf("login reset pair_lang to %q", user.PairLang)
}
// Falling back to the email keeps the sidebar from showing an empty name.
if err := users.Upsert("sub-456", "them@example.com", ""); err != nil {
t.Fatal(err)
}
if u, _ := users.Get("sub-456"); u.DisplayName != "them@example.com" {
t.Fatalf("display name = %q, want the email fallback", u.DisplayName)
}
if err := users.Upsert("", "nobody@example.com", "Nobody"); err == nil {
t.Fatal("a login with no subject was accepted")
}
}
// Sessions belong to their account: deleting a user takes their logins with it.
func TestSessionsCascadeWithUser(t *testing.T) {
sessions, _, database := newStores(t)
token, err := sessions.Create("bob", "")
if err != nil {
t.Fatal(err)
}
if _, err := database.Exec(`DELETE FROM users WHERE id = 'bob'`); err != nil {
t.Fatal(err)
}
if _, err := sessions.Resolve(withCookie(token)); !errors.Is(err, ErrNoSession) {
t.Fatalf("session outlived its account (err=%v)", err)
}
}
func TestAllowlist(t *testing.T) {
// No list configured = anyone the IdP authenticates, which is the right
// default for a household instance.
if !ParseAllowlist("").Permits("anyone", "anyone@example.com") {
t.Fatal("an empty allowlist turned someone away")
}
if !ParseAllowlist(" ").Permits("anyone", "anyone@example.com") {
t.Fatal("a whitespace-only allowlist turned someone away")
}
list := ParseAllowlist(" sub-123 , Her@Example.com ,, ")
cases := []struct {
sub, email string
want bool
}{
{"sub-123", "someone@example.com", true}, // by subject
{"sub-999", "her@example.com", true}, // by email
{"sub-999", "HER@EXAMPLE.COM", true}, // case-insensitively
{"sub-999", "stranger@example.com", false}, // neither
{"", "", false}, // no claims at all
{"sub-1", "", false}, // a near-miss subject
}
for _, c := range cases {
if got := list.Permits(c.sub, c.email); got != c.want {
t.Fatalf("Permits(%q, %q) = %v, want %v", c.sub, c.email, got, c.want)
}
}
}
internal/auth/users.go
+180
View File
@@ -0,0 +1,180 @@
package auth
import (
"database/sql"
"encoding/json"
"errors"
"net/http"
"strings"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/httputil"
)
// UserStore provisions and reads accounts. Petal has no signup flow: a row
// appears the first time someone Authentik vouches for signs in, and that is
// the only way one is ever created.
type UserStore struct {
db *sql.DB
}
// NewUserStore returns a store backed by the given database.
func NewUserStore(sqlDB *sql.DB) *UserStore { return &UserStore{db: sqlDB} }
// Upsert records the account behind an OIDC login, keyed by the issuer's
// subject id.
//
// The subject is the id — not the email, which people change and which
// Authentik does not promise is stable. Email and display name are refreshed on
// every login so a rename upstream shows up here; pair_lang is deliberately not
// touched, because it is Petal's own setting rather than the IdP's.
func (u *UserStore) Upsert(sub, email, displayName string) error {
if sub == "" {
return errors.New("oidc: empty subject")
}
if displayName == "" {
displayName = email
}
_, err := u.db.Exec(
`INSERT INTO users (id, email, display_name) VALUES (?, ?, ?)
ON CONFLICT(id) DO UPDATE SET
email = excluded.email,
display_name = excluded.display_name`,
sub, email, displayName,
)
return err
}
// Get loads one account.
func (u *UserStore) Get(id string) (db.User, error) {
var user db.User
err := u.db.QueryRow(
`SELECT id, email, COALESCE(display_name, ''), created_at, pair_lang
FROM users WHERE id = ?`, id,
).Scan(&user.ID, &user.Email, &user.DisplayName, &user.CreatedAt, &user.PairLang)
return user, err
}
// MeHandler reports who the caller is. The frontend uses it to namespace
// per-account browser state and to show the signed-in writer; it sits behind
// the auth middleware, so reaching it at all already proves a valid session.
func (u *UserStore) MeHandler() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
user, err := u.Get(UserID(r.Context()))
if err != nil {
httputil.ErrorJSON(w, http.StatusUnauthorized, "not signed in")
return
}
httputil.WriteJSON(w, http.StatusOK, user)
}
}
// The pairs a writer may actually choose, in the order the picker offers them.
//
// This is deliberately *not* internal/llm's list of languages. That one names
// every pair the prompts know how to talk about, which is a cheap thing to add;
// this one names the pairs Petal can render itself in, which requires a langpack
// on the frontend. Accepting a code with no pack would leave her looking at
// Chinese with no way back except another guess, so the server refuses it. es
// joins this list on the day its pack lands, not before.
var shippedPairs = []string{"zh", "pt-PT", "fr"}
func pairIsShipped(lang string) bool {
for _, p := range shippedPairs {
if p == lang {
return true
}
}
return false
}
// SetPairLang moves an account to another (English + X) pair.
func (u *UserStore) SetPairLang(id, lang string) error {
if !pairIsShipped(lang) {
return errors.New("auth: unshipped pair language " + lang)
}
res, err := u.db.Exec(`UPDATE users SET pair_lang = ? WHERE id = ?`, lang, id)
if err != nil {
return err
}
if n, err := res.RowsAffected(); err == nil && n == 0 {
return sql.ErrNoRows
}
return nil
}
// UpdateMeHandler changes the caller's own settings — today, the one setting
// there is: which language Petal speaks alongside her English.
//
// It answers with the whole updated user rather than an empty 204 so the client
// has one shape to trust: /api/me and this return the same thing, and the app
// re-reads the pair from the response instead of assuming its request took.
//
// The pair language reaches further than the UI copy — it picks her Hunspell
// dictionary, her read-aloud voice, which word-lookup provider answers, and the
// language the prompts ask the model to explain in. All of those read
// `users.pair_lang` at use time, so all of them follow from this one write.
func (u *UserStore) UpdateMeHandler() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
var body struct {
PairLang string `json:"pair_lang"`
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
httputil.BadRequest(w, "invalid request body")
return
}
lang := strings.TrimSpace(body.PairLang)
if !pairIsShipped(lang) {
// Name the ones that work. A writer who lands here has picked from a
// stale client, and "not a language" tells her nothing.
httputil.BadRequest(w, "unsupported language pair — Petal speaks "+strings.Join(shippedPairs, ", "))
return
}
id := UserID(r.Context())
if err := u.SetPairLang(id, lang); err != nil {
if errors.Is(err, sql.ErrNoRows) {
httputil.ErrorJSON(w, http.StatusUnauthorized, "not signed in")
return
}
httputil.ServerError(w, err)
return
}
user, err := u.Get(id)
if err != nil {
httputil.ServerError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, user)
}
}
// Allowlist decides which of Authentik's users may write in this Petal.
// Authentik fronts several applications; being a valid user there does not mean
// being a user here.
//
// An entry matches a subject id or an email address, case-insensitively. Both
// are accepted on purpose: a subject is an opaque uuid nobody can know before
// that person's first login, so a subject-only list means the operator must let
// someone in, read a log line, and edit config — whereas an email is knowable in
// advance. An empty list allows everyone the IdP authenticates, which is the
// right default for a single-household instance.
type Allowlist map[string]bool
// ParseAllowlist builds an Allowlist from a comma-separated env value.
func ParseAllowlist(raw string) Allowlist {
list := Allowlist{}
for _, part := range strings.Split(raw, ",") {
if p := strings.ToLower(strings.TrimSpace(part)); p != "" {
list[p] = true
}
}
return list
}
// Permits reports whether this login may proceed.
func (a Allowlist) Permits(sub, email string) bool {
if len(a) == 0 {
return true
}
return a[strings.ToLower(sub)] || (email != "" && a[strings.ToLower(email)])
}
internal/config/config.go
+106 -11
View File
@@ -2,6 +2,7 @@ package config
import (
"os"
"strings"
"time"
)
@@ -12,6 +13,12 @@ type Config struct {
BaseURL string
DatabasePath string
ImageDir string // on-disk store for editor image uploads
// DictPath is DreamDict's built dict.db, read-only, sitting beside
// petal.db. It is what gives Petal French, European Portuguese and Spanish
// word lookups; without it only the embedded English/Chinese datasets
// exist, which is exactly how a laptop checkout runs. A missing file is
// therefore not an error — see lexicon.OpenDreamDict.
DictPath string
// LLM
LLMBackend string // "vllm" | "ollama"
@@ -23,19 +30,49 @@ type Config struct {
// TTS (read-aloud). Off unless TTSEndpoint is set — when empty, the /api/tts
// route isn't mounted and the frontend falls back to the browser's Web Speech
// API. Endpoint points at a local Piper HTTP server.
TTSEndpoint string // Piper instance serving the English voice
TTSEndpointZH string // Piper instance serving the Chinese voice; empty = zh falls back to Web Speech
TTSVoiceEN string // Piper voice id for English (e.g. en_US-amy-medium)
TTSVoiceZH string // Piper voice id for Chinese (e.g. zh_CN-huayan-medium)
TTSEndpoint string // Piper instance serving the English voice; also the on/off switch
// TTSVoices is every language Petal can read aloud, keyed by base language
// tag ("en", "zh", "pt", …). Each Piper server loads exactly one model, so
// a language *is* an instance — and the instances are discovered from the
// environment rather than named in this struct: one
// TTS_ENDPOINT_<LANG>/TTS_VOICE_<LANG> pair per language, so the fr and es
// pairs cost a compose service and two lines of .env rather than a code
// change. English keeps the unsuffixed TTS_ENDPOINT/TTS_VOICE_EN it has
// always had.
TTSVoices map[string]TTSVoice
// TTSPath is the path Piper serves synthesis on. Piper moved it from "/" to
// "/synthesize" in 1.6.0 with an unchanged request body, so this is a
// version knob, not a feature: millenia's older server keeps the default,
// the containerised sidecars set "/synthesize".
TTSPath string
TTSCacheDir string // on-disk store for synthesized clips (content-addressed)
TTSTimeout time.Duration
TTSFormat string // mp3 | opus | wav — mp3/opus transcode Piper's WAV via ffmpeg
// Auth (deferred — not wired in the local-dev build, kept for later)
AuthentikURL string
// Auth. OIDC against Authentik. Login is enabled only when the issuer, the
// client id and the secret are all present; with any of them missing Petal
// falls back to the single hardcoded local user, which is what local
// development wants and what every deployment did before Phase 16.
AuthentikURL string // issuer URL of the Petal provider in Authentik
AuthentikClientID string
AuthentikClientSecret string
SessionSecret string
// AllowedSubs gates who may sign in, as a comma-separated list of OIDC
// subject ids and/or email addresses. Empty means everyone Authentik
// authenticates — right for a single-household instance, wrong the moment
// the IdP serves an audience wider than Petal's.
AllowedSubs string
}
// TTSVoice is one Piper instance and the single voice it has loaded.
type TTSVoice struct {
Endpoint string
Voice string
}
// AuthEnabled reports whether real logins are configured. When false, Petal
// resolves every request to the local user.
func (c *Config) AuthEnabled() bool {
return c.AuthentikURL != "" && c.AuthentikClientID != "" && c.AuthentikClientSecret != ""
}
// Load reads configuration from the environment, applying sane local-dev defaults.
@@ -45,6 +82,7 @@ func Load() *Config {
BaseURL: env("BASE_URL", "http://localhost:8080"),
DatabasePath: env("DATABASE_PATH", "./data/petal.db"),
ImageDir: env("IMAGE_DIR", "./data/images"),
DictPath: env("DICT_PATH", "./data/dict.db"),
LLMBackend: env("LLM_BACKEND", "vllm"),
LLMEndpoint: env("LLM_ENDPOINT", "http://localhost:8000"),
@@ -53,9 +91,8 @@ func Load() *Config {
LLMTimeout: envDuration("LLM_TIMEOUT", 30*time.Second),
TTSEndpoint: env("TTS_ENDPOINT", ""),
TTSEndpointZH: env("TTS_ENDPOINT_ZH", ""),
TTSVoiceEN: env("TTS_VOICE_EN", "en_US-amy-medium"),
TTSVoiceZH: env("TTS_VOICE_ZH", "zh_CN-huayan-medium"),
TTSVoices: ttsVoices(os.Environ()),
TTSPath: env("TTS_PATH", "/"),
TTSCacheDir: env("TTS_CACHE_DIR", "./data/tts"),
TTSTimeout: envDuration("TTS_TIMEOUT", 15*time.Second),
TTSFormat: env("TTS_AUDIO_FORMAT", "mp3"),
@@ -63,10 +100,68 @@ func Load() *Config {
AuthentikURL: env("AUTHENTIK_URL", ""),
AuthentikClientID: env("AUTHENTIK_CLIENT_ID", ""),
AuthentikClientSecret: env("AUTHENTIK_CLIENT_SECRET", ""),
SessionSecret: env("SESSION_SECRET", "dev-insecure-secret-change-me"),
AllowedSubs: env("PETAL_ALLOWED_SUBS", ""),
}
}
// ttsVoices reads the Piper instances out of an environment slice (as returned
// by os.Environ) into a map keyed by base language tag.
//
// English is the unsuffixed pair, TTS_ENDPOINT + TTS_VOICE_EN, because that is
// what every deployment already sets and read-aloud has always been English
// first. Every other language is a TTS_ENDPOINT_<LANG>/TTS_VOICE_<LANG> pair,
// discovered rather than enumerated — TTS_ENDPOINT_ZH is what millenia and the
// VPS already use, and TTS_ENDPOINT_PT is all the Portuguese pair needs.
//
// <LANG> is the *base* tag: an environment variable name cannot hold the hyphen
// in "pt-PT", and the handler routes on the base tag anyway (a request for
// pt-PT, pt-BR or bare pt reaches the same instance, because there is only one
// Portuguese voice loaded). A pair is ignored unless both halves are set: half
// a configuration should read as "no voice for this language" and fall back to
// the browser, not as an instance that answers every request with an error.
func ttsVoices(environ []string) map[string]TTSVoice {
vals := make(map[string]string, len(environ))
for _, kv := range environ {
if k, v, ok := strings.Cut(kv, "="); ok {
vals[k] = v
}
}
voices := map[string]TTSVoice{}
add := func(lang, endpoint, voice string) {
endpoint = strings.TrimRight(strings.TrimSpace(endpoint), "/")
voice = strings.TrimSpace(voice)
if endpoint == "" || voice == "" {
return
}
voices[lang] = TTSVoice{Endpoint: endpoint, Voice: voice}
}
// The two languages that shipped before this was a map keep their voice
// defaults, so an existing deployment that names only the endpoints (as
// millenia's unit does) sounds exactly as it did.
voiceOr := func(key, fallback string) string {
if v := strings.TrimSpace(vals[key]); v != "" {
return v
}
return fallback
}
add("en", vals["TTS_ENDPOINT"], voiceOr("TTS_VOICE_EN", "en_US-amy-medium"))
for k, endpoint := range vals {
suffix, ok := strings.CutPrefix(k, "TTS_ENDPOINT_")
if !ok || suffix == "" {
continue
}
voice := vals["TTS_VOICE_"+suffix]
if suffix == "ZH" {
voice = voiceOr("TTS_VOICE_ZH", "zh_CN-huayan-medium")
}
add(strings.ToLower(suffix), endpoint, voice)
}
return voices
}
func env(key, fallback string) string {
if v := os.Getenv(key); v != "" {
return v
internal/config/config_test.go
+88
View File
@@ -0,0 +1,88 @@
package config
import "testing"
// The Piper instances are discovered from the environment rather than named in
// code, so that a new pair costs a compose service and two .env lines. These
// assert the discovery rule, including the two shapes that already exist in the
// wild (millenia's systemd unit and the VPS compose file).
func TestTTSVoicesDiscovery(t *testing.T) {
voices := ttsVoices([]string{
"TTS_ENDPOINT=http://piper-en:5000",
"TTS_VOICE_EN=en_US-amy-medium",
"TTS_ENDPOINT_ZH=http://piper-zh:5000/",
"TTS_VOICE_ZH=zh_CN-huayan-medium",
"TTS_ENDPOINT_PT=http://piper-pt:5000",
"TTS_VOICE_PT=pt_PT-tugão-medium",
"TTS_ENDPOINT_FR=http://piper-fr:5000",
"TTS_VOICE_FR=fr_FR-siwis-medium",
// Noise that must not become a language.
"TTS_PATH=/synthesize",
"PATH=/usr/bin",
})
want := map[string]TTSVoice{
"en": {"http://piper-en:5000", "en_US-amy-medium"},
// The trailing slash is trimmed here so the synthesis path concatenates
// cleanly rather than producing a double slash at every call site.
"zh": {"http://piper-zh:5000", "zh_CN-huayan-medium"},
"pt": {"http://piper-pt:5000", "pt_PT-tugão-medium"},
// Phase 24's whole TTS change: a fourth language costs two lines here
// and a compose service, and no Go at all.
"fr": {"http://piper-fr:5000", "fr_FR-siwis-medium"},
}
if len(voices) != len(want) {
t.Fatalf("discovered %v, want %v", voices, want)
}
for lang, w := range want {
if voices[lang] != w {
t.Errorf("%s = %+v, want %+v", lang, voices[lang], w)
}
}
}
// Half a configuration is not a language. An endpoint with no voice (or the
// reverse) must read as "no voice for this language" — a 404 the client answers
// by falling back to Web Speech — rather than as an instance that exists and
// errors on every request.
func TestTTSVoicesIgnoresHalfConfiguredLanguages(t *testing.T) {
voices := ttsVoices([]string{
"TTS_ENDPOINT=http://piper-en:5000",
"TTS_VOICE_EN=en_US-amy-medium",
"TTS_ENDPOINT_FR=http://piper-fr:5000", // no TTS_VOICE_FR
"TTS_VOICE_ES=es_ES-davefx-medium", // no TTS_ENDPOINT_ES
})
if _, ok := voices["fr"]; ok {
t.Errorf("fr routed with no voice configured")
}
if _, ok := voices["es"]; ok {
t.Errorf("es routed with no endpoint configured")
}
if len(voices) != 1 {
t.Errorf("discovered %v, want English only", voices)
}
}
// A deployment that predates the map names only the endpoints and relies on the
// voice defaults; it must sound exactly as it did.
func TestTTSVoicesKeepsTheOriginalDefaults(t *testing.T) {
voices := ttsVoices([]string{
"TTS_ENDPOINT=http://127.0.0.1:5005",
"TTS_ENDPOINT_ZH=http://127.0.0.1:5006",
})
if got := voices["en"].Voice; got != "en_US-amy-medium" {
t.Errorf("en voice = %q, want the default", got)
}
if got := voices["zh"].Voice; got != "zh_CN-huayan-medium" {
t.Errorf("zh voice = %q, want the default", got)
}
}
// Read-aloud is off when no English instance is configured; nothing else may
// switch it on. (tts.New gates on TTSEndpoint, so a stray TTS_ENDPOINT_PT with
// no English sibling must not produce a routable map that outlives that gate.)
func TestTTSVoicesEmptyWithoutEndpoints(t *testing.T) {
if voices := ttsVoices([]string{"TTS_VOICE_EN=en_US-amy-medium"}); len(voices) != 0 {
t.Errorf("discovered %v, want none", voices)
}
}
internal/db/backup.go
+79
View File
@@ -0,0 +1,79 @@
package db
import (
"database/sql"
"fmt"
"os"
"path/filepath"
)
// Backup writes a consistent copy of the database at srcPath to destPath using
// SQLite's `VACUUM INTO`.
//
// Why not copy the file: Petal runs in WAL mode, so at any instant the newest
// committed pages may live in petal.db-wal rather than petal.db. Copying the
// three files separately can capture them mid-checkpoint and produce a backup
// that is subtly torn. `VACUUM INTO` runs inside a read transaction, so it sees
// one coherent snapshot including the WAL, and emits a single defragmented file
// with no -wal/-shm companions — exactly what you want to ship off-box.
//
// It takes no write lock, so this is safe to run against the live database
// while someone is writing.
//
// destPath must not already exist: SQLite refuses to overwrite, which keeps a
// failed run from destroying the previous good backup.
func Backup(srcPath, destPath string) error {
if _, err := os.Stat(srcPath); err != nil {
return fmt.Errorf("source database: %w", err)
}
if _, err := os.Stat(destPath); err == nil {
return fmt.Errorf("destination %s already exists", destPath)
}
if dir := filepath.Dir(destPath); dir != "" && dir != "." {
if err := os.MkdirAll(dir, 0o755); err != nil {
return fmt.Errorf("create backup dir: %w", err)
}
}
// Opened directly rather than through Open: a backup must never migrate or
// seed the database it is copying.
conn, err := sql.Open("sqlite", dsn(srcPath))
if err != nil {
return fmt.Errorf("open source: %w", err)
}
defer conn.Close()
conn.SetMaxOpenConns(1)
if err := conn.Ping(); err != nil {
return fmt.Errorf("ping source: %w", err)
}
// The path is interpolated because VACUUM INTO takes a literal, not a bound
// parameter. Quotes are doubled so a path containing one can't break out.
quoted := "'" + escapeSQLiteString(destPath) + "'"
if _, err := conn.Exec("VACUUM INTO " + quoted); err != nil {
return fmt.Errorf("vacuum into %s: %w", destPath, err)
}
// A zero-byte result would mean the vacuum silently produced nothing; catch
// it here rather than discovering it during a restore.
info, err := os.Stat(destPath)
if err != nil {
return fmt.Errorf("stat backup: %w", err)
}
if info.Size() == 0 {
return fmt.Errorf("backup %s is empty", destPath)
}
return nil
}
func escapeSQLiteString(s string) string {
out := make([]byte, 0, len(s))
for i := 0; i < len(s); i++ {
if s[i] == '\'' {
out = append(out, '\'')
}
out = append(out, s[i])
}
return string(out)
}
internal/db/backup_test.go
+108
View File
@@ -0,0 +1,108 @@
package db
import (
"database/sql"
"os"
"path/filepath"
"testing"
)
// The point of VACUUM INTO over a file copy is that it captures rows still
// sitting in the WAL. This writes with the source connection open (so the WAL
// is hot and unlikely to have been checkpointed) and asserts the backup has
// them.
func TestBackupCapturesLiveWrites(t *testing.T) {
dir := t.TempDir()
src := filepath.Join(dir, "petal.db")
dest := filepath.Join(dir, "backups", "petal-backup.db")
d, err := Open(src)
if err != nil {
t.Fatalf("open source: %v", err)
}
defer d.Close()
if _, err := d.Exec(
`INSERT INTO documents (id, user_id, title, content_text) VALUES ('d1', ?, '春天', 'hello 春天')`,
LocalUserID,
); err != nil {
t.Fatalf("insert: %v", err)
}
if err := Backup(src, dest); err != nil {
t.Fatalf("backup: %v", err)
}
// VACUUM INTO emits a single self-contained file — no -wal/-shm to ship
// alongside it.
for _, suffix := range []string{"-wal", "-shm"} {
if _, err := os.Stat(dest + suffix); err == nil {
t.Errorf("backup left a %s companion file behind", suffix)
}
}
copyConn, err := sql.Open("sqlite", dsn(dest))
if err != nil {
t.Fatalf("open backup: %v", err)
}
defer copyConn.Close()
var title string
if err := copyConn.QueryRow(`SELECT title FROM documents WHERE id = 'd1'`).Scan(&title); err != nil {
t.Fatalf("row missing from backup: %v", err)
}
if title != "春天" {
t.Errorf("title = %q, want 春天", title)
}
// The seeded user has to come across too, or a restore would orphan every
// document's foreign key.
var users int
if err := copyConn.QueryRow(`SELECT COUNT(*) FROM users WHERE id = ?`, LocalUserID).Scan(&users); err != nil {
t.Fatalf("count users: %v", err)
}
if users != 1 {
t.Errorf("users in backup = %d, want 1", users)
}
}
// A second run to the same path must fail loudly rather than clobber or
// half-write the previous good backup.
func TestBackupRefusesExistingDestination(t *testing.T) {
dir := t.TempDir()
src := filepath.Join(dir, "petal.db")
dest := filepath.Join(dir, "petal-backup.db")
d, err := Open(src)
if err != nil {
t.Fatalf("open source: %v", err)
}
defer d.Close()
if err := Backup(src, dest); err != nil {
t.Fatalf("first backup: %v", err)
}
before, err := os.ReadFile(dest)
if err != nil {
t.Fatalf("read backup: %v", err)
}
if err := Backup(src, dest); err == nil {
t.Fatal("second backup to the same path succeeded; want an error")
}
after, err := os.ReadFile(dest)
if err != nil {
t.Fatalf("re-read backup: %v", err)
}
if len(before) != len(after) {
t.Errorf("existing backup was modified: %d bytes → %d", len(before), len(after))
}
}
func TestBackupMissingSource(t *testing.T) {
dir := t.TempDir()
if err := Backup(filepath.Join(dir, "nope.db"), filepath.Join(dir, "out.db")); err == nil {
t.Fatal("backup of a nonexistent database succeeded; want an error")
}
}
internal/db/db.go
+132
View File
@@ -365,6 +365,138 @@ SELECT id, doc_id, from_pos, to_pos, original, replacement, explanation, type, s
DROP TABLE suggestions;
ALTER TABLE suggestions_new RENAME TO suggestions;
CREATE INDEX idx_suggestions_doc_id ON suggestions(doc_id);
`,
},
{
// Writing passport: evidence that a document was written, not pasted.
//
// `preserve_history` opts a document out of auto-snapshot pruning. The
// 40-snapshot cap is right for recovery (you want recent states) but
// wrong for provenance (you want the *whole* span, oldest included), so
// a writer who may need to defend authorship flags the doc and keeps
// every snapshot.
//
// `content_hash`/`prev_hash` chain each snapshot to the one before it:
// hash = sha256(prev_hash | doc_id | created_at | word_count | text).
// This proves the local history is internally consistent — no snapshot
// was edited, reordered, or removed after the fact without breaking
// every link downstream. It is NOT third-party attestation: anyone with
// the DB and the algorithm could forge a fresh chain. It raises the cost
// of a doctored history from "edit one row" to "rebuild all of them".
// Pre-existing snapshots keep empty hashes and are reported as
// unverifiable rather than as failures.
name: "0009_writing_passport",
stmt: `
ALTER TABLE documents ADD COLUMN preserve_history INTEGER NOT NULL DEFAULT 0;
ALTER TABLE document_versions ADD COLUMN content_hash TEXT NOT NULL DEFAULT '';
ALTER TABLE document_versions ADD COLUMN prev_hash TEXT NOT NULL DEFAULT '';
`,
},
{
// Real accounts. Three separate things land together because they are
// one change: Petal can now tell users apart.
//
// `sessions` backs server-side login state. The cookie carries an opaque
// random token and this table stores only its SHA-256 — a leaked database
// copy therefore yields no usable session, the same reason passwords are
// hashed. Server-side rows (rather than a signed stateless cookie) are
// what make logout and revocation actually revoke.
//
// `images` gives the content-addressed image store an owner. Until now it
// was a flat directory with no database row at all: any caller holding a
// hash could fetch anyone's image, which is capability-URL security, not
// access control. The primary key is (name, user_id), so the same picture
// uploaded by two people is still stored once on disk and simply has two
// rows — deduplication survives; the file is deleted only with its last
// row. Rows for images already on disk are backfilled at startup by the
// images package, which is the only code that knows the storage path.
//
// `users.pair_lang` is the writer's language pair (English + X). It is
// unused until the langpack work, but it belongs to provisioning and
// costs nothing to add while the users table is already being touched.
name: "0010_sessions_images_and_pair_lang",
stmt: `
CREATE TABLE sessions (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
expires_at DATETIME NOT NULL,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
user_agent TEXT NOT NULL DEFAULT ''
);
CREATE INDEX idx_sessions_user_id ON sessions(user_id);
CREATE TABLE images (
name TEXT NOT NULL,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
content_type TEXT NOT NULL DEFAULT '',
size INTEGER NOT NULL DEFAULT 0,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (name, user_id)
);
CREATE INDEX idx_images_user_id ON images(user_id);
ALTER TABLE users ADD COLUMN pair_lang TEXT NOT NULL DEFAULT 'zh';
`,
},
{
// The personal spelling dictionary moves off the browser. It used to be a
// single `petal.spell.personal` key in localStorage, which meant two
// people sharing a device shared a word list built from one person's
// private writing — and one person writing on two devices had two
// unrelated lists.
//
// `lang` is the *dictionary's* language, not the writer's: a word is only
// ever added while a particular Hunspell dictionary flagged it, and an
// en-US personal word must not silence a pt-PT flag (or vice versa) once
// the second pair ships. `word` is stored as typed; matching is exact,
// because case carries meaning to a speller ("polish" vs "Polish").
name: "0011_personal_dictionary",
stmt: `
CREATE TABLE personal_words (
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
lang TEXT NOT NULL,
word TEXT NOT NULL,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (user_id, lang, word)
);
`,
},
{
// The growth journal reads the suggestions table as a record of what the
// writer has been learning, and that reading only works if a row is dated
// by *her decision* rather than by the model's proposal. `created_at` is
// when a checkpoint offered the edit; a suggestion offered in April and
// accepted in June is June's growth, not April's.
//
// Existing rows are backfilled to created_at — which is exactly the
// approximation the journal would have had to make anyway, and is very
// nearly right in practice since edits are settled minutes after a
// checkpoint. Only pending rows keep a NULL: nothing has been decided.
name: "0012_suggestion_resolved_at",
stmt: `
ALTER TABLE suggestions ADD COLUMN resolved_at DATETIME;
UPDATE suggestions SET resolved_at = created_at WHERE status != 'pending';
CREATE INDEX idx_suggestions_resolved ON suggestions(status, resolved_at);
`,
},
{
// Which engine proposed a row. Until now `type` doubled as that answer —
// 'mechanics' meant "the offline rule pack found this" and everything else
// meant "the model did". That breaks the moment an offline rule proposes a
// *collocation*: the miscollocation list (SUGGESTIONS §6) is the same
// family, the same rail and the same warm phrasing as the LLM coach, and it
// must stay type='collocation' so an accepted chunk still plants in the
// garden and still counts in the journal. With type no longer naming the
// engine, the two passes could not scope their own DELETEs — the coach
// would wipe the offline flags, and the offline pass would leave the
// coach's behind to accumulate.
//
// Existing mechanics rows are local by definition; everything else came
// from a model.
name: "0013_suggestion_source",
stmt: `
ALTER TABLE suggestions ADD COLUMN source TEXT NOT NULL DEFAULT 'llm';
UPDATE suggestions SET source = 'local' WHERE type = 'mechanics';
`,
},
}
internal/db/db_test.go
+144
View File
@@ -83,3 +83,147 @@ func TestOpenMigratesAndSeeds(t *testing.T) {
t.Errorf("expected exactly 1 local user after reopen, got %d", users)
}
}
// TestResolvedAtBackfill runs migration 0012 against a database that predates
// it, which is the only shape that matters: on the live box the suggestions
// table is years of settled edits with no resolved_at to their name. Backfilling
// to created_at is exactly the approximation the growth journal would otherwise
// have had to make, and a pending row must stay NULL — nothing has been decided.
func TestResolvedAtBackfill(t *testing.T) {
path := filepath.Join(t.TempDir(), "old.db")
d, err := Open(path)
if err != nil {
t.Fatalf("open: %v", err)
}
// Rewind to the state before 0012: drop the column and forget the migration.
if _, err := d.Exec(`DROP INDEX idx_suggestions_resolved`); err != nil {
t.Fatalf("rewind index: %v", err)
}
if _, err := d.Exec(`ALTER TABLE suggestions DROP COLUMN resolved_at`); err != nil {
t.Fatalf("rewind schema: %v", err)
}
if _, err := d.Exec(`DELETE FROM schema_migrations WHERE name = '0012_suggestion_resolved_at'`); err != nil {
t.Fatalf("rewind migration record: %v", err)
}
if _, err := d.Exec(`INSERT INTO documents (id, user_id) VALUES ('d1', ?)`, LocalUserID); err != nil {
t.Fatalf("insert document: %v", err)
}
for _, s := range []struct{ id, status string }{
{"s-old", "accepted"},
{"s-open", "pending"},
} {
if _, err := d.Exec(
`INSERT INTO suggestions (id, doc_id, from_pos, to_pos, original, replacement, explanation, type, status, created_at)
VALUES (?, 'd1', 0, 3, 'teh', 'the', 'x', 'grammar', ?, '2026-01-02 03:04:05')`,
s.id, s.status,
); err != nil {
t.Fatalf("seed %s: %v", s.id, err)
}
}
d.Close()
d2, err := Open(path)
if err != nil {
t.Fatalf("reopen (migrate): %v", err)
}
defer d2.Close()
// Compared against created_at read back the same way: the driver renders a
// DATETIME column itself, so the assertion is "the same instant", not a
// particular text format.
var settled, created *string
if err := d2.QueryRow(
`SELECT resolved_at, created_at FROM suggestions WHERE id = 's-old'`,
).Scan(&settled, &created); err != nil {
t.Fatalf("read settled row: %v", err)
}
if settled == nil || created == nil || *settled != *created {
t.Errorf("resolved_at = %v, want it backfilled from created_at (%v)", settled, created)
}
var pending *string
if err := d2.QueryRow(`SELECT resolved_at FROM suggestions WHERE id = 's-open'`).Scan(&pending); err != nil {
t.Fatalf("read pending row: %v", err)
}
if pending != nil {
t.Errorf("pending row got resolved_at = %v, want NULL — nothing was decided", *pending)
}
}
// TestSuggestionSourceBackfill runs migration 0013 against a database that
// predates it — the shape the live box is actually in. `source` is the column
// that lets the offline rule pack and the model share the collocation family
// without deleting each other's rows, and it can only do that if the existing
// rows are labelled correctly on the way in: everything the old deterministic
// pass wrote is local, and everything else came from a model.
func TestSuggestionSourceBackfill(t *testing.T) {
path := filepath.Join(t.TempDir(), "old.db")
d, err := Open(path)
if err != nil {
t.Fatalf("open: %v", err)
}
// Rewind to the state before 0013.
if _, err := d.Exec(`ALTER TABLE suggestions DROP COLUMN source`); err != nil {
t.Fatalf("rewind schema: %v", err)
}
if _, err := d.Exec(`DELETE FROM schema_migrations WHERE name = '0013_suggestion_source'`); err != nil {
t.Fatalf("rewind migration record: %v", err)
}
if _, err := d.Exec(`INSERT INTO documents (id, user_id) VALUES ('d1', ?)`, LocalUserID); err != nil {
t.Fatalf("insert document: %v", err)
}
for _, s := range []struct{ id, typ string }{
{"s-mech", SuggestionTypeMechanics},
{"s-gram", SuggestionTypeGrammar},
{"s-coll", SuggestionTypeCollocation},
} {
if _, err := d.Exec(
`INSERT INTO suggestions (id, doc_id, from_pos, to_pos, original, replacement, explanation, type)
VALUES (?, 'd1', 0, 3, 'teh', 'the', 'x', ?)`,
s.id, s.typ,
); err != nil {
t.Fatalf("seed %s: %v", s.id, err)
}
}
d.Close()
d2, err := Open(path)
if err != nil {
t.Fatalf("reopen (migrate): %v", err)
}
defer d2.Close()
// A pre-0013 collocation row can only have come from the coach — the offline
// miscollocation list did not exist yet — so it must NOT be claimed as local.
for id, want := range map[string]string{
"s-mech": SuggestionSourceLocal,
"s-gram": SuggestionSourceLLM,
"s-coll": SuggestionSourceLLM,
} {
var got string
if err := d2.QueryRow(`SELECT source FROM suggestions WHERE id = ?`, id).Scan(&got); err != nil {
t.Fatalf("read %s: %v", id, err)
}
if got != want {
t.Errorf("%s: source = %q, want %q", id, got, want)
}
}
// And a row written after the migration defaults to the model, so a code path
// that forgets to name a source can never silently claim to be offline.
if _, err := d2.Exec(
`INSERT INTO suggestions (id, doc_id, from_pos, to_pos, original, replacement, explanation, type)
VALUES ('s-new', 'd1', 0, 3, 'teh', 'the', 'x', 'grammar')`,
); err != nil {
t.Fatalf("insert new row: %v", err)
}
var fresh string
if err := d2.QueryRow(`SELECT source FROM suggestions WHERE id = 's-new'`).Scan(&fresh); err != nil {
t.Fatalf("read new row: %v", err)
}
if fresh != SuggestionSourceLLM {
t.Errorf("default source = %q, want %q", fresh, SuggestionSourceLLM)
}
}
internal/db/models.go
+29 -3
View File
@@ -2,14 +2,19 @@ package db
import "time"
// User is an account. With auth deferred, the app runs as a single hardcoded
// `local` user (see LocalUserID); the user_id columns and this type exist so
// real auth can drop in later without a schema migration.
// User is an account. Its ID is the OIDC subject for anyone who signed in, or
// LocalUserID for the pre-auth single user (and for local development, where
// StaticResolver still hands out that id).
type User struct {
ID string `json:"id"`
Email string `json:"email"`
DisplayName string `json:"display_name"`
CreatedAt time.Time `json:"created_at"`
// PairLang is the X in this writer's (English + X) language pair — "zh"
// today, "pt-PT"/"fr"/"es" once the langpacks land. It selects the UI copy
// and dictionary set, not the language they may type in.
PairLang string `json:"pair_lang"`
}
// Document is a single piece of writing. `Content` is the Tiptap JSON document
@@ -25,6 +30,10 @@ type Document struct {
WordCount int `json:"word_count"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
// PreserveHistory opts this document out of auto-snapshot pruning so its
// full writing trail survives as authorship evidence (see the passport).
PreserveHistory bool `json:"preserve_history"`
}
// DocumentVersion is a point-in-time snapshot of a document's body, captured so
@@ -42,6 +51,13 @@ type DocumentVersion struct {
WordCount int `json:"word_count"`
Kind string `json:"kind"` // auto | manual | pre_restore
CreatedAt time.Time `json:"created_at"`
// ContentHash chains this snapshot to the previous one (PrevHash), so a
// history that was edited or thinned after the fact fails verification.
// Both are empty for snapshots taken before the chain existed. Omitted from
// list responses; the passport loads them explicitly.
ContentHash string `json:"content_hash,omitempty"`
PrevHash string `json:"prev_hash,omitempty"`
}
// Document version kinds, mirrored from the schema CHECK constraint.
@@ -90,6 +106,10 @@ type Suggestion struct {
Explanation string `json:"explanation"`
Type string `json:"type"` // grammar | phrasing | idiom | clarity | voice | collocation
Status string `json:"status"` // pending | accepted | rejected
// Source names the engine that proposed the edit, not its family: an offline
// rule and the model can both propose a collocation, and the writer is never
// told which one spoke. It exists so each pass can replace its own rows.
Source string `json:"source"` // llm | local
CreatedAt time.Time `json:"created_at"`
}
@@ -103,6 +123,12 @@ const (
SuggestionTypeCollocation = "collocation"
SuggestionTypeMechanics = "mechanics" // deterministic rule-based pass (no LLM)
// Who proposed it. The offline rule pack ('local') runs on every edit inside
// the browser and survives a VPN-down box; the model ('llm') adds the long
// tail when it is reachable.
SuggestionSourceLLM = "llm"
SuggestionSourceLocal = "local"
SuggestionStatusPending = "pending"
SuggestionStatusAccepted = "accepted"
SuggestionStatusRejected = "rejected"
internal/docs/export.go
+3 -2
View File
@@ -13,6 +13,7 @@ import (
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/httputil"
)
@@ -46,7 +47,7 @@ func (h *Handler) exportAll(w http.ResponseWriter, r *http.Request) {
FROM documents
WHERE user_id = ?
ORDER BY updated_at DESC`,
db.LocalUserID,
auth.UserID(r.Context()),
)
if err != nil {
httputil.ServerError(w, err)
@@ -143,7 +144,7 @@ func (h *Handler) export(w http.ResponseWriter, r *http.Request) {
return
}
doc, err := h.fetch(chi.URLParam(r, "id"))
doc, err := h.fetch(auth.UserID(r.Context()), chi.URLParam(r, "id"))
if errors.Is(err, sql.ErrNoRows) {
notFound(w)
return
internal/docs/handlers.go
+28 -14
View File
@@ -1,6 +1,7 @@
// Package docs implements the document CRUD HTTP handlers — the create / list /
// read / update / delete surface that backs the editor and its 1.5s auto-save.
// All access is scoped to the single hardcoded local user while auth is deferred.
// Every query is scoped to the caller resolved by the auth middleware, so a
// document is only ever reachable by the user who owns it.
package docs
import (
@@ -12,6 +13,7 @@ import (
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/httputil"
)
@@ -52,15 +54,16 @@ type docSummary struct {
Tags []db.Tag `json:"tags"`
}
// list returns the local user's documents, most-recently-updated first, each
// list returns the caller's documents, most-recently-updated first, each
// decorated with its tags.
func (h *Handler) list(w http.ResponseWriter, r *http.Request) {
userID := auth.UserID(r.Context())
rows, err := h.DB.Query(
`SELECT id, title, word_count, updated_at
FROM documents
WHERE user_id = ?
ORDER BY updated_at DESC`,
db.LocalUserID,
userID,
)
if err != nil {
httputil.ServerError(w, err)
@@ -84,7 +87,7 @@ func (h *Handler) list(w http.ResponseWriter, r *http.Request) {
return
}
byDoc, err := h.tagsByDoc(ids)
byDoc, err := h.tagsByDoc(userID, ids)
if err != nil {
httputil.ServerError(w, err)
return
@@ -104,7 +107,7 @@ func (h *Handler) create(w http.ResponseWriter, r *http.Request) {
err := h.DB.QueryRow(
`INSERT INTO documents (user_id) VALUES (?)
RETURNING id, user_id, title, content, content_text, tone, word_count, created_at, updated_at`,
db.LocalUserID,
auth.UserID(r.Context()),
).Scan(
&doc.ID, &doc.UserID, &doc.Title, &doc.Content, &doc.ContentText,
&doc.Tone, &doc.WordCount, &doc.CreatedAt, &doc.UpdatedAt,
@@ -118,7 +121,7 @@ func (h *Handler) create(w http.ResponseWriter, r *http.Request) {
// get returns a single full document by id.
func (h *Handler) get(w http.ResponseWriter, r *http.Request) {
doc, err := h.fetch(chi.URLParam(r, "id"))
doc, err := h.fetch(auth.UserID(r.Context()), chi.URLParam(r, "id"))
if errors.Is(err, sql.ErrNoRows) {
notFound(w)
return
@@ -139,12 +142,17 @@ type updateRequest struct {
ContentText *string `json:"content_text"`
Tone *string `json:"tone"`
WordCount *int `json:"word_count"`
// PreserveHistory toggles the passport's keep-everything mode. Sent alone
// by the History panel's toggle, never by the auto-save path.
PreserveHistory *bool `json:"preserve_history"`
}
// update applies the provided fields to a document and returns the saved row.
// content and content_text are kept in sync by the client and written together.
func (h *Handler) update(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
userID := auth.UserID(r.Context())
var req updateRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
@@ -159,9 +167,11 @@ func (h *Handler) update(w http.ResponseWriter, r *http.Request) {
content_text = COALESCE(?, content_text),
tone = COALESCE(?, tone),
word_count = COALESCE(?, word_count),
preserve_history = COALESCE(?, preserve_history),
updated_at = CURRENT_TIMESTAMP
WHERE id = ? AND user_id = ?`,
req.Title, req.Content, req.ContentText, req.Tone, req.WordCount, id, db.LocalUserID,
req.Title, req.Content, req.ContentText, req.Tone, req.WordCount,
req.PreserveHistory, id, userID,
)
if err != nil {
httputil.ServerError(w, err)
@@ -172,7 +182,7 @@ func (h *Handler) update(w http.ResponseWriter, r *http.Request) {
return
}
doc, err := h.fetch(id)
doc, err := h.fetch(userID, id)
if err != nil {
httputil.ServerError(w, err)
return
@@ -194,7 +204,7 @@ func (h *Handler) update(w http.ResponseWriter, r *http.Request) {
func (h *Handler) delete(w http.ResponseWriter, r *http.Request) {
res, err := h.DB.Exec(
`DELETE FROM documents WHERE id = ? AND user_id = ?`,
chi.URLParam(r, "id"), db.LocalUserID,
chi.URLParam(r, "id"), auth.UserID(r.Context()),
)
if err != nil {
httputil.ServerError(w, err)
@@ -207,17 +217,21 @@ func (h *Handler) delete(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNoContent)
}
// fetch loads one full document scoped to the local user.
func (h *Handler) fetch(id string) (db.Document, error) {
// fetch loads one full document, scoped to its owner. Callers pass the id from
// [auth.UserID]; a document belonging to anyone else comes back as
// sql.ErrNoRows, which handlers surface as a 404 rather than a 403 (a stranger's
// document should be indistinguishable from one that doesn't exist).
func (h *Handler) fetch(userID, id string) (db.Document, error) {
var doc db.Document
err := h.DB.QueryRow(
`SELECT id, user_id, title, content, content_text, tone, word_count, created_at, updated_at
`SELECT id, user_id, title, content, content_text, tone, word_count,
created_at, updated_at, preserve_history
FROM documents
WHERE id = ? AND user_id = ?`,
id, db.LocalUserID,
id, userID,
).Scan(
&doc.ID, &doc.UserID, &doc.Title, &doc.Content, &doc.ContentText,
&doc.Tone, &doc.WordCount, &doc.CreatedAt, &doc.UpdatedAt,
&doc.Tone, &doc.WordCount, &doc.CreatedAt, &doc.UpdatedAt, &doc.PreserveHistory,
)
return doc, err
}
internal/docs/handlers_test.go
+12 -2
View File
@@ -8,10 +8,14 @@ import (
"path/filepath"
"testing"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// newTestServer spins up an isolated on-disk database and the docs router.
// newTestServer spins up an isolated on-disk database and the docs router,
// behind the same auth middleware main.go installs. Tests must go through it:
// handlers read the caller from the request context, so a router mounted bare
// would see an empty user id and match no rows.
func newTestServer(t *testing.T) http.Handler {
t.Helper()
database, err := db.Open(filepath.Join(t.TempDir(), "test.db"))
@@ -19,7 +23,13 @@ func newTestServer(t *testing.T) http.Handler {
t.Fatalf("open db: %v", err)
}
t.Cleanup(func() { database.Close() })
return New(database).Routes()
return withAuth(New(database).Routes())
}
// withAuth wraps a router so every test request arrives authenticated as the
// seeded local user — the stand-in for a real session until Authentik lands.
func withAuth(h http.Handler) http.Handler {
return auth.Middleware(auth.StaticResolver(db.LocalUserID))(h)
}
func do(t *testing.T, srv http.Handler, method, path, body string) *httptest.ResponseRecorder {
internal/docs/isolation_test.go
+222
View File
@@ -0,0 +1,222 @@
package docs
import (
"encoding/json"
"net/http"
"path/filepath"
"testing"
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// This file is the point of the auth plumbing: it proves that swapping the
// hardcoded user for a request-scoped one actually isolates accounts. Every
// handler resolves its user from the request, so mounting the same routers twice
// behind two different resolvers gives us two "logged-in" users over one
// database — which is exactly the situation a real login will create.
// newTwoUserServer opens one database holding two users and returns a router for
// each, identical but for who the auth middleware says is calling.
func newTwoUserServer(t *testing.T) (alice, bob http.Handler) {
t.Helper()
database, err := db.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatalf("open db: %v", err)
}
t.Cleanup(func() { database.Close() })
// db.Open seeds the local user; add a second so both sides have a valid FK.
if _, err := database.Exec(
`INSERT INTO users (id, email, display_name) VALUES (?, ?, ?)`,
"bob", "bob@petal.local", "Bob",
); err != nil {
t.Fatalf("seed second user: %v", err)
}
mount := func(userID string) http.Handler {
h := New(database)
r := chi.NewRouter()
r.Mount("/docs", h.Routes())
r.Mount("/tags", h.TagRoutes())
r.Mount("/search", h.SearchRoutes())
return auth.Middleware(auth.StaticResolver(userID))(r)
}
return mount(db.LocalUserID), mount("bob")
}
// TestDocumentIsolation walks every read and write path that takes a document id
// and asserts Bob cannot reach Alice's document through any of them. A stranger's
// document must be indistinguishable from a nonexistent one — 404, never 403.
func TestDocumentIsolation(t *testing.T) {
alice, bob := newTwoUserServer(t)
docID := createDoc(t, alice, "Alice's diary", "a private sentence about my day")
t.Run("not in list", func(t *testing.T) {
rec := do(t, bob, http.MethodGet, "/docs", "")
var out []docSummary
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatalf("decode list: %v", err)
}
if len(out) != 0 {
t.Fatalf("bob sees %d of alice's documents, want 0", len(out))
}
})
t.Run("not in search", func(t *testing.T) {
rec := do(t, bob, http.MethodGet, "/search?q=private", "")
var out []searchResult
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatalf("decode search: %v", err)
}
if len(out) != 0 {
t.Fatalf("search leaked %d of alice's documents", len(out))
}
})
// The FTS index is a separate table joined back to documents; a missing
// user_id filter there would leak content even though the list query is
// scoped, so assert the owner still finds her own document.
t.Run("owner still finds it", func(t *testing.T) {
rec := do(t, alice, http.MethodGet, "/search?q=private", "")
var out []searchResult
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatalf("decode search: %v", err)
}
if len(out) != 1 {
t.Fatalf("alice found %d results for her own document, want 1", len(out))
}
})
for _, tc := range []struct {
name, method, path, body string
}{
{"get", http.MethodGet, "/docs/" + docID, ""},
{"update", http.MethodPut, "/docs/" + docID, `{"title":"defaced"}`},
{"delete", http.MethodDelete, "/docs/" + docID, ""},
{"export", http.MethodGet, "/docs/" + docID + "/export?format=md", ""},
{"passport", http.MethodGet, "/docs/" + docID + "/passport", ""},
{"snapshot", http.MethodPost, "/docs/" + docID + "/versions", ""},
} {
t.Run(tc.name, func(t *testing.T) {
rec := do(t, bob, tc.method, tc.path, tc.body)
if rec.Code != http.StatusNotFound {
t.Fatalf("%s %s as bob = %d, want 404 (body: %s)",
tc.method, tc.path, rec.Code, rec.Body)
}
})
}
// The document must have survived every attempt above unchanged.
rec := do(t, alice, http.MethodGet, "/docs/"+docID, "")
if rec.Code != http.StatusOK {
t.Fatalf("alice lost access to her own document: %d %s", rec.Code, rec.Body)
}
var doc db.Document
if err := json.Unmarshal(rec.Body.Bytes(), &doc); err != nil {
t.Fatalf("decode doc: %v", err)
}
if doc.Title != "Alice's diary" {
t.Fatalf("title = %q, want %q — bob's update went through", doc.Title, "Alice's diary")
}
}
// TestVersionIsolation covers the history endpoints, which scope through a join
// to documents rather than a direct user_id column — an easy place to forget the
// filter, and one where the leak would be the full text of every draft.
func TestVersionIsolation(t *testing.T) {
alice, bob := newTwoUserServer(t)
docID := createDoc(t, alice, "Draft", "the first version of my essay")
rec := do(t, alice, http.MethodPost, "/docs/"+docID+"/versions", "")
if rec.Code != http.StatusCreated {
t.Fatalf("snapshot: %d %s", rec.Code, rec.Body)
}
var v db.DocumentVersion
if err := json.Unmarshal(rec.Body.Bytes(), &v); err != nil {
t.Fatalf("decode version: %v", err)
}
t.Run("list is empty for stranger", func(t *testing.T) {
rec := do(t, bob, http.MethodGet, "/docs/"+docID+"/versions", "")
var out []db.DocumentVersion
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatalf("decode: %v", err)
}
if len(out) != 0 {
t.Fatalf("bob sees %d of alice's snapshots, want 0", len(out))
}
})
for _, tc := range []struct{ name, method, path string }{
{"preview", http.MethodGet, "/docs/" + docID + "/versions/" + v.ID},
{"restore", http.MethodPost, "/docs/" + docID + "/versions/" + v.ID + "/restore"},
} {
t.Run(tc.name, func(t *testing.T) {
rec := do(t, bob, tc.method, tc.path, "")
if rec.Code != http.StatusNotFound {
t.Fatalf("%s as bob = %d, want 404 (body: %s)", tc.name, rec.Code, rec.Body)
}
})
}
}
// TestTagIsolation checks the tag roster and, more importantly, that a document
// and a tag can't be cross-linked across accounts — the assignment endpoint takes
// two ids from different tables and must own-check both.
func TestTagIsolation(t *testing.T) {
alice, bob := newTwoUserServer(t)
docID := createDoc(t, alice, "Essay", "some words")
rec := do(t, alice, http.MethodPost, "/tags", `{"name":"school","color":"mint"}`)
if rec.Code != http.StatusCreated {
t.Fatalf("create tag: %d %s", rec.Code, rec.Body)
}
var aliceTag db.Tag
if err := json.Unmarshal(rec.Body.Bytes(), &aliceTag); err != nil {
t.Fatalf("decode tag: %v", err)
}
rec = do(t, bob, http.MethodPost, "/tags", `{"name":"bobs","color":"sky"}`)
var bobTag db.Tag
if err := json.Unmarshal(rec.Body.Bytes(), &bobTag); err != nil {
t.Fatalf("decode bob tag: %v", err)
}
t.Run("roster is per user", func(t *testing.T) {
rec := do(t, bob, http.MethodGet, "/tags", "")
var out []db.Tag
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatalf("decode: %v", err)
}
if len(out) != 1 || out[0].Name != "bobs" {
t.Fatalf("bob's roster = %+v, want just his own tag", out)
}
})
t.Run("cannot tag a stranger's document", func(t *testing.T) {
body, _ := json.Marshal(map[string]string{"tag_id": bobTag.ID})
rec := do(t, bob, http.MethodPost, "/docs/"+docID+"/tags", string(body))
if rec.Code != http.StatusNotFound {
t.Fatalf("bob tagging alice's doc = %d, want 404", rec.Code)
}
})
t.Run("cannot rename a stranger's tag", func(t *testing.T) {
rec := do(t, bob, http.MethodPatch, "/tags/"+aliceTag.ID, `{"name":"stolen"}`)
if rec.Code != http.StatusNotFound {
t.Fatalf("bob renaming alice's tag = %d, want 404", rec.Code)
}
})
t.Run("cannot delete a stranger's tag", func(t *testing.T) {
rec := do(t, bob, http.MethodDelete, "/tags/"+aliceTag.ID, "")
if rec.Code != http.StatusNotFound {
t.Fatalf("bob deleting alice's tag = %d, want 404", rec.Code)
}
})
}
internal/docs/passport.go
+283
View File
@@ -0,0 +1,283 @@
package docs
// Writing passport: a standalone, printable report showing *how* a document was
// written — when each snapshot landed, how the word count grew, how the work
// broke into sessions. It exists because automated "AI detector" verdicts are
// unreliable and skew against non-native English writers, so the useful thing to
// hand someone who doubts your authorship is not a score but a record.
//
// The report is deliberately modest about what it proves (see passportLimits):
// it evidences a plausible writing process, it does not certify one.
import (
"crypto/sha256"
"database/sql"
"encoding/hex"
"errors"
"fmt"
"net/http"
"time"
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/httputil"
)
// Passport tuning.
const (
// sessionGap is the idle time that separates one writing session from the
// next. Auto-snapshots fire at most every 3 minutes while typing, so any
// gap far above that means the writer stepped away. 45 minutes keeps a
// coffee break inside one session but splits morning from evening work.
sessionGap = 45 * time.Minute
// jumpNoteThreshold is the share of the final word count a single
// snapshot-to-snapshot increase must exceed before the report calls it out.
// A large jump is the first thing a skeptical reader will ask about, so the
// report raises it rather than leaving it to be discovered.
jumpNoteThreshold = 0.25
)
// chainHash links a snapshot to its predecessor. Covering prev_hash makes each
// hash depend on the entire history before it, so altering any earlier snapshot
// invalidates every later one; covering created_at means a row cannot be
// silently backdated.
//
// This detects tampering with the local database. It is not third-party
// attestation — someone with the database and this function could regenerate a
// consistent chain from scratch.
func chainHash(prevHash, docID string, createdAt time.Time, wordCount int, text string) string {
h := sha256.New()
fmt.Fprintf(h, "%s\x00%s\x00%d\x00%d\x00%s",
prevHash, docID, createdAt.UTC().UnixNano(), wordCount, text)
return hex.EncodeToString(h.Sum(nil))
}
// --- report model -----------------------------------------------------------
// passportSession is one continuous stretch of work — snapshots with no
// sessionGap-sized pause between them.
type passportSession struct {
Start, End time.Time
Snapshots int
WordsAdded int // net change across the session; negative when trimming
}
// Duration is the observed length of the session: first snapshot to last. A
// single-snapshot session reports zero, which is why total active time is
// described as a lower bound.
func (s passportSession) Duration() time.Duration { return s.End.Sub(s.Start) }
// chain verification outcomes, in the order the report prefers to report them.
const (
chainVerified = "verified" // every hash recomputes and every link holds
chainGaps = "gaps" // hashes valid, links broken — consistent with pruning
chainPartial = "partial" // some snapshots predate the hash chain
chainUnverifiable = "unverifiable" // no snapshot carries a hash
chainBroken = "broken" // a hash does not match its own contents
)
// passportData is everything the template renders.
type passportData struct {
Doc db.Document
Versions []db.DocumentVersion // ascending by time
Sessions []passportSession
FirstAt time.Time
LastAt time.Time
Span time.Duration // wall-clock first snapshot → last
ActiveTime time.Duration // summed session durations; a lower bound
LargestJump int // biggest single snapshot-to-snapshot word increase
LargestJumpAt time.Time
LargestJumpIdx int // index into Versions, so the chart can mark it
NoteJump bool // jump is large enough to be worth pre-empting
ChainStatus string
UnhashedCount int
GeneratedAt time.Time
}
// buildPassport derives the report from a document and its snapshots, which must
// be ordered oldest-first. It assumes nothing about snapshot spacing.
func buildPassport(doc db.Document, versions []db.DocumentVersion) passportData {
d := passportData{
Doc: doc,
Versions: versions,
GeneratedAt: time.Now(),
}
if len(versions) == 0 {
d.ChainStatus = chainUnverifiable
return d
}
d.FirstAt = versions[0].CreatedAt
d.LastAt = versions[len(versions)-1].CreatedAt
d.Span = d.LastAt.Sub(d.FirstAt)
cur := passportSession{Start: versions[0].CreatedAt, End: versions[0].CreatedAt, Snapshots: 1}
// Baseline for the running session's net-words figure. Later sessions
// measure from the *previous* session's final count, not from their own
// first snapshot, because that first snapshot already contains the few
// minutes of typing that preceded it — measuring from it would drop that
// work. The first session is the exception: it measures from its own first
// snapshot rather than from zero, so a history whose early snapshots were
// pruned understates session one instead of reporting the words it never
// saw as a sudden addition.
startWords := versions[0].WordCount
for i := 1; i < len(versions); i++ {
v, prev := versions[i], versions[i-1]
if delta := v.WordCount - prev.WordCount; delta > d.LargestJump {
d.LargestJump, d.LargestJumpAt, d.LargestJumpIdx = delta, v.CreatedAt, i
}
if v.CreatedAt.Sub(prev.CreatedAt) > sessionGap {
cur.WordsAdded = prev.WordCount - startWords
d.Sessions = append(d.Sessions, cur)
cur = passportSession{Start: v.CreatedAt, End: v.CreatedAt, Snapshots: 1}
startWords = prev.WordCount
continue
}
cur.End = v.CreatedAt
cur.Snapshots++
}
cur.WordsAdded = versions[len(versions)-1].WordCount - startWords
d.Sessions = append(d.Sessions, cur)
for _, s := range d.Sessions {
d.ActiveTime += s.Duration()
}
final := versions[len(versions)-1].WordCount
d.NoteJump = final > 0 && float64(d.LargestJump)/float64(final) > jumpNoteThreshold
d.ChainStatus, d.UnhashedCount = verifyChain(doc, versions)
return d
}
// verifyChain recomputes every snapshot's hash and checks that each links to the
// one before it. Returns the outcome and how many snapshots predate the chain.
//
// Broken *links* are not evidence of tampering on their own: auto-snapshot
// pruning legitimately removes rows from the middle of the history, which severs
// the links across the hole. So a link break is reported as a gap unless the
// document is in preserve-history mode, where nothing should ever be removed. A
// hash that fails to match its *own* contents is unambiguous, and always broken.
func verifyChain(doc db.Document, versions []db.DocumentVersion) (status string, unhashed int) {
var (
hashed int
linkBreak bool
prevHash string
havePrev bool
)
for _, v := range versions {
if v.ContentHash == "" {
unhashed++
havePrev = false // can't vouch for what follows an unhashed row
continue
}
hashed++
want := chainHash(v.PrevHash, v.DocID, v.CreatedAt, v.WordCount, v.ContentText)
if want != v.ContentHash {
return chainBroken, unhashed
}
if havePrev && v.PrevHash != prevHash {
linkBreak = true
}
prevHash, havePrev = v.ContentHash, true
}
switch {
case hashed == 0:
return chainUnverifiable, unhashed
case linkBreak && doc.PreserveHistory:
// Nothing should have been removed from a preserved history.
return chainBroken, unhashed
case linkBreak:
return chainGaps, unhashed
case unhashed > 0:
return chainPartial, unhashed
default:
return chainVerified, unhashed
}
}
// --- HTTP -------------------------------------------------------------------
// passport renders the report for one document as a standalone HTML download.
// HTML rather than PDF for the same reason as the other exports: a CJK-safe PDF
// needs an embedded Unicode font or a headless browser. The page is styled for
// printing, so "Save as PDF" in the browser produces the handoff artifact.
func (h *Handler) passport(w http.ResponseWriter, r *http.Request) {
docID := chi.URLParam(r, "id")
userID := auth.UserID(r.Context())
doc, err := h.fetch(userID, docID)
if errors.Is(err, sql.ErrNoRows) {
notFound(w)
return
}
if err != nil {
httputil.ServerError(w, err)
return
}
versions, err := h.passportVersions(userID, docID)
if err != nil {
httputil.ServerError(w, err)
return
}
body := renderPassport(buildPassport(doc, versions))
filename := sanitizeFilename(doc.Title)
if filename == "" {
filename = "untitled"
}
filename += " - writing passport.html"
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.Header().Set("Content-Disposition",
fmt.Sprintf("attachment; filename*=UTF-8''%s", urlEscapeFilename(filename)))
w.Header().Set("Content-Length", fmt.Sprintf("%d", len(body)))
_, _ = w.Write(body)
}
// passportVersions loads every snapshot oldest-first with the fields the report
// and the chain check need — including content_text, which the list endpoint
// omits as too heavy but verification cannot do without.
func (h *Handler) passportVersions(userID, docID string) ([]db.DocumentVersion, error) {
rows, err := h.DB.Query(
`SELECT v.id, v.doc_id, v.title, v.content_text, v.word_count, v.kind,
v.created_at, v.content_hash, v.prev_hash
FROM document_versions v
JOIN documents d ON d.id = v.doc_id
WHERE v.doc_id = ? AND d.user_id = ?
ORDER BY v.created_at ASC, v.rowid ASC`,
docID, userID,
)
if err != nil {
return nil, err
}
defer rows.Close()
var out []db.DocumentVersion
for rows.Next() {
var v db.DocumentVersion
if err := rows.Scan(
&v.ID, &v.DocID, &v.Title, &v.ContentText, &v.WordCount, &v.Kind,
&v.CreatedAt, &v.ContentHash, &v.PrevHash,
); err != nil {
return nil, err
}
out = append(out, v)
}
return out, rows.Err()
}
internal/docs/passport_render.go
+387
View File
@@ -0,0 +1,387 @@
package docs
// HTML rendering for the writing passport. Self-contained (no external assets)
// and styled for print, so the browser's "Save as PDF" turns it into the file a
// writer actually hands over.
import (
"fmt"
"math"
"strings"
"time"
)
// Chart geometry. The plot is wide and short on purpose: the report's question
// is "what shape did this document grow in", and a wide aspect makes a steady
// climb read as steady rather than dramatic.
const (
chartW, chartH = 760, 260
padL, padR, padT, padB = 52, 20, 18, 34
plotW, plotH = chartW - padL - padR, chartH - padT - padB
minBandW = 2.0 // so a single-snapshot session still shows
// gutterSlots is the space between sessions, in snapshot-slot widths. Wide
// enough to read as a break and to seat its duration label.
gutterSlots = 2.5
)
// Palette — the export stylesheet's tokens, reused so a passport looks like it
// came from the same application as the document it describes.
const (
rose = "#b04a6a"
roseLight = "#f6d6e0"
roseWash = "#fdeef3"
surface = "#fffafb"
)
func renderPassport(d passportData) []byte {
var b strings.Builder
fmt.Fprintf(&b, passportHead, htmlEscape(d.Doc.Title))
fmt.Fprintf(&b, `<header>
<p class="eyebrow">Writing passport</p>
<h1>%s</h1>
<p class="sub">Generated %s</p>
</header>
`, htmlEscape(d.Doc.Title), htmlEscape(formatWhen(d.GeneratedAt)))
if len(d.Versions) == 0 {
b.WriteString(`<p class="empty">This document has no saved history yet, so there is
nothing to report. History builds up automatically as you write.</p>
</body></html>`)
return []byte(b.String())
}
b.WriteString(renderStats(d))
b.WriteString(renderChart(d))
b.WriteString(renderSessions(d))
b.WriteString(renderIntegrity(d))
b.WriteString(passportLimits)
b.WriteString("</body></html>\n")
return []byte(b.String())
}
// renderStats is the headline row — the numbers a reader wants before deciding
// whether to study the chart.
func renderStats(d passportData) string {
final := d.Versions[len(d.Versions)-1].WordCount
tiles := []struct{ value, label string }{
{fmt.Sprintf("%d", len(d.Versions)), "snapshots saved"},
{humanDuration(d.Span), "from first to last edit"},
{fmt.Sprintf("%d", len(d.Sessions)), pluralize(len(d.Sessions), "writing session", "writing sessions")},
{humanDuration(d.ActiveTime), "spent actively editing"},
{fmt.Sprintf("%d", final), "words in the final draft"},
}
var b strings.Builder
b.WriteString(`<section class="stats">`)
for _, t := range tiles {
fmt.Fprintf(&b, `<div class="tile"><span class="v">%s</span><span class="l">%s</span></div>`,
htmlEscape(t.value), htmlEscape(t.label))
}
b.WriteString("</section>\n")
return b.String()
}
// renderChart draws word count as a step line — the count is only known at
// snapshot moments, and a step says that honestly where a smooth curve would
// invent values in between. Shaded bands mark writing sessions.
//
// The x axis is snapshot order, not wall-clock time, and this is deliberate. On
// a linear time axis an essay written in three half-hour sittings across three
// days renders as three vertical cliffs separated by empty space: all the actual
// writing is crushed into one percent of the width, and the result looks exactly
// like text pasted in three chunks — the opposite of what happened. Because
// auto-snapshots are throttled to roughly one per few minutes of *active*
// editing, snapshot order is already close to proportional to time spent
// writing. So the plot gives its width to the writing and compresses the breaks,
// which are drawn as explicit labelled gaps rather than silently removed.
//
// One series, so no legend: the heading names it.
func renderChart(d passportData) string {
maxW := 0
for _, v := range d.Versions {
if v.WordCount > maxW {
maxW = v.WordCount
}
}
yTop := niceCeil(maxW)
y := func(words int) float64 {
if yTop <= 0 {
return padT + plotH
}
return padT + plotH - float64(words)/float64(yTop)*plotH
}
// Lay snapshots out in slots: one per snapshot, plus a gutter between
// sessions for the break marker.
slots := float64(len(d.Versions)) + gutterSlots*float64(len(d.Sessions)-1)
sw := plotW / slots
xs := make([]float64, len(d.Versions))
bandStart := make([]float64, len(d.Sessions))
bandEnd := make([]float64, len(d.Sessions))
cursor, vi := 0.0, 0
for si, s := range d.Sessions {
if si > 0 {
cursor += gutterSlots
}
bandStart[si] = padL + cursor*sw
for k := 0; k < s.Snapshots; k++ {
xs[vi] = padL + (cursor+0.5)*sw
cursor++
vi++
}
bandEnd[si] = padL + cursor*sw
}
var b strings.Builder
fmt.Fprintf(&b, `<section class="chart">
<h2>How the draft grew</h2>
<svg viewBox="0 0 %d %d" role="img" aria-label="Word count at each saved snapshot, grouped into writing sessions">
`, chartW, chartH)
// Session bands sit behind everything; each carries a native tooltip.
for i, s := range d.Sessions {
w := bandEnd[i] - bandStart[i]
if w < minBandW {
w = minBandW
}
fmt.Fprintf(&b, `<rect x="%.1f" y="%d" width="%.1f" height="%d" fill="%s" rx="3"><title>Session %d: %s, %s, %d snapshots</title></rect>
`, bandStart[i], padT, w, plotH, roseWash, i+1,
htmlEscape(formatWhen(s.Start)), htmlEscape(humanDuration(s.Duration())), s.Snapshots)
}
// Recessive gridlines with y labels at 0 / half / top.
for _, gv := range []int{0, yTop / 2, yTop} {
gy := y(gv)
fmt.Fprintf(&b, `<line x1="%d" y1="%.1f" x2="%d" y2="%.1f" stroke="%s" stroke-width="1"/>
<text x="%d" y="%.1f" class="axis" text-anchor="end">%d</text>
`, padL, gy, chartW-padR, gy, roseLight, padL-8, gy+4, gv)
}
// Break markers in the gutters, so compressed time is stated, not hidden.
for i := 1; i < len(d.Sessions); i++ {
mid := (bandEnd[i-1] + bandStart[i]) / 2
gap := d.Sessions[i].Start.Sub(d.Sessions[i-1].End)
fmt.Fprintf(&b, `<line x1="%.1f" y1="%d" x2="%.1f" y2="%d" stroke="%s" stroke-width="1" stroke-dasharray="3 3"/>
<text x="%.1f" y="%d" class="gap" text-anchor="middle">%s</text>
`, mid, padT, mid, padT+plotH, roseLight, mid, padT+plotH+13, htmlEscape(humanDuration(gap)+" away"))
}
// Step path: hold the previous value until the next snapshot lands.
var path strings.Builder
fmt.Fprintf(&path, "M %.1f %.1f", xs[0], y(d.Versions[0].WordCount))
for i := 1; i < len(d.Versions); i++ {
fmt.Fprintf(&path, " L %.1f %.1f L %.1f %.1f",
xs[i], y(d.Versions[i-1].WordCount), xs[i], y(d.Versions[i].WordCount))
}
fmt.Fprintf(&b, `<path d="%s" fill="none" stroke="%s" stroke-width="2" stroke-linejoin="round"/>
`, path.String(), rose)
// Pre-empt the obvious question: label the largest single jump when it is a
// big share of the finished draft, rather than letting a reader find it.
if d.NoteJump && d.LargestJumpIdx < len(xs) {
jx, jy := xs[d.LargestJumpIdx], y(d.Versions[d.LargestJumpIdx].WordCount)
// Flip the label inboard near the right edge so it can't overflow, and
// push it below the point when the point sits near the top.
anchor, dx := "start", 9.0
if jx > float64(chartW)*0.6 {
anchor, dx = "end", -9.0
}
ly := jy - 10
if ly < padT+12 {
ly = jy + 18
}
fmt.Fprintf(&b, `<circle cx="%.1f" cy="%.1f" r="4" fill="%s" stroke="%s" stroke-width="2"/>
<text x="%.1f" y="%.1f" class="note" text-anchor="%s">largest single addition: +%d words</text>
`, jx, jy, rose, surface, jx+dx, ly, anchor, d.LargestJump)
}
fmt.Fprintf(&b, `</svg>
<p class="caption">Each shaded band is one writing session, %s to %s. Width follows
snapshots saved, so time spent writing gets the space and breaks are compressed to
the labelled gaps.</p>
</section>
`, htmlEscape(formatDay(d.FirstAt)), htmlEscape(formatDay(d.LastAt)))
return b.String()
}
func renderSessions(d passportData) string {
var b strings.Builder
b.WriteString(`<section>
<h2>Writing sessions</h2>
<table>
<thead><tr><th>#</th><th>Started</th><th>Length</th><th>Snapshots</th><th>Net words</th></tr></thead>
<tbody>
`)
for i, s := range d.Sessions {
fmt.Fprintf(&b, `<tr><td>%d</td><td>%s</td><td>%s</td><td>%d</td><td>%+d</td></tr>
`, i+1, htmlEscape(formatWhen(s.Start)), htmlEscape(humanDuration(s.Duration())),
s.Snapshots, s.WordsAdded)
}
b.WriteString("</tbody></table>\n</section>\n")
return b.String()
}
// renderIntegrity explains the hash chain in plain language, including when it
// cannot vouch for something.
func renderIntegrity(d passportData) string {
var headline, detail string
switch d.ChainStatus {
case chainVerified:
headline = "History intact"
detail = "Every snapshot matches its own contents and links correctly to the one before it. Nothing in this history has been altered or removed since it was recorded."
case chainGaps:
headline = "History intact, with gaps"
detail = "Every snapshot matches its own contents, but some older automatic snapshots have been cleared to save space, so the record is not continuous. Turn on “keep full history” for this document to stop that happening."
case chainPartial:
headline = "Partly verifiable"
detail = fmt.Sprintf("%d snapshot(s) were recorded before this document started tracking integrity, so they cannot be checked. Everything recorded since then matches.", d.UnhashedCount)
case chainUnverifiable:
headline = "Not verifiable"
detail = "These snapshots were recorded before integrity tracking existed. The timeline above is still the record that was saved as you wrote; it simply cannot be checked for later alteration."
case chainBroken:
headline = "Integrity check failed"
detail = "At least one snapshot does not match what was recorded for it. This can mean the history was edited after the fact, or that the database was restored from a backup or copied between machines."
}
return fmt.Sprintf(`<section class="integrity %s">
<h2>%s</h2>
<p>%s</p>
</section>
`, htmlEscape(d.ChainStatus), htmlEscape(headline), htmlEscape(detail))
}
// passportLimits states plainly what the report does and does not establish.
// Overclaiming would be worse than useless: a reader who catches the report
// overstating its case discounts the whole thing.
const passportLimits = `<section class="limits">
<h2>How to read this</h2>
<p>A document written over time leaves a trail: many snapshots, uneven growth,
words added and cut and added again across separate sittings. A document that was
pasted in from elsewhere tends to arrive nearly whole, in one or two snapshots,
with little revision after.</p>
<p>What this report shows is the record Petal saved automatically while the
document was open, roughly every few minutes of active editing.</p>
<p><strong>What it does not show.</strong> It cannot prove who was at the
keyboard, and it cannot tell whether text typed into the editor was composed
there or copied from another window. It is evidence of a writing process, not a
certificate of authorship. It is most useful read alongside the drafts
themselves.</p>
</section>
`
// --- formatting helpers -----------------------------------------------------
func formatWhen(t time.Time) string { return t.Local().Format("2 Jan 2006, 3:04 PM") }
func formatDay(t time.Time) string { return t.Local().Format("2 Jan 2006") }
// humanDuration renders a span at the coarsest useful precision — a reader cares
// that a session ran "2h 40m", never that it ran 2h40m12s.
func humanDuration(d time.Duration) string {
if d < time.Minute {
return "under a minute"
}
days := int(d.Hours()) / 24
hours := int(d.Hours()) % 24
mins := int(d.Minutes()) % 60
switch {
case days > 0 && hours > 0:
return fmt.Sprintf("%dd %dh", days, hours)
case days > 0:
return fmt.Sprintf("%dd", days)
case hours > 0 && mins > 0:
return fmt.Sprintf("%dh %dm", hours, mins)
case hours > 0:
return fmt.Sprintf("%dh", hours)
default:
return fmt.Sprintf("%dm", mins)
}
}
func pluralize(n int, one, many string) string {
if n == 1 {
return one
}
return many
}
// niceCeil rounds a maximum up to a round number so gridlines land on values a
// reader can hold in their head.
func niceCeil(n int) int {
if n <= 0 {
return 0
}
mag := math.Pow(10, math.Floor(math.Log10(float64(n))))
return int(math.Ceil(float64(n)/(mag/2)) * (mag / 2))
}
// passportHead is the page shell: one %s for the title. Print rules keep the
// chart and the caveats on the page rather than letting them break across sheets.
const passportHead = `<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Writing passport — %s</title>
<style>
:root { color-scheme: light; }
body {
font-family: "Georgia", "Songti SC", "Noto Serif CJK SC", "Source Han Serif SC", serif;
line-height: 1.7; color: #463a3f; background: #fffafb;
max-width: 48rem; margin: 3rem auto; padding: 0 1.5rem;
}
header { border-bottom: 2px solid #f6d6e0; padding-bottom: 1rem; margin-bottom: 2rem; }
.eyebrow { text-transform: uppercase; letter-spacing: .12em; font-size: .72rem;
color: #b04a6a; margin: 0 0 .3rem; }
h1 { font-size: 1.8rem; color: #b04a6a; margin: 0; line-height: 1.3; }
h2 { font-size: 1.05rem; color: #b04a6a; margin: 0 0 .75rem; }
.sub, .axis, .l { color: #6b5860; }
.sub { margin: .4rem 0 0; font-size: .9rem; }
section { margin: 2.25rem 0; }
.stats { display: flex; flex-wrap: wrap; gap: 1.25rem 2rem; margin: 2rem 0; }
.tile { display: flex; flex-direction: column; min-width: 7rem; }
.tile .v { font-size: 1.6rem; color: #b04a6a; line-height: 1.1; }
.tile .l { font-size: .8rem; margin-top: .15rem; }
.chart svg { width: 100%%; height: auto; }
.axis { font-size: 11px; fill: #6b5860; font-family: system-ui, sans-serif; }
.note { font-size: 11px; fill: #463a3f; font-family: system-ui, sans-serif; }
.gap { font-size: 10px; fill: #6b5860; font-family: system-ui, sans-serif; }
.caption { font-size: .8rem; color: #6b5860; margin: .5rem 0 0; }
table { border-collapse: collapse; width: 100%%; font-size: .9rem; }
th, td { text-align: left; padding: .45rem .6rem; border-bottom: 1px solid #f3cdd9; }
th { color: #6b5860; font-weight: normal; font-size: .78rem;
text-transform: uppercase; letter-spacing: .06em; }
.integrity { background: #fff2f6; border-left: 3px solid #f3b6c8;
padding: 1rem 1.25rem; border-radius: .4rem; }
.integrity.broken { border-left-color: #c2410c; }
.integrity p { margin: 0; font-size: .92rem; }
.limits { font-size: .88rem; color: #6b5860; border-top: 1px solid #f3cdd9;
padding-top: 1.25rem; }
.limits strong { color: #463a3f; }
.empty { color: #6b5860; }
@media print {
body { margin: 0; max-width: none; }
section, .chart svg, table { break-inside: avoid; }
}
</style>
</head>
<body>
`
internal/docs/passport_test.go
+376
View File
@@ -0,0 +1,376 @@
package docs
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"regexp"
"strconv"
"strings"
"testing"
"time"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// chained builds a valid hash-chained snapshot run from (minutes-offset, words)
// pairs, so tests can describe a writing history in terms a reader recognises
// and get correct hashes for free.
func chained(docID string, base time.Time, points ...[2]int) []db.DocumentVersion {
var (
out []db.DocumentVersion
prev string
)
for i, p := range points {
at := base.Add(time.Duration(p[0]) * time.Minute)
text := strings.Repeat("word ", p[1])
v := db.DocumentVersion{
ID: fmt.Sprintf("v%d", i),
DocID: docID,
ContentText: text,
WordCount: p[1],
Kind: db.VersionKindAuto,
CreatedAt: at,
PrevHash: prev,
}
v.ContentHash = chainHash(prev, docID, at, p[1], text)
prev = v.ContentHash
out = append(out, v)
}
return out
}
func TestBuildPassportSessions(t *testing.T) {
base := time.Date(2026, 3, 2, 9, 0, 0, 0, time.UTC)
doc := db.Document{ID: "d1", Title: "Essay"}
// Two sittings: 09:00–09:30, then a three-hour break, then 12:30–13:00.
vs := chained("d1", base,
[2]int{0, 40}, [2]int{15, 120}, [2]int{30, 210},
[2]int{210, 260}, [2]int{240, 330},
)
d := buildPassport(doc, vs)
if len(d.Sessions) != 2 {
t.Fatalf("sessions = %d, want 2", len(d.Sessions))
}
if got := d.Sessions[0].Duration(); got != 30*time.Minute {
t.Errorf("session 1 duration = %v, want 30m", got)
}
if got := d.Sessions[1].Snapshots; got != 2 {
t.Errorf("session 2 snapshots = %d, want 2", got)
}
if got := d.Span; got != 4*time.Hour {
t.Errorf("span = %v, want 4h", got)
}
// Active time counts only time inside sessions, never the break.
if got := d.ActiveTime; got != 60*time.Minute {
t.Errorf("active time = %v, want 60m", got)
}
// Session 2 measures from session 1's final count (210 → 330).
if got := d.Sessions[1].WordsAdded; got != 120 {
t.Errorf("session 2 words = %d, want 120", got)
}
if d.ChainStatus != chainVerified {
t.Errorf("chain = %q, want %q", d.ChainStatus, chainVerified)
}
}
func TestBuildPassportFlagsLargeJump(t *testing.T) {
base := time.Date(2026, 3, 2, 9, 0, 0, 0, time.UTC)
doc := db.Document{ID: "d1"}
t.Run("steady growth is not flagged", func(t *testing.T) {
vs := chained("d1", base, [2]int{0, 100}, [2]int{5, 200}, [2]int{10, 300}, [2]int{15, 400})
if d := buildPassport(doc, vs); d.NoteJump {
t.Errorf("even growth flagged a jump of %d", d.LargestJump)
}
})
t.Run("a paste-shaped jump is flagged", func(t *testing.T) {
vs := chained("d1", base, [2]int{0, 20}, [2]int{5, 40}, [2]int{10, 900})
d := buildPassport(doc, vs)
if !d.NoteJump {
t.Fatal("large jump not flagged")
}
if d.LargestJump != 860 {
t.Errorf("largest jump = %d, want 860", d.LargestJump)
}
if !d.LargestJumpAt.Equal(base.Add(10 * time.Minute)) {
t.Errorf("jump at %v, want +10m", d.LargestJumpAt)
}
})
}
func TestVerifyChain(t *testing.T) {
base := time.Date(2026, 3, 2, 9, 0, 0, 0, time.UTC)
good := func() []db.DocumentVersion {
return chained("d1", base, [2]int{0, 50}, [2]int{5, 90}, [2]int{10, 160})
}
t.Run("intact chain verifies", func(t *testing.T) {
got, _ := verifyChain(db.Document{ID: "d1"}, good())
if got != chainVerified {
t.Errorf("got %q, want %q", got, chainVerified)
}
})
t.Run("edited content breaks it", func(t *testing.T) {
vs := good()
vs[1].ContentText = "something else entirely"
if got, _ := verifyChain(db.Document{ID: "d1"}, vs); got != chainBroken {
t.Errorf("got %q, want %q", got, chainBroken)
}
})
t.Run("backdating breaks it", func(t *testing.T) {
vs := good()
vs[2].CreatedAt = base.Add(-time.Hour)
if got, _ := verifyChain(db.Document{ID: "d1"}, vs); got != chainBroken {
t.Errorf("got %q, want %q", got, chainBroken)
}
})
t.Run("a removed snapshot reads as a gap when pruning is allowed", func(t *testing.T) {
vs := good()
pruned := []db.DocumentVersion{vs[0], vs[2]} // middle snapshot gone
got, _ := verifyChain(db.Document{ID: "d1"}, pruned)
if got != chainGaps {
t.Errorf("got %q, want %q", got, chainGaps)
}
})
t.Run("a removed snapshot is tampering when history is preserved", func(t *testing.T) {
vs := good()
pruned := []db.DocumentVersion{vs[0], vs[2]}
got, _ := verifyChain(db.Document{ID: "d1", PreserveHistory: true}, pruned)
if got != chainBroken {
t.Errorf("got %q, want %q", got, chainBroken)
}
})
t.Run("pre-chain snapshots are partial, not failures", func(t *testing.T) {
vs := good()
vs[0].ContentHash, vs[0].PrevHash = "", ""
got, unhashed := verifyChain(db.Document{ID: "d1"}, vs)
if got != chainPartial {
t.Errorf("got %q, want %q", got, chainPartial)
}
if unhashed != 1 {
t.Errorf("unhashed = %d, want 1", unhashed)
}
})
t.Run("no hashes at all is unverifiable", func(t *testing.T) {
vs := good()
for i := range vs {
vs[i].ContentHash, vs[i].PrevHash = "", ""
}
if got, _ := verifyChain(db.Document{ID: "d1"}, vs); got != chainUnverifiable {
t.Errorf("got %q, want %q", got, chainUnverifiable)
}
})
}
// The report must survive the degenerate histories — one snapshot, an empty
// document — rather than dividing by a zero span or a zero maximum.
func TestRenderPassportEdgeCases(t *testing.T) {
base := time.Date(2026, 3, 2, 9, 0, 0, 0, time.UTC)
t.Run("no history", func(t *testing.T) {
out := string(renderPassport(buildPassport(db.Document{Title: "Empty"}, nil)))
if !strings.Contains(out, "no saved history") {
t.Errorf("missing empty-state copy:\n%s", out)
}
})
t.Run("single snapshot", func(t *testing.T) {
vs := chained("d1", base, [2]int{0, 12})
out := string(renderPassport(buildPassport(db.Document{ID: "d1", Title: "One"}, vs)))
if strings.Contains(out, "NaN") || strings.Contains(out, "+Inf") {
t.Errorf("degenerate geometry leaked into output:\n%s", out)
}
})
t.Run("empty document", func(t *testing.T) {
vs := chained("d1", base, [2]int{0, 0}, [2]int{5, 0})
out := string(renderPassport(buildPassport(db.Document{ID: "d1"}, vs)))
if strings.Contains(out, "NaN") {
t.Errorf("zero word count produced NaN:\n%s", out)
}
})
t.Run("title is escaped", func(t *testing.T) {
doc := db.Document{ID: "d1", Title: `<script>alert(1)</script>`}
out := string(renderPassport(buildPassport(doc, chained("d1", base, [2]int{0, 5}))))
if strings.Contains(out, "<script>") {
t.Error("title was not escaped")
}
})
}
// The chart must give its width to the writing, not to the gaps between
// sittings. Three short sessions spread over three days is the case that a
// wall-clock x axis renders as three vertical cliffs — visually identical to
// pasted text, and wrong.
func TestChartGivesWidthToWriting(t *testing.T) {
base := time.Date(2026, 3, 2, 9, 0, 0, 0, time.UTC)
// ~30 minutes of work on each of three consecutive days.
var pts [][2]int
words := 0
for day := 0; day < 3; day++ {
for k := 0; k < 6; k++ {
words += 50
pts = append(pts, [2]int{day*1440 + k*5, words})
}
}
d := buildPassport(db.Document{ID: "d1"}, chained("d1", base, pts...))
if len(d.Sessions) != 3 {
t.Fatalf("sessions = %d, want 3", len(d.Sessions))
}
out := renderChart(d)
// Every session band should be a substantial share of the plot, not a sliver.
widths := regexp.MustCompile(`<rect [^>]*width="([0-9.]+)"`).FindAllStringSubmatch(out, -1)
if len(widths) != 3 {
t.Fatalf("session bands = %d, want 3", len(widths))
}
for i, m := range widths {
w, err := strconv.ParseFloat(m[1], 64)
if err != nil {
t.Fatal(err)
}
if w < plotW*0.15 {
t.Errorf("session %d band is %.1fpx of %dpx plot — writing got crushed", i+1, w, plotW)
}
}
// The compressed breaks must be stated, not silently removed.
if got := strings.Count(out, `class="gap"`); got != 2 {
t.Errorf("break labels = %d, want 2", got)
}
if !strings.Contains(out, "away") {
t.Error("break labels do not name their duration")
}
}
// Chart coordinates must stay inside the viewBox whatever the history looks like.
func TestChartStaysInBounds(t *testing.T) {
base := time.Date(2026, 3, 2, 9, 0, 0, 0, time.UTC)
histories := map[string][][2]int{
"single snapshot": {{0, 30}},
"two sessions": {{0, 30}, {5, 90}, {600, 140}, {605, 210}},
"words removed": {{0, 400}, {5, 380}, {10, 120}},
"all zero": {{0, 0}, {5, 0}},
"many snapshots": func() (p [][2]int) {
for i := 0; i < 60; i++ {
p = append(p, [2]int{i * 4, i * 20})
}
return
}(),
}
num := regexp.MustCompile(`(?:x|cx)="([0-9.-]+)"`)
for name, pts := range histories {
t.Run(name, func(t *testing.T) {
out := renderChart(buildPassport(db.Document{ID: "d1"}, chained("d1", base, pts...)))
if strings.Contains(out, "NaN") || strings.Contains(out, "Inf") {
t.Fatalf("degenerate geometry:\n%s", out)
}
for _, m := range num.FindAllStringSubmatch(out, -1) {
v, err := strconv.ParseFloat(m[1], 64)
if err != nil {
t.Fatal(err)
}
if v < 0 || v > chartW {
t.Errorf("x coordinate %.1f outside 0..%d", v, chartW)
}
}
})
}
}
func TestHumanDuration(t *testing.T) {
cases := []struct {
in time.Duration
want string
}{
{0, "under a minute"},
{30 * time.Second, "under a minute"},
{18 * time.Minute, "18m"},
{2 * time.Hour, "2h"},
{2*time.Hour + 40*time.Minute, "2h 40m"},
{50 * time.Hour, "2d 2h"},
}
for _, c := range cases {
if got := humanDuration(c.in); got != c.want {
t.Errorf("humanDuration(%v) = %q, want %q", c.in, got, c.want)
}
}
}
// --- endpoint / persistence -------------------------------------------------
func TestPassportEndpoint(t *testing.T) {
srv := newTestServer(t)
id := newDoc(t, srv)
do(t, srv, http.MethodPut, "/"+id,
`{"content":"{}","content_text":"the first draft","word_count":3}`)
rec := do(t, srv, http.MethodGet, "/"+id+"/passport", "")
if rec.Code != http.StatusOK {
t.Fatalf("passport: code=%d body=%s", rec.Code, rec.Body)
}
if ct := rec.Header().Get("Content-Type"); !strings.HasPrefix(ct, "text/html") {
t.Errorf("content-type = %q, want text/html", ct)
}
body := rec.Body.String()
if !strings.Contains(body, "Writing passport") {
t.Errorf("report body missing heading:\n%s", body)
}
// Snapshots written through the real insert path must verify.
if !strings.Contains(body, "History intact") {
t.Errorf("live-written history did not verify:\n%s", body)
}
if rec := do(t, srv, http.MethodGet, "/does-not-exist/passport", ""); rec.Code != http.StatusNotFound {
t.Errorf("missing doc: code = %d, want 404", rec.Code)
}
}
func TestPreserveHistoryExemptsFromPruning(t *testing.T) {
srv := newTestServer(t)
id := newDoc(t, srv)
rec := do(t, srv, http.MethodPut, "/"+id, `{"preserve_history":true}`)
if rec.Code != http.StatusOK {
t.Fatalf("set preserve_history: code=%d body=%s", rec.Code, rec.Body)
}
decodeDoc := func(rec *httptest.ResponseRecorder) db.Document {
t.Helper()
var doc db.Document
if err := json.Unmarshal(rec.Body.Bytes(), &doc); err != nil {
t.Fatalf("decode doc: %v", err)
}
return doc
}
if !decodeDoc(rec).PreserveHistory {
t.Fatal("preserve_history did not persist")
}
// An ordinary body save must not clear the flag.
rec = do(t, srv, http.MethodPut, "/"+id,
`{"content":"{}","content_text":"hello there","word_count":2}`)
if !decodeDoc(rec).PreserveHistory {
t.Error("a normal save cleared preserve_history")
}
}
internal/docs/search.go
+6 -4
View File
@@ -7,6 +7,7 @@ import (
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/httputil"
)
@@ -50,12 +51,13 @@ func (h *Handler) SearchRoutes() chi.Router {
return r
}
// search runs a cross-document full-text search for the local user. Queries of
// search runs a cross-document full-text search for the caller. Queries of
// three or more runes use the trigram FTS index (fast, ranked); shorter queries
// fall back to a LIKE scan so 2-character Chinese words still resolve. Either way
// the snippet is built in Go from the original text, for clean word boundaries
// and a uniform highlight format.
func (h *Handler) search(w http.ResponseWriter, r *http.Request) {
userID := auth.UserID(r.Context())
q := strings.TrimSpace(r.URL.Query().Get("q"))
if q == "" {
httputil.WriteJSON(w, http.StatusOK, []searchResult{})
@@ -80,7 +82,7 @@ func (h *Handler) search(w http.ResponseWriter, r *http.Request) {
WHERE documents_fts MATCH ? AND d.user_id = ?
ORDER BY rank
LIMIT ?`,
phrase, db.LocalUserID, maxSearchResults,
phrase, userID, maxSearchResults,
)
if err != nil {
httputil.ServerError(w, err)
@@ -110,7 +112,7 @@ func (h *Handler) search(w http.ResponseWriter, r *http.Request) {
AND (title LIKE ? ESCAPE '\' OR content_text LIKE ? ESCAPE '\')
ORDER BY updated_at DESC
LIMIT ?`,
db.LocalUserID, like, like, maxSearchResults,
userID, like, like, maxSearchResults,
)
if err != nil {
httputil.ServerError(w, err)
@@ -144,7 +146,7 @@ func (h *Handler) search(w http.ResponseWriter, r *http.Request) {
ids = append(ids, rw.id)
}
byDoc, err := h.tagsByDoc(ids)
byDoc, err := h.tagsByDoc(userID, ids)
if err != nil {
httputil.ServerError(w, err)
return
internal/docs/tags.go
+20 -17
View File
@@ -7,6 +7,7 @@ import (
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/httputil"
)
@@ -55,7 +56,7 @@ func (h *Handler) listTags(w http.ResponseWriter, r *http.Request) {
WHERE t.user_id = ?
GROUP BY t.id
ORDER BY t.name COLLATE NOCASE`,
db.LocalUserID,
auth.UserID(r.Context()),
)
if err != nil {
httputil.ServerError(w, err)
@@ -104,7 +105,7 @@ func (h *Handler) createTag(w http.ResponseWriter, r *http.Request) {
`INSERT INTO tags (user_id, name, color) VALUES (?, ?, ?)
ON CONFLICT(user_id, name) DO UPDATE SET name = excluded.name
RETURNING id, name, color`,
db.LocalUserID, name, normalizeColor(req.Color),
auth.UserID(r.Context()), name, normalizeColor(req.Color),
).Scan(&t.ID, &t.Name, &t.Color)
if err != nil {
httputil.ServerError(w, err)
@@ -117,6 +118,7 @@ func (h *Handler) createTag(w http.ResponseWriter, r *http.Request) {
// a recolor needn't resend the name.
func (h *Handler) updateTag(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
userID := auth.UserID(r.Context())
var req struct {
Name *string `json:"name"`
@@ -145,7 +147,7 @@ func (h *Handler) updateTag(w http.ResponseWriter, r *http.Request) {
SET name = COALESCE(?, name),
color = COALESCE(?, color)
WHERE id = ? AND user_id = ?`,
namePtr, colorPtr, id, db.LocalUserID,
namePtr, colorPtr, id, userID,
)
if err != nil {
httputil.ServerError(w, err)
@@ -159,7 +161,7 @@ func (h *Handler) updateTag(w http.ResponseWriter, r *http.Request) {
var t db.Tag
if err := h.DB.QueryRow(
`SELECT id, name, color FROM tags WHERE id = ? AND user_id = ?`,
id, db.LocalUserID,
id, userID,
).Scan(&t.ID, &t.Name, &t.Color); err != nil {
httputil.ServerError(w, err)
return
@@ -171,7 +173,7 @@ func (h *Handler) updateTag(w http.ResponseWriter, r *http.Request) {
func (h *Handler) deleteTag(w http.ResponseWriter, r *http.Request) {
res, err := h.DB.Exec(
`DELETE FROM tags WHERE id = ? AND user_id = ?`,
chi.URLParam(r, "id"), db.LocalUserID,
chi.URLParam(r, "id"), auth.UserID(r.Context()),
)
if err != nil {
httputil.ServerError(w, err)
@@ -184,10 +186,11 @@ func (h *Handler) deleteTag(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNoContent)
}
// assignTag attaches a tag to a document. Both must belong to the local user;
// assignTag attaches a tag to a document. Both must belong to the caller;
// the assignment is idempotent (re-assigning is a no-op, not an error).
func (h *Handler) assignTag(w http.ResponseWriter, r *http.Request) {
docID := chi.URLParam(r, "id")
userID := auth.UserID(r.Context())
var req struct {
TagID string `json:"tag_id"`
@@ -203,11 +206,11 @@ func (h *Handler) assignTag(w http.ResponseWriter, r *http.Request) {
// Verify both the doc and the tag belong to the user before linking, so a
// stray id can't cross-link another account's rows.
if !h.ownsDoc(docID) {
if !h.ownsDoc(userID, docID) {
notFound(w)
return
}
if !h.ownsTag(req.TagID) {
if !h.ownsTag(userID, req.TagID) {
notFoundMsg(w, "tag not found")
return
}
@@ -228,7 +231,7 @@ func (h *Handler) unassignTag(w http.ResponseWriter, r *http.Request) {
docID := chi.URLParam(r, "id")
tagID := chi.URLParam(r, "tagId")
if !h.ownsDoc(docID) {
if !h.ownsDoc(auth.UserID(r.Context()), docID) {
notFound(w)
return
}
@@ -242,22 +245,22 @@ func (h *Handler) unassignTag(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNoContent)
}
// ownsDoc reports whether a document belongs to the local user.
func (h *Handler) ownsDoc(docID string) bool {
// ownsDoc reports whether a document belongs to the given user.
func (h *Handler) ownsDoc(userID, docID string) bool {
var exists bool
_ = h.DB.QueryRow(
`SELECT EXISTS(SELECT 1 FROM documents WHERE id = ? AND user_id = ?)`,
docID, db.LocalUserID,
docID, userID,
).Scan(&exists)
return exists
}
// ownsTag reports whether a tag belongs to the local user.
func (h *Handler) ownsTag(tagID string) bool {
// ownsTag reports whether a tag belongs to the given user.
func (h *Handler) ownsTag(userID, tagID string) bool {
var exists bool
_ = h.DB.QueryRow(
`SELECT EXISTS(SELECT 1 FROM tags WHERE id = ? AND user_id = ?)`,
tagID, db.LocalUserID,
tagID, userID,
).Scan(&exists)
return exists
}
@@ -265,7 +268,7 @@ func (h *Handler) ownsTag(tagID string) bool {
// tagsByDoc loads the tags for a set of documents in one query and groups them
// by doc id. Used to decorate the document list and search results without an
// N+1 of per-doc queries. Returns an empty (non-nil) map when ids is empty.
func (h *Handler) tagsByDoc(ids []string) (map[string][]db.Tag, error) {
func (h *Handler) tagsByDoc(userID string, ids []string) (map[string][]db.Tag, error) {
out := map[string][]db.Tag{}
if len(ids) == 0 {
return out, nil
@@ -277,7 +280,7 @@ func (h *Handler) tagsByDoc(ids []string) (map[string][]db.Tag, error) {
for _, id := range ids {
args = append(args, id)
}
args = append(args, db.LocalUserID)
args = append(args, userID)
rows, err := h.DB.Query(
`SELECT dt.doc_id, t.id, t.name, t.color
internal/docs/tags_test.go
+1 -1
View File
@@ -27,7 +27,7 @@ func newFullServer(t *testing.T) http.Handler {
r.Mount("/docs", h.Routes())
r.Mount("/tags", h.TagRoutes())
r.Mount("/search", h.SearchRoutes())
return r
return withAuth(r)
}
// createDoc makes a document with the given title/body and returns its id.
internal/docs/versions.go
+71 -15
View File
@@ -8,6 +8,7 @@ import (
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/httputil"
)
@@ -33,6 +34,7 @@ func (h *Handler) versionRoutes(r chi.Router) {
r.Post("/{id}/versions", h.createVersion) // explicit "save a restore point"
r.Get("/{id}/versions/{vid}", h.getVersion) // full body for preview
r.Post("/{id}/versions/{vid}/restore", h.restoreVersion)
r.Get("/{id}/passport", h.passport) // authorship report over that history
}
// listVersions returns the document's snapshots, newest first, without the heavy
@@ -48,7 +50,7 @@ func (h *Handler) listVersions(w http.ResponseWriter, r *http.Request) {
JOIN documents d ON d.id = v.doc_id
WHERE v.doc_id = ? AND d.user_id = ?
ORDER BY v.created_at DESC`,
docID, db.LocalUserID,
docID, auth.UserID(r.Context()),
)
if err != nil {
httputil.ServerError(w, err)
@@ -74,7 +76,7 @@ func (h *Handler) listVersions(w http.ResponseWriter, r *http.Request) {
// getVersion returns one snapshot in full (including content) for preview.
func (h *Handler) getVersion(w http.ResponseWriter, r *http.Request) {
v, err := h.fetchVersion(chi.URLParam(r, "id"), chi.URLParam(r, "vid"))
v, err := h.fetchVersion(auth.UserID(r.Context()), chi.URLParam(r, "id"), chi.URLParam(r, "vid"))
if errors.Is(err, sql.ErrNoRows) {
notFoundMsg(w, "version not found")
return
@@ -91,7 +93,7 @@ func (h *Handler) getVersion(w http.ResponseWriter, r *http.Request) {
func (h *Handler) createVersion(w http.ResponseWriter, r *http.Request) {
docID := chi.URLParam(r, "id")
doc, err := h.fetch(docID)
doc, err := h.fetch(auth.UserID(r.Context()), docID)
if errors.Is(err, sql.ErrNoRows) {
notFound(w)
return
@@ -115,8 +117,9 @@ func (h *Handler) createVersion(w http.ResponseWriter, r *http.Request) {
func (h *Handler) restoreVersion(w http.ResponseWriter, r *http.Request) {
docID := chi.URLParam(r, "id")
vid := chi.URLParam(r, "vid")
userID := auth.UserID(r.Context())
v, err := h.fetchVersion(docID, vid)
v, err := h.fetchVersion(userID, docID, vid)
if errors.Is(err, sql.ErrNoRows) {
notFoundMsg(w, "version not found")
return
@@ -126,7 +129,7 @@ func (h *Handler) restoreVersion(w http.ResponseWriter, r *http.Request) {
return
}
current, err := h.fetch(docID)
current, err := h.fetch(userID, docID)
if err != nil {
httputil.ServerError(w, err)
return
@@ -141,7 +144,7 @@ func (h *Handler) restoreVersion(w http.ResponseWriter, r *http.Request) {
SET title = ?, content = ?, content_text = ?, word_count = ?,
updated_at = CURRENT_TIMESTAMP
WHERE id = ? AND user_id = ?`,
v.Title, v.Content, v.ContentText, v.WordCount, docID, db.LocalUserID,
v.Title, v.Content, v.ContentText, v.WordCount, docID, userID,
)
if err != nil {
httputil.ServerError(w, err)
@@ -152,7 +155,7 @@ func (h *Handler) restoreVersion(w http.ResponseWriter, r *http.Request) {
return
}
doc, err := h.fetch(docID)
doc, err := h.fetch(userID, docID)
if err != nil {
httputil.ServerError(w, err)
return
@@ -202,20 +205,73 @@ func (h *Handler) maybeAutoSnapshot(doc db.Document) error {
// insertVersion writes a snapshot row of the given kind and returns it (without
// the heavy content fields, matching the list shape).
//
// The row is linked into the document's hash chain: it carries the previous
// snapshot's hash, and its own hash covers that link plus its content. The hash
// can only be computed once the database has assigned created_at, so the insert
// and the hash write share a transaction — a snapshot is never visible with a
// hash that doesn't cover its own timestamp.
func (h *Handler) insertVersion(doc db.Document, kind string) (db.DocumentVersion, error) {
tx, err := h.DB.Begin()
if err != nil {
return db.DocumentVersion{}, err
}
defer tx.Rollback() //nolint:errcheck // no-op once committed
// Chain onto the newest existing snapshot. created_at has second
// granularity, so rowid breaks ties in true insertion order; verification
// walks the same ordering in reverse.
var prevHash string
err = tx.QueryRow(
`SELECT content_hash FROM document_versions
WHERE doc_id = ? ORDER BY created_at DESC, rowid DESC LIMIT 1`,
doc.ID,
).Scan(&prevHash)
if err != nil && !errors.Is(err, sql.ErrNoRows) {
return db.DocumentVersion{}, err
}
var v db.DocumentVersion
err := h.DB.QueryRow(
`INSERT INTO document_versions (doc_id, title, content, content_text, word_count, kind)
VALUES (?, ?, ?, ?, ?, ?)
if err := tx.QueryRow(
`INSERT INTO document_versions (doc_id, title, content, content_text, word_count, kind, prev_hash)
VALUES (?, ?, ?, ?, ?, ?, ?)
RETURNING id, doc_id, title, word_count, kind, created_at`,
doc.ID, doc.Title, doc.Content, doc.ContentText, doc.WordCount, kind,
).Scan(&v.ID, &v.DocID, &v.Title, &v.WordCount, &v.Kind, &v.CreatedAt)
return v, err
doc.ID, doc.Title, doc.Content, doc.ContentText, doc.WordCount, kind, prevHash,
).Scan(&v.ID, &v.DocID, &v.Title, &v.WordCount, &v.Kind, &v.CreatedAt); err != nil {
return db.DocumentVersion{}, err
}
v.PrevHash = prevHash
v.ContentHash = chainHash(prevHash, v.DocID, v.CreatedAt, v.WordCount, doc.ContentText)
if _, err := tx.Exec(
`UPDATE document_versions SET content_hash = ? WHERE id = ?`, v.ContentHash, v.ID,
); err != nil {
return db.DocumentVersion{}, err
}
if err := tx.Commit(); err != nil {
return db.DocumentVersion{}, err
}
return v, nil
}
// pruneAutoVersions trims a document's 'auto' snapshots to the newest
// maxAutoVersions, leaving 'manual' and 'pre_restore' restore points intact.
//
// Documents flagged preserve_history are exempt entirely: their history is
// authorship evidence, and evidence with the oldest entries dropped is exactly
// the part a reader would want — the early, sparse, figuring-it-out edits that
// distinguish writing from pasting.
func (h *Handler) pruneAutoVersions(docID string) error {
var preserve bool
if err := h.DB.QueryRow(
`SELECT preserve_history FROM documents WHERE id = ?`, docID,
).Scan(&preserve); err != nil {
return err
}
if preserve {
return nil
}
_, err := h.DB.Exec(
`DELETE FROM document_versions
WHERE doc_id = ? AND kind = 'auto'
@@ -230,14 +286,14 @@ func (h *Handler) pruneAutoVersions(docID string) error {
}
// fetchVersion loads one full snapshot, scoped to its owner via the parent doc.
func (h *Handler) fetchVersion(docID, vid string) (db.DocumentVersion, error) {
func (h *Handler) fetchVersion(userID, docID, vid string) (db.DocumentVersion, error) {
var v db.DocumentVersion
err := h.DB.QueryRow(
`SELECT v.id, v.doc_id, v.title, v.content, v.content_text, v.word_count, v.kind, v.created_at
FROM document_versions v
JOIN documents d ON d.id = v.doc_id
WHERE v.id = ? AND v.doc_id = ? AND d.user_id = ?`,
vid, docID, db.LocalUserID,
vid, docID, userID,
).Scan(
&v.ID, &v.DocID, &v.Title, &v.Content, &v.ContentText,
&v.WordCount, &v.Kind, &v.CreatedAt,
internal/images/handler.go
+174 -13
View File
@@ -2,20 +2,35 @@
// images from the editor, they're saved to disk under the configured directory,
// and served back by hashed filename. Content addressing means the same image
// pasted twice is stored once, and URLs are stable and cacheable forever.
//
// Each stored file also has one row per owner in the `images` table, and a fetch
// joins on the caller. Before that, the store was a flat directory with no
// database presence at all: any authenticated user holding a sha256 could fetch
// anyone else's image. Hashes aren't guessable, so it was never an emergency —
// but "unguessable filename" is not access control, and images pasted into a
// private journal are exactly the content that shouldn't depend on it.
//
// One row per owner (rather than one owner per file) is what keeps deduplication:
// the same picture uploaded by two people is stored once and simply has two rows.
// The file is removed only with its last row.
package images
import (
"crypto/sha256"
"database/sql"
"encoding/hex"
"encoding/json"
"errors"
"io"
"log"
"net/http"
"os"
"path/filepath"
"strings"
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
)
// maxUploadBytes caps a single image at 10 MiB — generous for a writing tool,
@@ -32,25 +47,96 @@ var extByContentType = map[string]string{
"image/svg+xml": ".svg",
}
// Handler serves the upload + fetch endpoints, backed by a directory on disk.
// Handler serves the upload + fetch endpoints, backed by a directory on disk and
// an ownership table.
type Handler struct {
dir string
db *sql.DB
}
// New constructs a Handler, ensuring the storage directory exists.
func New(dir string) (*Handler, error) {
// New constructs a Handler, ensuring the storage directory exists and that every
// file already in it has an owner.
func New(dir string, database *sql.DB, backfillOwner string) (*Handler, error) {
if err := os.MkdirAll(dir, 0o755); err != nil {
return nil, err
}
return &Handler{dir: dir}, nil
h := &Handler{dir: dir, db: database}
if err := h.backfill(backfillOwner); err != nil {
return nil, err
}
return h, nil
}
// backfill claims pre-existing files for one user. Images uploaded before
// ownership existed have no row, and a row is now what makes them fetchable —
// so without this every picture already pasted into a document would 404.
// Attributing them to the account that has been the only one until now is the
// only answer the data supports. Idempotent: files that already have an owner
// are left alone.
func (h *Handler) backfill(owner string) error {
if owner == "" {
return nil
}
// The owner may not exist — after the `local` account has been migrated onto
// a real one, it doesn't. Claiming for a missing user would violate the
// foreign key, and this runs during startup, so the error would take the
// whole app down. There is nothing left to claim in that case anyway: the
// migration moves the image rows along with everything else.
var ownerExists bool
if err := h.db.QueryRow(
`SELECT EXISTS(SELECT 1 FROM users WHERE id = ?)`, owner,
).Scan(&ownerExists); err != nil {
return err
}
if !ownerExists {
return nil
}
entries, err := os.ReadDir(h.dir)
if err != nil {
return err
}
claimed := 0
for _, e := range entries {
if e.IsDir() {
continue
}
var exists bool
if err := h.db.QueryRow(
`SELECT EXISTS(SELECT 1 FROM images WHERE name = ?)`, e.Name(),
).Scan(&exists); err != nil {
return err
}
if exists {
continue
}
var size int64
if info, err := e.Info(); err == nil {
size = info.Size()
}
if _, err := h.db.Exec(
`INSERT INTO images (name, user_id, content_type, size) VALUES (?, ?, '', ?)
ON CONFLICT DO NOTHING`,
e.Name(), owner, size,
); err != nil {
return err
}
claimed++
}
if claimed > 0 {
log.Printf("images: claimed %d pre-existing image(s) for %s", claimed, owner)
}
return nil
}
// Routes mounts the image endpoints. Mount under "/images" so the full paths are
// POST /api/images (upload) and GET /api/images/{name} (fetch).
// POST /api/images (upload), GET /api/images/{name} (fetch) and
// DELETE /api/images/{name} (drop your copy).
func (h *Handler) Routes() chi.Router {
r := chi.NewRouter()
r.Post("/", h.upload)
r.Get("/{name}", h.serve)
r.Delete("/{name}", h.remove)
return r
}
@@ -79,7 +165,7 @@ func (h *Handler) upload(w http.ResponseWriter, r *http.Request) {
ext, ok := extByContentType[ct]
if !ok {
if looksLikeSVG(data) {
ext, ok = ".svg", true
ct, ext, ok = "image/svg+xml", ".svg", true
}
}
if !ok {
@@ -99,28 +185,103 @@ func (h *Handler) upload(w http.ResponseWriter, r *http.Request) {
}
}
// Record the caller as an owner. Re-uploading your own image is a no-op;
// uploading someone else's identical image adds a second row over one file.
if _, err := h.db.Exec(
`INSERT INTO images (name, user_id, content_type, size) VALUES (?, ?, ?, ?)
ON CONFLICT (name, user_id) DO NOTHING`,
name, auth.UserID(r.Context()), ct, len(data),
); err != nil {
log.Printf("images: could not record ownership of %s: %v", name, err)
http.Error(w, "could not store image", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]string{"url": "/api/images/" + name})
}
// serve returns a stored image by its hashed filename. The filename is validated
// to be a bare name (no path separators) so it can't escape the storage dir, and
// served with a long-lived cache header since content-addressed URLs never change.
// serve returns a stored image by its hashed filename, but only to someone who
// owns it. The filename is validated to be a bare name (no path separators) so
// it can't escape the storage dir, and served with a long-lived cache header
// since content-addressed URLs never change.
//
// Someone else's image is a 404, not a 403: whether a hash exists is itself
// information the caller has no business learning.
func (h *Handler) serve(w http.ResponseWriter, r *http.Request) {
name := chi.URLParam(r, "name")
if name == "" || name != filepath.Base(name) || strings.ContainsAny(name, `/\`) {
name, ok := safeName(chi.URLParam(r, "name"))
if !ok || !h.owns(name, auth.UserID(r.Context())) {
http.NotFound(w, r)
return
}
path := filepath.Join(h.dir, filepath.Base(name))
path := filepath.Join(h.dir, name)
if _, err := os.Stat(path); err != nil {
http.NotFound(w, r)
return
}
w.Header().Set("Cache-Control", "public, max-age=31536000, immutable")
// Private: a shared cache must never hand one writer's image to another.
w.Header().Set("Cache-Control", "private, max-age=31536000, immutable")
http.ServeFile(w, r, path)
}
// remove drops the caller's claim on an image, and deletes the file itself once
// nobody is left holding it.
func (h *Handler) remove(w http.ResponseWriter, r *http.Request) {
name, ok := safeName(chi.URLParam(r, "name"))
if !ok {
http.NotFound(w, r)
return
}
res, err := h.db.Exec(`DELETE FROM images WHERE name = ? AND user_id = ?`,
name, auth.UserID(r.Context()))
if err != nil {
http.Error(w, "could not delete image", http.StatusInternalServerError)
return
}
if n, _ := res.RowsAffected(); n == 0 {
http.NotFound(w, r)
return
}
var others bool
if err := h.db.QueryRow(
`SELECT EXISTS(SELECT 1 FROM images WHERE name = ?)`, name,
).Scan(&others); err != nil {
// The row is gone either way; leaving an orphaned file behind is a
// wasted block, not a correctness problem.
log.Printf("images: could not check remaining owners of %s: %v", name, err)
w.WriteHeader(http.StatusNoContent)
return
}
if !others {
if err := os.Remove(filepath.Join(h.dir, name)); err != nil && !errors.Is(err, os.ErrNotExist) {
log.Printf("images: could not remove %s: %v", name, err)
}
}
w.WriteHeader(http.StatusNoContent)
}
// owns reports whether userID has a claim on a stored image.
func (h *Handler) owns(name, userID string) bool {
var ok bool
if err := h.db.QueryRow(
`SELECT EXISTS(SELECT 1 FROM images WHERE name = ? AND user_id = ?)`, name, userID,
).Scan(&ok); err != nil {
log.Printf("images: ownership check failed for %s: %v", name, err)
return false
}
return ok
}
// safeName rejects anything that isn't a bare filename, so a request can't walk
// out of the storage directory.
func safeName(name string) (string, bool) {
if name == "" || name != filepath.Base(name) || strings.ContainsAny(name, `/\`) {
return "", false
}
return name, true
}
// looksLikeSVG does a cheap check for an <svg root tag near the start of the
// file, since DetectContentType doesn't recognize SVG.
func looksLikeSVG(data []byte) bool {
internal/images/handler_test.go
+176 -31
View File
@@ -6,8 +6,13 @@ import (
"mime/multipart"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// a 1x1 transparent PNG.
@@ -19,6 +24,39 @@ var pngBytes = []byte{
0x42, 0x60, 0x82,
}
// another 1x1 PNG, differing in one pixel byte, so it hashes elsewhere.
var otherPNG = append(append([]byte{}, pngBytes[:len(pngBytes)-8]...),
0x01, 0x00, 0x00, 0x00, 0x49, 0x45, 0x4e, 0x44)
// newStore returns a handler over a fresh directory and database, plus a router
// per user: identical but for who the auth middleware says is calling. Two users
// over one store is the situation that ownership exists to handle.
func newStore(t *testing.T) (dir string, alice, bob http.Handler) {
t.Helper()
dir = t.TempDir()
database, err := db.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatalf("open db: %v", err)
}
t.Cleanup(func() { database.Close() })
if _, err := database.Exec(
`INSERT INTO users (id, email, display_name) VALUES (?, ?, ?)`,
"bob", "bob@petal.local", "Bob",
); err != nil {
t.Fatalf("seed second user: %v", err)
}
h, err := New(dir, database.DB, db.LocalUserID)
if err != nil {
t.Fatalf("new store: %v", err)
}
mount := func(userID string) http.Handler {
return auth.Middleware(auth.StaticResolver(userID))(h.Routes())
}
return dir, mount(db.LocalUserID), mount("bob")
}
func uploadReq(t *testing.T, field string, data []byte) *http.Request {
t.Helper()
var buf bytes.Buffer
@@ -34,16 +72,11 @@ func uploadReq(t *testing.T, field string, data []byte) *http.Request {
return req
}
func TestUploadAndServe(t *testing.T) {
h, err := New(t.TempDir())
if err != nil {
t.Fatal(err)
}
r := h.Routes()
// Upload a PNG → expect a JSON url under /api/images/.
// upload posts an image and returns its stored name.
func upload(t *testing.T, h http.Handler, data []byte) string {
t.Helper()
rec := httptest.NewRecorder()
r.ServeHTTP(rec, uploadReq(t, "image", pngBytes))
h.ServeHTTP(rec, uploadReq(t, "image", data))
if rec.Code != http.StatusOK {
t.Fatalf("upload code=%d body=%s", rec.Code, rec.Body)
}
@@ -54,44 +87,156 @@ func TestUploadAndServe(t *testing.T) {
if !strings.HasPrefix(resp.URL, "/api/images/") || !strings.HasSuffix(resp.URL, ".png") {
t.Fatalf("unexpected url %q", resp.URL)
}
return strings.TrimPrefix(resp.URL, "/api/images/")
}
func get(t *testing.T, h http.Handler, name string) *httptest.ResponseRecorder {
t.Helper()
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/"+name, nil))
return rec
}
func TestUploadAndServe(t *testing.T) {
_, alice, _ := newStore(t)
name := upload(t, alice, pngBytes)
// The same content uploaded again dedupes to the same URL.
rec2 := httptest.NewRecorder()
r.ServeHTTP(rec2, uploadReq(t, "image", pngBytes))
var resp2 struct{ URL string }
json.Unmarshal(rec2.Body.Bytes(), &resp2)
if resp2.URL != resp.URL {
t.Fatalf("expected dedup to same url, got %q vs %q", resp2.URL, resp.URL)
if again := upload(t, alice, pngBytes); again != name {
t.Fatalf("expected dedup to same name, got %q vs %q", again, name)
}
// Fetch it back.
name := strings.TrimPrefix(resp.URL, "/api/images/")
rec3 := httptest.NewRecorder()
r.ServeHTTP(rec3, httptest.NewRequest(http.MethodGet, "/"+name, nil))
if rec3.Code != http.StatusOK {
t.Fatalf("serve code=%d", rec3.Code)
rec := get(t, alice, name)
if rec.Code != http.StatusOK {
t.Fatalf("serve code=%d", rec.Code)
}
if !bytes.Equal(rec3.Body.Bytes(), pngBytes) {
if !bytes.Equal(rec.Body.Bytes(), pngBytes) {
t.Fatal("served bytes differ from uploaded")
}
}
func TestUploadRejectsNonImage(t *testing.T) {
h, _ := New(t.TempDir())
r := h.Routes()
// The point of the ownership table: a hash is not a capability.
func TestImageIsolation(t *testing.T) {
_, alice, bob := newStore(t)
name := upload(t, alice, pngBytes)
if rec := get(t, bob, name); rec.Code != http.StatusNotFound {
t.Fatalf("bob fetched alice's image: code=%d", rec.Code)
}
// Nor can he delete it out from under her.
rec := httptest.NewRecorder()
r.ServeHTTP(rec, uploadReq(t, "image", []byte("this is plainly not an image at all")))
bob.ServeHTTP(rec, httptest.NewRequest(http.MethodDelete, "/"+name, nil))
if rec.Code != http.StatusNotFound {
t.Fatalf("bob deleted alice's image: code=%d", rec.Code)
}
if got := get(t, alice, name); got.Code != http.StatusOK {
t.Fatalf("alice's image disappeared: code=%d", got.Code)
}
}
// Deduplication has to survive ownership: one file, one row each.
func TestDedupAcrossUsers(t *testing.T) {
dir, alice, bob := newStore(t)
name := upload(t, alice, pngBytes)
if bobName := upload(t, bob, pngBytes); bobName != name {
t.Fatalf("expected the same stored name, got %q vs %q", bobName, name)
}
entries, _ := os.ReadDir(dir)
if len(entries) != 1 {
t.Fatalf("expected 1 file on disk, found %d", len(entries))
}
for _, h := range []http.Handler{alice, bob} {
if rec := get(t, h, name); rec.Code != http.StatusOK {
t.Fatalf("owner could not fetch shared image: code=%d", rec.Code)
}
}
// Alice dropping her copy must not take Bob's picture away with it.
rec := httptest.NewRecorder()
alice.ServeHTTP(rec, httptest.NewRequest(http.MethodDelete, "/"+name, nil))
if rec.Code != http.StatusNoContent {
t.Fatalf("delete code=%d", rec.Code)
}
if got := get(t, alice, name); got.Code != http.StatusNotFound {
t.Fatalf("alice still sees a deleted image: code=%d", got.Code)
}
if got := get(t, bob, name); got.Code != http.StatusOK {
t.Fatalf("bob lost his image when alice deleted hers: code=%d", got.Code)
}
if _, err := os.Stat(filepath.Join(dir, name)); err != nil {
t.Fatalf("file removed while still owned: %v", err)
}
// The last owner leaving takes the file with them.
rec2 := httptest.NewRecorder()
bob.ServeHTTP(rec2, httptest.NewRequest(http.MethodDelete, "/"+name, nil))
if rec2.Code != http.StatusNoContent {
t.Fatalf("delete code=%d", rec2.Code)
}
if _, err := os.Stat(filepath.Join(dir, name)); !os.IsNotExist(err) {
t.Fatalf("file survived its last owner: %v", err)
}
}
// Images that predate ownership must not vanish from documents that use them.
func TestBackfillClaimsExistingFiles(t *testing.T) {
dir := t.TempDir()
database, err := db.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatalf("open db: %v", err)
}
defer database.Close()
orphan := "deadbeefdeadbeefdeadbeefdeadbeef.png"
if err := os.WriteFile(filepath.Join(dir, orphan), pngBytes, 0o644); err != nil {
t.Fatal(err)
}
h, err := New(dir, database.DB, db.LocalUserID)
if err != nil {
t.Fatalf("new store: %v", err)
}
alice := auth.Middleware(auth.StaticResolver(db.LocalUserID))(h.Routes())
if rec := get(t, alice, orphan); rec.Code != http.StatusOK {
t.Fatalf("pre-existing image not claimed: code=%d", rec.Code)
}
// Re-running the backfill (i.e. a restart) must not double up or reassign.
if _, err := New(dir, database.DB, "bob"); err != nil {
t.Fatalf("second backfill: %v", err)
}
// And an owner who no longer exists — which is what the `local` account
// becomes once it has been migrated onto a real one — must be skipped, not
// turned into a foreign-key error that takes startup down with it.
if _, err := New(dir, database.DB, "nobody-at-all"); err != nil {
t.Fatalf("backfill for a missing owner should be a no-op, got: %v", err)
}
var owners int
if err := database.QueryRow(`SELECT COUNT(*) FROM images WHERE name = ?`, orphan).Scan(&owners); err != nil {
t.Fatal(err)
}
if owners != 1 {
t.Fatalf("expected the backfill to be idempotent, got %d owners", owners)
}
}
func TestUploadRejectsNonImage(t *testing.T) {
_, alice, _ := newStore(t)
rec := httptest.NewRecorder()
alice.ServeHTTP(rec, uploadReq(t, "image", []byte("this is plainly not an image at all")))
if rec.Code != http.StatusUnsupportedMediaType {
t.Fatalf("expected 415, got %d", rec.Code)
}
}
func TestServeMissing(t *testing.T) {
h, _ := New(t.TempDir())
r := h.Routes()
rec := httptest.NewRecorder()
r.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/deadbeef.png", nil))
if rec.Code != http.StatusNotFound {
_, alice, _ := newStore(t)
if rec := get(t, alice, "deadbeef.png"); rec.Code != http.StatusNotFound {
t.Fatalf("expected 404, got %d", rec.Code)
}
}
internal/lexicon/dreamdict.go
+333
View File
@@ -0,0 +1,333 @@
package lexicon
import (
"errors"
"fmt"
"io/fs"
"os"
"sort"
"strings"
"unicode/utf8"
"github.com/prosolis/dreamdict/dictionary"
)
// DreamDict is a read-only handle on a built dict.db — one SQLite file holding
// English, French, European Portuguese, Spanish and Mandarin.
//
// It is a second database beside petal.db and is never written to: the file is
// built by DreamDict's own import CLI a few times a year, and Petal only reads
// it. That is what makes importing the package the right shape rather than
// running DreamDict as a service — a hover gloss should not depend on a second
// process being up, still less on one reachable across a VPN.
type DreamDict struct {
d *dictionary.Dictionary
}
// OpenDreamDict opens dict.db read-only.
//
// A missing file returns (nil, nil), not an error. Petal is expected to run
// without dict.db — a laptop checkout has never had one, and the zh pair does
// not need one — so "the file isn't there" is a deployment state the caller
// handles by carrying on. A file that is *present but unusable* (corrupt, or
// never imported) does return an error, because that one is a mistake someone
// should hear about.
func OpenDreamDict(path string) (*DreamDict, error) {
if strings.TrimSpace(path) == "" {
return nil, nil
}
if _, err := os.Stat(path); err != nil {
if errors.Is(err, fs.ErrNotExist) {
return nil, nil
}
return nil, err
}
d, err := dictionary.NewReadOnly(path)
if err != nil {
return nil, err
}
return &DreamDict{d: d}, nil
}
// Close releases the dict.db handle. Safe on a nil DreamDict, so a caller that
// never got one can defer it unconditionally.
func (dd *DreamDict) Close() error {
if dd == nil {
return nil
}
return dd.d.Close()
}
// Contents reports how many words the open dict.db holds per language, so
// startup can log what it actually got.
//
// It counts rows rather than returning DreamDict's list of supported languages.
// Those are not the same thing and the difference is the whole point: a
// database built before Spanish existed still *supports* Spanish, and a log
// line naming the supported set would have said so cheerfully while every
// Spanish lookup came back empty. Counting rows is the question worth asking of
// a file somebody had to copy onto the box by hand.
func (dd *DreamDict) Contents() string {
counts, err := dd.d.WordCount()
if err != nil {
return "unreadable: " + err.Error()
}
langs := make([]string, 0, len(counts))
for lang := range counts {
langs = append(langs, lang)
}
sort.Strings(langs)
parts := make([]string, 0, len(langs))
for _, lang := range langs {
parts = append(parts, fmt.Sprintf("%s=%d", lang, counts[lang]))
}
if len(parts) == 0 {
return "no words"
}
return strings.Join(parts, " ")
}
// dreamProvider serves one writer: English lookups from dict.db, glossed into
// native. The struct is a value, created per request by [Set.For] — it holds no
// state beyond the shared handle and the language to translate into.
type dreamProvider struct {
dict *DreamDict
native string // the writer's language, e.g. "pt-PT"
}
// maxEtymology caps the free-form Wiktionary etymology. It is the one field
// with no natural length: some entries are a clause, some are four paragraphs
// tracing a word through three dead languages. The popover wants a line.
const maxEtymology = 220
// Lookup fills a Result from dict.db.
//
// The word is de-inflected with the same [candidates] walk the embedded
// datasets use, because dict.db stores headwords: "running" has no definitions
// row of its own. The first candidate that *has* definitions becomes the
// headword every other field is then read from, so a single popover never
// mixes "running"'s frequency with "run"'s definitions.
//
// The gloss is walked separately. A word can be absent from the definitions
// table and still have a translation (and vice versa), and the hover tooltip
// asks for the gloss alone — so tying it to the definition headword would lose
// glosses for no benefit.
func (p dreamProvider) Lookup(word string) (Result, error) {
res := Result{Word: word, Definitions: []Meaning{}, Synonyms: []string{}, Difficulty: unknownDifficulty}
norm := strings.ToLower(strings.TrimSpace(word))
if norm == "" {
return res, nil
}
head := norm
for _, c := range candidates(norm) {
defs, err := p.dict.d.Define(c, langEN)
if err != nil {
return Result{}, err
}
if len(defs) == 0 {
continue
}
head = c
for _, d := range defs {
// DreamDict orders by source priority, so the curated senses
// (WordNet, WOLF) are already ahead of the Wiktionary tail — taking
// the first few is taking the best few.
res.Definitions = append(res.Definitions, Meaning{PartOfSpeech: d.POS, Definition: d.Gloss})
if len(res.Definitions) >= maxDefinitions {
break
}
}
break
}
gloss, err := p.translate(norm)
if err != nil {
return Result{}, err
}
res.Gloss = gloss
syns, err := p.dict.d.Synonyms(head, langEN)
if err != nil {
return Result{}, err
}
if len(syns) > maxSynonyms {
syns = syns[:maxSynonyms]
}
res.Synonyms = append(res.Synonyms, syns...)
prons, err := p.dict.d.Pronunciation(head, langEN)
if err != nil {
return Result{}, err
}
res.Phonetic = pickIPA(prons)
if res.Frequency, err = p.dict.d.Frequency(head, langEN); err != nil {
return Result{}, err
}
if res.Difficulty, err = p.dict.d.Difficulty(head, langEN); err != nil {
return Result{}, err
}
ety, err := p.dict.d.Etymology(head, langEN)
if err != nil {
return Result{}, err
}
res.Etymology = trimEtymology(ety)
if res.Reverse, err = p.reverse(norm); err != nil {
return Result{}, err
}
return res, nil
}
// reverse reads the token as a word of the writer's own language, and returns
// nil when it isn't one — which is the answer for almost every word she looks
// up, since she is writing English.
//
// The English de-inflection walk is deliberately *not* applied here. [candidates]
// knows about -s, -ed and -ing; running it over Portuguese would turn "vinhas"
// into "vinha" by an English rule that happens to be right and "cantava" into
// nothing by rules that are simply irrelevant. dict.db stores headwords, so an
// inflected Portuguese form finds nothing and the card shows only the English
// reading — the same outcome as today, rather than a confidently wrong one.
func (p dreamProvider) reverse(norm string) (*Reverse, error) {
back, err := p.dict.d.Equivalents(norm, p.native, langEN)
if err != nil {
return nil, err
}
defs, err := p.dict.d.Define(norm, p.native)
if err != nil {
return nil, err
}
if len(back) == 0 && len(defs) == 0 {
return nil, nil
}
rev := &Reverse{Lang: p.native}
if len(back) > maxGlossSenses {
back = back[:maxGlossSenses]
}
rev.Gloss = strings.Join(back, "; ")
for _, d := range defs {
rev.Definitions = append(rev.Definitions, Meaning{PartOfSpeech: d.POS, Definition: d.Gloss})
if len(rev.Definitions) >= maxReverseDefinitions {
break
}
}
prons, err := p.dict.d.Pronunciation(norm, p.native)
if err != nil {
return nil, err
}
rev.Phonetic = pickIPA(prons)
return rev, nil
}
// maxReverseDefinitions is smaller than [maxDefinitions]: the reverse reading is
// the second half of a card that already has an English one, and it is there to
// say "this is also a Portuguese word, and here is what it means" rather than to
// be a dictionary entry in its own right.
const maxReverseDefinitions = 2
// Gloss returns the writer's-language translation alone — the hover tooltip's
// fast path, one indexed query per candidate form and nothing else.
func (p dreamProvider) Gloss(word string) (GlossResult, error) {
norm := strings.ToLower(strings.TrimSpace(word))
if norm == "" {
return GlossResult{Word: word}, nil
}
gloss, err := p.translate(norm)
if err != nil {
return GlossResult{}, err
}
res := GlossResult{Word: word, Gloss: gloss}
// The tooltip carries only the reverse *gloss*, not the whole reading: it is
// a one-line bubble under a resting pointer, and the popover is one click
// away for anyone who wants the rest.
back, err := p.dict.d.Equivalents(norm, p.native, langEN)
if err != nil {
return GlossResult{}, err
}
if len(back) > maxGlossSenses {
back = back[:maxGlossSenses]
}
res.Reverse = strings.Join(back, "; ")
return res, nil
}
// maxGlossSenses caps how many translations are strung together. One is often
// too thin to disambiguate; the whole list is a wall of words in a tooltip.
const maxGlossSenses = 3
// translate walks the candidate forms and returns the first that has an
// equivalent in the writer's language, joined into one line.
//
// It asks for Equivalents rather than Translate on the strength of measuring
// both against the real dict.db: Wiktionary's en→pt-PT translation table
// answers for 17% of the 2,000 commonest English words, and the shared-synset
// path answers for 62%. The plan assumed Translate would do — the database
// says otherwise, and a gloss that is absent five times out of six is not a
// gloss. Equivalents falls back to Translate internally, so nothing is lost.
//
// A language dict.db was built without simply has no rows, so this returns "" —
// which is exactly what an unglossed word returns, and the popover already
// renders that case. Spanish was precisely this until the database was rebuilt
// with it on 2026-07-27; the code path did not change, the file did.
func (p dreamProvider) translate(norm string) (string, error) {
for _, c := range candidates(norm) {
trs, err := p.dict.d.Equivalents(c, langEN, p.native)
if err != nil {
return "", err
}
if len(trs) == 0 {
continue
}
if len(trs) > maxGlossSenses {
trs = trs[:maxGlossSenses]
}
return strings.Join(trs, "; "), nil
}
return "", nil
}
// pickIPA chooses what to show beside the read-aloud button. IPA is the only
// form worth showing a learner — CMU's "IH0 F EH1 M ER0 AH0 L" is a machine
// format, and printing it would be noise dressed up as help. If there's no IPA,
// there's no phonetic line.
func pickIPA(prons []dictionary.Pronunciation) string {
for _, p := range prons {
if strings.EqualFold(p.Format, "ipa") && strings.TrimSpace(p.Value) != "" {
return strings.Trim(strings.TrimSpace(p.Value), "/[]")
}
}
return ""
}
// trimEtymology cuts Wiktionary's prose down to a line, preferring to stop at a
// sentence boundary so the result reads as a finished thought rather than a
// truncation.
func trimEtymology(text string) string {
text = strings.Join(strings.Fields(text), " ")
if utf8.RuneCountInString(text) <= maxEtymology {
return text
}
// Counted and cut in runes, not bytes. An etymology is the one field that
// is *mostly* not English — ἐφήμερος, ephemerus, 短暫 — and a byte slice
// through the middle of one of those characters is invalid UTF-8 in the
// JSON response.
cut := string([]rune(text)[:maxEtymology-1])
// Stop at a sentence when one ends late enough to be worth keeping. An
// early full stop ("From Latin. …") is not a summary, it's a discarded
// paragraph, so that case falls through to the word-boundary cut.
if i := strings.LastIndex(cut, ". "); i > len(cut)/2 {
return cut[:i+1]
}
if i := strings.LastIndex(cut, " "); i > 0 {
cut = cut[:i]
}
return cut + "…"
}
internal/lexicon/dreamdict_test.go
+664
View File
@@ -0,0 +1,664 @@
package lexicon
import (
"database/sql"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"unicode/utf8"
"github.com/go-chi/chi/v5"
"github.com/prosolis/dreamdict/dictionary"
_ "modernc.org/sqlite"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// The fixture is a real dict.db on disk rather than an in-memory handle, so
// these tests exercise the path production takes: stat the file, open it
// read-only, find it seeded. A fake would have skipped every one of those.
//
// "ephemeral" is the worked example throughout: it has definitions only under
// its own headword, translations into two languages, IPA alongside a CMU
// pronunciation Petal must not show, a frequency, a difficulty and an etymology
// long enough to need trimming.
func writeFixture(t *testing.T, seeded bool) string {
t.Helper()
path := filepath.Join(t.TempDir(), "dict.db")
sqldb, err := sql.Open("sqlite", path)
if err != nil {
t.Fatalf("open fixture: %v", err)
}
defer sqldb.Close()
if err := dictionary.BootstrapSchema(sqldb); err != nil {
t.Fatalf("bootstrap: %v", err)
}
if !seeded {
return path
}
exec := func(q string, args ...any) {
t.Helper()
if _, err := sqldb.Exec(q, args...); err != nil {
t.Fatalf("seed %q: %v", q, err)
}
}
exec(`INSERT INTO meta (key, value) VALUES ('schema_version', '2')`)
exec(`INSERT INTO words (id, word, lang, pos, frequency, difficulty) VALUES
(1, 'ephemeral', 'en', 'adjective', 50, 0.72),
(2, 'run', 'en', 'verb', 900, 0.05),
(3, 'plain', 'en', 'adjective', 0, NULL),
(4, 'efémero', 'pt-PT', 'adjective', 12, 0.6)`)
exec(`INSERT INTO definitions (word_id, pos, gloss, source, priority) VALUES
(1, 'adjective', 'lasting a very short time', 'wordnet', 10),
(1, 'adjective', 'short-lived', 'wiktionary', 99),
(1, 'adjective', 'transitory', 'wiktionary', 99),
(1, 'adjective', 'fleeting', 'wiktionary', 99),
(1, 'adjective', 'evanescent', 'wiktionary', 99),
(2, 'verb', 'move fast on foot', 'wordnet', 10),
(3, 'adjective', 'without decoration', 'wordnet', 10)`)
exec(`INSERT INTO synonyms (word_id, synonym, source) VALUES
(1, 'fleeting', 'wordnet'), (1, 'transient', 'wordnet'),
(2, 'sprint', 'wordnet')`)
exec(`INSERT INTO translations (word_id, translation, target_lang, source) VALUES
(1, 'efémero', 'pt-PT', 'kaikki'),
(1, 'passageiro','pt-PT', 'kaikki'),
(1, 'éphémère', 'fr', 'kaikki'),
(1, '短暂的', 'zh', 'cedict'),
(2, 'correr', 'pt-PT', 'kaikki')`)
// CMU is listed first deliberately: picking the first row would show a
// learner "IH0 F EH1 M ER0 AH0 L", which is a machine format, not help.
exec(`INSERT INTO pronunciations (word_id, format, value, source) VALUES
(1, 'cmu', 'IH0 F EH1 M ER0 AH0 L', 'cmudict'),
(1, 'ipa', '/ɪˈfɛm.ər.əl/', 'wiktionary')`)
// "brief" carries no translation row at all — only a shared WordNet synset
// with two pt-PT words. On the real database that is the *usual* case, not
// the exotic one, so Petal must reach a gloss this way or the pt-PT pair
// has almost no glosses. "breve" is the commoner of the two and leads.
exec(`INSERT INTO words (id, word, lang, pos, frequency) VALUES
(5, 'brief', 'en', 'adjective', 400),
(6, 'breve', 'pt-PT', 'adjective', 300),
(7, 'sucinto', 'pt-PT', 'adjective', 20)`)
exec(`INSERT INTO definitions (word_id, pos, gloss, source, priority) VALUES
(5, 'adjective', 'of short duration', 'wordnet', 10)`)
exec(`INSERT INTO synsets (id, synset_id, pos) VALUES (1, '00751145-a', 'adjective')`)
exec(`INSERT INTO word_synsets (word_id, synset_id, source) VALUES
(5, 1, 'wordnet'), (6, 1, 'omw'), (7, 1, 'omw')`)
// "data" is the collision the Latin pairs create and the zh pair never did:
// a real English word and a real Portuguese one, spelled identically and
// meaning different things. There is no honest way to look at it in a mixed
// document and know which was meant, so Petal shows both readings.
exec(`INSERT INTO words (id, word, lang, pos, frequency) VALUES
(8, 'data', 'en', 'noun', 800),
(9, 'data', 'pt-PT', 'noun', 700),
(10, 'date', 'en', 'noun', 750)`)
exec(`INSERT INTO definitions (word_id, pos, gloss, source, priority) VALUES
(8, 'noun', 'facts collected for reference', 'wordnet', 10),
(9, 'noun', 'dia do mês', 'wiktionary', 20),
(9, 'noun', 'momento no tempo', 'wiktionary', 30),
(9, 'noun', 'um terceiro sentido', 'wiktionary', 40)`)
exec(`INSERT INTO translations (word_id, translation, target_lang, source) VALUES
(9, 'date', 'en', 'kaikki')`)
exec(`INSERT INTO pronunciations (word_id, format, value, source) VALUES
(9, 'ipa', '/ˈdatɐ/', 'wiktionary')`)
exec(`INSERT INTO etymology (word_id, text, source) VALUES
(1, 'From Medieval Latin ephemerus, from Ancient Greek ἐφήμερος (ephḗmeros, "lasting only a day"), from ἐπί (epí, "upon") and ἡμέρα (hēméra, "day"). The sense of transience is attested in English from the late sixteenth century onwards.', 'wiktionary')`)
return path
}
func openFixture(t *testing.T) *DreamDict {
t.Helper()
dd, err := OpenDreamDict(writeFixture(t, true))
if err != nil {
t.Fatalf("OpenDreamDict: %v", err)
}
if dd == nil {
t.Fatal("OpenDreamDict returned no dictionary for a seeded file")
}
t.Cleanup(func() { dd.Close() })
return dd
}
func TestOpenMissingFileIsNotAnError(t *testing.T) {
dd, err := OpenDreamDict(filepath.Join(t.TempDir(), "absent.db"))
if err != nil {
t.Fatalf("a missing dict.db must not be an error: %v", err)
}
if dd != nil {
t.Fatal("a missing dict.db must yield no dictionary")
}
// An unset path is the laptop default and must behave the same way.
if dd, err := OpenDreamDict(""); err != nil || dd != nil {
t.Fatalf(`OpenDreamDict("") = %v, %v; want nil, nil`, dd, err)
}
// Close on the nil handle is what main.go defers unconditionally.
if err := dd.Close(); err != nil {
t.Fatalf("Close on absent dictionary: %v", err)
}
}
func TestOpenPresentButUnseededIsAnError(t *testing.T) {
// A file that exists but was never imported is somebody's mistake — a
// half-finished deploy — and must be loud, unlike a file that isn't there.
dd, err := OpenDreamDict(writeFixture(t, false))
if err == nil {
dd.Close()
t.Fatal("an unseeded dict.db must report an error")
}
if dd != nil {
t.Fatal("an unseeded dict.db must not yield a usable dictionary")
}
}
func TestOpenUnreadablePathIsAnError(t *testing.T) {
// Not a missing file: a directory where dict.db should be. Distinguishing
// this from ErrNotExist is the whole point of the stat.
dir := filepath.Join(t.TempDir(), "dict.db")
if err := os.Mkdir(dir, 0o755); err != nil {
t.Fatal(err)
}
if _, err := OpenDreamDict(dir); err == nil {
t.Fatal("a directory in place of dict.db must report an error")
}
}
func TestDreamLookupFillsEveryField(t *testing.T) {
p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
res, err := p.Lookup("ephemeral")
if err != nil {
t.Fatalf("Lookup: %v", err)
}
if res.Gloss != "efémero; passageiro" {
t.Errorf("Gloss = %q, want the pt-PT translations joined", res.Gloss)
}
if res.Phonetic != "ɪˈfɛm.ər.əl" {
t.Errorf("Phonetic = %q, want the IPA without its slashes", res.Phonetic)
}
if len(res.Definitions) != maxDefinitions {
t.Fatalf("Definitions = %d, want them capped at %d", len(res.Definitions), maxDefinitions)
}
if res.Definitions[0].Definition != "lasting a very short time" {
t.Errorf("first definition = %q, want the curated (wordnet) sense first",
res.Definitions[0].Definition)
}
if res.Definitions[0].PartOfSpeech != "adjective" {
t.Errorf("part of speech = %q, want adjective", res.Definitions[0].PartOfSpeech)
}
if len(res.Synonyms) != 2 {
t.Errorf("Synonyms = %v, want both", res.Synonyms)
}
if res.Frequency != 50 {
t.Errorf("Frequency = %d, want 50", res.Frequency)
}
if res.Difficulty != 0.72 {
t.Errorf("Difficulty = %v, want 0.72", res.Difficulty)
}
if !strings.HasPrefix(res.Etymology, "From Medieval Latin ephemerus") {
t.Errorf("Etymology = %q, want the Wiktionary text", res.Etymology)
}
if len(res.Etymology) > maxEtymology {
t.Errorf("Etymology not trimmed: %d chars", len(res.Etymology))
}
}
func TestDreamGlossFollowsTheWriterNotTheWord(t *testing.T) {
dd := openFixture(t)
for lang, want := range map[string]string{
"pt-PT": "efémero; passageiro",
"fr": "éphémère",
"es": "", // DreamDict supports Spanish; this database wasn't built with it
"de": "", // never a Petal pair, and must not silently borrow another's
} {
got, err := dreamProvider{dict: dd, native: lang}.Gloss("ephemeral")
if err != nil {
t.Fatalf("Gloss(%s): %v", lang, err)
}
if got.Gloss != want {
t.Errorf("Gloss for %s = %q, want %q", lang, got.Gloss, want)
}
if got.Word != "ephemeral" {
t.Errorf("Word = %q, want the word as asked", got.Word)
}
}
}
func TestDreamGlossesThroughSharedSynsets(t *testing.T) {
// The measurement that drove this: on the real dict.db, Wiktionary's
// en→pt-PT translation table answers for 17% of the 2,000 commonest English
// words and the shared-synset path answers for 62%. A word with no
// translation row must still get a gloss, commonest sense first.
p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
res, err := p.Lookup("brief")
if err != nil {
t.Fatalf("Lookup: %v", err)
}
if res.Gloss != "breve; sucinto" {
t.Errorf("Gloss = %q, want the synset equivalents, commonest first", res.Gloss)
}
}
func TestDreamDeinflectsToTheHeadword(t *testing.T) {
// dict.db stores headwords: "running" has no row of its own. The candidate
// walk is what makes a right-click on real prose work at all.
p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
res, err := p.Lookup("running")
if err != nil {
t.Fatalf("Lookup: %v", err)
}
if len(res.Definitions) == 0 || res.Definitions[0].Definition != "move fast on foot" {
t.Fatalf("Definitions = %+v, want run's", res.Definitions)
}
// Every other field must come from the same headword — a popover that mixed
// "running"'s (absent) frequency with "run"'s definitions would be lying.
if res.Frequency != 900 {
t.Errorf("Frequency = %d, want run's 900", res.Frequency)
}
if res.Difficulty != 0.05 {
t.Errorf("Difficulty = %v, want run's 0.05", res.Difficulty)
}
if res.Synonyms[0] != "sprint" {
t.Errorf("Synonyms = %v, want run's", res.Synonyms)
}
if res.Gloss != "correr" {
t.Errorf("Gloss = %q, want run's", res.Gloss)
}
}
func TestDreamMissIsAnEmptyResultNotAnError(t *testing.T) {
p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
res, err := p.Lookup("zzzxqqq")
if err != nil {
t.Fatalf("Lookup: %v", err)
}
if len(res.Definitions) != 0 || len(res.Synonyms) != 0 || res.Gloss != "" {
t.Errorf("expected an empty result, got %+v", res)
}
// The frontend renders [] and never null.
if res.Definitions == nil || res.Synonyms == nil {
t.Errorf("empty slices must be non-nil: %+v", res)
}
if res.Difficulty != unknownDifficulty {
t.Errorf("Difficulty = %v, want the unknown sentinel", res.Difficulty)
}
// Empty input is a miss, not a crash.
if res, err := p.Lookup(" "); err != nil || res.Gloss != "" {
t.Errorf("Lookup(blank) = %+v, %v", res, err)
}
if res, err := p.Gloss(""); err != nil || res.Gloss != "" {
t.Errorf("Gloss(empty) = %+v, %v", res, err)
}
}
func TestDreamUnscoredWordKeepsTheUnknownSentinel(t *testing.T) {
// "plain" is in the database with no frequency and a NULL difficulty. The
// popover must be able to tell that apart from "difficulty 0.0, the easiest
// word there is" — which is why the sentinel is -1 and not omitempty.
p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
res, err := p.Lookup("plain")
if err != nil {
t.Fatalf("Lookup: %v", err)
}
if len(res.Definitions) == 0 {
t.Fatal("expected plain to be found")
}
if res.Difficulty != unknownDifficulty {
t.Errorf("Difficulty = %v, want the unknown sentinel for a NULL score", res.Difficulty)
}
if res.Frequency != 0 {
t.Errorf("Frequency = %d, want 0 for no count", res.Frequency)
}
}
func TestPickIPASkipsMachineFormats(t *testing.T) {
if got := pickIPA([]dictionary.Pronunciation{{Format: "cmu", Value: "K AE1 T"}}); got != "" {
t.Errorf("pickIPA on CMU alone = %q, want empty — CMU is not for a reader", got)
}
if got := pickIPA(nil); got != "" {
t.Errorf("pickIPA(nil) = %q", got)
}
if got := pickIPA([]dictionary.Pronunciation{{Format: "IPA", Value: " [kæt] "}}); got != "kæt" {
t.Errorf("pickIPA = %q, want the bare IPA regardless of case or brackets", got)
}
}
func TestTrimEtymologyPrefersASentence(t *testing.T) {
// A sentence that ends past halfway is the good cut: keep it, drop the rest.
long := strings.Repeat("padding word ", 14) + "end. " + strings.Repeat("more ", 40)
got := trimEtymology(long)
if utf8.RuneCountInString(got) > maxEtymology {
t.Errorf("not trimmed: %d runes", utf8.RuneCountInString(got))
}
if !strings.HasSuffix(got, "end.") {
t.Errorf("trimEtymology = %q, want it to stop at the sentence", got)
}
// An early full stop is not a summary — cutting there would throw away
// almost the whole line — so this falls through to a word boundary.
got = trimEtymology("From Latin. " + strings.Repeat("padding word ", 40))
if strings.HasSuffix(got, "Latin.") {
t.Errorf("trimEtymology = %q, want more than the first four words", got)
}
if !strings.HasSuffix(got, "…") {
t.Errorf("trimEtymology = %q, want an ellipsis when cut mid-thought", got)
}
// Multi-byte text must be cut on rune boundaries: a byte slice through
// ἐφήμερος would put invalid UTF-8 in the JSON.
greek := trimEtymology(strings.Repeat("ἐφήμερος ", 60))
if !utf8.ValidString(greek) {
t.Errorf("trimEtymology produced invalid UTF-8: %q", greek)
}
if n := utf8.RuneCountInString(greek); n > maxEtymology {
t.Errorf("trimmed to %d runes, want at most %d", n, maxEtymology)
}
if strings.Contains(got, " ") || strings.Contains(trimEtymology("a\n b"), "\n") {
t.Error("whitespace should be collapsed to a single line")
}
// Short text passes through untouched.
if got := trimEtymology("From Old English."); got != "From Old English." {
t.Errorf("trimEtymology = %q", got)
}
}
// --- the Set: which provider answers, and what happens when dict.db is absent
func TestSetRoutesByPairLanguage(t *testing.T) {
set := NewSet(openFixture(t))
if !set.HasDreamDict() {
t.Fatal("HasDreamDict = false with a dictionary open")
}
// zh — and an empty column, which is what a pre-auth row reads as — stays
// on the embedded datasets until the two have been compared on real
// lookups. This test is the guard on that decision.
for _, lang := range []string{"", LangZh} {
if _, ok := set.For(lang).(*Lexicon); !ok {
t.Errorf("For(%q) = %T, want the embedded Lexicon", lang, set.For(lang))
}
}
for _, lang := range []string{"pt-PT", "fr", "es"} {
p, ok := set.For(lang).(dreamProvider)
if !ok {
t.Fatalf("For(%q) = %T, want DreamDict", lang, set.For(lang))
}
if p.native != lang {
t.Errorf("For(%q) glosses into %q", lang, p.native)
}
}
}
func TestSetWithoutDictKeepsTheEnglishHalf(t *testing.T) {
// The interesting degradation: dict.db never got deployed. A pt-PT writer
// should still get definitions, synonyms and phonetics — all compiled into
// the binary and all correct for her — and lose only the translation.
set := NewSet(nil)
if set.HasDreamDict() {
t.Fatal("HasDreamDict = true with no dictionary")
}
p := set.For("pt-PT")
res, err := p.Lookup("happy")
if err != nil {
t.Fatalf("Lookup: %v", err)
}
if len(res.Definitions) == 0 || len(res.Synonyms) == 0 {
t.Error("expected the embedded English half to survive a missing dict.db")
}
if res.Gloss != "" {
t.Errorf("Gloss = %q — a pt-PT writer must never be handed the Chinese gloss", res.Gloss)
}
g, err := p.Gloss("happy")
if err != nil {
t.Fatalf("Gloss: %v", err)
}
if g.Gloss != "" {
t.Errorf("Gloss = %q, want empty", g.Gloss)
}
if g.Word != "happy" {
t.Errorf("Word = %q, want the word as asked", g.Word)
}
// The zh writer is untouched by any of this.
zh, err := set.For(LangZh).Lookup("happy")
if err != nil {
t.Fatalf("Lookup(zh): %v", err)
}
if zh.Gloss == "" {
t.Error("the zh pair must keep its embedded gloss with no dict.db")
}
}
// --- the handler: the pair language is read per request, from the caller's row
func mountLexicon(t *testing.T, set *Set) (*chi.Mux, *db.DB) {
t.Helper()
database, err := db.Open(filepath.Join(t.TempDir(), "petal.db"))
if err != nil {
t.Fatalf("db.Open: %v", err)
}
t.Cleanup(func() { database.Close() })
for _, u := range []struct{ id, lang string }{
{"alice", LangZh}, {"bob", "pt-PT"},
} {
if _, err := database.Exec(
`INSERT INTO users (id, email, display_name, pair_lang) VALUES (?, ?, ?, ?)`,
u.id, u.id+"@example.com", u.id, u.lang,
); err != nil {
t.Fatalf("seed user: %v", err)
}
}
h := NewHandler(database.DB, set)
r := chi.NewMux()
r.Mount("/word", h.Routes())
r.Mount("/gloss", h.GlossRoutes())
return r, database
}
func getAs(t *testing.T, r http.Handler, userID, path string) Result {
t.Helper()
req := httptest.NewRequest(http.MethodGet, path, nil)
req = req.WithContext(auth.WithUser(req.Context(), userID))
rec := httptest.NewRecorder()
r.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("GET %s as %s = %d: %s", path, userID, rec.Code, rec.Body)
}
var res Result
if err := json.Unmarshal(rec.Body.Bytes(), &res); err != nil {
t.Fatalf("decode: %v", err)
}
return res
}
func TestHandlerGlossesInTheCallersLanguage(t *testing.T) {
r, _ := mountLexicon(t, NewSet(openFixture(t)))
// Same URL, two writers, two languages. This is why the response is no
// longer cacheable as `public`.
bob := getAs(t, r, "bob", "/word/ephemeral")
if bob.Gloss != "efémero; passageiro" {
t.Errorf("bob's gloss = %q, want pt-PT", bob.Gloss)
}
alice := getAs(t, r, "alice", "/word/ephemeral")
if !strings.ContainsAny(alice.Gloss, "短暂的") && alice.Gloss != "" {
// alice is on the embedded ECDICT dataset, not the fixture's zh row —
// what matters is that she is *not* served bob's Portuguese.
t.Logf("alice's embedded gloss: %q", alice.Gloss)
}
if alice.Gloss == bob.Gloss && bob.Gloss != "" {
t.Error("the zh writer was served the pt-PT gloss")
}
if strings.Contains(alice.Gloss, "efémero") {
t.Errorf("alice's gloss = %q, want the embedded Chinese one", alice.Gloss)
}
}
func TestHandlerUnknownCallerFallsBackRatherThanFailing(t *testing.T) {
// No session, or a user row that has gone: the lookup still answers, from
// the embedded datasets. A dictionary that fails closed would be worse than
// one that answers in the wrong language, because nothing at all is not a
// dictionary.
r, _ := mountLexicon(t, NewSet(openFixture(t)))
res := getAs(t, r, "nobody", "/word/happy")
if len(res.Definitions) == 0 {
t.Error("expected the embedded fallback to answer for an unknown caller")
}
}
func TestHandlerCachesPrivately(t *testing.T) {
r, _ := mountLexicon(t, NewSet(nil))
req := httptest.NewRequest(http.MethodGet, "/gloss/happy", nil)
req = req.WithContext(auth.WithUser(req.Context(), "alice"))
rec := httptest.NewRecorder()
r.ServeHTTP(rec, req)
if got := rec.Header().Get("Cache-Control"); !strings.HasPrefix(got, "private") {
t.Errorf("Cache-Control = %q — a per-writer gloss must not go in a shared cache", got)
}
}
func TestHandlerDecodesPunctuatedWords(t *testing.T) {
r, _ := mountLexicon(t, NewSet(openFixture(t)))
res := getAs(t, r, "bob", "/word/"+"caf%C3%A9")
if res.Word != "café" {
t.Errorf("Word = %q, want the decoded word", res.Word)
}
}
func TestContentsCountsRowsNotSupportedLanguages(t *testing.T) {
// The fixture is seeded with English and pt-PT only. DreamDict *supports*
// French, Spanish and Chinese too — and a startup line that reported the
// supported set would have named all five while every French lookup came
// back empty. That is the failure this log line exists to catch, so it must
// count rows.
got := NewSet(openFixture(t))
summary := got.Contents()
if !strings.Contains(summary, "en=") || !strings.Contains(summary, "pt-PT=") {
t.Errorf("Contents = %q, want the languages the fixture actually holds", summary)
}
for _, absent := range []string{"fr=", "es=", "zh="} {
if strings.Contains(summary, absent) {
t.Errorf("Contents = %q, must not name %q — no rows exist for it", summary, absent)
}
}
// No dictionary at all still has to answer something printable.
if s := NewSet(nil).Contents(); s == "" {
t.Error("Contents with no dictionary must still say something")
}
}
// The Latin+Latin wrinkle (SUGGESTIONS.md §3a). An English+Chinese pair never
// had to decide which language a word was in — the script decided. An
// English+Portuguese pair has no script boundary, and "data", "sale", "comum"
// and "tarde" are real words on both sides of it. Petal asks both directions
// and shows whatever answers, which needs no language detector and therefore
// cannot be wrong about somebody's writing.
func TestDreamLookupShowsBothReadingsOnACollision(t *testing.T) {
p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
res, err := p.Lookup("data")
if err != nil {
t.Fatalf("Lookup: %v", err)
}
// The English reading is unchanged and still leads.
if len(res.Definitions) == 0 || res.Definitions[0].Definition != "facts collected for reference" {
t.Fatalf("Definitions = %v, want the English sense first", res.Definitions)
}
if res.Reverse == nil {
t.Fatal("Reverse = nil; a word that exists in both languages must carry both readings")
}
if res.Reverse.Lang != "pt-PT" {
t.Errorf("Reverse.Lang = %q, want the writer's language", res.Reverse.Lang)
}
if res.Reverse.Gloss != "date" {
t.Errorf("Reverse.Gloss = %q, want the English meaning of the Portuguese word", res.Reverse.Gloss)
}
if res.Reverse.Phonetic != "ˈdatɐ" {
t.Errorf("Reverse.Phonetic = %q, want the Portuguese IPA without slashes", res.Reverse.Phonetic)
}
// The reverse reading is a footnote on a card that already has an English
// half, so it is capped harder than the main entry.
if len(res.Reverse.Definitions) != maxReverseDefinitions {
t.Fatalf("Reverse.Definitions = %d, want %d", len(res.Reverse.Definitions), maxReverseDefinitions)
}
if res.Reverse.Definitions[0].Definition != "dia do mês" {
t.Errorf("Reverse.Definitions[0] = %q, want the Portuguese sense",
res.Reverse.Definitions[0].Definition)
}
}
func TestDreamLookupHasNoReverseForAnEnglishOnlyWord(t *testing.T) {
// Which is almost every word she looks up: she is writing English. A
// second block under every card would make the collision case invisible.
p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
res, err := p.Lookup("ephemeral")
if err != nil {
t.Fatalf("Lookup: %v", err)
}
if res.Reverse != nil {
t.Fatalf("Reverse = %+v, want none for a word that is only English", res.Reverse)
}
}
func TestDreamGlossCarriesTheReverseReading(t *testing.T) {
// The hover tooltip takes the same both-directions rule in one line less
// space: the reverse *gloss* only, never the definitions.
p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
g, err := p.Gloss("data")
if err != nil {
t.Fatalf("Gloss: %v", err)
}
if g.Reverse != "date" {
t.Errorf("Gloss.Reverse = %q, want the English meaning of the Portuguese word", g.Reverse)
}
g, err = p.Gloss("ephemeral")
if err != nil {
t.Fatalf("Gloss: %v", err)
}
if g.Reverse != "" {
t.Errorf("Gloss.Reverse = %q, want none for an English-only word", g.Reverse)
}
if g.Gloss != "efémero; passageiro" {
t.Errorf("Gloss = %q, want the forward gloss untouched", g.Gloss)
}
}
func TestReverseIsSilentForTheEmbeddedProviders(t *testing.T) {
// The zh pair has no collisions and no DreamDict, and a writer with no
// dict.db at all falls through to `glossless`. Neither may start emitting a
// reverse block: the card would then claim a Chinese reading of an English
// word, which is worse than saying nothing.
set := NewSet(nil)
res, err := set.For(LangZh).Lookup("river")
if err != nil {
t.Fatalf("Lookup: %v", err)
}
if res.Reverse != nil {
t.Errorf("embedded Reverse = %+v, want none", res.Reverse)
}
res, err = set.For("pt-PT").Lookup("river")
if err != nil {
t.Fatalf("Lookup: %v", err)
}
if res.Reverse != nil {
t.Errorf("glossless Reverse = %+v, want none", res.Reverse)
}
}
internal/lexicon/handlers.go
+69 -37
View File
@@ -1,20 +1,27 @@
package lexicon
import (
"context"
"database/sql"
"encoding/json"
"net/http"
"net/url"
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
)
// Handler serves the word-lookup endpoint backed by a single shared Lexicon.
// Handler serves the word-lookup endpoints. It holds the shared provider Set
// and the database, because which provider answers depends on who is asking.
type Handler struct {
Lex *Lexicon
Set *Set
db *sql.DB
}
// New constructs a Handler with a fresh (lazily-loaded) Lexicon.
func NewHandler() *Handler { return &Handler{Lex: New()} }
// NewHandler constructs a Handler over a provider Set. db is used for one
// thing: reading the caller's pair language.
func NewHandler(db *sql.DB, set *Set) *Handler { return &Handler{Set: set, db: db} }
// Routes returns the router mounted at /api/word. The word is a path segment so
// "/api/word/happy" reads naturally; it's URL-decoded to tolerate the rare
@@ -26,56 +33,81 @@ func (h *Handler) Routes() chi.Router {
}
// GlossRoutes returns the router mounted at /api/gloss — the lightweight
// Chinese-only lookup behind the inline hover/select gloss. It shares the
// Handler's Lexicon, so the datasets still load just once.
// translation-only lookup behind the inline hover/select gloss. It shares the
// Handler's Set, so the embedded datasets and dict.db are still opened once.
func (h *Handler) GlossRoutes() chi.Router {
r := chi.NewRouter()
r.Get("/{word}", h.gloss)
return r
}
// lookup returns the definition + synonyms for one word. A word found in neither
// dataset still returns 200 with empty lists, so the popover can show a friendly
// "nothing found" rather than an error state.
// providerFor returns the provider for the caller's language pair.
//
// The pair language is read here rather than threaded down because a word
// lookup has no other query to piggyback on — unlike the document handlers,
// which take pair_lang from the row-scoped query that already proves
// ownership. It is one indexed primary-key read against a local SQLite file,
// which costs less than encoding the response it feeds.
//
// A read that fails, or a caller with no user row, resolves to the empty
// language, and [Set.For] maps that to today's embedded behaviour. Falling back
// to a working dictionary beats failing the lookup.
func (h *Handler) providerFor(ctx context.Context) Provider {
var lang string
if h.db != nil {
_ = h.db.QueryRowContext(ctx,
`SELECT COALESCE(pair_lang, '') FROM users WHERE id = ?`,
auth.UserID(ctx),
).Scan(&lang)
}
return h.Set.For(lang)
}
// pathWord reads the {word} segment, URL-decoded.
func pathWord(r *http.Request) string {
word := chi.URLParam(r, "word")
if decoded, err := url.PathUnescape(word); err == nil {
word = decoded
}
return word
}
// lookup returns the definition + synonyms for one word. A word found in no
// dataset still returns 200 with empty lists, so the popover can show a
// friendly "nothing found" rather than an error state.
func (h *Handler) lookup(w http.ResponseWriter, r *http.Request) {
word := chi.URLParam(r, "word")
if decoded, err := url.PathUnescape(word); err == nil {
word = decoded
}
res, err := h.Lex.Lookup(word)
res, err := h.providerFor(r.Context()).Lookup(pathWord(r))
if err != nil {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusInternalServerError)
_ = json.NewEncoder(w).Encode(map[string]string{"error": err.Error()})
writeLookupErr(w, err)
return
}
w.Header().Set("Content-Type", "application/json")
// Word lookups are static for the life of the build; let the browser cache
// them so repeated right-clicks on the same word are instant.
w.Header().Set("Cache-Control", "public, max-age=86400")
_ = json.NewEncoder(w).Encode(res)
writeLookup(w, res)
}
// gloss returns just the Chinese translation for one word. Like lookup, a miss
// is a 200 with an empty gloss so the hover tooltip can quietly skip rather than
// error.
// gloss returns just the translation for one word. Like lookup, a miss is a 200
// with an empty gloss so the hover tooltip can quietly skip rather than error.
func (h *Handler) gloss(w http.ResponseWriter, r *http.Request) {
word := chi.URLParam(r, "word")
if decoded, err := url.PathUnescape(word); err == nil {
word = decoded
}
res, err := h.Lex.Gloss(word)
res, err := h.providerFor(r.Context()).Gloss(pathWord(r))
if err != nil {
writeLookupErr(w, err)
return
}
writeLookup(w, res)
}
func writeLookupErr(w http.ResponseWriter, err error) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusInternalServerError)
_ = json.NewEncoder(w).Encode(map[string]string{"error": err.Error()})
return
}
}
func writeLookup(w http.ResponseWriter, v any) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Cache-Control", "public, max-age=86400")
_ = json.NewEncoder(w).Encode(res)
// A lookup is stable for the life of the deployment, so let the browser
// keep it — repeated right-clicks on the same word are then instant. It is
// `private` rather than `public` because the gloss is now in *her*
// language: a shared cache keyed on the URL alone would hand one writer
// another writer's language.
w.Header().Set("Cache-Control", "private, max-age=86400")
_ = json.NewEncoder(w).Encode(v)
}
internal/lexicon/lexicon.go
+56 -1
View File
@@ -29,14 +29,69 @@ type Result struct {
Phonetic string `json:"phonetic"` // IPA for the English word; "" when absent
Definitions []Meaning `json:"definitions"`
Synonyms []string `json:"synonyms"`
// The fields below only ever come from DreamDict; the embedded datasets
// leave them at their unknown values, and the popover hides them.
// Frequency is how common the word is (higher = more common). 0 means
// unknown, which is DreamDict's own convention — a word it carries but has
// no corpus count for is indistinguishable from a word it doesn't carry,
// and the popover treats both the same way.
Frequency int `json:"frequency"`
// Difficulty runs 0.0 (easiest) to 1.0 (hardest); -1 means unknown. It is a
// sentinel rather than an omitted field because 0.0 is a real, meaningful
// score and `omitempty` would erase it.
Difficulty float64 `json:"difficulty"`
// Etymology is free-form Wiktionary prose, trimmed to a line. Where the
// word came from is a real hook for a writer whose own language shares
// Latin roots with English — "ephemeral" is much easier to keep once you
// have seen efémero next to it.
Etymology string `json:"etymology"`
// Reverse is the same token read as a word of the writer's own language,
// present only when it is one. Absent for every writer whose pair is not
// Latin-script, and for the overwhelming majority of words in one that is.
Reverse *Reverse `json:"reverse,omitempty"`
}
// Reverse is a lookup in the other direction: the token treated as a word of the
// writer's language, translated into English.
//
// It exists because a Latin-script pair has no script boundary to tell the two
// halves apart. In English+Chinese, "which language is this word?" answers
// itself. In English+Portuguese it does not: *sale*, *casa*, *comum*, *tarde*
// and *ali* are all real words on both sides, and *chat* and *pain* are the
// French versions of the same trap.
//
// Petal does not guess. It asks both directions and shows whatever comes back,
// which needs no detector, cannot be wrong about someone's writing, and — for a
// learner — is more interesting than a correct guess would have been.
type Reverse struct {
// Lang is the language this reading is in, so the card can label it.
Lang string `json:"lang"`
// Gloss is the English meaning of the native-language word.
Gloss string `json:"gloss"`
// Definitions are the word's senses as written in the writer's own
// language — the monolingual half, for when the English gloss isn't enough.
Definitions []Meaning `json:"definitions,omitempty"`
// Phonetic is IPA for the native-language pronunciation; "" when absent.
Phonetic string `json:"phonetic,omitempty"`
}
// unknownDifficulty is the [Result.Difficulty] value meaning "no score",
// matching DreamDict's own -1 return.
const unknownDifficulty = -1
// GlossResult is the lightweight payload for the inline hover/select gloss: just
// the word and its Chinese translation, no definitions or synonyms. Kept small
// so the hover tooltip is instant and trivially cacheable.
type GlossResult struct {
Word string `json:"word"`
Gloss string `json:"gloss"`
// Reverse is the English meaning of the word read as one of the writer's
// own language — the tooltip's half of the both-directions rule (see
// [Reverse]). Empty unless the token is a word in her language too.
Reverse string `json:"reverse,omitempty"`
}
// maxSynonyms caps how many synonyms we hand the popover, even though the dataset
@@ -95,7 +150,7 @@ func (l *Lexicon) Lookup(word string) (Result, error) {
}
norm := strings.ToLower(strings.TrimSpace(word))
res := Result{Word: word, Definitions: []Meaning{}, Synonyms: []string{}}
res := Result{Word: word, Definitions: []Meaning{}, Synonyms: []string{}, Difficulty: unknownDifficulty}
if norm == "" {
return res, nil
}
internal/lexicon/provider.go
+111
View File
@@ -0,0 +1,111 @@
package lexicon
// A word lookup used to mean exactly one thing: the embedded datasets, which
// speak English and Mandarin and nothing else. That was fine while Petal had
// one writer. It stops being fine the moment a pt-PT writer right-clicks a
// word and gets a Chinese gloss.
//
// So the lookup becomes a seam. A [Provider] answers the same two questions the
// popover and the hover tooltip have always asked; which provider answers them
// depends on the writer's language pair, and [Set.For] is the only place that
// decision is made.
// Provider answers word lookups for one writer. The embedded datasets and
// DreamDict both satisfy it, and both treat a word they don't carry as an empty
// result rather than an error — a miss is an ordinary outcome of looking a word
// up, not a failure.
type Provider interface {
// Lookup returns the full popover payload: gloss, phonetic, definitions,
// synonyms, and whatever extras the provider carries.
Lookup(word string) (Result, error)
// Gloss returns just the writer's-language translation. It is the hover
// tooltip's fast path and skips everything else.
Gloss(word string) (GlossResult, error)
}
// LangZh is the one pair language still served by the embedded datasets. Every
// other pair goes to DreamDict — see [Set.For] for why zh is held back.
const LangZh = "zh"
// langEN is the language DreamDict is asked about for definitions, synonyms and
// pronunciation. English is always the *target* language of the pair — what
// varies is the language the gloss is written in.
const langEN = "en"
// Set holds every provider Petal can serve a lookup from and picks between them
// by pair language. One Set is shared by the whole process: the embedded
// datasets load once, and dict.db is one read-only handle.
type Set struct {
embedded *Lexicon
// dream is nil when dict.db was not deployed. That is a supported state,
// not an error — see [Set.For].
dream *DreamDict
}
// NewSet returns a Set backed by the embedded datasets and, when dream is
// non-nil, DreamDict. Passing a nil dream is how Petal runs without dict.db.
func NewSet(dream *DreamDict) *Set {
return &Set{embedded: New(), dream: dream}
}
// HasDreamDict reports whether a dict.db is open. Only startup logging and
// tests care; a handler never asks, because [Set.For] always returns something
// usable.
func (s *Set) HasDreamDict() bool { return s.dream != nil }
// Contents describes what the open dict.db actually holds, for the startup log.
// With no dictionary it says so rather than returning an empty string, because
// a blank in a log line is indistinguishable from a bug in the log line.
func (s *Set) Contents() string {
if s.dream == nil {
return "no dict.db — embedded datasets only"
}
return s.dream.Contents()
}
// For returns the provider that should answer lookups for a writer whose pair
// language is lang.
//
// Three rules, in order:
//
// zh — and an empty code, which is what a pre-Phase-16 row reads as — stays on
// the embedded ECDICT gloss. Not because DreamDict lacks Chinese (it has
// CC-CEDICT), but because that path is in daily use by a real writer and the
// two have not yet been compared on her actual lookups. Switching it is a
// quality decision, and it hasn't been made.
//
// Any other pair goes to DreamDict, which is the only source that has pt-PT,
// French or Spanish at all.
//
// If dict.db was never deployed, a non-zh writer falls back to the embedded
// datasets with the gloss suppressed. This is the interesting case: the naive
// "no data" answer would blank the popover entirely, when in fact the English
// half of it — definitions, synonyms, phonetic — is compiled into the binary
// and perfectly correct for her. Only the translation is missing, so only the
// translation goes missing. A failed dictionary deploy costs her the gloss, not
// the dictionary.
func (s *Set) For(lang string) Provider {
if lang == "" || lang == LangZh {
return s.embedded
}
if s.dream != nil {
return dreamProvider{dict: s.dream, native: lang}
}
return glossless{s.embedded}
}
// glossless serves the embedded datasets with the Chinese gloss stripped, for a
// writer who does not read Chinese. Handing her the zh gloss would be worse
// than handing her nothing: an empty field reads as "not found", where the
// wrong language reads as Petal being broken.
type glossless struct{ inner Provider }
func (g glossless) Lookup(word string) (Result, error) {
res, err := g.inner.Lookup(word)
res.Gloss = ""
return res, err
}
func (g glossless) Gloss(word string) (GlossResult, error) {
return GlossResult{Word: word}, nil
}
internal/llm/checkpoint.go
+1 -1
View File
@@ -41,7 +41,7 @@ type checkpointResponse struct {
// RunCheckpoint sends the grammar checkpoint and parses the JSON result. It
// applies the latency-guard truncation and the checkpoint sampling parameters
// from the spec.
func RunCheckpoint(ctx context.Context, client LLMClient, contentText, tone string) ([]RawSuggestion, error) {
func RunCheckpoint(ctx context.Context, client LLMClient, contentText, tone string, _ Lang) ([]RawSuggestion, error) {
raw, err := client.Complete(ctx, CompletionRequest{
Messages: CheckpointMessages(TruncateDoc(contentText), tone),
MaxTokens: checkpointMaxTokens,
internal/llm/collocation.go
+4 -3
View File
@@ -18,10 +18,11 @@ const CollocationInterval = 25 * time.Second
// reflect the full piece. Each flag carries a native replacement to apply.
//
// The tone argument is accepted for a uniform pass signature and passed through
// to the prompt so a hint can prefer a register-appropriate pairing.
func RunCollocation(ctx context.Context, client LLMClient, contentText, tone string) ([]RawSuggestion, error) {
// to the prompt so a hint can prefer a register-appropriate pairing. `lang` is
// the writer's pair language — the one each hint's short gloss is written in.
func RunCollocation(ctx context.Context, client LLMClient, contentText, tone string, lang Lang) ([]RawSuggestion, error) {
raw, err := client.Complete(ctx, CompletionRequest{
Messages: CollocationMessages(contentText, tone),
Messages: CollocationMessages(contentText, tone, lang),
MaxTokens: 2048,
Temperature: 0.3,
RepetitionPenalty: 1.15,
internal/llm/lang.go
+52
View File
@@ -0,0 +1,52 @@
package llm
import "strings"
// The pair language, as the prompts need to talk about it.
//
// Three of Petal's prompts name the writer's first language rather than merely
// being written in English: the collocation coach asks for a gloss in it, Ask
// Petal offers to answer in it, and the explanation translator renders into it.
// Before Phase 19 all three said "Simplified Chinese" outright, which made the
// zh pair the only one that could ever work.
//
// A Lang is not a translation of the prompt — the instructions stay in English,
// which is what the model follows best. It is the name the model should use for
// her language, plus the one word it should watch for when she writes in it.
type Lang struct {
// Code matches users.pair_lang.
Code string
// Name is how the prompt refers to the language, spelled the way a model
// recognises it. Regional precision matters here: "European Portuguese" is
// not "Portuguese" to a model that has read far more pt-BR than pt-PT.
Name string
// Why asks the same thing she would ask in her own language. It goes into
// the Ask Petal prompt as an example, so a model that answers only to
// English "why" still recognises the question when she types it her way.
Why string
}
// langs holds every pair Petal can currently be a partner in. A language with a
// frontend langpack but no entry here still works — it falls back to zh's
// behaviour of the prompts, which is wrong but not broken — so keep the two in
// step when a pair ships.
var langs = map[string]Lang{
"zh": {Code: "zh", Name: "Simplified Chinese (Mandarin)", Why: "为什么"},
"pt-PT": {Code: "pt-PT", Name: "European Portuguese (pt-PT, never Brazilian Portuguese)", Why: "porquê"},
"fr": {Code: "fr", Name: "French", Why: "pourquoi"},
"es": {Code: "es", Name: "Spanish", Why: "por qué"},
}
// DefaultLang is the pair assumed when none is known — the column's default, and
// the only pair that existed before Phase 19.
var DefaultLang = langs["zh"]
// LangFor resolves a users.pair_lang value. An empty or unrecognised code falls
// back to the default rather than erroring: a prompt is not the place to
// discover a configuration problem, and the writing still has to be checked.
func LangFor(code string) Lang {
if l, ok := langs[strings.TrimSpace(code)]; ok {
return l
}
return DefaultLang
}
internal/llm/lang_test.go
+85
View File
@@ -0,0 +1,85 @@
package llm
import (
"strings"
"testing"
)
func TestLangForFallsBackToDefault(t *testing.T) {
if got := LangFor("zh"); got.Code != "zh" {
t.Fatalf("LangFor(zh) = %+v", got)
}
if got := LangFor("pt-PT"); got.Code != "pt-PT" {
t.Fatalf("LangFor(pt-PT) = %+v", got)
}
// A blank column, a stray value, and stray whitespace all resolve rather
// than erroring — a prompt is the wrong place to discover a config problem.
for _, in := range []string{"", " ", "klingon", "ZH"} {
if got := LangFor(in); got.Code != DefaultLang.Code {
t.Fatalf("LangFor(%q) = %q, want the default %q", in, got.Code, DefaultLang.Code)
}
}
if got := LangFor(" zh "); got.Code != "zh" {
t.Fatalf("LangFor with padding = %+v", got)
}
}
// The three prompts that name the writer's language must actually name *hers*.
// Before Phase 19 all three said "Simplified Chinese" outright, which is the
// bug this guards: a pt-PT writer asking "porquê" would have been answered in
// Mandarin.
func TestPromptsNameTheWritersLanguage(t *testing.T) {
pt := LangFor("pt-PT")
collocation := CollocationMessages("The rain was strong.", "casual", pt)[0].Content
if !strings.Contains(collocation, "European Portuguese") {
t.Fatalf("collocation prompt doesn't ask for a pt-PT gloss:\n%s", collocation)
}
if strings.Contains(collocation, "Simplified Chinese") {
t.Fatalf("collocation prompt still hardcodes Chinese:\n%s", collocation)
}
// The tone steering must survive alongside the language — they share one
// format string, and getting the verbs in the wrong order silently drops one.
if !strings.Contains(collocation, "relaxed, friendly, and conversational") {
t.Fatalf("collocation prompt lost its tone guidance:\n%s", collocation)
}
translate := TranslateMessages("Try a shorter sentence here.", pt)[0].Content
if !strings.Contains(translate, "European Portuguese") || strings.Contains(translate, "Chinese") {
t.Fatalf("translate prompt targets the wrong language:\n%s", translate)
}
ask := AskPetalSystemPrompt("origin", "replacement", "grammar", "explanation", "paragraph", pt)
if !strings.Contains(ask, "European Portuguese") || strings.Contains(ask, "Mandarin") {
t.Fatalf("ask-petal prompt targets the wrong language:\n%s", ask)
}
if !strings.Contains(ask, "porquê") {
t.Fatalf("ask-petal prompt doesn't recognise her word for \"why\":\n%s", ask)
}
// The suggestion context is positional in that template; a mis-numbered
// verb would quietly blank one of these fields.
for _, want := range []string{"origin", "replacement", "grammar", "explanation", "paragraph"} {
if !strings.Contains(ask, want) {
t.Fatalf("ask-petal prompt dropped %q:\n%s", want, ask)
}
}
if strings.Contains(ask, "%!") {
t.Fatalf("ask-petal prompt has a formatting error:\n%s", ask)
}
}
// The zh pair is in daily use and must be untouched by the extraction: its
// prompts should read exactly as they did when they were hardcoded.
func TestDefaultPairStillReadsAsBefore(t *testing.T) {
zh := LangFor("zh")
if got := CollocationMessages("x", "", zh)[0].Content; !strings.Contains(got, "Simplified Chinese (Mandarin) gloss in parentheses") {
t.Fatalf("zh collocation gloss changed:\n%s", got)
}
if got := TranslateMessages("x", zh)[0].Content; !strings.Contains(got, "natural, friendly Simplified Chinese (Mandarin)") {
t.Fatalf("zh translate target changed:\n%s", got)
}
if got := AskPetalSystemPrompt("a", "b", "c", "d", "e", zh); !strings.Contains(got, "为什么") {
t.Fatalf("zh ask-petal lost its Mandarin \"why\":\n%s", got)
}
}
internal/llm/prompts.go
+30 -27
View File
@@ -97,8 +97,8 @@ func VoiceMessages(contentText string) []Message {
// just non-native ("do a decision" → "make a decision", "strong rain" → "heavy
// rain"), and explicitly DEFERS real grammar/spelling errors to the grammar
// checkpoint so the two families don't overlap. Every explanation is framed as a
// warm "natives usually say…" note with a short Mandarin gloss — never
// "error/wrong" — because these are stylistic, not mistakes. It is a distinct
// warm "natives usually say…" note with a short gloss in the writer's own
// language — never "error/wrong" — because these are stylistic, not mistakes. It is a distinct
// pass from the grammar checkpoint (do not bundle them). `replacement` carries
// the natural pairing the writer can accept in one tap.
const collocationSystemPrompt = `You are a warm, encouraging writing assistant helping someone who speaks English as a second language. ` +
@@ -113,7 +113,7 @@ Identify up to 5 such non-native word pairings. For each, give the natural pairi
`Be gentle and specific. Do NOT flag grammar errors, spelling mistakes, or unclear sentences — those are handled ` +
`elsewhere. Only flag word pairings that are correct but sound non-native.%s
Phrase every explanation warmly as "Natives usually say…" and include a brief Simplified Chinese gloss in parentheses. ` +
Phrase every explanation warmly as "Natives usually say…" and include a brief %s gloss in parentheses. ` +
`Never use the words "error", "wrong", or "mistake" — these are friendly polish, not corrections.
Respond ONLY with valid JSON. No preamble, no markdown fences. Format:
@@ -132,10 +132,11 @@ If every pairing already sounds natural, return: {"suggestions": []}`
// CollocationMessages builds the message array for a collocation pass over the
// WHOLE document (no truncation), gently steered toward the document's tone so a
// hint can prefer a register-appropriate pairing.
func CollocationMessages(contentText, tone string) []Message {
// hint can prefer a register-appropriate pairing. The parenthetical gloss is
// written in the writer's own language.
func CollocationMessages(contentText, tone string, lang Lang) []Message {
return []Message{
{Role: "system", Content: fmt.Sprintf(collocationSystemPrompt, toneGuidance(tone))},
{Role: "system", Content: fmt.Sprintf(collocationSystemPrompt, toneGuidance(tone), lang.Name)},
{Role: "user", Content: contentText},
}
}
@@ -147,26 +148,27 @@ const askPetalSystemTemplate = `You are Petal, a warm and patient English writin
`as a second language. You are currently discussing a specific writing suggestion.
Suggestion context:
- Original text: "%s"
- Suggested replacement: "%s"
- Issue type: %s
- Initial explanation: "%s"
- Surrounding paragraph: "%s"
- Original text: "%[1]s"
- Suggested replacement: "%[2]s"
- Issue type: %[3]s
- Initial explanation: "%[4]s"
- Surrounding paragraph: "%[5]s"
The user wants to understand this suggestion better. Detect the language of the user's message ` +
`and respond in that same language. If they write in Mandarin Chinese, respond entirely in ` +
`Mandarin. If they write in English, respond in English. Never mix languages in a single response.
`and respond in that same language. If they write in %[6]s, respond entirely in ` +
`%[6]s. If they write in English, respond in English. Never mix languages in a single response.
Explain clearly and kindly. Use simple language appropriate to the user's message. Give examples ` +
`when helpful. If they ask "why" (or "为什么"), explain the grammar rule or idiom behind it. ` +
`when helpful. If they ask "why" (or "%[7]s"), explain the grammar rule or idiom behind it. ` +
`If they suggest an alternative phrasing, evaluate it honestly.
Keep responses concise (2-4 sentences). This is a chat, not an essay. Be encouraging — ` +
`learning a language is hard and they're doing great.`
// AskPetalSystemPrompt fills the tutor prompt with one suggestion's context.
func AskPetalSystemPrompt(original, replacement, suggestionType, explanation, paragraph string) string {
return fmt.Sprintf(askPetalSystemTemplate, original, replacement, suggestionType, explanation, paragraph)
// AskPetalSystemPrompt fills the tutor prompt with one suggestion's context and
// the writer's pair language, which is the one she may ask her question in.
func AskPetalSystemPrompt(original, replacement, suggestionType, explanation, paragraph string, lang Lang) string {
return fmt.Sprintf(askPetalSystemTemplate, original, replacement, suggestionType, explanation, paragraph, lang.Name, lang.Why)
}
// rewriteSystemTemplate drives the "say it more naturally" / tone-rewrite tool.
@@ -215,22 +217,23 @@ func RewriteMessages(text, style string) []Message {
}
// translateSystemPrompt drives the explanation translator: it renders a
// suggestion's English explanation into Simplified Chinese so an ESL reader sees
// the "why" in her first language. Strict about returning ONLY the translation
// (no quotes, no pinyin, no English echo) so it can drop straight into the chat
// bubble. Kept warm and plain — these are short, friendly one-liners.
// suggestion's English explanation into the writer's own language so an ESL
// reader sees the "why" in her first language. Strict about returning ONLY the
// translation (no quotes, no romanisation, no English echo) so it can drop
// straight into the chat bubble. Kept warm and plain — these are short, friendly
// one-liners.
const translateSystemPrompt = `You are Petal, a warm writing assistant. Translate the English text the user ` +
`sends into natural, friendly Simplified Chinese (Mandarin). It is a short explanation of a writing ` +
`suggestion, written for a native Chinese speaker learning English.
`sends into natural, friendly %[1]s. It is a short explanation of a writing ` +
`suggestion, written for a native %[1]s speaker learning English.
Respond with ONLY the Simplified Chinese translation. No quotation marks, no pinyin, no English, no preamble — ` +
Respond with ONLY the %[1]s translation. No quotation marks, no romanisation, no English, no preamble — ` +
`just the translated sentence.`
// TranslateMessages builds the message array for translating one short English
// explanation into Simplified Chinese.
func TranslateMessages(text string) []Message {
// explanation into the writer's own language.
func TranslateMessages(text string, lang Lang) []Message {
return []Message{
{Role: "system", Content: translateSystemPrompt},
{Role: "system", Content: fmt.Sprintf(translateSystemPrompt, lang.Name)},
{Role: "user", Content: text},
}
}
internal/llm/translate.go
+5 -5
View File
@@ -4,13 +4,13 @@ import (
"context"
)
// RunTranslate renders a short English explanation into Simplified Chinese. It
// is a one-shot Complete (the result seeds the Ask Petal bubble), kept at a low
// temperature so the translation is faithful rather than creative. Output is
// RunTranslate renders a short English explanation into the writer's own
// language. It is a one-shot Complete (the result seeds the Ask Petal bubble),
// kept at a low temperature so the translation is faithful rather than creative. Output is
// trimmed of any stray surrounding quotes the model may add.
func RunTranslate(ctx context.Context, client LLMClient, text string) (string, error) {
func RunTranslate(ctx context.Context, client LLMClient, text string, lang Lang) (string, error) {
out, err := client.Complete(ctx, CompletionRequest{
Messages: TranslateMessages(text),
Messages: TranslateMessages(text, lang),
MaxTokens: 512,
Temperature: 0.2,
TopP: 0.9,
internal/llm/vllm.go
+8
View File
@@ -28,6 +28,12 @@ type vllmRequest struct {
TopP float64 `json:"top_p"`
Stop []string `json:"stop,omitempty"`
Stream bool `json:"stream"`
// ChatTemplateKwargs is a vLLM extension. Qwen3-family models reason by
// default and prepend a plain-text preamble ("Here's a thinking process:")
// ahead of the answer — not a <think> block, so it cannot be stripped after
// the fact. Every Petal pass parses a JSON object out of the completion, so
// an unsuppressed preamble fails the parse outright.
ChatTemplateKwargs map[string]any `json:"chat_template_kwargs,omitempty"`
}
func (c *VLLMClient) body(req CompletionRequest) vllmRequest {
@@ -40,6 +46,8 @@ func (c *VLLMClient) body(req CompletionRequest) vllmRequest {
TopP: req.TopP,
Stop: req.Stop,
Stream: req.Stream,
ChatTemplateKwargs: map[string]any{"enable_thinking": false},
}
}
internal/llm/voice.go
+1 -1
View File
@@ -19,7 +19,7 @@ const VoiceInterval = 20 * time.Second
// The tone argument is accepted for a uniform pass signature but ignored: voice
// consistency is judged against the document's own established voice, not an
// externally-chosen register.
func RunVoice(ctx context.Context, client LLMClient, contentText, _ string) ([]RawSuggestion, error) {
func RunVoice(ctx context.Context, client LLMClient, contentText, _ string, _ Lang) ([]RawSuggestion, error) {
raw, err := client.Complete(ctx, CompletionRequest{
Messages: VoiceMessages(contentText),
MaxTokens: 2048,
internal/spell/handlers.go
+209
View File
@@ -0,0 +1,209 @@
// Package spell owns the personal spelling dictionary — the words a writer has
// told Petal to stop flagging.
//
// It lived in the browser's localStorage until Phase 18, which was wrong twice
// over: two people sharing a device shared one list (built from one person's
// private writing), and one person writing on a laptop and a tablet had two
// lists that never met. It is a small amount of state, but it is *her* state,
// so it belongs to her account rather than to a browser profile.
//
// Everything here is scoped by `lang` as well as by user. That is the language
// of the *dictionary* the word was accepted against, not the writer's own.
//
// Phase 18 justified that key by saying an en-US personal word must not silence
// a pt-PT flag once the second pair shipped. Phase 21 shipped it and the
// justification did not survive: under the both-dictionaries rule
// (SUGGESTIONS.md §3a) a word is only ever flagged when *every* loaded
// dictionary rejected it, so there is no such thing as a pt-PT flag an English
// exception could silence. What the key is actually good for is narrower and
// still worth having — the rows say which dictionary each acceptance was made
// against, so a pair that later loses or gains a dictionary keeps a truthful
// record instead of one merged list of unknown provenance. The browser writes a
// row per loaded dictionary when she accepts a word; see useSpellChecker.
package spell
import (
"encoding/json"
"net/http"
"strings"
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/httputil"
)
// DefaultLang is the dictionary assumed when a caller doesn't name one. English
// is in every pair, so it is the safe assumption; pt-PT is named explicitly by
// the pt-PT pair's second dictionary.
const DefaultLang = "en"
// MaxWordLen bounds a single entry. A personal dictionary holds words, and a
// pasted paragraph is a bug (or an attempt to use the table as storage).
const MaxWordLen = 80
// MaxBatch bounds one request. The only bulk caller is the one-time adoption of
// a browser's pre-Phase-18 list, which is realistically tens of words.
const MaxBatch = 500
// Handler owns the /api/spell routes.
type Handler struct {
DB *db.DB
}
func New(database *db.DB) *Handler { return &Handler{DB: database} }
// Routes mounts the personal-dictionary endpoints under /api/spell.
func (h *Handler) Routes() chi.Router {
r := chi.NewRouter()
r.Get("/words", h.list)
r.Post("/words", h.add)
r.Delete("/words", h.remove)
return r
}
type wordsResponse struct {
Lang string `json:"lang"`
Words []string `json:"words"`
}
type addRequest struct {
Lang string `json:"lang"`
// Word and Words are both accepted so the everyday "add this one word" call
// stays obvious while the one-shot migration of a browser's old list is a
// single request rather than one per word.
Word string `json:"word"`
Words []string `json:"words"`
}
// normLang keeps the dictionary tag in one canonical shape so "EN", "en" and a
// missing value can never split one list into three.
func normLang(lang string) string {
lang = strings.ToLower(strings.TrimSpace(lang))
if lang == "" {
return DefaultLang
}
return lang
}
// list returns the caller's words for one dictionary, alphabetically so the
// order is stable between requests.
func (h *Handler) list(w http.ResponseWriter, r *http.Request) {
lang := normLang(r.URL.Query().Get("lang"))
words, err := h.fetch(auth.UserID(r.Context()), lang)
if err != nil {
httputil.ServerError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, wordsResponse{Lang: lang, Words: words})
}
// add inserts one or more words, idempotently, and answers with the resulting
// full list — so the client never has to merge two views of the same set.
func (h *Handler) add(w http.ResponseWriter, r *http.Request) {
var req addRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
httputil.BadRequest(w, "invalid JSON body")
return
}
lang := normLang(req.Lang)
incoming := req.Words
if req.Word != "" {
incoming = append(incoming, req.Word)
}
clean := make([]string, 0, len(incoming))
for _, word := range incoming {
word = strings.TrimSpace(word)
if word == "" || len([]rune(word)) > MaxWordLen {
continue
}
clean = append(clean, word)
}
if len(clean) == 0 {
httputil.BadRequest(w, "no word given")
return
}
if len(clean) > MaxBatch {
httputil.BadRequest(w, "too many words in one request")
return
}
userID := auth.UserID(r.Context())
tx, err := h.DB.Begin()
if err != nil {
httputil.ServerError(w, err)
return
}
defer func() { _ = tx.Rollback() }()
for _, word := range clean {
if _, err := tx.Exec(
`INSERT INTO personal_words (user_id, lang, word) VALUES (?, ?, ?)
ON CONFLICT(user_id, lang, word) DO NOTHING`,
userID, lang, word,
); err != nil {
httputil.ServerError(w, err)
return
}
}
if err := tx.Commit(); err != nil {
httputil.ServerError(w, err)
return
}
words, err := h.fetch(userID, lang)
if err != nil {
httputil.ServerError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, wordsResponse{Lang: lang, Words: words})
}
// remove forgets one word. Deleting something that was never there is a success:
// the caller's intent — "this word is not in my dictionary" — already holds.
func (h *Handler) remove(w http.ResponseWriter, r *http.Request) {
word := strings.TrimSpace(r.URL.Query().Get("word"))
if word == "" {
httputil.BadRequest(w, "no word given")
return
}
lang := normLang(r.URL.Query().Get("lang"))
userID := auth.UserID(r.Context())
if _, err := h.DB.Exec(
`DELETE FROM personal_words WHERE user_id = ? AND lang = ? AND word = ?`,
userID, lang, word,
); err != nil {
httputil.ServerError(w, err)
return
}
words, err := h.fetch(userID, lang)
if err != nil {
httputil.ServerError(w, err)
return
}
httputil.WriteJSON(w, http.StatusOK, wordsResponse{Lang: lang, Words: words})
}
// fetch reads one (user, dictionary) list. Both keys are always bound — an
// unscoped read here would hand one writer another's private vocabulary.
func (h *Handler) fetch(userID, lang string) ([]string, error) {
rows, err := h.DB.Query(
`SELECT word FROM personal_words WHERE user_id = ? AND lang = ? ORDER BY word`,
userID, lang,
)
if err != nil {
return nil, err
}
defer rows.Close()
words := []string{} // never nil: the client expects a list, not null
for rows.Next() {
var word string
if err := rows.Scan(&word); err != nil {
return nil, err
}
words = append(words, word)
}
return words, rows.Err()
}
internal/spell/handlers_test.go
+213
View File
@@ -0,0 +1,213 @@
package spell
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"path/filepath"
"testing"
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// newTestServer mounts the routes behind the same auth middleware main.go
// installs — a bare router resolves no caller, so every scoped query would
// silently match nothing.
func newTestServer(t *testing.T) (http.Handler, *db.DB) {
t.Helper()
database, err := db.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatalf("open db: %v", err)
}
t.Cleanup(func() { database.Close() })
r := chi.NewRouter()
r.Mount("/spell", New(database).Routes())
return auth.Middleware(auth.StaticResolver(db.LocalUserID))(r), database
}
func do(t *testing.T, srv http.Handler, method, path, body string) *httptest.ResponseRecorder {
t.Helper()
var r *http.Request
if body != "" {
r = httptest.NewRequest(method, path, bytes.NewBufferString(body))
} else {
r = httptest.NewRequest(method, path, nil)
}
rec := httptest.NewRecorder()
srv.ServeHTTP(rec, r)
return rec
}
func decodeWords(t *testing.T, rec *httptest.ResponseRecorder) wordsResponse {
t.Helper()
if rec.Code != http.StatusOK {
t.Fatalf("code=%d body=%s", rec.Code, rec.Body)
}
var got wordsResponse
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("decode: %v (body %s)", err, rec.Body)
}
return got
}
func equal(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
// TestLifecycle walks add → list → re-add → delete, and asserts the two
// properties the client relies on: adds are idempotent, and every response
// carries the full resulting list so the browser never has to merge.
func TestLifecycle(t *testing.T) {
srv, _ := newTestServer(t)
// Empty to start, and a list, never null.
got := decodeWords(t, do(t, srv, http.MethodGet, "/spell/words", ""))
if got.Lang != "en" || len(got.Words) != 0 {
t.Fatalf("fresh list = %+v, want empty en", got)
}
if !bytes.Contains(
do(t, srv, http.MethodGet, "/spell/words", "").Body.Bytes(), []byte(`"words":[]`),
) {
t.Fatal("empty list encoded as null, not []")
}
// One word, then a bulk add (the shape the browser's one-time adoption uses).
got = decodeWords(t, do(t, srv, http.MethodPost, "/spell/words", `{"word":"Petal"}`))
if !equal(got.Words, []string{"Petal"}) {
t.Fatalf("after add = %v", got.Words)
}
got = decodeWords(t, do(t, srv, http.MethodPost, "/spell/words",
`{"words":["hanfu","qipao","Petal"]}`))
if !equal(got.Words, []string{"Petal", "hanfu", "qipao"}) {
t.Fatalf("after bulk add = %v, want sorted and de-duplicated", got.Words)
}
// Re-adding an existing word must not error or duplicate it.
got = decodeWords(t, do(t, srv, http.MethodPost, "/spell/words", `{"word":"hanfu"}`))
if !equal(got.Words, []string{"Petal", "hanfu", "qipao"}) {
t.Fatalf("re-add changed the list: %v", got.Words)
}
// Delete, then delete again — forgetting a word Petal never knew is a
// success, since the caller's intent already holds.
got = decodeWords(t, do(t, srv, http.MethodDelete, "/spell/words?word=qipao", ""))
if !equal(got.Words, []string{"Petal", "hanfu"}) {
t.Fatalf("after delete = %v", got.Words)
}
got = decodeWords(t, do(t, srv, http.MethodDelete, "/spell/words?word=qipao", ""))
if !equal(got.Words, []string{"Petal", "hanfu"}) {
t.Fatalf("repeat delete = %v", got.Words)
}
}
// TestLanguagesDoNotMerge is the reason `lang` is in the primary key: a word the
// writer excused in English must not silence the pt-PT dictionary too.
func TestLanguagesDoNotMerge(t *testing.T) {
srv, _ := newTestServer(t)
do(t, srv, http.MethodPost, "/spell/words", `{"word":"tarde"}`)
got := decodeWords(t, do(t, srv, http.MethodPost, "/spell/words",
`{"lang":"pt-PT","word":"tarde"}`))
if !equal(got.Words, []string{"tarde"}) || got.Lang != "pt-pt" {
t.Fatalf("pt list = %+v", got)
}
// Removing it from one dictionary leaves the other alone.
do(t, srv, http.MethodDelete, "/spell/words?lang=pt-PT&word=tarde", "")
if got = decodeWords(t, do(t, srv, http.MethodGet, "/spell/words?lang=pt-PT", "")); len(got.Words) != 0 {
t.Fatalf("pt list after delete = %v", got.Words)
}
if got = decodeWords(t, do(t, srv, http.MethodGet, "/spell/words", "")); !equal(got.Words, []string{"tarde"}) {
t.Fatalf("en list collaterally damaged: %v", got.Words)
}
// Case and whitespace in the tag must not split one list into three.
got = decodeWords(t, do(t, srv, http.MethodGet, "/spell/words?lang=EN", ""))
if !equal(got.Words, []string{"tarde"}) {
t.Fatalf("uppercase lang tag saw a different list: %v", got.Words)
}
}
func TestRejectsJunk(t *testing.T) {
srv, _ := newTestServer(t)
cases := []struct{ name, method, path, body string }{
{"empty word", http.MethodPost, "/spell/words", `{"word":" "}`},
{"no word at all", http.MethodPost, "/spell/words", `{"lang":"en"}`},
{"not json", http.MethodPost, "/spell/words", `nonsense`},
{"delete without a word", http.MethodDelete, "/spell/words", ""},
}
for _, c := range cases {
if rec := do(t, srv, c.method, c.path, c.body); rec.Code != http.StatusBadRequest {
t.Errorf("%s: code=%d, want 400", c.name, rec.Code)
}
}
// An over-long entry is dropped rather than stored — a pasted paragraph is
// not a word. Dropping the only entry leaves nothing to add, hence 400.
long := `{"word":"` + string(bytes.Repeat([]byte("a"), MaxWordLen+1)) + `"}`
if rec := do(t, srv, http.MethodPost, "/spell/words", long); rec.Code != http.StatusBadRequest {
t.Errorf("over-long word: code=%d, want 400", rec.Code)
}
}
// TestTwoUsersDoNotShare is the standing rule for every user-scoped endpoint:
// mount the same routes twice behind two resolvers over one database.
func TestTwoUsersDoNotShare(t *testing.T) {
database, err := db.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatalf("open db: %v", err)
}
t.Cleanup(func() { database.Close() })
if _, err := database.Exec(
`INSERT INTO users (id, email, display_name) VALUES (?, ?, ?)`,
"bob", "bob@petal.local", "Bob",
); err != nil {
t.Fatalf("seed second user: %v", err)
}
mount := func(userID string) http.Handler {
r := chi.NewRouter()
r.Mount("/spell", New(database).Routes())
return auth.Middleware(auth.StaticResolver(userID))(r)
}
alice, bob := mount(db.LocalUserID), mount("bob")
do(t, alice, http.MethodPost, "/spell/words", `{"words":["Xiaolan","hanfu"]}`)
// Bob sees none of it — a personal dictionary is built from private writing.
if got := decodeWords(t, do(t, bob, http.MethodGet, "/spell/words", "")); len(got.Words) != 0 {
t.Fatalf("bob sees alice's words: %v", got.Words)
}
// Bob's own identical word is his own row, and deleting it leaves hers.
do(t, bob, http.MethodPost, "/spell/words", `{"word":"hanfu"}`)
do(t, bob, http.MethodDelete, "/spell/words?word=hanfu", "")
if got := decodeWords(t, do(t, alice, http.MethodGet, "/spell/words", "")); !equal(got.Words, []string{"Xiaolan", "hanfu"}) {
t.Fatalf("bob's delete reached alice's list: %v", got.Words)
}
// Deleting the account takes the dictionary with it.
if _, err := database.Exec(`DELETE FROM users WHERE id = 'bob'`); err != nil {
t.Fatalf("delete user: %v", err)
}
var n int
if err := database.QueryRow(
`SELECT COUNT(*) FROM personal_words WHERE user_id = 'bob'`).Scan(&n); err != nil {
t.Fatalf("count: %v", err)
}
if n != 0 {
t.Fatalf("%d orphaned rows after the user was deleted", n)
}
}
internal/suggestions/chat.go
+8 -5
View File
@@ -9,7 +9,7 @@ import (
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/httputil"
"gitea.parodia.dev/drwily/petal/internal/llm"
)
@@ -40,14 +40,17 @@ func (h *Handler) chat(w http.ResponseWriter, r *http.Request) {
original, replacement, explanation, typ string
fromPos int
contentText string
pairLang string
)
err := h.DB.QueryRow(
`SELECT s.original, s.replacement, s.explanation, s.type, s.from_pos, d.content_text
`SELECT s.original, s.replacement, s.explanation, s.type, s.from_pos, d.content_text,
COALESCE(u.pair_lang, '')
FROM suggestions s
JOIN documents d ON d.id = s.doc_id
JOIN users u ON u.id = d.user_id
WHERE s.id = ? AND d.user_id = ?`,
sugID, db.LocalUserID,
).Scan(&original, &replacement, &explanation, &typ, &fromPos, &contentText)
sugID, auth.UserID(r.Context()),
).Scan(&original, &replacement, &explanation, &typ, &fromPos, &contentText, &pairLang)
if errors.Is(err, sql.ErrNoRows) {
httputil.ErrorJSON(w, http.StatusNotFound, "suggestion not found")
return
@@ -58,7 +61,7 @@ func (h *Handler) chat(w http.ResponseWriter, r *http.Request) {
}
paragraph := surroundingParagraph(contentText, fromPos)
systemPrompt := llm.AskPetalSystemPrompt(original, replacement, typ, explanation, paragraph)
systemPrompt := llm.AskPetalSystemPrompt(original, replacement, typ, explanation, paragraph, llm.LangFor(pairLang))
// SSE requires an unbuffered, flushable writer. chi's middleware writers pass
// Flush through; bail with a plain error if somehow they don't.
internal/suggestions/growth.go
+230
View File
@@ -0,0 +1,230 @@
package suggestions
import (
"net/http"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/httputil"
"gitea.parodia.dev/drwily/petal/internal/vocab"
)
// The growth journal.
//
// The suggestions table already records everything this needs — it is purely a
// read-side view, with no new capture and no model call. Two framing rules
// decide what may appear here, and they are enforced in the SQL rather than left
// to the copy:
//
// 1. It reports growth, never an error tally. Nothing counts what she got
// wrong this month; the signals are things that *stopped* happening and
// phrasing that *stuck*.
// 2. It only ever compares the writer to her own past self. There is no
// target, no average, no other user anywhere in these queries.
//
// A quiet month is quiet: every signal below is omitted rather than softened
// when the data isn't there, because an invented milestone is worse than none.
// Journal is one writer's growth over the recent windows.
type Journal struct {
// Kept / KeptBefore are edits she took on board in the last 30 days and in
// the 30 before that — her own past self, the only comparison offered.
Kept int `json:"kept"`
KeptBefore int `json:"kept_before"`
// Stuck: phrasing she was given that now turns up across her own writing.
Stuck []Chunk `json:"stuck"`
// Faded: things she used to need fixing and hasn't, recently.
Faded []Fade `json:"faded"`
}
// Chunk is a phrase that has stuck: it appears in Docs of her documents now.
type Chunk struct {
Phrase string `json:"phrase"`
Docs int `json:"docs"`
}
// Fade is a pattern that has stopped appearing. Times is how often it came up
// during the earlier window — context for "and not since", never a scoreboard.
type Fade struct {
Pattern string `json:"pattern"`
Times int `json:"times"`
}
// Journal windows, in days. `recent` is the month being reported on; `history`
// reaches back far enough that a pattern's absence means something (one quiet
// fortnight doesn't).
const (
recentDays = 30
historyDays = 120
maxSignals = 3 // per list: a journal is a couple of warm lines, not a report
)
// growth serves GET /api/suggestions/growth.
func (h *Handler) growth(w http.ResponseWriter, r *http.Request) {
userID := auth.UserID(r.Context())
j := Journal{Stuck: []Chunk{}, Faded: []Fade{}}
err := h.DB.QueryRow(
`SELECT
sum(CASE WHEN s.resolved_at >= datetime('now', '-30 days') THEN 1 ELSE 0 END),
sum(CASE WHEN s.resolved_at < datetime('now', '-30 days')
AND s.resolved_at >= datetime('now', '-60 days') THEN 1 ELSE 0 END)
FROM suggestions s JOIN documents d ON d.id = s.doc_id
WHERE d.user_id = ? AND s.status = 'accepted' AND s.resolved_at IS NOT NULL`,
userID,
).Scan(&nullInt{&j.Kept}, &nullInt{&j.KeptBefore})
if err != nil {
httputil.ServerError(w, err)
return
}
stuck, err := h.stuck(userID)
if err != nil {
httputil.ServerError(w, err)
return
}
j.Stuck = stuck
faded, err := h.faded(userID)
if err != nil {
httputil.ServerError(w, err)
return
}
j.Faded = faded
httputil.WriteJSON(w, http.StatusOK, j)
}
// stuck finds accepted phrasing that now appears in more than one of her own
// documents. One document is just the edit itself, still sitting where it was
// applied; a second is her reaching for the phrase on her own, which is the
// whole claim the line makes.
func (h *Handler) stuck(userID string) ([]Chunk, error) {
rows, err := h.DB.Query(
`SELECT DISTINCT s.replacement
FROM suggestions s JOIN documents d ON d.id = s.doc_id
WHERE d.user_id = ? AND s.status = 'accepted'
AND s.resolved_at >= datetime('now', '-120 days')
AND trim(s.replacement) != ''
ORDER BY s.resolved_at DESC
LIMIT 40`,
userID,
)
if err != nil {
return nil, err
}
defer rows.Close()
// vocab.PhraseKey is the same definition of "a learnable chunk" the garden
// plants, so the journal and the garden can never disagree about what counts.
var phrases []string
for rows.Next() {
var replacement string
if err := rows.Scan(&replacement); err != nil {
return nil, err
}
if key := vocab.PhraseKey(replacement); key != "" {
phrases = append(phrases, key)
}
}
if err := rows.Err(); err != nil {
return nil, err
}
out := []Chunk{}
for _, p := range phrases {
var docs int
if err := h.DB.QueryRow(
`SELECT count(*) FROM documents WHERE user_id = ? AND instr(lower(content_text), ?) > 0`,
userID, p,
).Scan(&docs); err != nil {
return nil, err
}
if docs >= 2 {
out = append(out, Chunk{Phrase: p, Docs: docs})
}
}
sortDesc(out, func(c Chunk) int { return c.Docs })
return trim(out, maxSignals), nil
}
// faded finds patterns she used to be corrected on during the earlier part of
// the history window and hasn't been since.
//
// The guard that makes this honest: it says nothing at all unless she has
// actually been writing lately. Without it, a month away from Petal would be
// reported back to her as progress, which is the one way this feature could lie.
func (h *Handler) faded(userID string) ([]Fade, error) {
var wroteRecently int
if err := h.DB.QueryRow(
`SELECT count(*) FROM suggestions s JOIN documents d ON d.id = s.doc_id
WHERE d.user_id = ? AND s.resolved_at >= datetime('now', '-30 days')`,
userID,
).Scan(&wroteRecently); err != nil {
return nil, err
}
if wroteRecently == 0 {
return []Fade{}, nil
}
rows, err := h.DB.Query(
`SELECT lower(trim(s.original)) AS pattern, count(*) AS times
FROM suggestions s JOIN documents d ON d.id = s.doc_id
WHERE d.user_id = ? AND s.status = 'accepted'
AND s.resolved_at < datetime('now', '-30 days')
AND s.resolved_at >= datetime('now', '-120 days')
AND trim(s.original) != ''
AND pattern NOT IN (
SELECT lower(trim(s2.original))
FROM suggestions s2 JOIN documents d2 ON d2.id = s2.doc_id
WHERE d2.user_id = ? AND s2.status = 'accepted'
AND s2.resolved_at >= datetime('now', '-30 days'))
GROUP BY pattern
HAVING times >= 2
ORDER BY times DESC
LIMIT 3`,
userID, userID,
)
if err != nil {
return nil, err
}
defer rows.Close()
out := []Fade{}
for rows.Next() {
var f Fade
if err := rows.Scan(&f.Pattern, &f.Times); err != nil {
return nil, err
}
out = append(out, f)
}
return out, rows.Err()
}
// nullInt scans a possibly-NULL aggregate into an int (SUM over no rows is
// NULL, which is a zero here, not an error).
type nullInt struct{ dst *int }
func (n *nullInt) Scan(v any) error {
switch t := v.(type) {
case int64:
*n.dst = int(t)
case nil:
*n.dst = 0
}
return nil
}
func sortDesc[T any](s []T, key func(T) int) {
for i := 1; i < len(s); i++ {
for j := i; j > 0 && key(s[j]) > key(s[j-1]); j-- {
s[j], s[j-1] = s[j-1], s[j]
}
}
}
func trim[T any](s []T, n int) []T {
if len(s) > n {
return s[:n]
}
return s
}
internal/suggestions/growth_test.go
+156
View File
@@ -0,0 +1,156 @@
package suggestions
import (
"encoding/json"
"net/http"
"strconv"
"testing"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// resolved seeds one already-settled suggestion, dated `daysAgo` at the moment
// she decided it (the journal reads decisions, not proposals).
func resolved(t *testing.T, h *Handler, docID, status, original, replacement string, daysAgo int) {
t.Helper()
_, err := h.DB.Exec(
`INSERT INTO suggestions (doc_id, from_pos, to_pos, original, replacement, explanation, type, status, created_at, resolved_at)
VALUES (?, 0, 0, ?, ?, '', 'collocation', ?, datetime('now', ?), datetime('now', ?))`,
docID, original, replacement, status,
"-"+strconv.Itoa(daysAgo)+" days", "-"+strconv.Itoa(daysAgo)+" days",
)
if err != nil {
t.Fatalf("seed resolved suggestion: %v", err)
}
}
func seedDoc(t *testing.T, h *Handler, userID, text string) string {
t.Helper()
var id string
if err := h.DB.QueryRow(
`INSERT INTO documents (user_id, content_text) VALUES (?, ?) RETURNING id`, userID, text,
).Scan(&id); err != nil {
t.Fatalf("seed doc: %v", err)
}
return id
}
func readJournal(t *testing.T, srv http.Handler) Journal {
t.Helper()
rec := do(t, srv, http.MethodGet, "/suggestions/growth", "")
if rec.Code != http.StatusOK {
t.Fatalf("growth: got %d, want 200 (body %s)", rec.Code, rec.Body.String())
}
var j Journal
if err := json.Unmarshal(rec.Body.Bytes(), &j); err != nil {
t.Fatalf("decode journal: %v", err)
}
return j
}
// TestJournalIsEmptyForANewWriter: nothing to report reports nothing. Empty
// lists, not nulls, so the frontend never has to guess.
func TestJournalIsEmptyForANewWriter(t *testing.T) {
srv, _, _ := newTestServer(t, &stubClient{})
j := readJournal(t, srv)
if j.Kept != 0 || j.KeptBefore != 0 || len(j.Stuck) != 0 || len(j.Faded) != 0 {
t.Fatalf("new writer got a journal: %+v", j)
}
}
// TestKeptComparesHerToHerOwnPastSelf.
func TestKeptCountsTwoWindows(t *testing.T) {
srv, docID, h := newTestServer(t, &stubClient{})
for i := 0; i < 3; i++ {
resolved(t, h, docID, "accepted", "do a decision", "make a decision", 5)
}
resolved(t, h, docID, "accepted", "big rain", "heavy rain", 40)
resolved(t, h, docID, "rejected", "no thanks", "no, thank you", 5) // decisions kept only
resolved(t, h, docID, "accepted", "long ago", "long since", 200) // outside both windows
j := readJournal(t, srv)
if j.Kept != 3 {
t.Errorf("Kept = %d, want 3", j.Kept)
}
if j.KeptBefore != 1 {
t.Errorf("KeptBefore = %d, want 1", j.KeptBefore)
}
}
// TestStuckNeedsASecondDocument: a phrase sitting in the one document it was
// applied to has not stuck — it's just the edit, where she left it. A second
// document is her reaching for it herself, which is the claim the line makes.
func TestStuckNeedsASecondDocument(t *testing.T) {
srv, docID, h := newTestServer(t, &stubClient{})
if _, err := h.DB.Exec(`UPDATE documents SET content_text = ? WHERE id = ?`,
"I had to make a decision.", docID); err != nil {
t.Fatalf("set content: %v", err)
}
resolved(t, h, docID, "accepted", "do a decision", "make a decision", 10)
resolved(t, h, docID, "accepted", "do a photo", "take a photo", 10)
if j := readJournal(t, srv); len(j.Stuck) != 0 {
t.Fatalf("one document counted as sticking: %+v", j.Stuck)
}
// She uses it again, elsewhere, on her own.
seedDoc(t, h, db.LocalUserID, "Later I had to Make A Decision about the flat.")
j := readJournal(t, srv)
if len(j.Stuck) != 1 {
t.Fatalf("Stuck = %+v, want just the phrase she reused", j.Stuck)
}
if j.Stuck[0].Phrase != "make a decision" || j.Stuck[0].Docs != 2 {
t.Errorf("Stuck[0] = %+v, want {make a decision 2} (case-insensitive)", j.Stuck[0])
}
}
// TestFadedNeedsRecentWriting is the guard that keeps this feature honest: a
// month away from Petal must never be reported back as progress.
func TestFadedNeedsRecentWriting(t *testing.T) {
srv, docID, h := newTestServer(t, &stubClient{})
resolved(t, h, docID, "accepted", "在 the morning", "in the morning", 60)
resolved(t, h, docID, "accepted", "在 the morning", "in the morning", 55)
if j := readJournal(t, srv); len(j.Faded) != 0 {
t.Fatalf("silence reported as growth: %+v", j.Faded)
}
// She has been writing again this month — now the absence means something.
resolved(t, h, docID, "accepted", "big rain", "heavy rain", 3)
j := readJournal(t, srv)
if len(j.Faded) != 1 || j.Faded[0].Pattern != "在 the morning" || j.Faded[0].Times != 2 {
t.Fatalf("Faded = %+v, want the pattern she stopped needing (twice, back then)", j.Faded)
}
}
// TestFadedExcludesWhatStillHappens: a pattern corrected again this month has
// not faded, however often it came up before.
func TestFadedExcludesWhatStillHappens(t *testing.T) {
srv, docID, h := newTestServer(t, &stubClient{})
resolved(t, h, docID, "accepted", "在 the morning", "in the morning", 60)
resolved(t, h, docID, "accepted", "在 the morning", "in the morning", 55)
resolved(t, h, docID, "accepted", "在 the morning", "in the morning", 2)
if j := readJournal(t, srv); len(j.Faded) != 0 {
t.Fatalf("Faded = %+v, want empty — it still happens", j.Faded)
}
}
// TestJournalIsPerWriter: another account's learning is never anyone else's
// journal, and the only comparison Petal draws is with her own past self.
func TestJournalIsPerWriter(t *testing.T) {
srv, _, h := newTestServer(t, &stubClient{})
if _, err := h.DB.Exec(`INSERT INTO users (id, email) VALUES ('bob', 'bob@example.com')`); err != nil {
t.Fatalf("seed user: %v", err)
}
bobDoc := seedDoc(t, h, "bob", "Bob had to make a decision.")
seedDoc(t, h, "bob", "Bob will make a decision again.")
resolved(t, h, bobDoc, "accepted", "do a decision", "make a decision", 5)
resolved(t, h, bobDoc, "accepted", "big rain", "heavy rain", 60)
resolved(t, h, bobDoc, "accepted", "big rain", "heavy rain", 55)
j := readJournal(t, srv)
if j.Kept != 0 || j.KeptBefore != 0 || len(j.Stuck) != 0 || len(j.Faded) != 0 {
t.Fatalf("bob's learning leaked into the local user's journal: %+v", j)
}
}
internal/suggestions/handlers.go
+172 -48
View File
@@ -11,14 +11,17 @@ import (
"encoding/json"
"errors"
"io"
"log"
"net/http"
"strings"
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/httputil"
"gitea.parodia.dev/drwily/petal/internal/llm"
"gitea.parodia.dev/drwily/petal/internal/vocab"
)
// Handler holds the dependencies for the checkpoint + suggestion routes. The
@@ -59,6 +62,10 @@ func (h *Handler) RegisterDocRoutes(r chi.Router) {
// actions.
func (h *Handler) Routes() chi.Router {
r := chi.NewRouter()
// The growth journal reads the same table these actions write, so it lives
// here rather than growing its own mount. A literal segment, so it can never
// be shadowed by an id.
r.Get("/growth", h.growth)
r.Post("/{id}/accept", h.accept)
r.Post("/{id}/dismiss", h.dismiss)
r.Post("/{id}/chat", h.chat)
@@ -83,6 +90,22 @@ type mechanicsFinding struct {
Original string `json:"original"`
Replacement string `json:"replacement"`
Explanation string `json:"explanation"`
// Which family this offline finding belongs to. Empty (the historical shape)
// means mechanics; the miscollocation rules send 'collocation' so a chunk the
// rule pack caught is indistinguishable from one the coach caught — same
// family, same rail, and the same planting into the garden on accept.
Type string `json:"type"`
}
// localType maps a client-supplied family onto the two an offline rule may claim.
// Anything else — including the empty string older clients send — is mechanics,
// so a stray label can never smuggle a row into an LLM family and survive that
// pass's DELETE.
func localType(t string) string {
if strings.ToLower(strings.TrimSpace(t)) == db.SuggestionTypeCollocation {
return db.SuggestionTypeCollocation
}
return db.SuggestionTypeMechanics
}
// maxMechanicsFindings caps a single submission so a runaway client can't flood
@@ -98,11 +121,11 @@ const maxMechanicsFindings = 500
func (h *Handler) mechanics(w http.ResponseWriter, r *http.Request) {
docID := chi.URLParam(r, "id")
// Confirm the document exists (and is the local user's) for clean 404s.
// Confirm the document exists (and belongs to the caller) for clean 404s.
var exists bool
err := h.DB.QueryRow(
`SELECT EXISTS(SELECT 1 FROM documents WHERE id = ? AND user_id = ?)`,
docID, db.LocalUserID,
docID, auth.UserID(r.Context()),
).Scan(&exists)
if err != nil {
httputil.ServerError(w, err)
@@ -129,7 +152,7 @@ func (h *Handler) mechanics(w http.ResponseWriter, r *http.Request) {
return
}
out, err := h.fetchPending(docID)
out, err := h.fetchPending(auth.UserID(r.Context()), docID)
if err != nil {
httputil.ServerError(w, err)
return
@@ -137,10 +160,16 @@ func (h *Handler) mechanics(w http.ResponseWriter, r *http.Request) {
httputil.WriteJSON(w, http.StatusOK, out)
}
// replaceMechanics swaps the document's pending mechanics rows for the supplied
// replaceMechanics swaps the document's pending offline rows for the supplied
// findings in one transaction, leaving the LLM families and actioned rows
// untouched. Findings the user already accepted or dismissed are suppressed (the
// detector has no memory between runs), and malformed spans are skipped.
//
// The DELETE is scoped by *source*, not by type: the rule pack owns both the
// mechanics family and its share of the collocation family, and every run is a
// full recompute of the document, so everything it wrote last time goes. Scoping
// by type instead would strand offline collocations the current text no longer
// warrants — the one row nobody would ever replace.
func (h *Handler) replaceMechanics(docID string, findings []mechanicsFinding) error {
tx, err := h.DB.Begin()
if err != nil {
@@ -149,8 +178,8 @@ func (h *Handler) replaceMechanics(docID string, findings []mechanicsFinding) er
defer tx.Rollback()
if _, err := tx.Exec(
`DELETE FROM suggestions WHERE doc_id = ? AND status = ? AND type = ?`,
docID, db.SuggestionStatusPending, db.SuggestionTypeMechanics,
`DELETE FROM suggestions WHERE doc_id = ? AND status = ? AND source = ?`,
docID, db.SuggestionStatusPending, db.SuggestionSourceLocal,
); err != nil {
return err
}
@@ -168,9 +197,10 @@ func (h *Handler) replaceMechanics(docID string, findings []mechanicsFinding) er
continue
}
if _, err := tx.Exec(
`INSERT INTO suggestions (doc_id, from_pos, to_pos, original, replacement, explanation, type)
VALUES (?, ?, ?, ?, ?, ?, ?)`,
docID, f.From, f.To, f.Original, f.Replacement, f.Explanation, db.SuggestionTypeMechanics,
`INSERT INTO suggestions (doc_id, from_pos, to_pos, original, replacement, explanation, type, source)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
docID, f.From, f.To, f.Original, f.Replacement, f.Explanation,
localType(f.Type), db.SuggestionSourceLocal,
); err != nil {
return err
}
@@ -193,9 +223,12 @@ func (h *Handler) collocation(w http.ResponseWriter, r *http.Request) {
}
// pass is the signature shared by the grammar checkpoint and the voice pass:
// given the document text and the document's tone it returns the model's raw
// suggestions. The voice pass ignores tone (see llm.RunVoice).
type pass func(ctx context.Context, client llm.LLMClient, contentText, tone string) ([]llm.RawSuggestion, error)
// given the document text, the document's tone and the writer's pair language it
// returns the model's raw suggestions. The voice pass ignores both extras (see
// llm.RunVoice) and the checkpoint ignores the language — only the collocation
// coach writes a word of it — but one signature keeps runPass free of special
// cases.
type pass func(ctx context.Context, client llm.LLMClient, contentText, tone string, lang llm.Lang) ([]llm.RawSuggestion, error)
// runPass is the shared body for both LLM passes. It loads the document text,
// enforces the pass's per-document rate limit, runs the model, swaps in the
@@ -203,12 +236,19 @@ type pass func(ctx context.Context, client llm.LLMClient, contentText, tone stri
// (both families) so the client always renders a unified picture.
func (h *Handler) runPass(w http.ResponseWriter, r *http.Request, limiter *llm.RateLimiter, run pass, scope pendingScope) {
docID := chi.URLParam(r, "id")
userID := auth.UserID(r.Context())
var contentText, tone string
// The writer's pair language rides along with the document rather than in a
// second query: it is read from the same row-scoped lookup that already
// proves she owns this document.
var contentText, tone, pairLang string
err := h.DB.QueryRow(
`SELECT content_text, tone FROM documents WHERE id = ? AND user_id = ?`,
docID, db.LocalUserID,
).Scan(&contentText, &tone)
`SELECT d.content_text, d.tone, COALESCE(u.pair_lang, '')
FROM documents d
JOIN users u ON u.id = d.user_id
WHERE d.id = ? AND d.user_id = ?`,
docID, userID,
).Scan(&contentText, &tone, &pairLang)
if errors.Is(err, sql.ErrNoRows) {
httputil.ErrorJSON(w, http.StatusNotFound, "document not found")
return
@@ -228,7 +268,7 @@ func (h *Handler) runPass(w http.ResponseWriter, r *http.Request, limiter *llm.R
if !ok {
// Throttled: return the existing pending set unchanged rather than an
// error, so the frontend keeps showing current suggestions.
existing, err := h.fetchPending(docID)
existing, err := h.fetchPending(userID, docID)
if err != nil {
httputil.ServerError(w, err)
return
@@ -237,7 +277,7 @@ func (h *Handler) runPass(w http.ResponseWriter, r *http.Request, limiter *llm.R
return
}
raw, err := run(r.Context(), h.Client, contentText, tone)
raw, err := run(r.Context(), h.Client, contentText, tone, llm.LangFor(pairLang))
if err != nil {
// Allow ran before the model call, so a failed pass would otherwise hold
// the per-document slot for the full interval — stranding the frontend's
@@ -255,7 +295,7 @@ func (h *Handler) runPass(w http.ResponseWriter, r *http.Request, limiter *llm.R
// Return the unified pending set (grammar + voice), not just this batch, so
// a grammar check never drops the voice highlights from the client and the
// throttle path above stays consistent with the success path.
out, err := h.fetchPending(docID)
out, err := h.fetchPending(userID, docID)
if err != nil {
httputil.ServerError(w, err)
return
@@ -272,17 +312,22 @@ type pendingScope struct {
forceType string // if set, every inserted row gets this type; else normalizeType
}
// Every scope below is confined to source='llm'. The offline rule pack replaces
// its own rows wholesale on each edit (see replaceMechanics) and its findings
// must survive all three model passes — including the collocation coach, which
// now shares the collocation family with it.
var (
// grammarScope owns the grammar/phrasing/idiom/clarity flags — everything but
// the other self-owned families (voice, collocation, mechanics), which run on
// their own cadence/pass and must survive a grammar checkpoint. Notably the
// deterministic mechanics pass writes its rows in the same /check request just
// before this DELETE runs, so excluding it here is what keeps them alive.
grammarScope = pendingScope{deleteWhere: "type NOT IN ('voice','collocation','mechanics')", forceType: ""}
// voiceScope owns the voice flags only.
voiceScope = pendingScope{deleteWhere: "type = 'voice'", forceType: db.SuggestionTypeVoice}
// collocationScope owns the collocation flags only.
collocationScope = pendingScope{deleteWhere: "type = 'collocation'", forceType: db.SuggestionTypeCollocation}
// the other self-owned families (voice, collocation), which run on their own
// cadence/pass and must survive a grammar checkpoint. Notably the offline pass
// writes its rows in the same /check request just before this DELETE runs, so
// the source clause is also what keeps them alive.
grammarScope = pendingScope{deleteWhere: "source = 'llm' AND type NOT IN ('voice','collocation')", forceType: ""}
// voiceScope owns the model's voice flags only.
voiceScope = pendingScope{deleteWhere: "source = 'llm' AND type = 'voice'", forceType: db.SuggestionTypeVoice}
// collocationScope owns the model's collocation flags only — the rule pack's
// share of the same family is left standing.
collocationScope = pendingScope{deleteWhere: "source = 'llm' AND type = 'collocation'", forceType: db.SuggestionTypeCollocation}
)
// replacePending swaps a document's pending suggestions within one family for a
@@ -324,9 +369,9 @@ func (h *Handler) replacePending(docID, contentText string, raw []llm.RawSuggest
}
from, to := locate(contentText, s.Original)
if _, err := tx.Exec(
`INSERT INTO suggestions (doc_id, from_pos, to_pos, original, replacement, explanation, type)
VALUES (?, ?, ?, ?, ?, ?, ?)`,
docID, from, to, s.Original, s.Replacement, s.Explanation, typ,
`INSERT INTO suggestions (doc_id, from_pos, to_pos, original, replacement, explanation, type, source)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
docID, from, to, s.Original, s.Replacement, s.Explanation, typ, db.SuggestionSourceLLM,
); err != nil {
return err
}
@@ -465,7 +510,7 @@ func buildSuppressor(tx *sql.Tx, docID string) (suppressor, error) {
// listForDoc returns the document's current pending suggestions (used when the
// editor loads a document, before any new checkpoint fires).
func (h *Handler) listForDoc(w http.ResponseWriter, r *http.Request) {
out, err := h.fetchPending(chi.URLParam(r, "id"))
out, err := h.fetchPending(auth.UserID(r.Context()), chi.URLParam(r, "id"))
if err != nil {
httputil.ServerError(w, err)
return
@@ -473,13 +518,19 @@ func (h *Handler) listForDoc(w http.ResponseWriter, r *http.Request) {
httputil.WriteJSON(w, http.StatusOK, out)
}
func (h *Handler) fetchPending(docID string) ([]db.Suggestion, error) {
// fetchPending loads a document's pending suggestions, joined through documents
// so the rows are only reachable by the document's owner. A suggestion quotes the
// sentence it corrects, so an unscoped read here would leak document text to
// anyone holding a doc id.
func (h *Handler) fetchPending(userID, docID string) ([]db.Suggestion, error) {
rows, err := h.DB.Query(
`SELECT id, doc_id, from_pos, to_pos, original, replacement, explanation, type, status, created_at
FROM suggestions
WHERE doc_id = ? AND status = ?
ORDER BY from_pos ASC, created_at ASC`,
docID, db.SuggestionStatusPending,
`SELECT s.id, s.doc_id, s.from_pos, s.to_pos, s.original, s.replacement,
s.explanation, s.type, s.status, s.source, s.created_at
FROM suggestions s
JOIN documents d ON d.id = s.doc_id
WHERE s.doc_id = ? AND d.user_id = ? AND s.status = ?
ORDER BY s.from_pos ASC, s.created_at ASC`,
docID, userID, db.SuggestionStatusPending,
)
if err != nil {
return nil, err
@@ -491,7 +542,7 @@ func (h *Handler) fetchPending(docID string) ([]db.Suggestion, error) {
var s db.Suggestion
if err := rows.Scan(
&s.ID, &s.DocID, &s.FromPos, &s.ToPos, &s.Original, &s.Replacement,
&s.Explanation, &s.Type, &s.Status, &s.CreatedAt,
&s.Explanation, &s.Type, &s.Status, &s.Source, &s.CreatedAt,
); err != nil {
return nil, err
}
@@ -503,11 +554,14 @@ func (h *Handler) fetchPending(docID string) ([]db.Suggestion, error) {
return dedupeSpans(out), nil
}
// dedupeSpans resolves collisions between the deterministic mechanics family and
// the LLM families: when a mechanics finding and an LLM suggestion fight over the
// same characters, mechanics wins and the LLM card is dropped. Its span is exact
// (the detector matched it), whereas the LLM positions are only advisory
// (re-anchored by string at render), so the precise fix should own the span.
// dedupeSpans resolves collisions between the offline rule pack and the model:
// when a local finding and an LLM suggestion fight over the same characters, the
// local one wins and the LLM card is dropped. Its span is exact (the detector
// matched it), whereas the LLM positions are only advisory (re-anchored by string
// at render), so the precise fix should own the span. This is why the split is by
// source rather than by type — an offline miscollocation is as exact as an
// offline comma, and the coach's fuzzy version of the same chunk shouldn't
// double up next to it.
//
// This deliberately does NOT dedupe LLM-vs-LLM overlaps: voice (awareness-only,
// no replacement) and collocation legitimately co-occupy the same span, and that
@@ -517,7 +571,7 @@ func dedupeSpans(in []db.Suggestion) []db.Suggestion {
type span struct{ from, to int }
var claimed []span
for _, s := range in {
if s.Type == db.SuggestionTypeMechanics && s.FromPos >= 0 {
if s.Source == db.SuggestionSourceLocal && s.FromPos >= 0 {
claimed = append(claimed, span{s.FromPos, s.ToPos})
}
}
@@ -527,7 +581,7 @@ func dedupeSpans(in []db.Suggestion) []db.Suggestion {
out := make([]db.Suggestion, 0, len(in))
for _, s := range in {
if s.Type != db.SuggestionTypeMechanics && s.FromPos >= 0 {
if s.Source != db.SuggestionSourceLocal && s.FromPos >= 0 {
overlaps := false
for _, sp := range claimed {
if s.FromPos < sp.to && sp.from < s.ToPos {
@@ -536,7 +590,7 @@ func dedupeSpans(in []db.Suggestion) []db.Suggestion {
}
}
if overlaps {
continue // an exact mechanics fix owns these characters
continue // an exact offline fix owns these characters
}
}
out = append(out, s)
@@ -554,10 +608,17 @@ func (h *Handler) dismiss(w http.ResponseWriter, r *http.Request) {
h.setStatus(w, r, db.SuggestionStatusRejected)
}
// setStatus accepts or dismisses one suggestion. The doc_id subquery scopes the
// write to the caller's own documents, so a stray (or guessed) suggestion id
// can't action a row belonging to another account; an unowned id simply affects
// no rows and surfaces as a 404.
func (h *Handler) setStatus(w http.ResponseWriter, r *http.Request, status string) {
res, err := h.DB.Exec(
`UPDATE suggestions SET status = ? WHERE id = ? AND status = ?`,
`UPDATE suggestions SET status = ?, resolved_at = datetime('now')
WHERE id = ? AND status = ?
AND doc_id IN (SELECT id FROM documents WHERE user_id = ?)`,
status, chi.URLParam(r, "id"), db.SuggestionStatusPending,
auth.UserID(r.Context()),
)
if err != nil {
httputil.ServerError(w, err)
@@ -567,9 +628,72 @@ func (h *Handler) setStatus(w http.ResponseWriter, r *http.Request, status strin
httputil.ErrorJSON(w, http.StatusNotFound, "pending suggestion not found")
return
}
if status == db.SuggestionStatusAccepted {
h.plant(chi.URLParam(r, "id"), auth.UserID(r.Context()))
}
w.WriteHeader(http.StatusNoContent)
}
// plant grows an accepted collocation into a vocabulary-garden phrase card. It
// runs after the status write and swallows its own errors: accepting an edit is
// the thing the writer asked for, and it must not fail — or even feel slower —
// because a flashcard couldn't be made.
//
// Only collocations are planted. The other families correct *this* sentence
// ("their" → "there", a comma, a clearer clause); a collocation is the one that
// hands over a reusable chunk, which is the only thing worth reviewing in a week.
func (h *Handler) plant(id, userID string) {
var s db.Suggestion
var contentText string
err := h.DB.QueryRow(
`SELECT s.type, s.original, s.replacement, s.explanation, s.doc_id, d.content_text
FROM suggestions s JOIN documents d ON d.id = s.doc_id
WHERE s.id = ? AND d.user_id = ?`,
id, userID,
).Scan(&s.Type, &s.Original, &s.Replacement, &s.Explanation, &s.DocID, &contentText)
if err != nil {
if !errors.Is(err, sql.ErrNoRows) {
log.Printf("suggestions: could not read %s for planting: %v", id, err)
}
return
}
if s.Type != db.SuggestionTypeCollocation || strings.TrimSpace(s.Replacement) == "" {
return
}
// The stored text is still the pre-accept draft — the client applies the
// replacement in the editor. Correct the sentence here so the flashcard
// quizzes the phrasing she is keeping, not the one she just left behind.
docID := s.DocID
if _, err := vocab.Plant(h.DB, userID, vocab.Phrase{
Text: s.Replacement,
Meaning: s.Explanation,
Example: correctedSentence(contentText, s.Original, s.Replacement),
DocID: &docID,
}); err != nil {
log.Printf("suggestions: could not plant %s: %v", id, err)
}
}
// correctedSentence returns the sentence of contentText containing original,
// with original swapped for replacement. Returns "" when the original isn't
// found (the draft moved on) — a card with no example still reviews, just
// without the cloze, so there's nothing to fall back to and nothing to guess.
func correctedSentence(contentText, original, replacement string) string {
idx := strings.Index(contentText, original)
if original == "" || idx < 0 {
return ""
}
start := strings.LastIndexAny(contentText[:idx], ".!?\n")
end := strings.IndexAny(contentText[idx+len(original):], ".!?\n")
if end < 0 {
end = len(contentText)
} else {
end += idx + len(original) + 1 // keep the terminator
}
sentence := strings.TrimSpace(contentText[start+1 : end])
return strings.Replace(sentence, original, replacement, 1)
}
// locate finds the plaintext offsets of original within contentText. Returns
// (-1, -1) when not found; the frontend anchors by string regardless, so a miss
// here is non-fatal.
internal/suggestions/handlers_test.go
+6 -1
View File
@@ -11,6 +11,7 @@ import (
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/llm"
)
@@ -56,7 +57,11 @@ func newTestServer(t *testing.T, client llm.LLMClient) (http.Handler, string, *H
r := chi.NewRouter()
r.Route("/docs", func(dr chi.Router) { h.RegisterDocRoutes(dr) })
r.Mount("/suggestions", h.Routes())
return r, docID, h
// Behind the same auth middleware main.go installs: handlers resolve the
// caller from the request context, so a bare router would see no user.
authed := auth.Middleware(auth.StaticResolver(db.LocalUserID))(r)
return authed, docID, h
}
func do(t *testing.T, srv http.Handler, method, path, body string) *httptest.ResponseRecorder {
internal/suggestions/isolation_test.go
+119
View File
@@ -0,0 +1,119 @@
package suggestions
import (
"encoding/json"
"net/http"
"path/filepath"
"testing"
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/llm"
)
// Suggestions are scoped indirectly: the table has no user_id of its own, only a
// doc_id, so every access has to reach the owner through the parent document. A
// forgotten join here is worse than it sounds — a suggestion quotes the sentence
// it corrects, so listing another account's suggestions leaks their prose.
// newTwoUserSuggestionServer seeds one document owned by the local user and
// returns routers for its owner and for a second, unrelated user.
func newTwoUserSuggestionServer(t *testing.T, client llm.LLMClient) (owner, stranger http.Handler, docID string) {
t.Helper()
database, err := db.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatalf("open db: %v", err)
}
t.Cleanup(func() { database.Close() })
if _, err := database.Exec(
`INSERT INTO users (id, email, display_name) VALUES (?, ?, ?)`,
"bob", "bob@petal.local", "Bob",
); err != nil {
t.Fatalf("seed second user: %v", err)
}
if err := database.QueryRow(
`INSERT INTO documents (user_id, content_text) VALUES (?, ?) RETURNING id`,
db.LocalUserID, "I has two apple.",
).Scan(&docID); err != nil {
t.Fatalf("seed doc: %v", err)
}
mount := func(userID string) http.Handler {
h := New(database, client)
r := chi.NewRouter()
r.Route("/docs", func(dr chi.Router) { h.RegisterDocRoutes(dr) })
r.Mount("/suggestions", h.Routes())
return auth.Middleware(auth.StaticResolver(userID))(r)
}
return mount(db.LocalUserID), mount("bob"), docID
}
func TestSuggestionIsolation(t *testing.T) {
client := &stubClient{response: `{"suggestions":[
{"original":"I has","replacement":"I have","explanation":"subject-verb agreement","type":"grammar"}
]}`}
owner, stranger, docID := newTwoUserSuggestionServer(t, client)
// The owner runs a checkpoint so there is a real pending suggestion to guard.
rec := do(t, owner, http.MethodPost, "/docs/"+docID+"/check", "")
if rec.Code != http.StatusOK {
t.Fatalf("check: %d %s", rec.Code, rec.Body)
}
var pending []db.Suggestion
if err := json.Unmarshal(rec.Body.Bytes(), &pending); err != nil {
t.Fatalf("decode: %v", err)
}
if len(pending) != 1 {
t.Fatalf("owner has %d suggestions, want 1", len(pending))
}
sugID := pending[0].ID
t.Run("cannot list a stranger's suggestions", func(t *testing.T) {
rec := do(t, stranger, http.MethodGet, "/docs/"+docID+"/suggestions", "")
var out []db.Suggestion
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
t.Fatalf("decode: %v", err)
}
if len(out) != 0 {
t.Fatalf("stranger read %d suggestions (leaking %q)", len(out), out[0].Original)
}
})
t.Run("cannot run a pass on a stranger's document", func(t *testing.T) {
rec := do(t, stranger, http.MethodPost, "/docs/"+docID+"/check", "")
if rec.Code != http.StatusNotFound {
t.Fatalf("stranger check = %d, want 404", rec.Code)
}
})
// accept/dismiss take a bare suggestion id with no document in the path, so
// the write has to scope itself through doc_id → documents.user_id.
for _, action := range []string{"accept", "dismiss"} {
t.Run("cannot "+action+" a stranger's suggestion", func(t *testing.T) {
rec := do(t, stranger, http.MethodPost, "/suggestions/"+sugID+"/"+action, "")
if rec.Code != http.StatusNotFound {
t.Fatalf("stranger %s = %d, want 404 (body: %s)", action, rec.Code, rec.Body)
}
})
}
// After every attempt the suggestion must still be pending for its owner.
rec = do(t, owner, http.MethodGet, "/docs/"+docID+"/suggestions", "")
var after []db.Suggestion
if err := json.Unmarshal(rec.Body.Bytes(), &after); err != nil {
t.Fatalf("decode: %v", err)
}
if len(after) != 1 || after[0].Status != db.SuggestionStatusPending {
t.Fatalf("owner's suggestion was altered by the stranger: %+v", after)
}
// And the owner can still action it — the scoping guards, it doesn't block.
rec = do(t, owner, http.MethodPost, "/suggestions/"+sugID+"/accept", "")
if rec.Code != http.StatusNoContent {
t.Fatalf("owner accept = %d, want 204 (body: %s)", rec.Code, rec.Body)
}
}
internal/suggestions/offline_test.go
+203
View File
@@ -0,0 +1,203 @@
package suggestions
import (
"encoding/json"
"net/http"
"testing"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// The offline rule pack and the LLM now share the collocation family, which is
// the point: the writer sees one rail and is never told which engine spoke. What
// makes that safe is `source` — each pass replaces only its own rows. These tests
// pin the two ways that could go wrong, both of which the old type-scoped DELETEs
// would have hit.
// pendingOfType counts the pending rows of one family in a response body.
func pendingOfType(got []db.Suggestion, typ string) []db.Suggestion {
var out []db.Suggestion
for _, s := range got {
if s.Type == typ {
out = append(out, s)
}
}
return out
}
// TestOfflineCollocationFilesAsCollocation proves a miscollocation the rule pack
// found is stored in the collocation family (so accepting it plants a garden
// card, exactly as the coach's would) while still being marked as locally found.
func TestOfflineCollocationFilesAsCollocation(t *testing.T) {
srv, docID, _ := newTestServer(t, &stubClient{response: `{"suggestions":[]}`})
got := postMechanics(t, srv, docID, `[
{"from":0,"to":13,"original":"do a decision","replacement":"make a decision","explanation":"pairing","type":"collocation"},
{"from":20,"to":27,"original":"the the","replacement":"the","explanation":"doubled word","type":"mechanics"}
]`)
if len(got) != 2 {
t.Fatalf("want both findings, got %+v", got)
}
coll := pendingOfType(got, db.SuggestionTypeCollocation)
if len(coll) != 1 {
t.Fatalf("want 1 collocation, got %+v", got)
}
if coll[0].Source != db.SuggestionSourceLocal {
t.Errorf("offline finding should be source=local, got %q", coll[0].Source)
}
if mech := pendingOfType(got, db.SuggestionTypeMechanics); len(mech) != 1 {
t.Fatalf("want 1 mechanics finding, got %+v", got)
}
}
// TestUnknownLocalTypeFallsBackToMechanics: a family the offline pass isn't
// allowed to claim (or an older client sending none at all) must land in
// mechanics. Otherwise a stray label would smuggle a row into an LLM family,
// where nothing would ever replace it.
func TestUnknownLocalTypeFallsBackToMechanics(t *testing.T) {
srv, docID, _ := newTestServer(t, &stubClient{response: `{"suggestions":[]}`})
got := postMechanics(t, srv, docID, `[
{"from":0,"to":5,"original":"aaaaa","replacement":"bbbbb","explanation":"x","type":"voice"},
{"from":6,"to":11,"original":"ccccc","replacement":"ddddd","explanation":"y"}
]`)
if len(got) != 2 {
t.Fatalf("want 2 findings, got %+v", got)
}
for _, s := range got {
if s.Type != db.SuggestionTypeMechanics {
t.Errorf("offline finding claimed family %q; only mechanics/collocation are allowed", s.Type)
}
}
}
// TestCoachDoesNotWipeOfflineCollocations is the collision the source column
// exists for: the LLM collocation pass replaces the collocation family, and the
// rule pack's share of that family has to survive it. Before `source`, running
// the coach silently deleted every offline chunk on the page.
func TestCoachDoesNotWipeOfflineCollocations(t *testing.T) {
client := &stubClient{response: `{"suggestions":[
{"original":"apple","replacement":"an apple","explanation":"article","type":"collocation"}
]}`}
srv, docID, _ := newTestServer(t, client)
// The seeded doc is "I has two apple." — the coach's flag anchors on "apple"
// at [10,15], so the offline finding is given a span well clear of it. Two
// findings fighting over the same characters is a different rule (see
// TestOfflineCardWinsSpanCollision); this test is about the DELETE.
postMechanics(t, srv, docID, `[
{"from":0,"to":5,"original":"do a decision","replacement":"make a decision","explanation":"pairing","type":"collocation"}
]`)
rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/collocation", "")
if rec.Code != http.StatusOK {
t.Fatalf("collocation pass: code=%d body=%s", rec.Code, rec.Body)
}
var got []db.Suggestion
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("decode: %v", err)
}
var local, llm int
for _, s := range pendingOfType(got, db.SuggestionTypeCollocation) {
if s.Source == db.SuggestionSourceLocal {
local++
} else {
llm++
}
}
if local != 1 {
t.Errorf("the coach wiped the offline collocation: local=%d, got %+v", local, got)
}
if llm != 1 {
t.Errorf("want the coach's own flag alongside it: llm=%d, got %+v", llm, got)
}
}
// TestOfflinePassReplacesItsOwnCollocations is the mirror: the rule pack
// recomputes the whole document every run, so a chunk the current text no longer
// warrants must go — and the coach's flags must stay. Scoping the offline DELETE
// by type instead of source would have stranded the first row forever.
func TestOfflinePassReplacesItsOwnCollocations(t *testing.T) {
client := &stubClient{response: `{"suggestions":[
{"original":"apple","replacement":"an apple","explanation":"article","type":"collocation"}
]}`}
srv, docID, _ := newTestServer(t, client)
// A coach flag, then an offline chunk, then a rerun that no longer finds it.
do(t, srv, http.MethodPost, "/docs/"+docID+"/collocation", "")
postMechanics(t, srv, docID, `[
{"from":0,"to":13,"original":"do a decision","replacement":"make a decision","explanation":"pairing","type":"collocation"}
]`)
got := postMechanics(t, srv, docID, `[]`)
for _, s := range got {
if s.Source == db.SuggestionSourceLocal {
t.Errorf("stale offline finding survived a recompute: %+v", s)
}
}
if len(pendingOfType(got, db.SuggestionTypeCollocation)) != 1 {
t.Fatalf("the coach's own flag should be untouched, got %+v", got)
}
}
// TestOfflineCollocationPlantsOnAccept closes the loop the family split was for:
// a chunk the rule pack found, accepted, becomes a vocabulary-garden card — with
// no model involved anywhere in the path.
func TestOfflineCollocationPlantsOnAccept(t *testing.T) {
srv, docID, h := newTestServer(t, &stubClient{response: `{"suggestions":[]}`})
if _, err := h.DB.Exec(
`UPDATE documents SET content_text = ? WHERE id = ?`,
"I had to do a decision about the job.", docID,
); err != nil {
t.Fatalf("set content: %v", err)
}
got := postMechanics(t, srv, docID, `[
{"from":9,"to":22,"original":"do a decision","replacement":"make a decision","explanation":"pairing","type":"collocation"}
]`)
if len(got) != 1 {
t.Fatalf("want the offline chunk, got %+v", got)
}
if rec := do(t, srv, http.MethodPost, "/suggestions/"+got[0].ID+"/accept", ""); rec.Code != http.StatusNoContent {
t.Fatalf("accept: code=%d body=%s", rec.Code, rec.Body)
}
cards := gardenCards(t, h)
if len(cards) != 1 || cards[0].word != "make a decision" {
t.Fatalf("want a planted phrase card, got %+v", cards)
}
// The example is the corrected sentence — the phrasing she kept, not the one
// she just left behind.
if cards[0].example != "I had to make a decision about the job." {
t.Errorf("example should be the corrected sentence, got %q", cards[0].example)
}
}
// TestOfflineCardWinsSpanCollision: the tiebreak is by engine, not by family. An
// offline miscollocation has an exact span; the coach's overlapping flag is only
// advisory, so it is the one that goes.
func TestOfflineCardWinsSpanCollision(t *testing.T) {
client := &stubClient{response: `{"suggestions":[
{"original":"do a decision about","replacement":"decide about","explanation":"wordy","type":"collocation"}
]}`}
srv, docID, h := newTestServer(t, client)
if _, err := h.DB.Exec(
`UPDATE documents SET content_text = ? WHERE id = ?`,
"I had to do a decision about the job.", docID,
); err != nil {
t.Fatalf("set content: %v", err)
}
do(t, srv, http.MethodPost, "/docs/"+docID+"/collocation", "")
got := postMechanics(t, srv, docID, `[
{"from":9,"to":22,"original":"do a decision","replacement":"make a decision","explanation":"pairing","type":"collocation"}
]`)
if len(got) != 1 {
t.Fatalf("want the overlapping coach flag dropped, got %+v", got)
}
if got[0].Source != db.SuggestionSourceLocal {
t.Errorf("the exact offline card should own the span, got %+v", got[0])
}
}
internal/suggestions/pairlang_test.go
+109
View File
@@ -0,0 +1,109 @@
package suggestions
import (
"net/http"
"path/filepath"
"strings"
"testing"
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/llm"
)
// The langpack decides what Petal says in the browser; users.pair_lang has to
// decide what the *model* says too, or a pt-PT writer gets a Mandarin gloss on
// an otherwise Portuguese screen. These tests follow the value from the column
// to the system prompt for each pass that names a language.
//
// This is the same failure mode the standing isolation rule guards against: the
// column is read in a query the handler already ran, so nothing fails loudly if
// the join is dropped — the prompt just quietly reverts to Mandarin.
// newPairServer seeds one writer on the given pair with a document of her own.
func newPairServer(t *testing.T, client llm.LLMClient, pairLang string) (http.Handler, string, *db.DB) {
t.Helper()
database, err := db.Open(filepath.Join(t.TempDir(), "pair.db"))
if err != nil {
t.Fatalf("open db: %v", err)
}
t.Cleanup(func() { database.Close() })
const userID = "writer-pt"
if _, err := database.Exec(
`INSERT INTO users (id, email, display_name, pair_lang) VALUES (?, ?, ?, ?)`,
userID, "w@example.com", "Writer", pairLang,
); err != nil {
t.Fatalf("seed user: %v", err)
}
var docID string
if err := database.QueryRow(
`INSERT INTO documents (user_id, content_text) VALUES (?, ?) RETURNING id`,
userID, "The rain was strong yesterday.",
).Scan(&docID); err != nil {
t.Fatalf("seed doc: %v", err)
}
h := New(database, client)
r := chi.NewRouter()
r.Route("/docs", func(dr chi.Router) { h.RegisterDocRoutes(dr) })
r.Mount("/suggestions", h.Routes())
return auth.Middleware(auth.StaticResolver(userID))(r), docID, database
}
func TestCollocationPromptUsesTheWritersPair(t *testing.T) {
client := &recordingClient{response: `{"suggestions":[]}`}
srv, docID, _ := newPairServer(t, client, "pt-PT")
rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/collocation", "")
if rec.Code != http.StatusOK {
t.Fatalf("collocation: code=%d body=%s", rec.Code, rec.Body)
}
system := client.last.Messages[0].Content
if !strings.Contains(system, "European Portuguese") {
t.Fatalf("collocation prompt ignored pair_lang:\n%s", system)
}
if strings.Contains(system, "Simplified Chinese") {
t.Fatalf("collocation prompt fell back to Mandarin:\n%s", system)
}
}
func TestTranslatePromptUsesTheWritersPair(t *testing.T) {
client := &recordingClient{response: "Chove muito."}
srv, docID, database := newPairServer(t, client, "pt-PT")
var sugID string
if err := database.QueryRow(
`INSERT INTO suggestions (doc_id, original, replacement, explanation, type, from_pos, to_pos)
VALUES (?, ?, ?, ?, ?, 0, 5) RETURNING id`,
docID, "strong rain", "heavy rain", "Natives usually say heavy rain.", "collocation",
).Scan(&sugID); err != nil {
t.Fatalf("seed suggestion: %v", err)
}
rec := do(t, srv, http.MethodPost, "/suggestions/"+sugID+"/translate", "")
if rec.Code != http.StatusOK {
t.Fatalf("translate: code=%d body=%s", rec.Code, rec.Body)
}
system := client.last.Messages[0].Content
if !strings.Contains(system, "European Portuguese") || strings.Contains(system, "Chinese") {
t.Fatalf("translate prompt ignored pair_lang:\n%s", system)
}
}
// A writer whose column still holds the default — every account today — must be
// answered exactly as before.
func TestDefaultPairIsUnchanged(t *testing.T) {
client := &recordingClient{response: `{"suggestions":[]}`}
srv, docID, _ := newPairServer(t, client, "zh")
if rec := do(t, srv, http.MethodPost, "/docs/"+docID+"/collocation", ""); rec.Code != http.StatusOK {
t.Fatalf("collocation: code=%d body=%s", rec.Code, rec.Body)
}
if system := client.last.Messages[0].Content; !strings.Contains(system, "Simplified Chinese (Mandarin) gloss") {
t.Fatalf("zh writer no longer gets a Mandarin gloss:\n%s", system)
}
}
internal/suggestions/plant_test.go
+186
View File
@@ -0,0 +1,186 @@
package suggestions
import (
"net/http"
"testing"
"gitea.parodia.dev/drwily/petal/internal/db"
)
// seedSuggestion writes one pending suggestion against the seeded doc, after
// replacing the doc's text so the sentence around `original` is under the test's
// control.
func seedSuggestion(t *testing.T, h *Handler, docID, text, sType, original, replacement, explanation string) string {
t.Helper()
if _, err := h.DB.Exec(`UPDATE documents SET content_text = ? WHERE id = ?`, text, docID); err != nil {
t.Fatalf("set content: %v", err)
}
var id string
err := h.DB.QueryRow(
`INSERT INTO suggestions (doc_id, from_pos, to_pos, original, replacement, explanation, type, status)
VALUES (?, 0, 0, ?, ?, ?, ?, 'pending') RETURNING id`,
docID, original, replacement, explanation, sType,
).Scan(&id)
if err != nil {
t.Fatalf("seed suggestion: %v", err)
}
return id
}
type card struct {
word, definition, example string
interval int
}
func gardenCards(t *testing.T, h *Handler) []card {
t.Helper()
rows, err := h.DB.Query(
`SELECT word, definition, example, interval_days FROM vocab_words WHERE user_id = ? ORDER BY word`,
db.LocalUserID,
)
if err != nil {
t.Fatalf("read garden: %v", err)
}
defer rows.Close()
var out []card
for rows.Next() {
var c card
if err := rows.Scan(&c.word, &c.definition, &c.example, &c.interval); err != nil {
t.Fatalf("scan: %v", err)
}
out = append(out, c)
}
return out
}
// TestAcceptedCollocationIsPlanted walks the whole hand-over: a collocation the
// writer accepts becomes a phrase card whose example is the *corrected*
// sentence, so the flashcard quizzes the phrasing she kept.
func TestAcceptedCollocationIsPlanted(t *testing.T) {
srv, docID, h := newTestServer(t, &stubClient{})
id := seedSuggestion(t, h, docID,
"Yesterday was hard. I had to do a decision about the job. Then I slept.",
db.SuggestionTypeCollocation, "do a decision", "make a decision",
"English pairs “make” with “decision”.")
if rec := do(t, srv, http.MethodPost, "/suggestions/"+id+"/accept", ""); rec.Code != http.StatusNoContent {
t.Fatalf("accept: got %d, want 204", rec.Code)
}
cards := gardenCards(t, h)
if len(cards) != 1 {
t.Fatalf("garden has %d cards, want 1: %+v", len(cards), cards)
}
got := cards[0]
if got.word != "make a decision" {
t.Errorf("word = %q, want %q", got.word, "make a decision")
}
if got.example != "I had to make a decision about the job." {
t.Errorf("example = %q — want the corrected sentence, bounded to its own sentence", got.example)
}
if got.definition != "English pairs “make” with “decision”." {
t.Errorf("definition = %q, want the explanation", got.definition)
}
if got.interval != 1 {
t.Errorf("interval_days = %d, want 1 (due tomorrow, like a fresh capture)", got.interval)
}
}
// TestOnlyCollocationsArePlanted: the other families correct this sentence and
// hand over nothing reusable. A dismissed collocation is not a lesson either.
func TestOnlyCollocationsArePlanted(t *testing.T) {
srv, docID, h := newTestServer(t, &stubClient{})
grammar := seedSuggestion(t, h, docID, "I has two apples.",
db.SuggestionTypeGrammar, "I has", "I have", "Subject–verb agreement.")
if rec := do(t, srv, http.MethodPost, "/suggestions/"+grammar+"/accept", ""); rec.Code != http.StatusNoContent {
t.Fatalf("accept grammar: got %d", rec.Code)
}
dismissed := seedSuggestion(t, h, docID, "We must take a photo of it.",
db.SuggestionTypeCollocation, "do a photo", "take a photo", "Photos are taken.")
if rec := do(t, srv, http.MethodPost, "/suggestions/"+dismissed+"/dismiss", ""); rec.Code != http.StatusNoContent {
t.Fatalf("dismiss: got %d", rec.Code)
}
if cards := gardenCards(t, h); len(cards) != 0 {
t.Fatalf("garden grew %d card(s) from a grammar fix and a dismissal: %+v", len(cards), cards)
}
}
// TestPlantingIsIdempotentAndNeverResets: accepting the same chunk again is
// evidence it's still being learned — the worst possible response is to wipe the
// card's first context and the schedule it has been climbing.
func TestPlantingIsIdempotentAndNeverResets(t *testing.T) {
srv, docID, h := newTestServer(t, &stubClient{})
first := seedSuggestion(t, h, docID, "I had to do a decision.",
db.SuggestionTypeCollocation, "do a decision", "make a decision", "First explanation.")
if rec := do(t, srv, http.MethodPost, "/suggestions/"+first+"/accept", ""); rec.Code != http.StatusNoContent {
t.Fatalf("accept: got %d", rec.Code)
}
// The card climbs a little.
if _, err := h.DB.Exec(
`UPDATE vocab_words SET reps = 3, interval_days = 7 WHERE user_id = ? AND word = 'make a decision'`,
db.LocalUserID,
); err != nil {
t.Fatalf("advance card: %v", err)
}
second := seedSuggestion(t, h, docID, "Later I must do a decision again.",
db.SuggestionTypeCollocation, "do a decision", "make a decision", "Second explanation.")
if rec := do(t, srv, http.MethodPost, "/suggestions/"+second+"/accept", ""); rec.Code != http.StatusNoContent {
t.Fatalf("accept again: got %d", rec.Code)
}
cards := gardenCards(t, h)
if len(cards) != 1 {
t.Fatalf("garden has %d cards, want 1 (one chunk, one card)", len(cards))
}
if cards[0].definition != "First explanation." {
t.Errorf("definition = %q — the existing card should win", cards[0].definition)
}
if cards[0].example != "I had to make a decision." {
t.Errorf("example = %q — the first context should survive", cards[0].example)
}
if cards[0].interval != 7 {
t.Errorf("interval_days = %d, want 7 — progress must not be reset", cards[0].interval)
}
}
// TestSentenceRewriteIsNotAPhraseCard: a "collocation" long enough to be a
// rewritten sentence makes a miserable flashcard, so it is dropped rather than
// planted — and the accept still succeeds.
func TestSentenceRewriteIsNotAPhraseCard(t *testing.T) {
srv, docID, h := newTestServer(t, &stubClient{})
long := "I would like to take this opportunity to thank you for everything"
id := seedSuggestion(t, h, docID, "I want thank you for everything.",
db.SuggestionTypeCollocation, "I want thank you for everything", long, "More natural.")
if rec := do(t, srv, http.MethodPost, "/suggestions/"+id+"/accept", ""); rec.Code != http.StatusNoContent {
t.Fatalf("accept: got %d, want 204 — a skipped card must never fail the accept", rec.Code)
}
if cards := gardenCards(t, h); len(cards) != 0 {
t.Fatalf("planted a sentence as a phrase card: %+v", cards)
}
}
func TestCorrectedSentence(t *testing.T) {
const text = "One thing. I had to do a decision fast! Another thing."
cases := []struct {
name, original, replacement, want string
}{
{"bounded to its sentence", "do a decision", "make a decision", "I had to make a decision fast!"},
{"original no longer present", "do a choice", "make a choice", ""},
{"empty original", "", "make a decision", ""},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := correctedSentence(text, tc.original, tc.replacement); got != tc.want {
t.Errorf("correctedSentence = %q, want %q", got, tc.want)
}
})
}
// A document with no terminator at all is one sentence, and still works.
if got := correctedSentence("i had to do a decision", "do a decision", "make a decision"); got != "i had to make a decision" {
t.Errorf("unterminated doc: got %q", got)
}
}
internal/suggestions/rewrite.go
+2 -2
View File
@@ -9,7 +9,7 @@ import (
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/httputil"
"gitea.parodia.dev/drwily/petal/internal/llm"
)
@@ -56,7 +56,7 @@ func (h *Handler) rewrite(w http.ResponseWriter, r *http.Request) {
var exists int
err := h.DB.QueryRow(
`SELECT 1 FROM documents WHERE id = ? AND user_id = ?`,
docID, db.LocalUserID,
docID, auth.UserID(r.Context()),
).Scan(&exists)
if errors.Is(err, sql.ErrNoRows) {
httputil.ErrorJSON(w, http.StatusNotFound, "document not found")
internal/suggestions/translate.go
+7 -6
View File
@@ -8,7 +8,7 @@ import (
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/httputil"
"gitea.parodia.dev/drwily/petal/internal/llm"
)
@@ -25,14 +25,15 @@ type translateResponse struct {
func (h *Handler) translate(w http.ResponseWriter, r *http.Request) {
sugID := chi.URLParam(r, "id")
var explanation string
var explanation, pairLang string
err := h.DB.QueryRow(
`SELECT s.explanation
`SELECT s.explanation, COALESCE(u.pair_lang, '')
FROM suggestions s
JOIN documents d ON d.id = s.doc_id
JOIN users u ON u.id = d.user_id
WHERE s.id = ? AND d.user_id = ?`,
sugID, db.LocalUserID,
).Scan(&explanation)
sugID, auth.UserID(r.Context()),
).Scan(&explanation, &pairLang)
if errors.Is(err, sql.ErrNoRows) {
httputil.ErrorJSON(w, http.StatusNotFound, "suggestion not found")
return
@@ -48,7 +49,7 @@ func (h *Handler) translate(w http.ResponseWriter, r *http.Request) {
return
}
out, err := llm.RunTranslate(r.Context(), h.Client, explanation)
out, err := llm.RunTranslate(r.Context(), h.Client, explanation, llm.LangFor(pairLang))
if err != nil {
httputil.ErrorJSON(w, http.StatusBadGateway, "translate failed: "+err.Error())
return
internal/tts/handler.go
+67 -18
View File
@@ -19,6 +19,7 @@ import (
"os"
"os/exec"
"path/filepath"
"sort"
"strings"
"time"
"unicode/utf8"
@@ -36,6 +37,12 @@ const maxTextBytes = 4000
// with the words, mirroring the old utterance.rate = 0.95. Higher = slower.
const lengthScale = 1.1
// slowLengthScale is the "say it slower" replay (SUGGESTIONS §5e): roughly 0.75×
// the normal pace, which is the speed listening drills have used for decades.
// Piper stretches durations rather than resampling, so the voice keeps its pitch
// instead of turning into a slowed tape.
const slowLengthScale = lengthScale / 0.75
// audioFormat describes one output encoding: the cache-file extension, the
// response Content-Type, and the ffmpeg args that turn Piper's WAV (on stdin)
// into this format (on stdout). A nil ffmpegArgs means "serve the WAV as-is".
@@ -56,6 +63,25 @@ var formats = map[string]audioFormat{
ffmpegArgs: []string{"-f", "ogg", "-c:a", "libopus", "-b:a", "32k", "-ac", "1"}},
}
// synthPath normalises TTS_PATH into a leading-slash path with no trailing
// slash, so it concatenates cleanly onto a route's endpoint.
//
// Piper moved synthesis from `POST /` to `POST /synthesize` in 1.6.0, and the
// request body is identical either side of that change. Rather than pinning
// every deployment to one Piper release, the path is configuration: millenia
// keeps the default `/` its installed server expects, and the containerised
// 1.6.0 sidecars on the VPS set `/synthesize`.
func synthPath(p string) string {
p = strings.TrimSpace(p)
if p == "" || p == "/" {
return "/"
}
if !strings.HasPrefix(p, "/") {
p = "/" + p
}
return strings.TrimRight(p, "/")
}
// route is the Piper instance and voice id serving one language. Each Piper
// HTTP server loads exactly one model, so distinct languages mean distinct
// endpoints (e.g. English on :5005, Chinese on :5006).
@@ -67,6 +93,7 @@ type route struct {
// Handler proxies synthesis to Piper and caches the result on disk.
type Handler struct {
routes map[string]route // base language (e.g. "en", "zh") -> Piper instance
synthURI string // path Piper serves synthesis on (see TTSPath)
cacheDir string
format audioFormat
client *http.Client
@@ -85,16 +112,14 @@ func New(cfg *config.Config) (*Handler, bool) {
format = formats["mp3"]
}
// Map by base language so en-US, en-GB, etc. all resolve to the English
// instance (the client sends BCP-47 tags like the old Web Speech path did).
// A language is only routable when both its endpoint and voice are set;
// otherwise the client falls back to Web Speech for that language.
// Keyed by base language so en-US, en-GB — and pt-PT, pt-BR, bare pt —
// resolve to the one instance that has that language's model loaded (the
// client sends BCP-47 tags, as the old Web Speech path did). Config has
// already dropped any language configured by halves, so an unroutable
// language reaches the client as a 404 and falls back to Web Speech.
routes := map[string]route{}
if cfg.TTSVoiceEN != "" {
routes["en"] = route{strings.TrimRight(cfg.TTSEndpoint, "/"), cfg.TTSVoiceEN}
}
if cfg.TTSEndpointZH != "" && cfg.TTSVoiceZH != "" {
routes["zh"] = route{strings.TrimRight(cfg.TTSEndpointZH, "/"), cfg.TTSVoiceZH}
for lang, v := range cfg.TTSVoices {
routes[lang] = route{endpoint: v.Endpoint, voice: v.Voice}
}
if err := os.MkdirAll(cfg.TTSCacheDir, 0o755); err != nil {
@@ -105,12 +130,26 @@ func New(cfg *config.Config) (*Handler, bool) {
return &Handler{
routes: routes,
synthURI: synthPath(cfg.TTSPath),
cacheDir: cfg.TTSCacheDir,
format: format,
client: &http.Client{Timeout: cfg.TTSTimeout},
}, true
}
// Languages lists the base language tags this handler can synthesize, sorted,
// each with the voice serving it — for the startup line, so a deployment says
// which sidecars it actually reached rather than which ones it was configured
// to want.
func (h *Handler) Languages() []string {
out := make([]string, 0, len(h.routes))
for lang, rt := range h.routes {
out = append(out, lang+"="+rt.voice)
}
sort.Strings(out)
return out
}
// Routes mounts the synthesis endpoint. Mount under "/tts" so the full path is
// POST /api/tts.
func (h *Handler) Routes() chi.Router {
@@ -119,11 +158,13 @@ func (h *Handler) Routes() chi.Router {
return r
}
// synthRequest is the body the editor posts: a passage and the BCP-47 language
// tag it's written in (e.g. "en-US", "zh-CN").
// synthRequest is the body the editor posts: a passage, the BCP-47 language tag
// it's written in (e.g. "en-US", "zh-CN", "pt-PT"), and whether to say it slowly
// — the replay a learner reaches for when the sentence went past too fast.
type synthRequest struct {
Text string `json:"text"`
Lang string `json:"lang"`
Slow bool `json:"slow"`
}
// synth resolves a voice for the requested language, returns cached audio when
@@ -159,9 +200,17 @@ func (h *Handler) synth(w http.ResponseWriter, r *http.Request) {
return
}
// Content-addressed: identical (voice, text) → identical clip. The format
// extension keeps encodings from colliding in the same dir.
sum := sha256.Sum256([]byte(rt.voice + "\n" + text))
scale := lengthScale
if req.Slow {
scale = slowLengthScale
}
// Content-addressed: identical (voice, pace, text) → identical clip. The pace
// belongs in the key — without it the slow replay of a word already heard at
// normal speed would be served from cache at normal speed, which is the one
// request where the difference is the whole point. The format extension keeps
// encodings from colliding in the same dir.
sum := sha256.Sum256([]byte(fmt.Sprintf("%s\n%.3f\n%s", rt.voice, scale, text)))
name := hex.EncodeToString(sum[:])[:32] + h.format.ext
path := filepath.Join(h.cacheDir, name)
@@ -170,7 +219,7 @@ func (h *Handler) synth(w http.ResponseWriter, r *http.Request) {
return
}
audio, err := h.synthesize(r.Context(), rt, text)
audio, err := h.synthesize(r.Context(), rt, text, scale)
if err != nil {
http.Error(w, "synthesis failed", http.StatusBadGateway)
fmt.Fprintf(os.Stderr, "tts: synthesize: %v\n", err)
@@ -201,13 +250,13 @@ func (h *Handler) serve(w http.ResponseWriter, r *http.Request, path string) {
// synthesize POSTs to the route's Piper instance, then transcodes the returned
// WAV when the configured format calls for it.
func (h *Handler) synthesize(ctx context.Context, rt route, text string) ([]byte, error) {
func (h *Handler) synthesize(ctx context.Context, rt route, text string, scale float64) ([]byte, error) {
body, _ := json.Marshal(map[string]any{
"text": text,
"voice": rt.voice,
"length_scale": lengthScale,
"length_scale": scale,
})
httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, rt.endpoint+"/", bytes.NewReader(body))
httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, rt.endpoint+h.synthURI, bytes.NewReader(body))
if err != nil {
return nil, err
}
internal/tts/handler_test.go
+135 -3
View File
@@ -22,6 +22,7 @@ func newStubPiper(t *testing.T, body []byte) (*httptest.Server, *int32, *synthEc
_ = json.NewDecoder(r.Body).Decode(&req)
last.voice, _ = req["voice"].(string)
last.text, _ = req["text"].(string)
last.scale, _ = req["length_scale"].(float64)
w.Header().Set("Content-Type", "audio/wav")
_, _ = w.Write(body)
}))
@@ -29,22 +30,74 @@ func newStubPiper(t *testing.T, body []byte) (*httptest.Server, *int32, *synthEc
return srv, &calls, last
}
type synthEcho struct{ voice, text string }
type synthEcho struct {
voice, text string
scale float64
}
// newHandler builds a wav-format handler (no ffmpeg) pointed at a stub server.
func newHandler(t *testing.T, endpoint string) *Handler {
t.Helper()
return &Handler{
routes: map[string]route{"en": {strings.TrimRight(endpoint, "/"), "en_US-amy-medium"}},
synthURI: synthPath("/"),
cacheDir: t.TempDir(),
format: formats["wav"],
client: http.DefaultClient,
}
}
// Piper 1.6.0 serves synthesis on /synthesize and 405s on /. The path is
// configuration (TTS_PATH) so one Petal build talks to either server version;
// this asserts the configured path is the one actually requested.
func TestSynthUsesConfiguredPath(t *testing.T) {
var gotPath string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if r.URL.Path != "/synthesize" {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
w.Header().Set("Content-Type", "audio/wav")
_, _ = w.Write([]byte("RIFF....fake-wav"))
}))
t.Cleanup(srv.Close)
h := newHandler(t, srv.URL)
h.synthURI = synthPath("/synthesize")
if rr := post(t, h, "hello there", "en-US"); rr.Code != http.StatusOK {
t.Fatalf("status = %d, want 200 (piper saw path %q)", rr.Code, gotPath)
}
if gotPath != "/synthesize" {
t.Errorf("piper path = %q, want /synthesize", gotPath)
}
}
func TestSynthPathNormalisation(t *testing.T) {
cases := map[string]string{
"": "/",
"/": "/",
"synthesize": "/synthesize",
"/synthesize": "/synthesize",
"/synthesize/": "/synthesize",
" /v1/tts ": "/v1/tts",
}
for in, want := range cases {
if got := synthPath(in); got != want {
t.Errorf("synthPath(%q) = %q, want %q", in, got, want)
}
}
}
func post(t *testing.T, h *Handler, text, lang string) *httptest.ResponseRecorder {
t.Helper()
b, _ := json.Marshal(synthRequest{Text: text, Lang: lang})
return postReq(t, h, synthRequest{Text: text, Lang: lang})
}
func postReq(t *testing.T, h *Handler, body synthRequest) *httptest.ResponseRecorder {
t.Helper()
b, _ := json.Marshal(body)
req := httptest.NewRequest(http.MethodPost, "/", bytes.NewReader(b))
rr := httptest.NewRecorder()
h.synth(rr, req)
@@ -148,8 +201,87 @@ func TestTextIsCapped(t *testing.T) {
}
}
// The slow replay is the whole of SUGGESTIONS §5e: same text, same voice, more
// time per phoneme.
func TestSlowRequestStretchesTheVoice(t *testing.T) {
srv, _, last := newStubPiper(t, []byte("RIFF....fake-wav"))
h := newHandler(t, srv.URL)
if rr := postReq(t, h, synthRequest{Text: "reception", Lang: "en-US"}); rr.Code != http.StatusOK {
t.Fatalf("status = %d, want 200", rr.Code)
}
if last.scale != lengthScale {
t.Fatalf("normal length_scale = %v, want %v", last.scale, lengthScale)
}
if rr := postReq(t, h, synthRequest{Text: "reception", Lang: "en-US", Slow: true}); rr.Code != http.StatusOK {
t.Fatalf("slow status = %d, want 200", rr.Code)
}
if last.scale != slowLengthScale {
t.Fatalf("slow length_scale = %v, want %v", last.scale, slowLengthScale)
}
if slowLengthScale <= lengthScale {
t.Fatalf("slowLengthScale %v is not slower than %v", slowLengthScale, lengthScale)
}
}
// The pace has to be part of the cache key. Without it, asking for the slow
// replay of a word already heard at normal speed serves the normal clip — the
// one request where hearing the difference is the entire point.
func TestSlowClipIsNotServedFromTheNormalCache(t *testing.T) {
srv, calls, last := newStubPiper(t, []byte("RIFF....fake-wav"))
h := newHandler(t, srv.URL)
postReq(t, h, synthRequest{Text: "reception", Lang: "en-US"})
postReq(t, h, synthRequest{Text: "reception", Lang: "en-US", Slow: true})
if *calls != 2 {
t.Fatalf("piper calls = %d, want 2 (the slow clip is a different clip)", *calls)
}
if last.scale != slowLengthScale {
t.Fatalf("second call length_scale = %v, want the slow one", last.scale)
}
// …and each pace still caches on its own.
postReq(t, h, synthRequest{Text: "reception", Lang: "en-US", Slow: true})
postReq(t, h, synthRequest{Text: "reception", Lang: "en-US"})
if *calls != 2 {
t.Fatalf("piper calls = %d, want 2 (both paces now cached)", *calls)
}
}
// A Portuguese request must reach the Portuguese instance on the base tag alone:
// env var names cannot hold the hyphen in pt-PT, so config keys the map on "pt"
// and the handler has to meet it there. pt-BR resolves to the same instance
// because there is only one Portuguese voice loaded — and it is the European one.
func TestPortugueseRoutesOnTheBaseTag(t *testing.T) {
enSrv, enCalls, _ := newStubPiper(t, []byte("EN-wav"))
ptSrv, ptCalls, ptLast := newStubPiper(t, []byte("PT-wav"))
h := &Handler{
routes: map[string]route{
"en": {strings.TrimRight(enSrv.URL, "/"), "en_US-amy-medium"},
"pt": {strings.TrimRight(ptSrv.URL, "/"), "pt_PT-tugão-medium"},
},
cacheDir: t.TempDir(),
format: formats["wav"],
client: http.DefaultClient,
}
// Distinct text per tag, so a cache hit can't stand in for a route.
for i, tag := range []string{"pt-PT", "pt", "pt-BR"} {
if rr := post(t, h, strings.Repeat("receção ", i+1), tag); rr.Code != http.StatusOK {
t.Fatalf("%s status = %d, want 200", tag, rr.Code)
}
}
if *ptCalls != 3 || *enCalls != 0 {
t.Fatalf("calls en=%d pt=%d, want en=0 pt=3", *enCalls, *ptCalls)
}
if ptLast.voice != "pt_PT-tugão-medium" {
t.Fatalf("pt voice = %q, want the European Portuguese voice", ptLast.voice)
}
}
func TestBaseLang(t *testing.T) {
cases := map[string]string{"en-US": "en", "EN_gb": "en", "zh-CN": "zh", "en": "en", "": ""}
cases := map[string]string{"en-US": "en", "EN_gb": "en", "zh-CN": "zh", "pt-PT": "pt", "en": "en", "": ""}
for in, want := range cases {
if got := baseLang(in); got != want {
t.Errorf("baseLang(%q) = %q, want %q", in, got, want)
internal/vocab/handlers.go
+17 -13
View File
@@ -11,6 +11,7 @@ import (
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
"gitea.parodia.dev/drwily/petal/internal/httputil"
)
@@ -68,15 +69,15 @@ func scanWord(s interface {
}
// list returns the full garden, newest blossoms first.
func (h *Handler) list(w http.ResponseWriter, _ *http.Request) {
func (h *Handler) list(w http.ResponseWriter, r *http.Request) {
h.queryList(w, `SELECT `+vocabColumns+` FROM vocab_words
WHERE user_id = ? ORDER BY created_at DESC`, db.LocalUserID)
WHERE user_id = ? ORDER BY created_at DESC`, auth.UserID(r.Context()))
}
// due returns only the cards whose review time has arrived, soonest first.
func (h *Handler) due(w http.ResponseWriter, _ *http.Request) {
func (h *Handler) due(w http.ResponseWriter, r *http.Request) {
h.queryList(w, `SELECT `+vocabColumns+` FROM vocab_words
WHERE user_id = ? AND due_at <= datetime('now') ORDER BY due_at ASC`, db.LocalUserID)
WHERE user_id = ? AND due_at <= datetime('now') ORDER BY due_at ASC`, auth.UserID(r.Context()))
}
func (h *Handler) queryList(w http.ResponseWriter, query string, args ...any) {
@@ -138,6 +139,8 @@ func clamp(s string, max int) string {
// schedule untouched but refreshes its gloss/phonetic/example/doc_id so the most
// recent context wins. Looking words up IS the data source — no extra effort.
func (h *Handler) capture(w http.ResponseWriter, r *http.Request) {
userID := auth.UserID(r.Context())
var req captureRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
httputil.ErrorJSON(w, http.StatusBadRequest, "invalid body")
@@ -169,7 +172,7 @@ func (h *Handler) capture(w http.ResponseWriter, r *http.Request) {
var ok int
err := h.DB.QueryRow(
`SELECT 1 FROM documents WHERE id = ? AND user_id = ?`,
*req.DocID, db.LocalUserID,
*req.DocID, userID,
).Scan(&ok)
if errors.Is(err, sql.ErrNoRows) {
httputil.ErrorJSON(w, http.StatusBadRequest, "unknown doc_id")
@@ -194,14 +197,14 @@ func (h *Handler) capture(w http.ResponseWriter, r *http.Request) {
phonetic = excluded.phonetic,
example = CASE WHEN excluded.example != '' THEN excluded.example ELSE vocab_words.example END,
doc_id = COALESCE(excluded.doc_id, vocab_words.doc_id)`,
db.LocalUserID, word, req.Gloss, req.Definition, req.Phonetic, req.Example, req.DocID,
userID, word, req.Gloss, req.Definition, req.Phonetic, req.Example, req.DocID,
)
if err != nil {
httputil.ServerError(w, err)
return
}
out, err := h.fetch(word)
out, err := h.fetch(userID, word)
if err != nil {
httputil.ServerError(w, err)
return
@@ -210,10 +213,10 @@ func (h *Handler) capture(w http.ResponseWriter, r *http.Request) {
}
// fetch loads one word row by its (user, word) key.
func (h *Handler) fetch(word string) (Word, error) {
func (h *Handler) fetch(userID, word string) (Word, error) {
return scanWord(h.DB.QueryRow(
`SELECT `+vocabColumns+` FROM vocab_words WHERE user_id = ? AND word = ?`,
db.LocalUserID, word,
userID, word,
))
}
@@ -225,6 +228,7 @@ type reviewRequest struct {
// scheduler; the new interval is applied as `due_at = now + interval days`.
func (h *Handler) review(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
userID := auth.UserID(r.Context())
var req reviewRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
httputil.ErrorJSON(w, http.StatusBadRequest, "invalid body")
@@ -249,7 +253,7 @@ func (h *Handler) review(w http.ResponseWriter, r *http.Request) {
var cur State
err = tx.QueryRow(
`SELECT reps, interval_days, ease, lapses FROM vocab_words WHERE id = ? AND user_id = ?`,
id, db.LocalUserID,
id, userID,
).Scan(&cur.Reps, &cur.Interval, &cur.Ease, &cur.Lapses)
if errors.Is(err, sql.ErrNoRows) {
httputil.ErrorJSON(w, http.StatusNotFound, "word not found")
@@ -269,14 +273,14 @@ func (h *Handler) review(w http.ResponseWriter, r *http.Request) {
reps = ?, interval_days = ?, ease = ?, lapses = ?,
last_reviewed = datetime('now'), due_at = datetime('now', ?)
WHERE id = ? AND user_id = ?`,
nxt.Reps, nxt.Interval, nxt.Ease, nxt.Lapses, offset, id, db.LocalUserID,
nxt.Reps, nxt.Interval, nxt.Ease, nxt.Lapses, offset, id, userID,
); err != nil {
httputil.ServerError(w, err)
return
}
out, err := scanWord(tx.QueryRow(
`SELECT `+vocabColumns+` FROM vocab_words WHERE id = ? AND user_id = ?`, id, db.LocalUserID,
`SELECT `+vocabColumns+` FROM vocab_words WHERE id = ? AND user_id = ?`, id, userID,
))
if err != nil {
httputil.ServerError(w, err)
@@ -293,7 +297,7 @@ func (h *Handler) review(w http.ResponseWriter, r *http.Request) {
func (h *Handler) remove(w http.ResponseWriter, r *http.Request) {
res, err := h.DB.Exec(
`DELETE FROM vocab_words WHERE id = ? AND user_id = ?`,
chi.URLParam(r, "id"), db.LocalUserID,
chi.URLParam(r, "id"), auth.UserID(r.Context()),
)
if err != nil {
httputil.ServerError(w, err)
internal/vocab/handlers_test.go
+7 -1
View File
@@ -11,6 +11,7 @@ import (
"github.com/go-chi/chi/v5"
"gitea.parodia.dev/drwily/petal/internal/auth"
"gitea.parodia.dev/drwily/petal/internal/db"
)
@@ -23,7 +24,12 @@ func newTestServer(t *testing.T) (http.Handler, *db.DB) {
t.Cleanup(func() { database.Close() })
r := chi.NewRouter()
r.Mount("/vocab", New(database).Routes())
return r, database
// Behind the same auth middleware main.go installs: handlers resolve the
// caller from the request context, so a bare router would see no user and
// every user-scoped query would match nothing.
authed := auth.Middleware(auth.StaticResolver(db.LocalUserID))(r)
return authed, database
}
func do(t *testing.T, srv http.Handler, method, path, body string) *httptest.ResponseRecorder {
internal/vocab/plant.go
+100
View File
@@ -0,0 +1,100 @@
package vocab
import (
"database/sql"
"strings"
"unicode"
)
// Planting: the garden's second source.
//
// Capture (handlers.go) records words the writer *sought out*. Planting records
// phrasing she was gently *given* — an accepted collocation like "make a
// decision" is a learnable chunk exactly like a looked-up word, and the SM-2-lite
// scheduler doesn't care that it's three words rather than one. Together the two
// halves make the garden a record of both sides of learning.
//
// Everything here is best-effort by design: planting hangs off accepting a
// suggestion, and that accept must succeed whether or not a card comes of it.
// Execer is the slice of *sql.DB (or *sql.Tx) that planting needs.
type Execer interface {
Exec(query string, args ...any) (sql.Result, error)
}
// Phrase is one chunk to plant.
type Phrase struct {
Text string // the corrected phrasing, e.g. "make a decision"
Meaning string // why it's better — the suggestion's explanation
Example string // the sentence she met it in, already corrected
DocID *string // where, so "where did I see this?" stays one tap
}
// Phrase-card caps. A collocation is a short chunk; anything longer is a
// rewritten sentence wearing a collocation's label, and a sentence makes a
// miserable flashcard. Both bounds are deliberately tight — the cost of
// skipping a real chunk is one missing card, the cost of planting a sentence is
// a garden the writer stops trusting.
const (
maxPhraseRunes = 60
maxPhraseWords = 6
minPhraseWords = 2
)
// PhraseKey normalizes a replacement into a garden key, or returns "" when the
// text isn't a plantable chunk.
//
// Lowercasing matches capture's normalization, so a phrase and a looked-up word
// share one UNIQUE(user_id, word) namespace rather than colliding sideways.
// Single words are rejected on purpose: a one-word fix is word choice, and word
// choice already reaches the garden through lookup — planting it here would give
// it a card with no gloss and no phonetic, which reviews badly.
func PhraseKey(text string) string {
// Collapse all whitespace (a replacement can carry a newline from the
// editor) so the key is stable and the word count is honest.
s := strings.Join(strings.Fields(strings.ToLower(text)), " ")
// Trim the punctuation a phrase picks up from the sentence around it, but
// leave inner marks alone: "can't afford" and "in one's own time" are chunks.
s = strings.Trim(s, `.,;:!?…"'“”‘’()[]`)
s = strings.TrimSpace(s)
if s == "" || len([]rune(s)) > maxPhraseRunes {
return ""
}
n := len(strings.Fields(s))
if n < minPhraseWords || n > maxPhraseWords {
return ""
}
// A chunk of pure digits or symbols ("12 000", "-- --") isn't vocabulary.
if !strings.ContainsFunc(s, unicode.IsLetter) {
return ""
}
return s
}
// Plant adds a phrase card to the garden, due tomorrow like any fresh capture.
// It reports whether a new card was created.
//
// ON CONFLICT DO NOTHING, unlike capture's refresh-the-context upsert: accepting
// the same collocation again months later is evidence the chunk is still being
// learned, and the last thing that should do is overwrite the card's first
// context or disturb a schedule it has been climbing. An existing card wins.
func Plant(ex Execer, userID string, p Phrase) (bool, error) {
key := PhraseKey(p.Text)
if key == "" {
return false, nil
}
res, err := ex.Exec(
`INSERT INTO vocab_words (user_id, word, gloss, definition, phonetic, example, doc_id, due_at, interval_days)
VALUES (?, ?, '', ?, '', ?, ?, datetime('now', '+1 day'), 1)
ON CONFLICT(user_id, word) DO NOTHING`,
userID, key,
clamp(strings.TrimSpace(p.Meaning), maxDefinitionLen),
clamp(strings.TrimSpace(p.Example), maxExampleLen),
p.DocID,
)
if err != nil {
return false, err
}
n, err := res.RowsAffected()
return n > 0, err
}
internal/vocab/plant_test.go
+74
View File
@@ -0,0 +1,74 @@
package vocab
import (
"path/filepath"
"testing"
"gitea.parodia.dev/drwily/petal/internal/db"
)
func TestPhraseKey(t *testing.T) {
cases := []struct {
in, want string
}{
{"make a decision", "make a decision"},
{"Make A Decision", "make a decision"}, // shares one namespace with lookups
{" make a\ndecision ", "make a decision"}, // the editor's whitespace
{"“make a decision.”", "make a decision"}, // punctuation from the sentence around it
{"can’t afford it", "can’t afford it"}, // inner marks are part of the chunk
{"decision", ""}, // word choice, not a chunk — lookup's job
{"", ""}, //
{"...", ""}, //
{"12 000", ""}, // digits aren't vocabulary
{"a b c d e f g", ""}, // a clause wearing a chunk's label
{"in one’s own good time again", "in one’s own good time again"}, // six words is still a chunk
}
for _, tc := range cases {
if got := PhraseKey(tc.in); got != tc.want {
t.Errorf("PhraseKey(%q) = %q, want %q", tc.in, got, tc.want)
}
}
// The length cap counts runes, not bytes — otherwise a Portuguese chunk well
// inside the limit would be dropped for being accented.
accented := "ãããããã ãããããã ãããããã ãããããã ãããããã" // 34 runes, 64 bytes
if got := PhraseKey(accented); got != accented {
t.Errorf("PhraseKey(%d runes / %d bytes) = %q, want it kept", len([]rune(accented)), len(accented), got)
}
long := "ãããããããããããã ãããããããããããã ãããããããããããã ãããããããããããã ãããããããããããã ãããããããããããã"
if got := PhraseKey(long); got != "" {
t.Errorf("PhraseKey(%d runes) = %q, want \"\"", len([]rune(long)), got)
}
}
func TestPlantCreatesOnceAndReportsIt(t *testing.T) {
database, err := db.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatalf("open db: %v", err)
}
t.Cleanup(func() { database.Close() })
p := Phrase{Text: "make a decision", Meaning: "why", Example: "I had to make a decision."}
created, err := Plant(database, db.LocalUserID, p)
if err != nil || !created {
t.Fatalf("first plant: created=%v err=%v", created, err)
}
created, err = Plant(database, db.LocalUserID, p)
if err != nil || created {
t.Fatalf("second plant: created=%v err=%v, want false", created, err)
}
// A card that isn't plantable is a silent no-op, not an error: planting hangs
// off accepting an edit, and that accept must never fail for a flashcard.
created, err = Plant(database, db.LocalUserID, Phrase{Text: "decision"})
if err != nil || created {
t.Fatalf("unplantable: created=%v err=%v", created, err)
}
var n int
if err := database.QueryRow(`SELECT count(*) FROM vocab_words WHERE user_id = ?`, db.LocalUserID).Scan(&n); err != nil {
t.Fatalf("count: %v", err)
}
if n != 1 {
t.Fatalf("garden has %d cards, want 1", n)
}
}
scripts/build_hunspell_dictionary.py
+450
View File
@@ -0,0 +1,450 @@
#!/usr/bin/env python3
"""Build one of Petal's browser spelling dictionaries from a Hunspell one.
Why this script exists at all
----------------------------
English is vendored the obvious way: `dictionary-en`'s `en.aff` + `en.dic` go
into web/public/dictionaries/en and nspell reads them in the browser. The plan
for Phase 21 said "Hunspell pt-PT vendored like en-US", and that turns out not to
work, for a measured reason.
nspell expands affixes **eagerly at load time** — it materialises every surface
form into a hash the moment you construct it. English gets away with this: ~50k
stems and a small rule set. European Portuguese does not. `pt_PT.aff` carries
1,340 affix rules (the full verb paradigm: six persons x a dozen tenses, plus
diminutives, plus productive prefixes) over 44,257 stems. Measured on this
machine, nspell needed ~340 MB of heap for the first 12,000 entries alone and had
not returned after three minutes on the whole file; extrapolated, it wants well
over a gigabyte. That is not something to hand a browser, still less a tablet.
French is larger again: 5,600 affix rules over 84,140 stems.
So the expansion happens **here**, once, at build time, and the browser gets a
flat word list it can load with no affix machinery at all. The runtime code path
is then *identical* to English — same nspell, same interface — which is the real
prize. The aff shipped alongside keeps only the suggestion-shaping directives
(TRY/KEY/REP/MAP), so corrections still know that "cao" wants "ção" and that a
missing acute accent is a near miss.
What Phase 24 had to add
------------------------
The build plan recorded that the pt-PT version of this script "generalizes" to
French. It did not. It handled single-character flags and plain PFX/SFX and
stopped on anything else — the right call, because `fr.aff` uses four of the
things it stopped on, and getting any of them wrong changes which words are
accepted:
* **`FLAG long`** — French flags are *two characters* (`S.`, `L'`, `Um`). The
pt-PT reader took `set(flagstr)`, one flag per character, which on a French
entry yields a bag of unrelated single letters: every entry would have been
expanded through the wrong paradigm. This is the one that fails silently.
* **Continuation flags** — pt-PT's affixes append plain text, so that script
dropped anything after a `/` and asserted the drop was safe. French really
does affix an affixed form: `PFX Um 0 0/S.` says the prefixed form then takes
the plural suffix, and the elision prefixes arrive the same way from the
other side (`SFX ... ait/n'q'l'm't's'`).
* **`NEEDAFFIX`** — French marks thousands of stems "not a word on its own"
(`Allemagne/S.()`), the bare form arriving instead through a zero-append
rule. Ignoring the flag accepts stems the real dictionary rejects.
* **`FULLSTRIP`** — a rule may strip the whole stem.
`CIRCUMFIX` and `FORBIDDENWORD` are *declared* in `fr.aff` and used by nothing,
which this script asserts rather than assumes: an upstream release that started
using either would otherwise change what is accepted without changing this file.
`KEEPCASE` and `NOSUGGEST` are honoured by being ignored on purpose — they shape
casing and suggestions, not membership, and a NOSUGGEST word is still a word.
Elision is handled at lookup, not here — and that is the size decision
----------------------------------------------------------------------
Most of French's affix machinery by volume is elision: `l'`, `d'`, `qu'`, `j'`,
`n'`, `s'`, `jusqu'`, `puisqu'`. Hunspell treats `l'arbre` as one word, so a
faithful expansion carries much of the language thirty-four times over — and
Petal's tokenizer keeps internal apostrophes, so `l'arbre` really does arrive at
the dictionary as one token and really would be underlined if it were absent.
Both halves were built and measured. Keeping the elided forms: **3,159,832 forms,
8.25 MB gzipped**, ~45 MB of text for nspell to hash on a tablet. Dropping them:
**473,326 forms, 1.19 MB gzipped**. The elided seven-eighths are not new words —
they are thirteen little words glued to words already in the list — so the third
option is the one taken: rules whose append carries an apostrophe are skipped
here (the count is printed), and `withElision` in `useSpellChecker.ts` splits a
token at a *known clitic* and checks the remainder. `l'arbre` costs one extra
lookup instead of seven megabytes, and `zzz'arbre` is still flagged because
`zzz` is not one of the thirteen.
Stems that carry an apostrophe of their own — `aujourd'hui`, `quelqu'un`,
`presqu'île`, `prud'homme` — are dictionary entries rather than affixed forms,
so they are kept verbatim and matched directly. `entr'aide` and `grand'mère` are
absent for the same reason they are absent from Dicollecte: modern French spells
them `entraide` and `grand-mère`.
Choosing the source
-------------------
Both languages have a trap here, and they are different traps.
**pt-PT: the wrong country.** npm's `dictionary-pt` is not European Portuguese.
Both it and `dictionary-pt-br` package VERO ("Verificador Ortográfico Livre",
Brasil), so vendoring the obvious npm name would have shipped Brazilian spellings
under a pt-PT label — the pt-BR drift SUGGESTIONS.md §3 warns about, arriving
through the packaging rather than through the model. The authentic dictionary is
the Projecto Natura one (Universidade do Minho) that LibreOffice ships and Debian
packages as `hunspell-pt-pt`; its aff declares `LANG pt_PT`.
**fr: the wrong side of an argument the French have not settled.** The regional
question turns out to be a non-question — Debian's `fr_FR`, `fr_CA`, `fr_BE`,
`fr_CH`, `fr_LU` and `fr_MC` are all symlinks to one `fr.dic`, so unlike pt there
is no country here to get wrong. What there is instead is the 1990 spelling
reform, packaged three ways: `hunspell-fr-classical` (traditional), `-revised`
(reform only) and `-comprehensive` (both). Petal ships **comprehensive**, because
Petal never corrects her French — the only thing this dictionary can do is
underline something. *coût* and *cout* are both correct French, taught in
different decades to different people, and a writing companion has no business
underlining one of them to take a side. The `fr` MUST_ACCEPT list is written to
*prove* which package was used: classical rejects `cout`, revised rejects `coût`,
and only comprehensive accepts both.
Licensing: pt-PT is GPL-2 or LGPL-2.1 or MPL-1.1, (c) José João de Almeida, Rui
Vilela, Alberto Simões. fr is MPL-2.0, (c) 2007-2018 the Dicollecte contributors
(grammalecte.net). The upstream copyright file is vendored beside each output.
Usage
-----
apt-get download hunspell-fr-comprehensive # or hunspell-pt-pt
dpkg-deb -x hunspell-fr-comprehensive_*.deb src
python3 scripts/build_hunspell_dictionary.py fr \\
src/usr/share/hunspell/fr.aff \\
src/usr/share/hunspell/fr.dic \\
web/public/dictionaries/fr
"""
import gzip
import os
import re
import sys
import unicodedata
# Directives worth keeping in the shipped aff. These shape *suggestions*, not
# membership: TRY orders the alphabet the corrector tries, KEY knows which keys
# are adjacent, REP holds the language's own confusions (cao/ção, ss/ç), and MAP
# says an accented vowel and its bare form are the same letter for scoring —
# which is most of what an ESL writer gets wrong in either language.
KEEP_DIRECTIVES = ("SET", "TRY", "KEY", "REP", "MAP", "WORDCHARS")
# Directives that would change which words are *accepted* and that this expander
# does not implement. If a future upstream release starts using one, the output
# would silently disagree with the real dictionary, so the build stops instead.
UNSUPPORTED = (
"COMPOUNDFLAG", "COMPOUNDMIN", "COMPOUNDRULE", "COMPOUNDBEGIN",
"ONLYINCOMPOUND", "PSEUDOROOT", "AF", "AM",
)
APOSTROPHES = "'’"
class Aff:
"""The parts of an .aff file that decide which words exist."""
def __init__(self):
self.pfx = {} # flag -> [(strip, append, condition, continuation)]
self.sfx = {}
# Cross-product, per flag — and kept per table, because PFX and SFX are
# separate flag namespaces in hunspell: the same flag may name a prefix
# table and a suffix table, with different cross-product settings. One
# shared dict let the second block silently overwrite the first.
self.cross_pfx = {} # flag -> bool
self.cross_sfx = {}
self.flag_kind = "char"
self.needaffix = None
self.circumfix = None
self.forbidden = None
self.dropped_apostrophe_rules = 0
def parse_flags(raw, kind):
"""Split a flag string into flags, per the aff's FLAG declaration."""
raw = raw.strip()
if not raw:
return set()
if kind == "long":
# Two characters per flag, exactly. An odd length is a malformed flag
# string, and silently dropping the trailing character would quietly
# expand an entry through the wrong paradigm — the failure mode this
# whole FLAG-aware rewrite exists to avoid.
if len(raw) % 2:
raise SystemExit(f"odd-length long flag string {raw!r}")
return {raw[i:i + 2] for i in range(0, len(raw), 2)}
if kind == "num":
return {f for f in raw.split(",") if f}
return set(raw)
def parse_aff(path):
with open(path, encoding="utf-8") as fh:
lines = fh.read().splitlines()
aff = Aff()
# FLAG has to be known before anything containing a flag is read, and it can
# sit anywhere in the file. So: header pass first, rules second.
for line in lines:
parts = line.split()
if not parts:
continue
head = parts[0]
if head in UNSUPPORTED:
raise SystemExit(
f"{path}: unsupported directive {head!r} — this expander handles "
"PFX/SFX affixation with continuation flags, and honouring "
f"{head} would change which words are accepted. Extend the "
"script before shipping."
)
if len(parts) < 2:
continue
if head == "FLAG":
aff.flag_kind = parts[1]
if aff.flag_kind not in ("long", "num", "UTF-8"):
raise SystemExit(f"{path}: unknown FLAG type {aff.flag_kind!r}")
elif head == "NEEDAFFIX":
aff.needaffix = parts[1]
elif head == "CIRCUMFIX":
aff.circumfix = parts[1]
elif head == "FORBIDDENWORD":
aff.forbidden = parts[1]
i = 0
while i < len(lines):
parts = lines[i].split()
if parts and parts[0] in ("PFX", "SFX"):
kind, flag, cross_flag, count = parts[0], parts[1], parts[2], int(parts[3])
table = aff.pfx if kind == "PFX" else aff.sfx
cross = aff.cross_pfx if kind == "PFX" else aff.cross_sfx
cross[flag] = cross_flag == "Y"
rules = table.setdefault(flag, [])
for j in range(1, count + 1):
p = lines[i + j].split()
strip = "" if p[2] == "0" else p[2]
append, _, cont_raw = p[3].partition("/")
if append == "0":
append = ""
# Elision. See the header: these forms are `l'` and its twelve
# siblings glued to words already in the list, they multiply the
# download by seven, and `withElision` reconstructs them at
# lookup for the cost of one extra hash probe.
if any(a in append for a in APOSTROPHES):
aff.dropped_apostrophe_rules += 1
continue
cont = parse_flags(cont_raw, aff.flag_kind)
if aff.circumfix and aff.circumfix in cont:
raise SystemExit(
f"{path}: CIRCUMFIX is used by a {kind} {flag} rule. It "
"was declared-but-unused when this expander was written "
"and is not implemented; honouring it would change which "
"words are accepted."
)
cond = p[4] if len(p) > 4 else "."
anchored = ("^" + cond) if kind == "PFX" else (cond + "$")
rules.append((strip, append, re.compile(anchored), cont))
i += count + 1
continue
i += 1
return aff
def apply_suffix(word, rules):
"""Every (form, continuation flags) a suffix table yields for `word`."""
out = []
for strip, append, cond, cont in rules:
if strip and not word.endswith(strip):
continue
if not cond.search(word):
continue
stem = word[: len(word) - len(strip)] if strip else word
out.append((stem + append, cont))
return out
def apply_prefix(word, rules):
out = []
for strip, append, cond, cont in rules:
if strip and not word.startswith(strip):
continue
if not cond.search(word):
continue
stem = word[len(strip):] if strip else word
out.append((append + stem, cont))
return out
def expand_entry(word, flags, aff, out):
"""Add every surface form of one dictionary entry to `out`.
Hunspell's model without compounding: a form is the stem plus at most one
prefix and at most one suffix. A flag reaches an affix either from the stem's
own flags or from the continuation flags of the affix applied on the other
side; when both sides apply, both rules must be declared cross-product.
NEEDAFFIX is why the bare form is not simply added: the flag says *this* form
is not a word, only whatever can be built from it — and it arrives both on
stems and on continuations.
"""
def is_word(carried):
return not (aff.needaffix and aff.needaffix in carried)
if is_word(flags):
out.add(word)
suffixed = [] # (form, flag, continuation flags)
for f in flags:
if f in aff.sfx:
for form, cont in apply_suffix(word, aff.sfx[f]):
suffixed.append((form, f, cont))
if is_word(cont):
out.add(form)
prefixed = []
for f in flags:
if f in aff.pfx:
for form, cont in apply_prefix(word, aff.pfx[f]):
prefixed.append((form, f, cont))
if is_word(cont):
out.add(form)
# Prefix then suffix. The suffix flag may come from the stem or from the
# prefix's own continuation (`PFX Um 0 0/S.`), and the suffix condition is
# matched against the whole prefixed word, which is what hunspell does.
# NEEDAFFIX is checked here too, exactly as on the single-affix paths above:
# a doubly-affixed form whose last continuation still carries the flag is
# "not a word on its own", and without compounding there is no third affix
# left to make it one.
for form, pf, pcont in prefixed:
if not aff.cross_pfx.get(pf):
continue
for f in flags | pcont:
if f in aff.sfx and aff.cross_sfx.get(f):
for full, fcont in apply_suffix(form, aff.sfx[f]):
if is_word(fcont):
out.add(full)
# Suffix then prefix — the same pair reached from the other side, which is
# how the elision prefixes arrive in French. Only the flags the suffix hands
# forward are new here; the stem's own were covered above.
for form, sf, scont in suffixed:
if not aff.cross_sfx.get(sf):
continue
for f in scont:
if f in aff.pfx and aff.cross_pfx.get(f):
for full, fcont in apply_prefix(form, aff.pfx[f]):
if is_word(fcont):
out.add(full)
def expand(aff_path, dic_path):
aff = parse_aff(aff_path)
forms = set()
needaffix_stems = 0
with open(dic_path, encoding="utf-8") as fh:
fh.readline() # leading entry count, not a word
for raw in fh:
# Morphological fields (po:nom is:fem) follow the entry, separated by
# a tab in pt-PT and by a space in fr.
entry = raw.strip().split("\t")[0].split(" ")[0]
if not entry:
continue
word, _, flagstr = entry.partition("/")
word = word.strip()
if not word:
continue
flags = parse_flags(flagstr, aff.flag_kind)
if aff.forbidden and aff.forbidden in flags:
raise SystemExit(
f"{dic_path}: FORBIDDENWORD is in use ({word!r}). It was "
"declared-but-unused when this expander was written; the "
"forms it removes would be wrongly accepted."
)
if aff.needaffix and aff.needaffix in flags:
needaffix_stems += 1
expand_entry(word, flags, aff, forms)
# NFC, because the aff's own ICONV table normalises decomposed accents on the
# way in and the browser hands nspell whatever the keyboard produced.
forms = {unicodedata.normalize("NFC", f) for f in forms}
return forms, aff, needaffix_stems
def shipped_aff(aff_path):
keep = []
for line in open(aff_path, encoding="utf-8").read().splitlines():
head = line.split()[0] if line.split() else ""
if head in KEEP_DIRECTIVES:
keep.append(line)
return "\n".join(keep) + "\n"
# Words the built list must accept, and must reject, before it is written. Each
# set is chosen to fail loudly on the *specific* wrong source that language has a
# packaged, plausible way of reaching — not to spot-check spelling in general.
PROFILES = {
# The pt-PT/pt-BR fault lines: post-Acordo spellings, the European lexicon,
# and the first-person-plural preterite accent that only pt-PT writes.
"pt-PT": {
"accept": ("receção", "húmido", "telemóvel", "autocarro", "comboio",
"ótimo", "pensámos", "escrevêssemos", "jardim"),
"reject": ("recepção", "úmido", "ônibus", "óptimo"),
"wrong": "this does not look like European Portuguese",
},
# Which of the three 1990-reform packagings this is. `coût`/`cout` and
# `paraître`/`paraitre` are each accepted by exactly one of classical and
# revised, so a build accepting all four is comprehensive and one that drops
# any of them is not. `Allemagne` is a NEEDAFFIX stem reachable only through
# a zero-append rule and `km` only through a prefix continuation, so between
# them they also check that this expander honoured the two features the
# pt-PT one refused.
"fr": {
"accept": ("coût", "cout", "paraître", "paraitre", "nénuphar", "nénufar",
"oignon", "ognon", "événement", "évènement", "jardin",
"Allemagne", "écrivissions", "km"),
"reject": ("jardinn", "écrivaitz", "xyzzyque"),
"wrong": "this does not look like the comprehensive French dictionary",
},
}
def main(lang, aff_path, dic_path, out_dir):
profile = PROFILES.get(lang)
if profile is None:
raise SystemExit(f"no profile for {lang!r}; known: {', '.join(PROFILES)}")
forms, aff, needaffix_stems = expand(aff_path, dic_path)
missing = [w for w in profile["accept"] if w not in forms]
present = [w for w in profile["reject"] if w in forms]
if missing or present:
raise SystemExit(
f"{profile['wrong']}: missing {missing}, unexpectedly present {present}"
)
os.makedirs(out_dir, exist_ok=True)
ordered = sorted(forms)
body = f"{len(ordered)}\n" + "\n".join(ordered) + "\n"
dic_out = os.path.join(out_dir, f"{lang}.dic.gz")
# mtime=0 so rebuilding identical input produces an identical file — a
# vendored asset that changes on every build is noise in the diff.
with gzip.GzipFile(dic_out, "wb", compresslevel=9, mtime=0) as fh:
fh.write(body.encode("utf-8"))
aff_out = os.path.join(out_dir, f"{lang}.aff")
with open(aff_out, "w", encoding="utf-8") as fh:
fh.write(shipped_aff(aff_path))
print(f"{len(ordered)} forms -> {dic_out} "
f"({os.path.getsize(dic_out) / 1e6:.2f} MB gzipped); "
f"{needaffix_stems} NEEDAFFIX stems, "
f"{aff.dropped_apostrophe_rules} elision rules skipped", file=sys.stderr)
if __name__ == "__main__":
if len(sys.argv) != 5:
raise SystemExit(
"usage: build_hunspell_dictionary.py <lang> <aff> <dic> <out-dir>\n"
f" lang is one of: {', '.join(PROFILES)}"
)
main(*sys.argv[1:5])
scripts/migrate_local_user.py
Executable
+227
View File
@@ -0,0 +1,227 @@
#!/usr/bin/env python3
"""Move the pre-auth `local` user's writing onto a real account.
Petal ran as a single hardcoded user (`users.id = 'local'`) for its whole life
before sign-in existed. Everything she has written is owned by that row. This
script re-points it at the account she now signs in as.
Why a script and not a startup migration: the destination is an OIDC subject
id, which is not knowable from inside the app — it belongs to the identity
provider. Running it deliberately, with the app stopped and a backup taken, is
also the only way to be sure nothing is writing to the database halfway through.
What moves: `documents`, `tags`, `vocab_words` and `images` carry `user_id`
directly. `document_versions`, `suggestions` and `document_tags` hang off their
parents and follow without being touched — which is exactly why this must be one
transaction with foreign keys off: re-pointing a parent while its children are
enforced would either fail or cascade.
Sessions belonging to the old identity are deleted rather than moved. A session
is proof that *someone signed in*, and nobody ever signed in as `local`.
Safety: dry-run unless --apply; refuses to run while anything else has the
database open; refuses if the destination already owns writing of its own; takes
a `VACUUM INTO` backup first; and verifies every row it expected to move
actually moved before it commits.
Usage:
# look at what would happen
python3 scripts/migrate_local_user.py data/petal.db --to <oidc-sub>
# do it
python3 scripts/migrate_local_user.py data/petal.db --to <oidc-sub> \\
--email her@example.com --name "Her Name" --apply
The subject id comes from the identity provider. For authentik with the default
`hashed_user_id` sub mode it is the user's `uid`, which is stable and knowable
before she has ever logged in:
docker exec authentik-server-1 ak shell -c \\
"from authentik.core.models import User; print(User.objects.get(username='claire').uid)"
"""
import argparse
import os
import sqlite3
import sys
import time
# The tables that name an owner directly. Everything else in the schema reaches
# its owner through one of these.
OWNED_TABLES = ("documents", "tags", "vocab_words", "images")
LOCAL_USER = "local"
def die(msg: str) -> None:
print(f"error: {msg}", file=sys.stderr)
sys.exit(1)
def open_db(path: str) -> sqlite3.Connection:
if not os.path.exists(path):
die(f"no database at {path}")
conn = sqlite3.connect(path, isolation_level=None)
conn.row_factory = sqlite3.Row
return conn
def table_exists(conn: sqlite3.Connection, name: str) -> bool:
row = conn.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (name,)
).fetchone()
return row is not None
def counts(conn: sqlite3.Connection, user_id: str) -> dict[str, int]:
out = {}
for table in OWNED_TABLES:
if not table_exists(conn, table):
continue
out[table] = conn.execute(
f"SELECT COUNT(*) FROM {table} WHERE user_id = ?", (user_id,)
).fetchone()[0]
return out
def require_app_stopped(path: str) -> None:
"""Fail unless nothing else has the database open.
`BEGIN EXCLUSIVE` is not enough, and quietly so: in WAL mode it only
conflicts with another *writer*, so an idle-but-running Petal sails straight
past it — which is precisely the case this guard exists to catch. Taking
`locking_mode = EXCLUSIVE` conflicts with any other connection at all,
because it locks the shared-memory index every WAL reader must map.
Checking for a `-wal` file would be no use either: WAL mode leaves one
behind whether or not anything is running.
This runs on its own connection, which is then closed. Setting locking_mode
back to NORMAL does not release the lock straight away — SQLite drops it on
the connection's next database access, which on a WAL database can leave the
file locked against everything else this script is about to do. Closing is
the only unambiguous way to let go of it.
"""
probe = sqlite3.connect(path, isolation_level=None)
try:
probe.execute("PRAGMA locking_mode = EXCLUSIVE")
probe.execute("BEGIN IMMEDIATE")
probe.execute("COMMIT")
except sqlite3.OperationalError as err:
die(
f"{path} is in use ({err}). Stop Petal first:\n"
" docker compose stop petal # VPS\n"
" systemctl --user stop petal # millenia"
)
finally:
probe.close()
def backup(path: str) -> str:
"""Snapshot the database with VACUUM INTO — one coherent file including the
WAL, taken without a write lock, and it refuses to overwrite."""
dest = f"{path}.pre-migrate-{time.strftime('%Y%m%d-%H%M%S')}"
conn = sqlite3.connect(path)
try:
conn.execute("VACUUM INTO ?", (dest,))
finally:
conn.close()
return dest
def main() -> None:
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
ap.add_argument("database", help="path to petal.db")
ap.add_argument("--to", required=True, metavar="SUB",
help="OIDC subject id of the destination account")
ap.add_argument("--from", dest="source", default=LOCAL_USER,
help=f"account to move from (default: {LOCAL_USER})")
ap.add_argument("--email", default="", help="email for the destination account, if it doesn't exist yet")
ap.add_argument("--name", default="", help="display name for the destination account")
ap.add_argument("--pair-lang", default="zh", help="language pair for a newly created account (default: zh)")
ap.add_argument("--apply", action="store_true", help="actually commit (default: dry run)")
ap.add_argument("--no-backup", action="store_true", help="skip the pre-migration snapshot")
args = ap.parse_args()
if args.to == args.source:
die("source and destination are the same account")
require_app_stopped(args.database)
conn = open_db(args.database)
src_counts = counts(conn, args.source)
if not any(src_counts.values()):
die(f"account {args.source!r} owns nothing in this database — wrong file, or already migrated?")
dst_counts = counts(conn, args.to)
if any(dst_counts.values()):
die(
f"account {args.to!r} already owns writing here "
f"({', '.join(f'{k}={v}' for k, v in dst_counts.items() if v)}). "
"Refusing to merge two accounts — that is not something this script can undo."
)
dst = conn.execute("SELECT id, email, display_name FROM users WHERE id = ?", (args.to,)).fetchone()
print(f"database: {args.database}")
print(f"moving from: {args.source}")
print(f"moving to: {args.to}" + ("" if dst else " (will be created)"))
for table, n in src_counts.items():
print(f" {table:<14} {n}")
sessions = 0
if table_exists(conn, "sessions"):
sessions = conn.execute(
"SELECT COUNT(*) FROM sessions WHERE user_id = ?", (args.source,)
).fetchone()[0]
print(f" {'sessions':<14} {sessions} (deleted, not moved)")
if not args.apply:
print("\ndry run — nothing changed. Re-run with --apply to commit.")
return
if not args.no_backup:
dest = backup(args.database)
print(f"\nbackup: {dest}")
# Foreign keys off for the duration: children reference the parents being
# re-pointed, and this is one atomic swap of an identity, not a data change.
conn.execute("PRAGMA foreign_keys = OFF")
conn.execute("BEGIN EXCLUSIVE")
try:
if not dst:
conn.execute(
"INSERT INTO users (id, email, display_name, pair_lang) VALUES (?, ?, ?, ?)",
(args.to, args.email, args.name or args.email, args.pair_lang),
)
for table in src_counts:
conn.execute(
f"UPDATE {table} SET user_id = ? WHERE user_id = ?", (args.to, args.source)
)
if table_exists(conn, "sessions"):
conn.execute("DELETE FROM sessions WHERE user_id = ?", (args.source,))
# Verify before committing: every row that was the source's is now the
# destination's, and the source owns nothing.
moved = counts(conn, args.to)
left = counts(conn, args.source)
if moved != src_counts or any(left.values()):
raise RuntimeError(
f"row counts do not match after the move (expected {src_counts}, "
f"got {moved}, source still holds {left})"
)
conn.execute("DELETE FROM users WHERE id = ?", (args.source,))
conn.execute("COMMIT")
except Exception as err: # noqa: BLE001 — any failure must roll the whole thing back
conn.execute("ROLLBACK")
die(f"migration rolled back: {err}")
finally:
conn.execute("PRAGMA foreign_keys = ON")
print("\nmigrated. Start Petal and sign in as the destination account.")
for table, n in src_counts.items():
print(f" {table:<14} {n}")
if __name__ == "__main__":
main()
web/index.html
+5 -1
View File
@@ -2,7 +2,11 @@
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'><text y='14' font-size='14'>🌸</text></svg>" />
<!-- A drawn sakura rather than the 🌸 emoji: the emoji renders as whatever
each platform's font decides, which on some is not pink and on others
is not a blossom. This one is Petal's own rose palette everywhere, and
it doubles as the app tile in Authentik. -->
<link rel="icon" type="image/svg+xml" href="/petal.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Petal</title>
<link rel="preconnect" href="https://fonts.googleapis.com" />
web/public/dictionaries/fr/LICENSE
+49
View File
@@ -0,0 +1,49 @@
French spelling dictionary
==========================
The word list in `fr.dic.gz` and the suggestion directives in `fr.aff` are
derived from the Dicollecte / Grammalecte Hunspell dictionary for French
(`fr.aff` / `fr.dic`), as packaged by Debian/Ubuntu in
`hunspell-fr-comprehensive`.
Copyright (C) 2007-2018 the Dicollecte contributors
(full list at
https://grammalecte.net/members.php?prj=fr)
Dictionary author: Olivier R.
License: MPL-2.0
This Source Code Form is subject to the terms of the Mozilla
Public License, v. 2.0. If a copy of the MPL was not distributed
with this file, You can obtain one at
http://mozilla.org/MPL/2.0/.
Upstream: https://grammalecte.net/home.php?prj=fr
Which of the three
------------------
Debian packages this dictionary three ways, by how it treats the 1990 spelling
reform: `hunspell-fr-classical` (traditional spellings), `hunspell-fr-revised`
(reform spellings) and `hunspell-fr-comprehensive` (both). Petal ships the
**comprehensive** one, because Petal never corrects her French — the only thing
this dictionary can do is underline something, and *coût* and *cout* are both
correct French. The regional packages (`fr_FR`, `fr_CA`, `fr_BE`, `fr_CH`,
`fr_LU`, `fr_MC`) are all symlinks to the same word list, so there is no
regional choice being made here.
What Petal changed
------------------
`scripts/build_hunspell_dictionary.py` applies the upstream affix rules ahead of
time — Hunspell's PFX/SFX expansion run once at build time instead of once per
browser — and writes the resulting 473,326 surface forms as a flat word list.
The shipped `.aff` keeps only upstream's TRY/KEY/REP/MAP/WORDCHARS lines, which
shape *corrections* rather than membership.
One thing about membership did change, and it is reversible at lookup rather
than lost: the elided forms (`l'arbre`, `qu'elle`, `jusqu'ici`) are **not** in
the word list. Expanding them costs 8.25 MB gzipped against 1.19 MB, and they
are not new words — they are thirteen clitics glued to words already present —
so `withElision` in `web/src/hooks/useSpellChecker.ts` splits the token and
checks the remainder instead. Stems that carry an apostrophe of their own
(`aujourd'hui`, `quelqu'un`, `presqu'île`, `prud'homme`) are kept verbatim.
web/public/dictionaries/fr/fr.aff
+141
View File
@@ -0,0 +1,141 @@
SET UTF-8
WORDCHARS -’'1234567890.
TRY esntiarulodcpmévqfgbhàxèjyêMILzACçôîPâùJFSûBVœRDGNETHXkïOwKWYUëQÉZŒüãÎáöóÈíæÅñäśńÿ
MAP 25
MAP aàâäAÀÂÄ
MAP eéèêëEÉÈÊË
MAP iîïyIÎÏY
MAP oôöOÔÖ
MAP uùûüUÙÛÜ
MAP cçCÇ
MAP bB
MAP dD
MAP fF
MAP gG
MAP hH
MAP jJ
MAP kK
MAP lL
MAP mM
MAP nN
MAP pP
MAP qQ
MAP rR
MAP sS
MAP tT
MAP vV
MAP wW
MAP xX
MAP zZ
REP 110
REP a â
REP â a
REP e é
REP é e
REP e ê
REP ê e
REP e è
REP è e
REP i î
REP î i
REP o ô
REP ô o
REP u û
REP û u
REP A Â
REP Â A
REP E É
REP É E
REP E Ê
REP Ê E
REP E È
REP È E
REP I Î
REP Î I
REP O Ô
REP Ô O
REP U Û
REP Û U
REP ^Ca$ Ça
REP ^l l'
REP ^d d'
REP ^n n'
REP ^s s'
REP ^j j'
REP ^m m'
REP ^t t'
REP ^c c'
REP f ph
REP ph f
REP c qu
REP qu c
REP k qu
REP qu k
REP x ct
REP ct x
REP bb b
REP b bb
REP cc c
REP c cc
REP ff f
REP f ff
REP ll l
REP l ll
REP mm m
REP m mm
REP nn n
REP n nn
REP pp p
REP p pp
REP rr r
REP r rr
REP ss s
REP s ss
REP ss c
REP c ss
REP ss ç
REP ç ss
REP tt t
REP t tt
REP œ oe
REP oe œ
REP æ ae
REP ae æ
REP ai é
REP é ai
REP ai è
REP è ai
REP ai ê
REP ê ai
REP ei é
REP é ei
REP ei è
REP è ei
REP ei ê
REP ê ei
REP o au
REP au o
REP o eau
REP eau o
REP ett èt
REP èt ett
REP ell èl
REP èl ell
REP t th
REP th t
REP ième$ e
REP ème$ e
REP è$ e
REP mn$ min
REP ogue$ ogiste
REP ogiste$ ogue
REP disez$ dites
REP fesez$ faites
REP faisez$ faites
REP puit puits
REP sanctionnable punissable
REP questionnable discutable
REP antitartre détartrant
REP email courriel
REP construirent construisirent
KEY azertyuiop|qsdfghjklmù|wxcvbn|aéz|yèu|iço|oàp|aqz|zse|edr|rft|tgy|yhu|uji|iko|olpm|qws|sxd|dcf|fvg|gbh|hnj
web/public/dictionaries/fr/fr.dic.gz
BIN
View File
Binary file not shown.
web/public/dictionaries/pt-PT/LICENSE
+32
View File
@@ -0,0 +1,32 @@
European Portuguese spelling dictionary
=======================================
The word list in `pt-PT.dic.gz` and the suggestion directives in `pt-PT.aff` are
derived from the LibreOffice/Projecto Natura Hunspell dictionary for European
Portuguese (`pt_PT.aff` / `pt_PT.dic`), as packaged by Debian/Ubuntu in
`hunspell-pt-pt`.
Copyright (C) 2006-2012 José João de Almeida <jj@di.uminho.pt>
Rui Vilela <ruivilela@di.uminho.pt>
Alberto Simões <ambs@di.uminho.pt>
Universidade do Minho — Projecto Natura
License: GPL-2 or LGPL-2.1 or MPL-1.1
(Petal redistributes it under the MPL-1.1 option.)
Upstream: https://natura.di.uminho.pt/ — via
https://git.libreoffice.org/dictionaries/+/refs/heads/master/pt_PT
What Petal changed
------------------
Nothing about which words are correct. `scripts/build_hunspell_dictionary.py`
applies the upstream affix rules ahead of time — Hunspell's PFX/SFX expansion
run once at build time instead of once per browser — and writes the resulting
1,039,058 surface forms as a flat word list. The shipped `.aff` keeps only
upstream's TRY/KEY/REP/MAP/WORDCHARS lines, which shape *corrections* rather
than membership. See that script's header for why the dictionary could not be
vendored in its original form.
Note that npm's `dictionary-pt` is *not* this dictionary: both it and
`dictionary-pt-br` package the Brazilian VERO word list.
web/public/dictionaries/pt-PT/pt-PT.aff
+42
View File
@@ -0,0 +1,42 @@
SET UTF-8
TRY aerisontcdmlupvgbfzáhçqjíxãóéêâúõACMPSBTELGRIFVDkHJONôywUKXZWQÁYÍÉàÓèÂÚ
KEY qwertyuiop|asdfghjkl|zxcvbnm
WORDCHARS -
REP 25
REP por pro
REP pre per
REP damente mente
REP mente damente
REP iz íz
REP cao ção
REP ç ss
REP ss ç
REP c ss
REP ss c
REP ch x
REP x ch
REP cç x
REP x cç
REP k qu
REP íti ití
REP ití íti
REP issí íssi
REP ilí íli
REP íli ilí
REP ífi ifí
REP ifí ífi
REP nume mune
REP coen quen
REP concerteza com_certeza
MAP 11
MAP aá
MAP aã
MAP aâ
MAP eé
MAP eê
MAP ií
MAP cç
MAP oó
MAP oô
MAP oõ
MAP uú
web/public/dictionaries/pt-PT/pt-PT.dic.gz
BIN
View File
Binary file not shown.
web/public/petal.svg
+58
View File
@@ -0,0 +1,58 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" width="64" height="64" role="img" aria-label="Petal">
<title>Petal</title>
<defs>
<!-- Each petal is lighter at the tip and deepens toward the middle, the way
a real sakura petal does. Petal's own rose tokens: #E8A0BF / #D98AAF. -->
<radialGradient id="petal-fill" cx="50%" cy="88%" r="82%">
<stop offset="0%" stop-color="#D98AAF" />
<stop offset="45%" stop-color="#E8A0BF" />
<stop offset="100%" stop-color="#FBDDE7" />
</radialGradient>
<radialGradient id="petal-heart" cx="50%" cy="42%" r="65%">
<stop offset="0%" stop-color="#FFF3D6" />
<stop offset="100%" stop-color="#F2C878" />
</radialGradient>
<!-- One petal, pointing up from the flower's centre, notched at the tip. -->
<path id="petal-leaf"
d="M32 34
C 21.5 31.5, 15.5 23, 18 14.6
C 20 8.2, 26 5.4, 29.6 9.8
L 32 12.8
L 34.4 9.8
C 38 5.4, 44 8.2, 46 14.6
C 48.5 23, 42.5 31.5, 32 34 Z" />
</defs>
<g>
<!-- Five petals around the centre, each a rotated copy of the same shape.
They overlap generously and each keeps its own outline, so the petals
stay individually readable rather than merging into one silhouette. -->
<g fill="url(#petal-fill)" stroke="#D98AAF" stroke-width="1.4" stroke-linejoin="round">
<use href="#petal-leaf" transform="rotate(0 32 32)" />
<use href="#petal-leaf" transform="rotate(72 32 32)" />
<use href="#petal-leaf" transform="rotate(144 32 32)" />
<use href="#petal-leaf" transform="rotate(216 32 32)" />
<use href="#petal-leaf" transform="rotate(288 32 32)" />
</g>
<!-- Stamens: little dots on short stalks, kept inside the centre. -->
<g stroke="#E7B15F" stroke-width="1.3" stroke-linecap="round">
<path d="M32 32 L32 25.2" />
<path d="M32 32 L38.4 27.4" />
<path d="M32 32 L36 35.6" />
<path d="M32 32 L28 35.6" />
<path d="M32 32 L25.6 27.4" />
</g>
<g fill="#FFE9AE">
<circle cx="32" cy="24.6" r="1.9" />
<circle cx="38.9" cy="26.9" r="1.9" />
<circle cx="36.4" cy="36.2" r="1.9" />
<circle cx="27.6" cy="36.2" r="1.9" />
<circle cx="25.1" cy="26.9" r="1.9" />
</g>
<!-- The flower's heart. -->
<circle cx="32" cy="32" r="5.1" fill="url(#petal-heart)" stroke="#E7B15F" stroke-width="1.1" />
<circle cx="30.3" cy="30.4" r="1.5" fill="#FFF8E6" opacity="0.9" />
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 2.5 KiB

web/src/App.tsx
+66 -6
View File
@@ -1,5 +1,5 @@
import { useCallback, useEffect, useRef, useState } from 'react'
import { api, type DocSummary, type Document, type Suggestion, type Tag, type TagColor } from './api/client'
import { api, type DocSummary, type DocUpdate, type Document, type Suggestion, type Tag, type TagColor } from './api/client'
import { useAutoSave } from './hooks/useAutoSave'
import { useCheckpoint } from './hooks/useCheckpoint'
import { useSpellChecker } from './hooks/useSpellChecker'
@@ -13,8 +13,12 @@ import { GardenPanel } from './components/Garden/GardenPanel'
import { StatusBar } from './components/StatusBar/StatusBar'
import { PetalCompanion } from './components/Companion/PetalCompanion'
import { UpdateBanner } from './components/UpdateBanner/UpdateBanner'
import { SignInOverlay } from './components/Auth/SignInOverlay'
import { useSession } from './hooks/useSession'
import { takeDraft } from './lib/drafts'
import { useVersionWatch } from './hooks/useVersionWatch'
import { PetalFall } from './effects/PetalFall'
import { usePack } from './i18n'
import { useNightMode } from './hooks/useNightMode'
import { playSuggestionSound } from './audio/sounds'
@@ -24,6 +28,13 @@ export default function App() {
// toggles the `petal-night` class on <html>; we pass the flag to the ambient
// layer so the petals become stars.
const night = useNightMode()
// Who's writing, and whether the server still recognises them. `signedOut`
// flips the moment any call comes back 401.
const { me, signedOut } = useSession()
const t = usePack()
// A real account to sign out of, as opposed to the hardcoded local user a
// build without auth configured runs as.
const account = me && me.id !== 'local' ? { name: me.display_name || me.email } : null
const [docs, setDocs] = useState<DocSummary[]>([])
const [currentDoc, setCurrentDoc] = useState<Document | null>(null)
const [title, setTitle] = useState('')
@@ -132,6 +143,35 @@ export default function App() {
[createTag, setDocTag],
)
// If the session lapsed while she was writing, the body that couldn't be
// saved was stashed on this device. Opening the document again is where it
// comes back: the stashed fields win over the server's older copy, and a save
// is scheduled straight away so it stops being local-only. Version history
// makes this safe to do silently — the server's copy is one restore away.
const pendingRescueRef = useRef<{ id: string; body: DocUpdate } | null>(null)
const rescueDraft = useCallback((doc: Document): Document => {
const stashed = takeDraft(doc.id)
if (!stashed) return doc
const patch = Object.fromEntries(
Object.entries(stashed.body).filter(([, v]) => v !== undefined),
)
if (Object.keys(patch).length === 0) return doc
const merged = { ...doc, ...patch } as Document
if (merged.content === doc.content && merged.title === doc.title) return doc
// Save it, but only once this really is the open document — the auto-save
// writes to whichever doc is current when its timer fires.
pendingRescueRef.current = { id: doc.id, body: stashed.body }
return merged
}, [])
useEffect(() => {
const rescued = pendingRescueRef.current
if (rescued && currentDoc?.id === rescued.id) {
pendingRescueRef.current = null
schedule(rescued.body)
}
}, [currentDoc?.id, schedule])
const openDoc = useCallback(
async (id: string) => {
setDrawerOpen(false) // close the mobile drawer when a doc is chosen
@@ -139,7 +179,7 @@ export default function App() {
const leaving = currentDocRef.current
const leavingBlank = isBlankDraft()
await saveNow() // flush any pending edits to the doc we're leaving
const doc = await api.getDoc(id)
const doc = rescueDraft(await api.getDoc(id))
setCurrentDoc(doc)
setTitle(doc.title)
setWordCount(doc.word_count)
@@ -201,7 +241,8 @@ export default function App() {
setDocText('')
}, [saveNow, isBlankDraft])
// Duplicate a document: copy its body/tone into a fresh doc titled "… (副本)",
// Duplicate a document: copy its body/tone into a fresh doc under the pack's
// "copy" title,
// then open the copy. Tags aren't carried over (a fresh start for the copy).
const handleDuplicate = useCallback(
async (id: string) => {
@@ -209,7 +250,7 @@ export default function App() {
await saveNow() // flush in case we're duplicating the open doc
const src = await api.getDoc(id)
const fresh = await api.createDoc()
const dupTitle = `${src.title?.trim() || 'Untitled'} (副本)`
const dupTitle = t.app.duplicateTitle(src.title?.trim() || 'Untitled')
const updated = await api.updateDoc(fresh.id, {
title: dupTitle,
content: src.content,
@@ -266,6 +307,18 @@ export default function App() {
[currentDoc, patchSummary, schedule],
)
// She took the kitten up on its daily invitation. The prompt becomes the
// blank page's title, so the question she agreed to answer stays in front of
// her while she answers it — rather than being said once and then gone the
// moment the bubble fades.
const handleAcceptInvitation = useCallback(
(prompt: string) => {
if (!currentDoc) return
handleTitleChange(prompt)
},
[currentDoc, handleTitleChange],
)
const handleEditorChange = useCallback(
(change: EditorChange) => {
setWordCount(change.word_count)
@@ -413,7 +466,7 @@ export default function App() {
}}
>
<span aria-hidden>🌷</span>
<span>词汇花园</span>
<span>{t.app.garden}</span>
<span style={{ color: 'var(--color-muted)' }}>· Garden</span>
</button>
</header>
@@ -432,6 +485,7 @@ export default function App() {
onDuplicate={handleDuplicate}
onToggleTag={handleToggleTag}
onCreateTag={handleCreateTag}
account={account}
/>
</div>
@@ -475,7 +529,7 @@ export default function App() {
}}
>
<span aria-hidden>🕘</span>
<span>历史</span>
<span>{t.app.history}</span>
<span style={{ color: 'var(--color-muted)' }}>· History</span>
</button>
<div className="petal-no-print">
@@ -543,6 +597,10 @@ export default function App() {
{updateAvailable && <UpdateBanner />}
{/* The session lapsed. The editor stays visible behind this — nothing has
been taken away — and anything unsaved is already on disk. */}
{signedOut && <SignInOverlay hasDraft={status === 'signed-out'} />}
<div className="petal-no-print">
<PetalCompanion
wordCount={wordCount}
@@ -551,6 +609,8 @@ export default function App() {
editTick={editTick}
acceptTick={acceptTick}
text={docText}
blankPage={wordCount === 0 && docText.trim() === ''}
onAcceptInvitation={handleAcceptInvitation}
/>
</div>
</div>
web/src/api/client.ts
+130 -3
View File
@@ -41,6 +41,9 @@ export interface Document {
word_count: number
created_at: string
updated_at: string
// When true, this document's automatic snapshots are never pruned, so its
// full writing trail survives as authorship evidence (see the passport).
preserve_history: boolean
}
// Fields the editor sends on auto-save. All optional so a rename can send title
@@ -51,6 +54,7 @@ export interface DocUpdate {
content_text?: string
tone?: string
word_count?: number
preserve_history?: boolean
}
// One sense of a word from the offline dictionary.
@@ -64,16 +68,37 @@ export interface WordMeaning {
// a list of synonyms. Any of these may be empty when the word isn't a headword.
export interface WordInfo {
word: string
gloss: string // Chinese translation; '' when the word isn't in the gloss set
gloss: string // translation into the writer's language; '' when absent
phonetic: string // IPA for the English word; '' when absent
definitions: WordMeaning[]
synonyms: string[]
// From DreamDict only; the embedded datasets leave them unknown. `frequency`
// is 0 and `difficulty` is -1 when the dictionary has no score — see
// wordBand, which turns the pair into a band or into nothing at all.
frequency: number
difficulty: number
etymology: string // free-form, already trimmed to a line by the server; '' when absent
// The same token read as a word of the writer's own language, when it is one.
// Absent for a zh-pair writer and for almost every word in a Latin pair — see
// lexicon.Reverse for why Petal asks both directions instead of guessing.
reverse?: WordReverse
}
// A word looked up in the other direction: the writer's language -> English.
export interface WordReverse {
lang: string
gloss: string
definitions?: WordMeaning[]
phonetic?: string
}
// The lightweight Chinese-only gloss behind the inline hover/select tooltip.
export interface Gloss {
word: string
gloss: string
// The English meaning of the token read as a word of her own language.
// Present only on a collision (Portuguese *sale*, French *chat*).
reverse?: string
}
export type SuggestionType = 'grammar' | 'phrasing' | 'idiom' | 'clarity' | 'voice' | 'collocation' | 'mechanics'
@@ -132,18 +157,79 @@ export interface Suggestion {
explanation: string
type: SuggestionType
status: 'pending' | 'accepted' | 'rejected'
// Which engine proposed it — the offline rule pack or the model. The rail
// deliberately renders both identically; this is here because the wire format
// carries it, not because the writer is ever shown it.
source?: 'llm' | 'local'
created_at: string
}
// The growth journal (GET /api/suggestions/growth). `kept`/`kept_before` are
// the last thirty days and the thirty before them — the only comparison Petal
// draws is with her own past self. `stuck` is phrasing she was given that now
// turns up across her own documents; `faded` is what she used to be corrected
// on and hasn't been lately. Both lists are empty when the data isn't there:
// nothing here is padded to fill a page.
export interface GrowthJournal {
kept: number
kept_before: number
stuck: { phrase: string; docs: number }[]
faded: { pattern: string; times: number }[]
}
// A deterministic, rule-based fix detected client-side (see Companion/prose.ts).
// The frontend owns mechanics detection; the backend only persists these as the
// 'mechanics' suggestion family. Spans are exact plaintext offsets.
// The frontend owns offline detection; the backend only persists these. Spans
// are exact plaintext offsets. `type` names the family the finding belongs to:
// 'mechanics' for a fix to this sentence, 'collocation' for the miscollocation
// rules, whose findings are chunks worth keeping and are filed — and planted in
// the garden on accept — exactly like the LLM coach's.
export interface MechanicsFinding {
from: number
to: number
original: string
replacement: string
explanation: string
type: 'mechanics' | 'collocation'
}
// One dictionary's worth of personal words — the ones she's excused from
// spell-check. Keyed by the dictionary's language, not the writer's.
export interface PersonalWords {
lang: string
words: string[]
}
// Who's writing. Mirrors the backend db.User.
export interface Me {
id: string
email: string
display_name: string
created_at: string
pair_lang: string
}
// Thrown when the server says the session is gone. Callers can tell it apart
// from a real failure — losing your session is not the same as a save going
// wrong, and the auto-save has to treat them very differently.
export class UnauthorizedError extends Error {
constructor() {
super('not signed in')
this.name = 'UnauthorizedError'
}
}
// Sessions expire, so *any* call can come back 401 — including the auto-save
// that fires 1.5s after every keystroke. One place notices, and the app reacts
// once, rather than each call site inventing its own answer.
let unauthorizedHandler: (() => void) | null = null
export function onUnauthorized(handler: () => void) {
unauthorizedHandler = handler
}
function signedOut(): UnauthorizedError {
unauthorizedHandler?.()
return new UnauthorizedError()
}
async function req<T>(path: string, init?: RequestInit): Promise<T> {
@@ -151,6 +237,7 @@ async function req<T>(path: string, init?: RequestInit): Promise<T> {
headers: { 'Content-Type': 'application/json' },
...init,
})
if (res.status === 401) throw signedOut()
if (!res.ok) {
const detail = await res.text().catch(() => '')
throw new Error(`${res.status} ${res.statusText}${detail ? `: ${detail}` : ''}`)
@@ -160,6 +247,17 @@ async function req<T>(path: string, init?: RequestInit): Promise<T> {
}
export const api = {
// The signed-in writer. With auth unconfigured (local development) this is
// the hardcoded local user, so the frontend needs no separate mode for it.
me: () => req<Me>('/me'),
// Move to another (English + X) pair. Answers with the whole updated user, so
// the caller re-reads the pair from the server rather than assuming its own
// request took — a code the server won't ship comes back 400 and the app is
// still on a language it can render.
setPairLang: (lang: string) =>
req<Me>('/me', { method: 'PATCH', body: JSON.stringify({ pair_lang: lang }) }),
listDocs: () => req<DocSummary[]>('/docs'),
createDoc: () => req<Document>('/docs', { method: 'POST' }),
getDoc: (id: string) => req<Document>(`/docs/${id}`),
@@ -197,6 +295,10 @@ export const api = {
// opening bubble (the explanation itself stays English in the card body).
translateSuggestion: (id: string) =>
req<{ translation: string }>(`/suggestions/${id}/translate`, { method: 'POST' }),
// The growth journal: her own accepted edits read back as patterns. Purely a
// read-side view of a table Petal already keeps, computed locally with no
// model call, so it costs nothing and leaves nothing.
growth: () => req<GrowthJournal>('/suggestions/growth'),
// Version history. listVersions returns metadata only (no bodies); getVersion
// loads one full snapshot for preview; snapshotDoc takes an explicit restore
@@ -219,6 +321,11 @@ export const api = {
// the given format. A one-click "download all my writing" safety net.
exportAllUrl: (format: ExportFormat) => `/api/docs/export-all?format=${format}`,
// Download URL for the writing passport: a standalone HTML report of how this
// document was written (timeline, growth, sessions), for showing someone who
// questions its authorship. Print to PDF from the browser to hand it over.
passportUrl: (id: string) => `/api/docs/${id}/passport`,
// Offline word lookup (gloss + definition + synonyms) for the right-click popover.
lookupWord: (word: string) => req<WordInfo>(`/word/${encodeURIComponent(word)}`),
// Lightweight Chinese-only gloss for the inline hover/select tooltip — instant
@@ -245,6 +352,7 @@ export const api = {
const form = new FormData()
form.append('image', file)
const res = await fetch('/api/images', { method: 'POST', body: form })
if (res.status === 401) throw signedOut()
if (!res.ok) {
const detail = await res.text().catch(() => '')
throw new Error(`${res.status} ${res.statusText}${detail ? `: ${detail}` : ''}`)
@@ -285,6 +393,24 @@ export const api = {
req<VocabWord>(`/vocab/${id}/review`, { method: 'POST', body: JSON.stringify({ grade }) }),
deleteVocab: (id: string) => req<void>(`/vocab/${id}`, { method: 'DELETE' }),
// The personal spelling dictionary — words she's told Petal to stop flagging.
// Server-side, so it belongs to her account and follows her between devices.
// `lang` is the *dictionary's* language: an English exception must not silence
// a pt-PT flag. Every call answers with the full resulting list, so the client
// never has to merge two views of the same set.
listPersonalWords: (lang: string) =>
req<PersonalWords>(`/spell/words?lang=${encodeURIComponent(lang)}`),
addPersonalWords: (lang: string, words: string[]) =>
req<PersonalWords>('/spell/words', {
method: 'POST',
body: JSON.stringify({ lang, words }),
}),
removePersonalWord: (lang: string, word: string) =>
req<PersonalWords>(
`/spell/words?lang=${encodeURIComponent(lang)}&word=${encodeURIComponent(word)}`,
{ method: 'DELETE' },
),
// Current deployed build id — changes whenever a new frontend ships. The
// app polls this to offer a refresh. Bypasses any cache so the answer is live.
version: () => req<{ version: string }>('/version', { cache: 'no-store' }),
@@ -361,6 +487,7 @@ export async function streamSuggestionChat(
body: JSON.stringify({ messages }),
signal,
})
if (res.status === 401) throw signedOut()
if (!res.ok || !res.body) {
const detail = await res.text().catch(() => '')
throw new Error(`${res.status} ${res.statusText}${detail ? `: ${detail}` : ''}`)
web/src/audio/sounds.ts
+16 -10
View File
@@ -13,7 +13,10 @@ import blockUrl from '../assets/sounds/block.mp3'
import baodingUrl from '../assets/sounds/baoding.mp3'
import milestoneUrl from '../assets/sounds/milestone.mp3'
import errorUrl from '../assets/sounds/error.mp3'
import { onPrefsScopeChange, readPref, writePref } from '../lib/prefs'
// The mute choice belongs to the account, not the browser — one person's
// silence must not mute the next writer to sign in here.
const STORAGE_KEY = 'petal.sound'
// Master volume — deliberately gentle. These are background delights, not alerts.
@@ -53,24 +56,27 @@ let enabled = readEnabled()
const listeners = new Set<(on: boolean) => void>()
function readEnabled(): boolean {
try {
return localStorage.getItem(STORAGE_KEY) !== 'off'
} catch {
return true
}
return readPref(STORAGE_KEY) !== 'off'
}
// This module reads its value at import time, before /api/me has answered.
// Re-read once the account is known, in case this writer's choice differs from
// whatever the browser was holding.
onPrefsScopeChange(() => {
const next = readEnabled()
if (next === enabled) return
enabled = next
listeners.forEach((fn) => fn(next))
if (next) void ensureContext()
})
export function isSoundEnabled(): boolean {
return enabled
}
export function setSoundEnabled(on: boolean): void {
enabled = on
try {
localStorage.setItem(STORAGE_KEY, on ? 'on' : 'off')
} catch {
/* private mode — choice just won't persist */
}
writePref(STORAGE_KEY, on ? 'on' : 'off')
listeners.forEach((fn) => fn(on))
// Touching the context on enable doubles as a user-gesture unlock + warm-up.
if (on) void ensureContext()
web/src/audio/speech.test.ts
+87
View File
@@ -0,0 +1,87 @@
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
import { nativeLang, speak, stopSpeech } from './speech'
import { resetPackForTests, setPackLang } from '../i18n'
// Read-aloud has two jobs beyond "make a sound": ask for the right pace, and ask
// in the right language. Both are decided at the call site and travel in the
// request body, so this checks the body — the part a component author can get
// wrong without anything failing loudly.
let bodies: Array<Record<string, unknown>>
beforeEach(() => {
bodies = []
vi.stubGlobal(
'fetch',
vi.fn((_url: string, init: RequestInit) => {
bodies.push(JSON.parse(String(init.body)))
// Never resolves to audio: the fallback path needs no window.Audio here,
// and rejecting would run the Web Speech branch instead of the server one.
return new Promise(() => {})
}),
)
})
afterEach(() => {
stopSpeech()
vi.unstubAllGlobals()
resetPackForTests()
})
describe('speak', () => {
it('asks for the normal pace by default', () => {
speak('reception')
expect(bodies).toHaveLength(1)
expect(bodies[0]).toMatchObject({ text: 'reception', lang: 'en-US', slow: false })
})
it('asks for the slow replay when the slow control is used', () => {
speak('reception', undefined, true)
expect(bodies[0]).toMatchObject({ text: 'reception', slow: true })
})
it('still detects Chinese by script, so a zh selection is never read in English', () => {
speak('你好世界')
expect(bodies[0]).toMatchObject({ lang: 'zh-CN' })
})
it('sends nothing for empty text', () => {
speak(' ')
expect(bodies).toHaveLength(0)
})
})
describe('nativeLang', () => {
// The voice for her own language comes from the pack, not from the letters.
// "comum" is spelled the same in both halves of the pt pair, so a detector
// would have to guess; the component that knows it is rendering her language
// says so instead.
it('follows the pair language', () => {
setPackLang('zh')
expect(nativeLang()).toBe('zh-CN')
setPackLang('pt-PT')
expect(nativeLang()).toBe('pt-PT')
setPackLang('fr')
expect(nativeLang()).toBe('fr-FR')
})
it('names a European Portuguese voice, never a Brazilian one', () => {
setPackLang('pt-PT')
expect(nativeLang()).not.toBe('pt-BR')
})
it('is what a Latin-pair lookup speaks the other reading in', () => {
setPackLang('pt-PT')
speak('comum', nativeLang())
expect(bodies[0]).toMatchObject({ text: 'comum', lang: 'pt-PT' })
})
it('speaks the French reading of a collision in French', () => {
// "chat" is the sharpest case in the fr pair: an English word, a French
// word, and the companion's own animal. Nothing about the letters says
// which — only the pack does.
setPackLang('fr')
speak('chat', nativeLang())
expect(bodies.at(-1)).toMatchObject({ text: 'chat', lang: 'fr-FR' })
})
})
web/src/audio/speech.ts
+26 -9
View File
@@ -6,6 +6,8 @@
// (TTS disabled) or unreachable, we fall back to the browser's Web Speech API so
// the buttons still do something. No model or network is strictly required.
import { pack } from '../i18n'
// speechSupported reports whether read-aloud can do anything at all. Audio
// playback is universal, so as long as we can construct an Audio element OR the
// Web Speech API exists, the buttons should show. The server path is tried at
@@ -51,8 +53,10 @@ function pickVoice(lang: string): SpeechSynthesisVoice | undefined {
}
// speakWebSpeech is the fallback: the browser's built-in synthesizer. A touch
// slower than default so learners can follow along.
function speakWebSpeech(text: string, lang: string): void {
// slower than default so learners can follow along, and slower still when the
// slow replay was asked for — the fallback should degrade in voice quality, not
// in what the button does.
function speakWebSpeech(text: string, lang: string, slow: boolean): void {
if (!webSpeechSupported()) return
const synth = window.speechSynthesis
synth.cancel()
@@ -60,7 +64,7 @@ function speakWebSpeech(text: string, lang: string): void {
utterance.lang = lang
const voice = pickVoice(lang)
if (voice) utterance.voice = voice
utterance.rate = 0.95
utterance.rate = slow ? 0.7 : 0.95
synth.speak(utterance)
}
@@ -74,13 +78,26 @@ export function detectLang(text: string): string {
return CJK.test(text) ? 'zh-CN' : 'en-US'
}
// nativeLang is the locale of the writer's own language — the voice for the
// *other* reading of a word that exists in both halves of a Latin pair.
//
// It is asked for explicitly rather than detected, and that is the point. A
// script boundary can be detected (the CJK test above); "comum" cannot. So the
// component that knows it is rendering her language says so, and everything
// rendering English lets the default stand. No guess, therefore no wrong guess
// about her writing — the same rule the both-directions gloss follows.
export function nativeLang(): string {
return pack().locale
}
// speak reads `text` aloud, cancelling anything already in flight so rapid taps
// don't queue up. `lang` defaults to a guess from the text (Chinese vs English)
// so callers can just pass the selection; pass an explicit locale to override.
// It tries the server's neural voice first and silently falls back to the browser
// voice if that's unavailable (route off, network error, or a 404 for a language
// with no configured voice).
export function speak(text: string, lang = detectLang(text)): void {
// `slow` asks for the stretched replay (SUGGESTIONS §5e) — the second tap on a
// sentence that went by too fast. It tries the server's neural voice first and
// silently falls back to the browser voice if that's unavailable (route off,
// network error, or a 404 for a language with no configured voice).
export function speak(text: string, lang = detectLang(text), slow = false): void {
if (!text.trim()) return
stopSpeech()
const seq = ++requestSeq
@@ -88,7 +105,7 @@ export function speak(text: string, lang = detectLang(text)): void {
fetch('/api/tts', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ text, lang }),
body: JSON.stringify({ text, lang, slow }),
})
.then((res) => {
if (!res.ok) throw new Error(`tts ${res.status}`)
@@ -115,6 +132,6 @@ export function speak(text: string, lang = detectLang(text)): void {
// Server TTS unavailable for this request — use the browser voice instead,
// unless a newer tap has already superseded this one.
if (seq !== requestSeq) return
speakWebSpeech(text, lang)
speakWebSpeech(text, lang, slow)
})
}
web/src/components/Auth/SignInOverlay.tsx
+70
View File
@@ -0,0 +1,70 @@
// SignInOverlay appears when the session has lapsed mid-session.
//
// The tone matters more than usual here. Being logged out of a writing app is
// alarming — the first thing anyone wants to know is whether their words
// survived — so the overlay leads with the reassurance and treats signing in
// again as an errand, not an error. The editor stays visible behind the scrim
// (dimmed, still there) for the same reason: nothing has been taken away.
import { usePack } from '../../i18n'
interface Props {
// Whether there is unsaved writing waiting on this device, which changes the
// reassurance from a promise to a statement of fact.
hasDraft: boolean
}
export function SignInOverlay({ hasDraft }: Props) {
const t = usePack()
return (
<div
role="dialog"
aria-modal="true"
aria-labelledby="petal-signin-title"
className="petal-no-print fixed inset-0 z-[60] flex items-center justify-center px-4"
style={{ background: 'color-mix(in srgb, var(--color-bg) 78%, transparent)', backdropFilter: 'blur(3px)' }}
>
<div
className="w-full max-w-md px-7 py-8 text-center"
style={{
background: 'var(--color-surface)',
border: '1px solid var(--color-border)',
borderRadius: 'var(--radius-lg)',
boxShadow: 'var(--shadow-soft)',
fontFamily: 'var(--font-ui)',
}}
>
<div aria-hidden style={{ fontSize: 34, lineHeight: 1 }}>
🌸
</div>
<h2
id="petal-signin-title"
className="mt-3 text-lg font-bold"
style={{ color: 'var(--color-plum)' }}
>
{t.auth.title}
</h2>
<p className="text-sm font-semibold" style={{ color: 'var(--color-muted)' }}>
{t.auth.titleEn}
</p>
<p className="mt-4 text-sm leading-relaxed" style={{ color: 'var(--color-plum)' }}>
{hasDraft ? t.auth.bodyWithDraft : t.auth.bodyPlain}
</p>
<p className="mt-1 text-xs leading-relaxed" style={{ color: 'var(--color-muted)' }}>
{hasDraft ? t.auth.bodyWithDraftEn : t.auth.bodyPlainEn}
</p>
<a
href="/auth/login"
className="mt-6 inline-block rounded-full px-6 py-2.5 text-sm font-bold text-white transition-colors"
style={{ background: 'var(--color-accent)' }}
onMouseEnter={(e) => (e.currentTarget.style.background = 'var(--color-accent-hover)')}
onMouseLeave={(e) => (e.currentTarget.style.background = 'var(--color-accent)')}
>
{t.auth.signIn}
</a>
</div>
</div>
)
}

Some files were not shown because too many files have changed in this diff Show More