Files

T

prosolis 97e9c269ec Phase 20: the dictionary stops being English and Chinese only

Word lookups now come from DreamDict's dict.db for every pair but Chinese —
opened read-only beside petal.db, no service, nothing over the VPN, because a
hover gloss has to answer in milliseconds.

`Provider` is the two questions the popover and the tooltip already asked, so
the embedded *Lexicon satisfies it with no changes at all; Set.For(lang) is the
single place the choice between them is made. The prerequisite in the dreamdict
repo turned out to be two things, not one: the module path was unfetchable
*and* the query layer sat in internal/, which no other module may import
whatever the module is called. Both fixed upstream.

The plan's central assumption did not survive the data. It mapped
Gloss ← Translate(word, "en", L1) one-to-one; against the real 452 MB database
that table answers for 17% of the 2,000 commonest English words into pt-PT.
Wiktionary's translation sections are thin in that direction — "ephemeral",
"think" and "quickly" have no en→pt-PT row at all. Shared WordNet synsets
answer for 61%, so DreamDict gained Equivalents() and Petal glosses through it.
Ordering those was wrong in an instructive way too: sorting by frequency
glosses "think" as lembrar, "remember", because lembrar is the commoner
Portuguese word even though pensar shares six of think's synsets to lembrar's
one. Counting sense agreement first asks the right question.

The same measurement is why zh stays on ECDICT: DreamDict reaches a Chinese
gloss for 53% of those words, ECDICT for nearly all of them. The plan said
converge only if quality holds. It didn't, so nothing converged.

Two decisions about failure worth keeping. A missing dict.db is not an error —
a laptop checkout has never had one — but a present-and-never-imported one is,
because that is a half-finished deploy. And a pt-PT writer with no dictionary
falls back to the embedded datasets with the gloss suppressed, keeping
definitions, synonyms and phonetics rather than blanking the popover: an empty
field reads as "not found", the wrong language reads as broken.

The new fields surface as an etymology line and a three-band chip. Three, not
five: the difficulty score separates "everyday" from "you'll have to explain
this" but cannot rank obfuscate against serendipity, and a finer scale would be
a confident-looking lie. An unscored word gets no chip.

Writing the tests found two bugs first — trimEtymology sliced by byte, which
would have emitted invalid UTF-8 for exactly the Greek and Latin etymologies
the feature exists for, and its ellipsis path overran its own cap.

go build/vet/test, tsc, vite, vitest 96/96 clean; live smoke against the real
dict.db with one instance flipped from zh to pt-PT mid-run.

Not deployed: go.mod still replaces github.com/prosolis/dreamdict with
../dreamdict, so the Docker build needs the two upstream commits pushed and the
replace dropped. The deployed dict.db also predates DreamDict's Spanish data.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua

2026-07-27 09:38:50 -07:00

19 KiB

Raw Blame History

Petal multi-user plan

Status: Phase 0 (identity plumbing) landed 2026-07-26 in 6901cdb. Phase A (authentication) + Phase C's image store built 2026-07-27 — in-app OIDC, server-side sessions, allowlist, provisioning, the frontend 401 path and per-owner images. Not yet configured against the live Authentik; see BUILD_PLAN.md Phase 16 and deploy/README.md §4. Phase B (migrating the local user) still waits on her first real login. All OPEN decisions ratified by the user 2026-07-26 (recommendations accepted as written) — see each OPEN for its settled answer. Execution phases live in BUILD_PLAN.md (Phase 15 onward); product rationale for the language work is in SUGGESTIONS.md.

Settled context that postdates the original draft: Petal will be hosted on the parodia.dev VPS, reaching vLLM on millenia over headscale VPN; Piper TTS is already installed on parodia (VPS-local, no VPN hop). The LLM is the only cross-VPN dependency, and per the LLM-minimalism principle (SUGGESTIONS.md §6) it must never gate essential functionality.

1. Where Petal is today

Single user, by construction. db.Open seeds one row (users.id = 'local') and, until this week, every query named that constant directly.

What was already right: the schema has been multi-user-shaped from day one. documents, tags, and vocab_words all carry user_id; document_versions and suggestions scope through their parent document. No migration is needed to support a second user — only a way to know which user is asking.

What Phase 0 changed

New internal/auth: Middleware(Resolver) resolves the caller once per API request and stores the id in the request context. Handlers read auth.UserID(r.Context()).
Resolver is a one-method interface — Resolve(*http.Request) (string, error) — and is the only thing a real identity provider has to implement.
StaticResolver(db.LocalUserID) supplies today's single user, so behavior is unchanged.
/api is split into a public group (/health, /version) and an authenticated group (everything else).
Two pre-existing access-control gaps fixed: setStatus (accept/dismiss) had no ownership check at all, and fetchPending read suggestions by doc_id alone — which leaked the quoted source sentences.
Two-user isolation test suites (docs, suggestions) mount the same routers twice behind two resolvers over one database.

The remaining work is not "make Petal multi-user." It is "authenticate someone, provision them, and clean up the three places where data is still global."

2. Goals and non-goals

Goals

Two or more people use one Petal instance without seeing each other's writing.
The existing local user's data survives, attached to a real account.
Adding a user is an operator action, not a code change.

Non-goals (explicitly out, unless a reviewer argues otherwise)

Sharing, collaboration, or multi-author documents. Petal is a private writing space; every feature to date assumes one reader. Sharing would change the passport's meaning (authorship evidence) and is a product decision, not an auth one.
Roles, permissions, or an admin UI.
Public signup. Accounts are provisioned deliberately.

3. Phase A — authentication

OPEN #1: forward-auth vs. in-app OIDC — SETTLED: Option B (in-app OIDC)

Ratified 2026-07-26. The deciding fact arrived with the deployment plan: Petal will live on the public parodia.dev VPS, which is exactly the environment where Option A's "must never be reachable except through Traefik" invariant is a footgun. Original analysis kept below for the record.

Option A — Traefik forward-auth to an Authentik outpost. Traefik is already in the deferred deploy bucket. Authentik's outpost terminates the login, and Petal receives a trusted header (X-authentik-uid, plus email and name). The Resolver becomes ~20 lines: read the header, map to a user id.

For: no OIDC library, no session store, no cookie handling, no redirect plumbing, no token refresh, no logout endpoint. Petal keeps zero auth code. Login/MFA/password reset are entirely Authentik's problem.
Against: Petal is only secure if it is never reachable except through Traefik. Anyone who can hit the container directly can forge the header and become any user. That's a deployment invariant enforced by network config, not by code — and it is exactly the kind of invariant that quietly breaks. It also makes local development awkward (no proxy → no identity), though StaticResolver covers that.

Option B — Petal is an OIDC client itself. github.com/coreos/go-oidc + golang.org/x/oauth2, a /auth/callback route, and a signed session cookie. The config fields already exist (AUTHENTIK_URL, AUTHENTIK_CLIENT_ID, AUTHENTIK_CLIENT_SECRET, SESSION_SECRET).

For: self-contained and safe to expose directly. No trust-the-proxy invariant. Works the same in dev and prod.
Against: meaningfully more code — a session table or signed-cookie scheme, CSRF on the callback, token expiry, logout. Two new dependencies in a project that currently has exactly two.

My recommendation: Option B, but not confidently. The deciding factor for me is that "must never be reachable directly" is a footgun that survives long after whoever set it up has forgotten, and Petal already holds someone's private journals. But if the deployment is definitively a single Traefik-fronted box on a LAN and will stay that way, Option A is much less code and I'd not object.

A reviewer should weigh: how likely is this instance ever exposed beyond the LAN? Does the operator want Petal to be independently deployable?

Session handling (assumes Option B)

Server-side sessions in a sessions table (id, user_id, expires_at, created_at, user_agent), cookie holds an opaque random id. Preferred over signed stateless cookies because it makes logout and revocation actually work — worth the one table.
Cookie: HttpOnly, SameSite=Lax, Secure when BASE_URL is https.
OPEN #2 — SETTLED: 30-day sliding expiry (ratified 2026-07-26). An editor that logs you out mid-draft is hostile, and auto-save makes a surprise 401 genuinely costly. Sliding: each authenticated request extends the session.

The 401 problem

Every frontend fetch currently assumes success. Once a session can expire, any call can return 401 mid-session — including the 1.5s auto-save, which is the one that must not fail silently.

Proposal: a single interceptor in web/src/api/client.ts that, on 401, halts auto-save, surfaces a warm bilingual "请重新登录 / Please sign in again" state rather than a raw error, and preserves unsaved editor content across the re-login (localStorage draft keyed by doc id). This is small but easy to forget, and getting it wrong means lost writing.

4. Phase B — user provisioning and migration

Provisioning

On first successful login, upsert a users row from the OIDC claims (sub → users.id, plus email and name). No signup flow; whoever Authentik lets in gets an account.

OPEN #3 — SETTLED: yes, allowlist (ratified 2026-07-26). Authentik may host other applications with a broader user set than Petal should have. Gate via a PETAL_ALLOWED_SUBS env var (or an Authentik group claim check — implementer's choice, env var is simpler); a valid login not on the list gets a warm bilingual "this Petal isn't yours to write in" page, not a 500.

Migrating the existing `local` user

The live database on millenia holds real writing under user_id = 'local'. That data must end up owned by the wife's real account.

Recommended: a migration that renames rather than copies — update the users.id and let ON UPDATE CASCADE… except SQLite FKs here are declared without ON UPDATE, so this needs either a deliberate multi-table update inside one transaction (documents, tags, vocab_words — versions and suggestions follow their parents) with PRAGMA foreign_keys=OFF around it, or an explicit one-off admin command.

I'd rather do this as a documented one-off script run with the app stopped and a copy of the DB taken first than as an automatic startup migration, because it depends on knowing the new OIDC subject id, which isn't available until that person logs in once. Sequence: deploy auth → she logs in → new empty account is created → stop app, back up, run script to move local's rows onto her real id, delete the empty row → restart.

OPEN #4 — SETTLED: documented one-off script (ratified 2026-07-26), run with the app stopped and a DB backup taken first, per the existing scripts/ convention. No admin endpoint.

5. Phase C — the data that is still global

Found during the Phase 0 audit. None of these break with two users; all of them leak or bleed.

Image store — the real one

internal/images is a flat content-addressed directory. There is no per-user association and no database row at all. Any authenticated user who knows a sha256 can fetch any other user's image.

That is capability-URL security. Hashes aren't guessable, so this is not an emergency — but "unguessable filename" is not access control, and images pasted into a private journal are exactly the content that shouldn't rely on it.

Proposal: an images table (hash, user_id, content_type, created_at, size) with the fetch handler joining on the caller. Content addressing is kept — the same image uploaded by two users is stored once on disk and simply has two rows, so deduplication survives. Deleting the last row referencing a hash removes the file.

OPEN #5 — SETTLED: fix it in the same phase as auth (ratified 2026-07-26). The moment a second account exists the exposure is real, and the fix requires a migration either way.

Frontend `localStorage`

petal.spell.personal (personal dictionary), petal.companion (chosen mascot), plus sound and petal-effect preferences are all per-browser. Two users on one device share them — and the personal dictionary is the one that matters, since it's built from someone's own writing.

Cheapest fix: namespace every key by user id once the client knows who it is. The honest fix for the dictionary is to move it server-side into a table, which also means it follows a user between devices — arguably a feature.

Not affected

export-all is correctly scoped. The TTS cache is content-addressed audio of text the requester supplied, no cross-user inference. The lexicon is a static dataset identical for everyone.

6. Phase D — per-user language (and DreamDict)

Tracked here because it lands on the same users row and shouldn't be designed twice.

English is always the target language. What varies is the user's native language — the one glosses and explanations are written in. Mandarin ships today; European Portuguese (pt-PT, explicitly not pt-BR) is wanted; French is possible.

OPEN #6 is answered: DreamDict

The original worry here was data sourcing — Petal's gloss comes from ECDICT (English↔Chinese), and a pt-PT equivalent of comparable quality and license looked like the blocker.

~/git/dreamdict already solves it, and more completely than expected. It supports en, fr, pt-PT, and zh (~136k/56k/136k/121k words), and its shape maps almost 1:1 onto lexicon.Result:

Petal field	DreamDict
`Gloss`	`Translate(word, "en", L1)`
`Phonetic`	pronunciation (CMU + IPA for en, Wiktionary IPA elsewhere)
`Definitions`	`Define(word, lang)` — curated sources ranked above Wiktionary
`Synonyms`	`Synonyms(word, lang)`

It also carries data Petal has no equivalent for and could use: Antonyms, Frequency, Difficulty, and Etymology.

Correction (2026-07-27, Phase 20). The Gloss row of that table is wrong. Translate(word, "en", L1) reads Wiktionary's translation sections, which are thin in the en→X direction: measured on the real dict.db, it answers for 17% of the 2,000 commonest English words into pt-PT and 16% into fr. Meaning has to come through shared WordNet synset ids instead (61%), which is what DreamDict's new Equivalents(word, from, to) does — falling back to the translations table, for 62% combined. The 1:1 mapping was assumed from the API surface and never checked against the data; it did not survive contact with it. Same measurement on zh reads 53% against ECDICT's near-total coverage of those words, which is why the zh pair did not converge.

So Phase D stops being gated on data and becomes an integration decision.

OPEN #6a (new): how to integrate — SETTLED: Option 3 (ratified 2026-07-26)

Import the package, open dict.db read-only. Prerequisite stands: DreamDict's module path must be renamed (or replace-directed) first — that change lives in the dreamdict repo, not this one. The migration caution below also stands: pt-PT/fr wire to DreamDict first; zh stays on ECDICT until compared on real lookups. Options kept below for the record.

Option 1 — HTTP client. Petal calls DreamDict on localhost:7777, exactly the pattern already used for Piper TTS (including graceful degradation when it's down).

For: zero coupling, DreamDict updates independently, all endpoints available.
Against: a second service Petal now depends on at runtime, and the gloss is a 350ms hover tooltip where "the dictionary service is down" is a visible regression from today's always-there embedded data.

Option 2 — build-time extraction. A script (sibling to the existing scripts/build_gloss.py) generates Petal's embedded .json.gz datasets per language from DreamDict's dict.db.

For: preserves the embedded/offline property exactly; no runtime dependency; no architectural change at all.
Against: every language multiplies the binary (the four current gz files are already ~11.6 MB); updating the dictionary means rebuilding and redeploying Petal; the richer fields are lost unless separately extracted.

Option 3 — import the package, open dict.db read-only. DreamDict's internal/dictionary is a plain library with NewReadOnly(dbPath), and its only dependency is modernc.org/sqlite — the same CGO-free driver Petal already uses. Petal opens dict.db as a second read-only handle beside petal.db.

For: no service, no HTTP, no new dependency, lookups stay local-file fast, all four languages at once, and it deletes ~11.6 MB of embedded gz plus the ECDICT build scripts. One dictionary, maintained once, shared with GogoBee.
Against: Petal stops being a self-contained binary in the "just run it" sense — dict.db has to be deployed alongside. In practice Petal already ships a data directory (petal.db, images, TTS cache), so this is a smaller loss than it first sounds.

My recommendation: Option 3. It is the only one that gets all four languages, keeps lookups offline and instant, and removes code rather than adding a subsystem. Option 1's runtime dependency buys flexibility Petal doesn't need for a dictionary that changes a few times a year.

Prerequisite: DreamDict's module path is currently module dreamdict, which isn't fetchable. Importing it needs the module renamed to something like gitea.parodia.dev/drwily/dreamdict (or a local replace directive for development). Small, but it must happen first.

Migration caution

Whichever option wins, the zh path is currently working and in daily use. The gloss quality difference between ECDICT and CC-CEDICT is unknown and matters more than the architecture.

Proposal: introduce DreamDict behind Petal's existing lexicon interface as a provider, wire pt-PT and fr to it first (nothing to regress — they don't exist yet), and keep zh on ECDICT until the two have been compared on real lookups from her actual documents. Converge only if quality holds. This also de-risks the whole change: if DreamDict turns out to be a poor fit, only the unshipped languages are affected.

Still per-user regardless

Native language becomes a users column, and these become per-user lookups: LLM prompt copy (internal/llm/prompts.go, currently Mandarin-first), companion tips (tips.ts), the L1 Piper voice (Piper has pt-PT voices), and the CJK font stacks (not needed for Latin-script L1). English-side machinery — nspell en-US, the phonetic dataset, the EN voice — is unaffected and stays shared.

7. Suggested sequence

~~Settle OPEN #1~~ Settled: in-app OIDC.
Deploy plumbing: Dockerfile, Traefik, real hostname on parodia.dev, HTTPS, headscale route to vLLM on millenia (bound to the headscale interface only), VPS-local Piper, off-VPS DB backup. Auth needs a stable BASE_URL and a redirect URI.
Auth itself: OIDC Resolver, sessions table (30-day sliding), allowlist, frontend 401 handling with draft preservation.
Image store table + migration (same phase, per OPEN #5).
Provision the second real account; migrate local's data (script, app stopped, backup first).
localStorage namespacing (key by user and language — see SUGGESTIONS.md §8).
Per-user language pair. No longer gated on data — DreamDict covers all four languages. Sequence within it: rename DreamDict's module path → wire it in as a lexicon provider → pt-PT/fr first → compare zh quality → converge if it holds. The pair model and langpack shape are specified in SUGGESTIONS.md §1–§3.

These are expanded into checkboxed execution phases in BUILD_PLAN.md (Phase 15 onward) — that file remains the source of truth for progress.

8. Risks

Silent unscoping. Phase 0 hit this exactly once: docs.fetch took a userID parameter and kept binding db.LocalUserID in the query. Unused parameters are legal Go — it compiled, vet was silent, and every existing test passed while the lookup stayed unscoped. Only the two-user isolation test caught it. Every new user-scoped endpoint should get an isolation case in the same commit; the existing suites are the template.
Migrating live data. The wife's real writing is the thing being moved. Back up first, run with the app stopped, verify counts before deleting anything.
A 401 mid-draft losing work. See Phase A.
Scope creep into sharing. Multi-user and collaboration are different products. Adding accounts should not quietly become adding sharing.

9. Questions for the reviewer — all answered 2026-07-26

~~Forward-auth or in-app OIDC?~~ In-app OIDC (OPEN #1).
~~Session lifetime?~~ 30-day sliding (OPEN #2).
~~Allowlist?~~ Yes, PETAL_ALLOWED_SUBS or group claim (OPEN #3).
~~Script vs. admin endpoint?~~ Script, app stopped, backup first (OPEN #4).
~~Image store timing?~~ Same phase as auth (OPEN #5).
~~pt-PT dictionary data?~~ DreamDict, integrated per Option 3 (import package, read-only dict.db; module rename is the prerequisite) (OPEN #6a).
~~zh gloss regression risk?~~ zh stays on ECDICT until compared against DreamDict on real lookups from her actual documents; converge only if quality holds.

19 KiB Raw Blame History