From dae1213c681c0873f081d13e8875df25a4e28588 Mon Sep 17 00:00:00 2001 From: prosolis <5590409+prosolis@users.noreply.github.com> Date: Sun, 26 Jul 2026 22:24:04 -0700 Subject: [PATCH] Ratify the language-learning direction and expand it into Phases 15-22 SUGGESTIONS.md (new): the product rationale. Every user gets one (English + X) pair, X in {zh, pt-PT, fr, maybe es} - bilingual UI in the pair, type in either language, direction inferred without a detector (both-dictionaries spellcheck, show-both gloss on collision). Langpacks keyed by X. LLM-minimalism as a standing principle: the LLM never gates essential functionality; grammar-lite rules and an embedded miscollocation list are planned as code-first layers beneath the LLM families. MULTIUSER_PLAN.md: all OPEN decisions settled - in-app OIDC (the parodia.dev VPS hosting plan decides it), 30-day sliding sessions, allowlist, migration-by-script, image-store ownership fixed alongside auth, DreamDict imported as a package reading dict.db (module rename prereq lives in the dreamdict repo), zh stays on ECDICT until compared. BUILD_PLAN.md: the deferred bucket becomes checkboxed Phases 15-22 (deploy plumbing / OIDC / local-user migration / client-state namespacing / langpack extraction / DreamDict provider / pt-PT pair / learning loop + code-first layers) with standing rules: isolation tests in the same commit as any user-scoped endpoint, LLM-minimalism, and bilingual aesthetic as acceptance criteria. --- BUILD_PLAN.md | 81 ++++++++++- MULTIUSER_PLAN.md | 117 ++++++++------- SUGGESTIONS.md | 364 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 510 insertions(+), 52 deletions(-) create mode 100644 SUGGESTIONS.md diff --git a/BUILD_PLAN.md b/BUILD_PLAN.md index fbb61e2..96577a5 100644 --- a/BUILD_PLAN.md +++ b/BUILD_PLAN.md @@ -136,9 +136,83 @@ Multi-session build. **Source of truth for what's done and what's next.** Update ### Deferred (post-v1-local) - [x] **Multi-user groundwork** (2026-07-26) — request-scoped identity. New `internal/auth`: `Middleware(Resolver)` resolves the caller once per API request and stores the id in the context; handlers read it via `auth.UserID(r.Context())` instead of naming `db.LocalUserID`. `StaticResolver(db.LocalUserID)` keeps Petal single-user today. **Auth itself is still deferred** — but every query is now scoped to whoever the resolver says is calling, so landing Authentik is a one-line change in `main.go` plus a `Resolver` implementation. -- [ ] Authentik OIDC auth + session middleware ← **next step: write a `Resolver` that validates a session cookie; the plumbing is in place** -- [ ] Copyleaks Tier-2 + webhook HMAC -- [ ] Dockerfile, docker-compose, Traefik, deploy to write.parodia.dev +- [ ] Copyleaks Tier-2 + webhook HMAC (still parked — needs a public webhook; revisit after Phase 15) +- Authentik OIDC + deploy: **no longer deferred — expanded into Phases 15–17 below** (decisions ratified 2026-07-26; see `MULTIUSER_PLAN.md`) + +## Execution phases 15–22 (added 2026-07-26) + +Decisions behind these are ratified in `MULTIUSER_PLAN.md` (all OPENs settled) and `SUGGESTIONS.md` (the *why*; Q1–Q3 settled). **Standing rules for every phase below:** +- **Isolation tests in the same commit** as any new user-scoped endpoint (the `docs/isolation_test.go` suites are the template — this discipline caught a real unscoped-query bug once already). +- **LLM-minimalism** (SUGGESTIONS §6): the LLM never gates essential functionality; new essential features are code+data first. +- **Aesthetic + bilingual-in-the-pair copy remain acceptance criteria** on every user-visible change. +- Verify per project convention: go build/vet/test, tsc, vite build, vitest, live smoke on a throwaway DB/port. + +### Phase 15 — Deploy plumbing (parodia.dev + headscale) +Petal hosted on the parodia.dev VPS; vLLM stays on millenia over headscale. Auth (Phase 16) needs the stable `BASE_URL`/redirect URI this phase creates. +- [ ] Dockerfile (multi-stage: `npm run build` → `go build`, single binary + data dir volume) + docker-compose +- [ ] Traefik route + HTTPS on the chosen hostname (e.g. `write.parodia.dev`); stable `BASE_URL` env +- [ ] `LLM_ENDPOINT` → millenia's **headscale** address; verify vLLM binds only to the headscale interface (not LAN/0.0.0.0); revisit `LLM_TIMEOUT` for the WAN+VPN round trip +- [ ] TTS → the Piper already installed on parodia (VPS-local; per-language instances as on millenia — EN now, zh; see `deploy/` artifacts for the service pattern) +- [ ] Off-VPS nightly backup of `petal.db` (+`-wal`/`-shm` coherently, e.g. `sqlite3 .backup`) to millenia over the VPN; document restore +- [ ] Migrate the live millenia data to the VPS (app stopped, backup first, counts verified) — or decide the millenia instance stays canonical until Phase 17 completes +- [ ] Acceptance: full editor works with the VPN link **down** (spell/gloss/garden/search/export/TTS all fine; checkpoint shows the warm 小助手在休息 state); `/api/health` public; HTTPS end-to-end + +### Phase 16 — Auth (in-app OIDC) + image-store ownership +Option B ratified. `go-oidc` + `x/oauth2`; config fields already exist. The `Resolver` seam from Phase 0 is the only integration point. +- [ ] OIDC login flow: `/auth/login` → Authentik → `/auth/callback` (state/CSRF checked) → provision user (upsert from `sub`/email/name) → session +- [ ] `sessions` table migration (id, user_id, expires_at, created_at, user_agent); opaque cookie (`HttpOnly`, `SameSite=Lax`, `Secure` when https); **30-day sliding expiry**; `/auth/logout` revokes server-side +- [ ] Allowlist: `PETAL_ALLOWED_SUBS` env (comma-separated); rejected valid logins get a warm bilingual page, not an error dump +- [ ] `SessionResolver` replaces `StaticResolver` in `main.go` (keep `StaticResolver` for dev via env flag) +- [ ] Frontend 401 interceptor in `api/client.ts`: halt auto-save, preserve the draft (localStorage keyed by doc id), warm bilingual "请重新登录 · Please sign in again" overlay, resume cleanly after re-login — **a 401 mid-draft must never lose writing** +- [ ] **Image store ownership** (same phase, per OPEN #5): `images` table migration (hash, user_id, content_type, size, created_at); fetch joins on caller; dedup preserved (one file, N rows); last-row delete removes the file; backfill existing images to the current user +- [ ] `users.pair_lang` column (default `'zh'`) in this phase's provisioning migration — read by Phase 19+, cheap to add now +- [ ] Isolation tests: sessions (expiry, revocation, cross-user), images, allowlist paths + +### Phase 17 — Migrate the `local` user +Script, app stopped, backup first (OPEN #4). Depends on: she logs in once so her OIDC `sub` exists. +- [ ] `scripts/migrate_local_user.*`: single transaction, `PRAGMA foreign_keys=OFF`, re-point `documents`/`tags`/`vocab_words` (versions/suggestions follow parents), delete the empty provisioned row, verify row counts before commit; refuses to run if the app is up or the target has data +- [ ] Runbook documented in the script header; dry-run mode + +### Phase 18 — Per-user, per-language client state +- [ ] Namespace `localStorage` keys by user id once known: `petal.spell.personal`, `petal.companion`, sound/petals prefs +- [ ] Personal spell dictionary keyed by **user + language** (en and pt-PT word lists must not merge); consider promoting it to a server table so it follows her across devices (nice-to-have, decide during) + +### Phase 19 — Langpack extraction (the copy chore) +Pure refactor, zero visible change; prerequisite for every new pair (SUGGESTIONS §2, Q2 settled). +- [ ] Extract the `中文 · English` strings from the ~29 frontend files + companion `tips.ts` into a langpack copy module keyed by the pair's X; today's strings become the `zh` pack **verbatim** +- [ ] Parameterize `internal/llm/prompts.go` bilingual copy the same way (explanation language, "natives usually say…" framing, both-directions preamble) +- [ ] Wire pack selection to `users.pair_lang`; acceptance: pixel-identical UI for the zh pair, vitest snapshots unchanged + +### Phase 20 — DreamDict as a lexicon provider +Option 3 ratified (import package, read-only `dict.db`). **Prerequisite in the dreamdict repo:** rename its module path (or add a `replace` for dev). +- [ ] Provider seam behind the existing `lexicon` interface; DreamDict provider opens `dict.db` read-only (modernc driver, second handle beside `petal.db`); graceful "no data" when the file is absent +- [ ] pt-PT + fr wired to DreamDict (nothing to regress); **zh stays on ECDICT** until compared on real lookups from her documents — converge only if quality holds +- [ ] Surface the new fields where cheap: frequency/difficulty chip in WordCard; etymology line (cognate hook for en-natives) +- [ ] Deploy: `dict.db` ships in the data dir alongside `petal.db` + +### Phase 21 — The pt-PT pair (first Latin pair, proves the model) +SUGGESTIONS §1/§3/§3a. French follows the same groove afterwards; Spanish stays gated on DreamDict es data. +- [ ] Hunspell pt-PT vendored like en-US; **both-dictionaries spellcheck** (flag only if wrong in both; pills from both) — the no-detector stance, Q1 settled +- [ ] Gloss/WordCard both directions; on en/pt collisions show both compactly, never hide either +- [ ] Prompts pinned to **European Portuguese, never pt-BR** (explicit in every prompt); pt-PT langpack copy written and **reviewed by a pt-PT speaker before trusted** +- [ ] Piper pt-PT voice instance on parodia; read-aloud + L1 voice wired; slow toggle (`length_scale`) while in there (SUGGESTIONS §5e) +- [ ] Companion tips/cheers/bedtime lines in the pt-PT pack (the kitten speaks pt+en to this user) +- [ ] Acceptance: a pt-PT-pair user gets the full experience end-to-end with the VPN down except LLM passes; zh-pair user sees zero change + +### Phase 22 — Learning loop + code-first layers +Each item independent and small; order within is free (SUGGESTIONS §5–§6). +- [ ] **Growth journal** (Q3 settled): local aggregation over accepted suggestions; growth-only, self-comparison-only framing; feeds companion cheers +- [ ] **Plant accepted collocations** in the vocabulary garden as phrase cards (scheduler unchanged) +- [ ] **Daily writing invitation** from the companion (no streaks, declining is fine) +- [ ] **False-friend list** per pair (curated data, WordCard heads-up + gentle flag) +- [ ] **Embedded miscollocation list** (code-first under the collocation family; LLM adds the long tail when reachable) +- [ ] **Grammar lite** rule-pack as a fourth suggestion family: instant, offline, precision-over-recall (near-certain or silent); per-pair L1-interference rules; sourcing per SUGGESTIONS Q6 (hand-curate vs mine LanguageTool's corpus — decide at build time) + +### Later / explicitly not now +- Learner-facing Chinese writing (the zh pair's second direction) — own phase with its own spec (SUGGESTIONS §4); only after Phases 19–21 prove the pair model +- Spanish pair — gated on DreamDict growing an es dataset +- Reactive-animation puppy companion — wishlist, low priority; `companions.ts` roster + mood engine is the drop-in point +- Copyleaks Tier-2 — revisit once Phase 15 provides a public webhook endpoint ### Next-up (post-v1 product, agreed with user 2026-06-26) - [x] **Phase 9 — ESL superpowers**: inline Chinese gloss on hover/select; "say it more naturally" / tone-rewrite. ✅ (see Phase 9 above) @@ -148,6 +222,7 @@ Multi-session build. **Source of truth for what's done and what's next.** Update - [x] **Phase 14 — companion warmth + bedtime nag + night mode**: more encouraging phrases, a gentle "go to bed" nudge after 11pm, and a calm dark theme + falling stars at night. ✅ (see Phase 14 above) ## Session log +- 2026-07-26: **Product direction + execution plan ratified** (user: "make it so, number one"). New `SUGGESTIONS.md` (product rationale for the language-learning direction): the **pair model** — every user gets one (English + X) pair, X ∈ {zh, pt-PT, fr, maybe es}, bilingual UI in the pair, type in either language, direction inferred (no detector: both-dictionaries spellcheck, show-both gloss on collision); **langpacks** keyed by X; **LLM-minimalism** as a standing principle (LLM is garnish, never a gatekeeper — grammar-lite rule pack + embedded miscollocation list planned as code-first layers). Deployment settled: Petal on the **parodia.dev VPS**, vLLM on millenia over **headscale** (the only cross-VPN dependency; Piper is VPS-local). All `MULTIUSER_PLAN.md` OPENs ratified: in-app OIDC (B), 30-day sliding sessions, allowlist, migration script, image-store fix with auth, DreamDict via package import (Option 3, module rename prereq in the dreamdict repo), zh stays on ECDICT until compared. Everything expanded into **Phases 15–22** above with standing rules (isolation tests same-commit, LLM-minimalism, bilingual aesthetic). Ready for implementation handoff starting at Phase 15. - 2026-07-26: **Multi-user groundwork** (user: "let's start preparing Petal for multi-user support"; scope agreed as plumbing-only, aimed at Authentik). New **`internal/auth`** package — context-carried identity (`WithUser`/`UserID`), a `Resolver` seam (`Resolve(*http.Request) (string, error)`), `StaticResolver` for today's single user, and `Middleware` that 401s anything unresolved. `main.go` splits `/api` into a **public group** (`/health`, `/version` — a monitoring probe must not need a session) and an **authenticated group** carrying everything else. All ~35 `db.LocalUserID` call sites across `docs`/`suggestions`/`vocab` now read the caller from the request; helpers that had no request in scope (`fetch`, `ownsDoc`, `ownsTag`, `tagsByDoc`, `fetchVersion`, `passportVersions`, `fetchPending`, vocab `fetch`) take an explicit `userID` param. `UserID` returns `""` rather than panicking when middleware is absent, so a mis-wired route **fails closed** (every query is `WHERE user_id = ?` → matches nothing). **Two real access-control gaps found and fixed while threading**: `setStatus` (accept/dismiss) updated a suggestion by bare id with **no ownership check at all**, and `fetchPending`/`listForDoc` read a document's suggestions by `doc_id` alone — a leak of the quoted source sentences. Both now scope through `documents.user_id`. **A third bug was caught by the new tests, not by the compiler**: `docs.fetch` gained a `userID` parameter but kept binding `db.LocalUserID` in the query — legal Go (unused params compile), silently unscoped, and it would have shipped. New tests: `internal/auth/auth_test.go` (round-trip, absent-context, both 401 paths) and **two-user isolation suites** (`docs/isolation_test.go`, `suggestions/isolation_test.go`) that mount the same routers twice behind two resolvers over one DB and assert a stranger gets 404 on get/update/delete/export/passport/snapshot/version-preview/restore/tag-assign/tag-rename/tag-delete/suggestion-accept/dismiss, sees nothing in list/search/version-list, and leaves the owner's data untouched. go build/vet/test all clean. **Still global, deliberately out of scope** (flagged for the auth phase): the image store is content-addressed with no per-user association or DB row — any authenticated user holding a hash can fetch any image (capability-URL security, needs a table + migration to fix); `export-all` is correctly scoped; frontend `localStorage` keys (`petal.spell.personal`, `petal.companion`, sound/petals prefs) are per-browser, not per-account, so they'd bleed across users sharing a device. - 2026-06-26: **Phases 12 + 13 complete** (collocation coach + vocabulary garden — "finish the rest of the build plan except Authentik/Traefik"). **Phase 12**: collocation drops in as a third suggestion family reusing the whole `runPass`/`pendingScope` machinery — `llm/collocation.go` (`RunCollocation`, 25s floor, reuses `ParseCheckpoint`), `collocationSystemPrompt`/`CollocationMessages` (warm "Natives usually say…" + Mandarin gloss, defers grammar elsewhere), migration `0005` **rebuilds** the suggestions table to extend the `type` CHECK (SQLite can't ALTER a CHECK), `collocationScope` + `CollocationLimit` + `POST /{id}/collocation`. **Caught a latent bug**: `grammarScope` was `type != 'voice'` → would wipe collocation flags; fixed to `type NOT IN ('voice','collocation')`. Frontend: `--color-blossom` pink, "Make it sound natural 🌸" toolbar pill, `collocating`/`runCollocation` in `useCheckpoint`, StatusBar dot — all into the existing rail/card. **Phase 13**: new `internal/vocab` package — migration `0006_vocab_garden` (`vocab_words`, SM-2-lite columns, doc_id `ON DELETE SET NULL`, `UNIQUE(user_id,word)`), `scheduler.go` (Leitner ladder 1/3/7/16/35 → geometric; gentle "again", no streak-shaming), `handlers.go` (capture-upsert/list/due/review/delete, all owner-scoped, time math via SQLite `datetime()` so stored values stay canonical-UTC). Auto-capture wired into `EditorCore.openWordLookup` (only dictionary-known words, captures the surrounding sentence + doc_id) + a 🤍/💚 toggle on `WordCard`. `GardenPanel` slide-over: blossom grid (bloom stage by reps), flashcard review (sentence blanked, flip, again/good/easy, direction alternates recognition↔production), sleepy-kitten footer; opened from a global 🌷 header button. Tests: `TestCollocationPassCoexists`, vocab `scheduler_test.go` + `handlers_test.go`, db CHECK test extended. go build/vet/test + tsc + vite + vitest (51/51) all clean; migration verified against a copy of the live DB; live backend smoke (throwaway DB) walked the full vocab lifecycle + the warm-502 collocation path. **Remaining: only the deferred infra bucket** — Authentik auth, Copyleaks Tier-2 (needs a public webhook), Docker/Traefik/deploy — all on hold per the user's "except Authentik/Traefik". - 2026-06-26: **Phase 14 complete** (companion warmth + bedtime nag + night mode). `tips.ts`: `ENCOURAGEMENTS` 5→10 lines; new `BEDTIME` array (4 lines, user-supplied English wit + gentle Mandarin leads). `useCompanion.ts`: bedtime branch in the 10s heartbeat (after idle-return + break, before the generic tip); only nudges while actively writing; own `lastBedtime` ref + 30min `BEDTIME_GAP`, respects `PROACTIVE_GAP`; new `'bedtime'` `BubbleTone` lingers ~4s longer. **Night mode** (added same session, user request): `lib/night.ts` centralizes `isBedtime()` + window (now shared by the nag too); `hooks/useNightMode.ts` toggles `petal-night` on `` (60s re-check); `index.css` `html.petal-night` re-points only the palette tokens → whole UI flips via `var()` (no component edits), 600ms dusk fade, print stays white; `PetalFall` gains a `night` prop → chunky cartoon power stars (`makeCartoonStar`, Mario/Kirby-style, 5 candy colors) mixed ~70/30 with small twinkle sparkles, gentle spin + shallow shimmer, effect re-inits on flip; App: `useNightMode()` → ``. tsc + vite clean, companion vitest 45/45; verified with real-browser Playwright screenshots (clock mocked to 23:30) — day petals/cream vs night stars/dark-plum, both pretty. Bedtime window is `BEDTIME_FROM`/`BEDTIME_TO` (local clock) for easy retune. diff --git a/MULTIUSER_PLAN.md b/MULTIUSER_PLAN.md index c2b52b3..37607f6 100644 --- a/MULTIUSER_PLAN.md +++ b/MULTIUSER_PLAN.md @@ -1,11 +1,16 @@ # Petal multi-user plan -**Status:** Phase 0 (identity plumbing) landed 2026-07-26 in `6901cdb`. Everything -below it is proposed, not agreed. +**Status:** Phase 0 (identity plumbing) landed 2026-07-26 in `6901cdb`. +**All OPEN decisions ratified by the user 2026-07-26** (recommendations +accepted as written) — see each OPEN for its settled answer. Execution phases +live in `BUILD_PLAN.md` (Phase 15 onward); product rationale for the language +work is in `SUGGESTIONS.md`. -This document exists to be argued with. Sections marked **OPEN** are real -decisions with real trade-offs, not rhetorical questions — a reviewer should -push back on them. Where I have a recommendation I say so and say why. +Settled context that postdates the original draft: Petal will be hosted on the +**parodia.dev VPS**, reaching vLLM on millenia over **headscale VPN**; Piper +TTS is already installed on parodia (VPS-local, no VPN hop). The LLM is the +only cross-VPN dependency, and per the LLM-minimalism principle +(SUGGESTIONS.md §6) it must never gate essential functionality. --- @@ -61,10 +66,12 @@ global." ## 3. Phase A — authentication -### OPEN #1: forward-auth vs. in-app OIDC +### OPEN #1: forward-auth vs. in-app OIDC — **SETTLED: Option B (in-app OIDC)** -This is the biggest fork and it should be settled first, because everything in -Phase B depends on it. +Ratified 2026-07-26. The deciding fact arrived with the deployment plan: Petal +will live on the public parodia.dev VPS, which is exactly the environment where +Option A's "must never be reachable except through Traefik" invariant is a +footgun. Original analysis kept below for the record. **Option A — Traefik forward-auth to an Authentik outpost.** Traefik is already in the deferred deploy bucket. Authentik's outpost terminates @@ -109,10 +116,10 @@ Does the operator want Petal to be independently deployable? Preferred over signed stateless cookies because it makes logout and revocation actually work — worth the one table. - Cookie: `HttpOnly`, `SameSite=Lax`, `Secure` when `BASE_URL` is https. -- **OPEN #2:** session lifetime. This is a personal writing tool used daily on - trusted devices; a 30-day sliding expiry is kind, an 8-hour one is - conventional. I lean long — an editor that logs you out mid-draft is hostile, - and the auto-save makes a surprise 401 genuinely costly. Needs a decision. +- **OPEN #2 — SETTLED: 30-day sliding expiry** (ratified 2026-07-26). An + editor that logs you out mid-draft is hostile, and auto-save makes a + surprise 401 genuinely costly. Sliding: each authenticated request extends + the session. ### The 401 problem @@ -136,9 +143,11 @@ On first successful login, upsert a `users` row from the OIDC claims (`sub` → `users.id`, plus email and name). No signup flow; whoever Authentik lets in gets an account. -**OPEN #3:** should there be an allowlist? Authentik may host other applications -with a broader user set than Petal should have. A `PETAL_ALLOWED_SUBS` env var, -or an Authentik group check on a claim, would gate it. Probably yes, cheaply. +**OPEN #3 — SETTLED: yes, allowlist** (ratified 2026-07-26). Authentik may +host other applications with a broader user set than Petal should have. Gate +via a `PETAL_ALLOWED_SUBS` env var (or an Authentik group claim check — +implementer's choice, env var is simpler); a valid login not on the list gets +a warm bilingual "this Petal isn't yours to write in" page, not a 500. ### Migrating the existing `local` user @@ -159,9 +168,9 @@ person logs in once. Sequence: deploy auth → she logs in → new empty account created → stop app, back up, run script to move `local`'s rows onto her real id, delete the empty row → restart. -**OPEN #4:** is that acceptable, or is a small admin endpoint preferable to a -script? The existing project convention (`scripts/`, hand-run Python) suggests a -script is in keeping. +**OPEN #4 — SETTLED: documented one-off script** (ratified 2026-07-26), run +with the app stopped and a DB backup taken first, per the existing `scripts/` +convention. No admin endpoint. --- @@ -186,10 +195,9 @@ image uploaded by two users is stored once on disk and simply has two rows, so deduplication survives. Deleting the last row referencing a hash removes the file. -**OPEN #5:** is this worth doing before real multi-user, or is it acceptable to -ship auth first and treat this as a known limitation? I lean toward doing it *in -the same phase as auth*, since the moment a second account exists the exposure is -real and the fix requires a migration either way. +**OPEN #5 — SETTLED: fix it in the same phase as auth** (ratified +2026-07-26). The moment a second account exists the exposure is real, and the +fix requires a migration either way. ### Frontend `localStorage` @@ -241,7 +249,13 @@ It also carries data Petal has no equivalent for and could use: `Antonyms`, So Phase D stops being gated on data and becomes an integration decision. -### OPEN #6a (new): how to integrate +### OPEN #6a (new): how to integrate — **SETTLED: Option 3** (ratified 2026-07-26) + +Import the package, open `dict.db` read-only. Prerequisite stands: DreamDict's +module path must be renamed (or `replace`-directed) first — **that change +lives in the dreamdict repo, not this one.** The migration caution below also +stands: pt-PT/fr wire to DreamDict first; zh stays on ECDICT until compared on +real lookups. Options kept below for the record. **Option 1 — HTTP client.** Petal calls DreamDict on localhost:7777, exactly the pattern already used for Piper TTS (including graceful degradation when it's @@ -305,18 +319,26 @@ the phonetic dataset, the EN voice — is unaffected and stays shared. ## 7. Suggested sequence -1. **Settle OPEN #1** (forward-auth vs. in-app OIDC). Everything else follows. -2. Deploy plumbing: Dockerfile, Traefik, real hostname, HTTPS. Auth needs a - stable `BASE_URL` and a redirect URI regardless of which option wins. -3. Auth itself: `Resolver` implementation, sessions if applicable, frontend 401 - handling. +1. ~~Settle OPEN #1~~ **Settled: in-app OIDC.** +2. Deploy plumbing: Dockerfile, Traefik, real hostname on parodia.dev, HTTPS, + headscale route to vLLM on millenia (bound to the headscale interface + only), VPS-local Piper, off-VPS DB backup. Auth needs a stable `BASE_URL` + and a redirect URI. +3. Auth itself: OIDC `Resolver`, sessions table (30-day sliding), allowlist, + frontend 401 handling with draft preservation. 4. Image store table + migration (same phase, per OPEN #5). -5. Provision the second real account; migrate `local`'s data. -6. `localStorage` namespacing. -7. Per-user language. **No longer gated on data** — DreamDict covers all four - languages. Sequence within it: rename DreamDict's module path → wire it in as - a lexicon provider → pt-PT/fr first → compare zh quality → converge if it - holds. +5. Provision the second real account; migrate `local`'s data (script, app + stopped, backup first). +6. `localStorage` namespacing (key by user **and** language — see + SUGGESTIONS.md §8). +7. Per-user language pair. **No longer gated on data** — DreamDict covers all + four languages. Sequence within it: rename DreamDict's module path → wire + it in as a lexicon provider → pt-PT/fr first → compare zh quality → + converge if it holds. The pair model and langpack shape are specified in + SUGGESTIONS.md §1–§3. + +These are expanded into checkboxed execution phases in `BUILD_PLAN.md` +(Phase 15 onward) — that file remains the source of truth for progress. --- @@ -337,19 +359,16 @@ the phonetic dataset, the EN voice — is unaffected and stays shared. --- -## 9. Questions for the reviewer +## 9. Questions for the reviewer — all answered 2026-07-26 -1. Forward-auth or in-app OIDC? (OPEN #1 — the one that matters most) -2. Session lifetime, given a daily-use editor with auto-save? (OPEN #2) -3. Allowlist Petal accounts separately from Authentik's user set? (OPEN #3) -4. Migration script vs. admin endpoint for moving `local`'s data? (OPEN #4) -5. Fix the image store alongside auth, or ship auth with it as a known - limitation? (OPEN #5) -6. ~~Is there a usable open English↔European-Portuguese dictionary dataset?~~ - **Answered: DreamDict**, which covers en/fr/pt-PT/zh. The live question is now - *how* to integrate it — HTTP service, build-time extraction, or importing the - package and opening `dict.db` read-only. (OPEN #6a; I recommend the third) -7. Does replacing the zh gloss (ECDICT → CC-CEDICT) risk regressing a feature in - daily use, and should zh stay on ECDICT until the two are compared? - -Anything above that reads as settled but shouldn't be is also fair game. +1. ~~Forward-auth or in-app OIDC?~~ **In-app OIDC** (OPEN #1). +2. ~~Session lifetime?~~ **30-day sliding** (OPEN #2). +3. ~~Allowlist?~~ **Yes**, `PETAL_ALLOWED_SUBS` or group claim (OPEN #3). +4. ~~Script vs. admin endpoint?~~ **Script**, app stopped, backup first (OPEN #4). +5. ~~Image store timing?~~ **Same phase as auth** (OPEN #5). +6. ~~pt-PT dictionary data?~~ **DreamDict**, integrated per **Option 3** + (import package, read-only `dict.db`; module rename is the prerequisite) + (OPEN #6a). +7. ~~zh gloss regression risk?~~ **zh stays on ECDICT** until compared against + DreamDict on real lookups from her actual documents; converge only if + quality holds. diff --git a/SUGGESTIONS.md b/SUGGESTIONS.md new file mode 100644 index 0000000..34b4538 --- /dev/null +++ b/SUGGESTIONS.md @@ -0,0 +1,364 @@ +# Petal — product suggestions: becoming essential for language learners + +**Status:** written 2026-07-26 against `feat/writing-passport`; **ratified by +the user 2026-07-26** (recommendations accepted — reviewer Q1–Q3 settled +below; Q4–Q6 remain genuinely open and don't block execution). This document +is the *why*; the checkboxed execution phases live in `BUILD_PLAN.md` +(Phase 15 onward). + +**The brief:** make Petal essential for two audiences — ESL writers (native +Mandarin / pt-PT / French → English), and English natives learning Mandarin, +European Portuguese, French, maybe Spanish. Preserve privacy and warmth. + +**The language model (settled by the user, 2026-07-26):** every user has +exactly one language **pair, with English always one half** — (en + X), +X ∈ {zh, pt-PT, fr, maybe es}. This is a deliberate scope decision: never an +X↔Y pair without English, never more than one pair per user. The UI is +bilingual in the pair everywhere (tips, pet responses, cards), and the user +may **type in either language of the pair**; Petal infers direction from the +text rather than asking. + +--- + +## 1. What the pair model implies + +The wife's zh setup is already exactly this — she writes Mandarin and English +mixed, the UI is zh+en bilingual, and Petal adapts per span (CJK is never +spellchecked, English words gloss to Chinese). So the pair model isn't a new +design; it's a *promotion of today's behavior to the spec*. Three consequences: + +- **Schema:** one column, `users.pair_lang` (the X half; default `'zh'`). + No per-document language, no target/native split. Add it in whatever + migration Phase B's provisioning touches — one column now vs. a real + migration later, the same logic that put `user_id` in the schema on day one. +- **Direction is inferred, not declared.** The zh pair gets inference for free + (script boundaries separate the languages). Latin pairs don't — see §3a, + which is the one genuinely new problem the pair model creates. +- **Every bilingual surface stays two-language**, just parameterized: the + `中文 · English` pattern becomes `X · English`. Nothing about the UI's shape + changes, which is why the copy extraction in §2 is safe to do early. + +## 2. Languages as data, not code ("langpacks") + +Adding pt-PT today means editing code in many places. A quick census: **29 +frontend files** carry hardcoded zh-first bilingual strings (`tips.ts`, +`GardenPanel`, `StatusBar`, `WordCard`, every popover…), plus the +Mandarin-first prompt copy in `internal/llm/prompts.go`. Adding each new +language by hunting through those files doesn't scale to four pairs and would +slowly erode the bilingual-copy quality that makes Petal feel cared-for. + +Because English is always one half, a **langpack is keyed by X alone** — one +pack per pair, holding everything that varies: + +- UI copy pairs — extract the existing `中文 · English` strings into a copy + module; the current strings become the `zh` pack verbatim, so nothing + visible changes. This is the biggest single chore in the whole effort; + better done once than per-language. +- Companion tip/cheer/bedtime lines (`tips.ts` is already data-shaped — + closest to done). +- LLM prompt copy: bilingual explanation phrasing, "natives usually say…" + example pairs, and the both-directions framing (the text may be English, X, + or mixed — respond appropriately). +- Hunspell dictionary for X where one exists (`pt-PT`, `fr`, `es` upstream; + zh has none — see §4). The en-US dictionary is shared by every pair. +- DreamDict wiring: gloss both directions (`en→X` and `X→en`), phonetics. +- Piper voices for X (both for reading X text aloud and as the L1 voice); + font stack (CJK stacks only for zh). + +Shared across all pairs, untouched: nspell en-US, the English IPA dataset, the +EN Piper voice, and all of the editor machinery. + +This is refactoring, not product, so it's tempting to skip. Don't: it's the +difference between "Spanish is a data drop" and "Spanish is a month." + +## 3. Sequencing: Latin-script targets first, and in this order + +**pt-PT → fr → es.** Everything needed for these exists already: Hunspell +dictionaries, Piper voices, DreamDict data (en/fr/pt-PT/zh), and — critically — +the entire decoration/anchoring machinery (`wordAt`, spell tokenizer, suggestion +re-anchoring) already works, because these languages are space-delimited and +Latin-script like English. + +Caveats worth writing down now: + +- **DreamDict has no Spanish.** "Maybe Spanish" is gated on adding es to + DreamDict first, or a separate dataset. Cheap to note, expensive to discover + later. +- **pt-BR drift is the main quality risk.** Qwen will default to Brazilian + Portuguese in both explanations and "natives say…" examples. Prompts must pin + European Portuguese explicitly, and the pt-PT pack should be reviewed by a + pt-PT speaker before it's trusted — same standard the zh copy got by being + written for a real reader. The multi-user plan's ECDICT-vs-DreamDict + compare-on-real-lookups discipline applies here too. +### 3a. The Latin+Latin wrinkle: inferring direction without a script boundary + +The zh pair gets "which language is this word?" for free — the script answers +it, and all of today's behavior (CJK never spellchecked, English words gloss +to Chinese) hangs off that. In an en+fr or en+pt pair, both halves are Latin +script, so the two per-word decisions need a real answer: + +- **Spellcheck:** load both Hunspell dictionaries and pass a token if *either* + accepts it; flag only words wrong in both. This never falsely squiggles + correct writing in either language — the failure mode is missing a French + word that happens to be a valid English word, which is the gentle direction + to fail in. Correction pills can offer both dictionaries' suggestions. +- **Gloss/WordCard:** look the word up in both directions via DreamDict; if it + exists in only one language, done. For collisions (*chat*, *pain*, *sale* + are all real words in both English and French), show both compactly — a + two-line card ("🇫🇷 chat → cat · 🇬🇧 chat → bavarder") is honest, needs no + detector, and is arguably *delightful* for a learner. A sentence-level + language guess can order the lines, but shouldn't hide either. + +No trained language detector, no heuristics that can be wrong about someone's +writing — both-dictionaries membership plus show-both-on-collision covers it. +The LLM passes need nothing: the prompt already sees the mixed text whole. + +- The Hunspell tokenizer's current rule "CJK is never tokenized" stays correct + for the zh pair unchanged. + +## 4. The zh pair's *other* direction is a separate epic — say so explicitly + +The en+zh pair already exists, but only one direction of it is built: today +Petal deliberately ignores typed hanzi (never tokenized, never flagged, never +glossed) — exactly right for a zh-native writer practicing English, and +exactly insufficient for an English native *learning* Chinese, for whom the +hanzi side is the whole point. Supporting that direction breaks assumptions +that are load-bearing everywhere: + +- No spaces → `wordAt`, the spell tokenizer, and word-boundary lookups need + real word segmentation (a jieba-style segmenter, client- or server-side). +- Hunspell has no concept of Chinese; "spellcheck" becomes wrong-character + (错别字) detection — a different problem, probably LLM-assisted. +- Smart-typography input rules and the IME interact; input rules are currently + ASCII-gated, which is correct, but selection/caret behavior mid-IME + composition needs testing. +- The learning aids that matter are different: pinyin annotation (useful here, + unlike for the current user who reads hanzi), tone-mark help, HSK-level word + difficulty, hanzi stroke/handwriting practice. + +None of this is unbuildable, but it is its **own phase with its own spec**, not +part of the langpack drop. Recommendation: ship the pt-PT/fr pairs first to +prove the pair model, and treat learner-facing Chinese writing as Petal's next +big product bet after that — it's also the most differentiated one (very few +warm, private tools exist for writing practice in Chinese). + +## 5. Deepening the learning loop (all local, all gentle) + +Petal's suggestion pipeline currently *corrects and forgets*. The vocabulary +garden proved that capturing what the user already does (lookups) creates a +learning surface for free. The same move is available twice more: + +### 5a. Growth journal (patterns from accepted suggestions) + +Accepted grammar/collocation suggestions are a record of what the writer is +learning. Aggregate them **locally** into gentle patterns: "this month you've +mostly stopped mixing 在/at" / "make a decision has stuck — you've used it +right 4 times since." Two framing rules that keep it warm: it reports *growth*, +never an error tally, and it only ever compares the writer to her own past +self. Feeds the companion's cheer pool with genuinely personal material +("上次你还问过这个词，这次自己用对了! 🌱"). Data is already in the +`suggestions` table (status + type + original/replacement); this is a read-side +feature, no new capture needed. + +### 5b. Plant accepted collocations in the garden + +An accepted collocation ("do a decision" → "make a decision") is a learnable +chunk, exactly like a looked-up word. Auto-capture it into the vocabulary +garden as a phrase card (the SM-2-lite scheduler doesn't care that it's two +words). The garden then reflects *both* halves of learning: words she sought +out, and phrasing she was gently given. + +### 5c. Companion as tutor-lite: a daily invitation to write + +The companion nudges about breaks and bedtime but never *invites writing*. A +once-a-day bilingual prompt ("写 50 个字：今天让你微笑的一件小事 · Write 50 +words: one small thing that made you smile today"), offered when a session +starts with no doc open. Explicitly **no streaks, no guilt** — the existing +no-streak-shaming ethos in the SR scheduler is the right precedent; a declined +prompt just gets a sleepy "好吧，我继续睡 😴". Prompt lists live in the +native-language pack. + +### 5d. Use DreamDict's richer fields + +The multi-user plan notes DreamDict carries `Frequency`, `Difficulty`, +`Antonyms`, `Etymology` with "no equivalent" in Petal. Three cheap, high-value +surfaces: +- A **frequency/difficulty chip** in the WordCard ("common word" / "advanced") + — helps a learner decide whether a word is worth gardening. +- **Etymology for the en-native audience**: Romance-language learners live on + cognates; a one-line "from Latin *decidere*, like English *decide*" is the + single best memory hook for pt/fr/es vocabulary. +- **False friends**: a small curated list per pair (en↔pt: *embarrassed* ≠ + *embaraçada*-adjacent traps, *actually*/*atualmente*; en↔fr likewise), + surfaced as a warm heads-up in the WordCard and as a collocation-style + gentle flag when one is used suspiciously. Tiny data, disproportionate + trust-building — this is the mistake that makes learners feel foolish, and + catching it kindly is very Petal. + +### 5e. Read-aloud, slower + +Piper voices exist per target language; wire the target-language voice into the +existing 🔊 surfaces, and add a **slow toggle** (Piper's `length_scale`) — +learners replaying a sentence at 0.75× is one of the oldest, most-loved +listening aids, and it's a query parameter away. + +## 6. LLM-minimalism: essential help in plain code, the model as garnish + +**Stated by the user (2026-07-26):** with the LLM on the far side of a VPN, +preserve as much essential functionality as possible in ordinary code inside +Petal, and rely on the LLM as little as possible. This deserves to be a +standing design principle, not just a deployment reaction — it's also what +keeps Petal instant (no 38-second checkpoints for things a lookup can answer) +and private by construction. + +Where Petal stands today, by dependency: + +| Already pure code (survives VPN-down) | LLM-only today | +|---|---| +| Spellcheck (Hunspell), gloss/definitions/synonyms/phonetics (embedded lexicon → DreamDict), thesaurus, vocabulary garden + SR review, search, tags, versions + writing passport, export, find/replace, typography, TTS (Piper, VPS-local) | Grammar checkpoint, collocation coach, voice pass, Ask Petal, tone rewrite | + +Everything in §5 lands in the left column by design (growth journal, garden +planting, daily prompts, DreamDict fields, false friends — all lookups and +local aggregation). The right column splits into two groups: + +**Worth a code-first layer (the essential two):** + +- **Grammar lite** — a rule-pack of high-precision, data-driven checks for + the classic ESL patterns: a/an before vowel sounds, uncountables + ("informations", "advices", "furnitures"), subject–verb agreement in simple + clauses, doubled comparatives, common preposition pairs ("depend of" → + "depend on"), per-pair L1-interference rules (zh: dropped articles, he/she + slips; pt/fr: "have X years" for age). These run instantly on every edit — + no debounce, no 30s rate limit — as a fourth suggestion family through the + existing rail. The bar is **precision over recall**: an offline rule must be + near-certain before it flags, because a wrong correction is colder than a + missed one. LanguageTool's open rule corpus is a mineable source for + vetted patterns (extract data, not the Java). +- **Collocation data** — the same curated-list move as false friends: the + do/make, say/tell, strong-tea/heavy-rain families that fill every ESL + collocation workbook are a few hundred entries of data, not a model. A + small embedded miscollocation list catches the top offenders offline; the + LLM pass, when reachable, adds the long tail. Same family, same rail, same + warm phrasing — the writer never needs to know which engine spoke. + +**Inherently LLM (degrade warmly, don't imitate):** Ask Petal, tone rewrite, +and the voice pass are open-ended language generation — a code fake would be +worse than the existing honest "小助手在休息" state. Leave them as the +garnish they are. + +The framing that falls out: **the LLM never holds essential functionality +hostage.** Every daily-writing need — spell, define, gloss, pronounce, catch +the common mistakes, review vocabulary, prove authorship — works on a +disconnected VPS. The model adds depth and conversation when the tunnel is up. + +## 7. The writing passport is an ESL flagship — treat it as one + +The passport exists because AI detectors misfire on non-native English (the +commit message cites the Stanford TOEFL finding). That's not a side feature — +for the ESL audience it may be *the* reason to adopt Petal over any other +editor: **the tool that protects you from being wrongly accused, instead of +scoring you.** No product change needed beyond making sure it works identically +for any target language (it should — it's language-agnostic snapshot history). +Worth a prominent place in the README/landing copy when Petal gets one. + +## 8. Ties into MULTIUSER_PLAN.md + +For the open questions there, this document's brief implies: + +- **OPEN #1 (auth):** Option B (in-app OIDC), and the planned deployment + settles it. The user's stated topology (2026-07-26) is: **Petal hosted on + the parodia.dev VPS, reaching vLLM on millenia over headscale VPN.** A + public-internet app is exactly the case where "must never be reachable + except through Traefik" is a footgun — one proxy misconfiguration on a VPS + and forged identity headers reach the app. In-app OIDC is safe to expose + directly. +- **Deployment topology consequences** worth writing into the plan's Phase 2 + (deploy plumbing): + - The LLM becomes the only cross-VPN runtime dependency (Piper is already + installed on parodia.dev, so TTS stays VPS-local). The warm + "小助手在休息" degradation path was built for a flaky co-tenant Ollama; a + VPN link-down hits the same path, so the architecture already fails + gently — but checkpoint latency now includes a WAN+VPN round trip, worth + a look at the 60s LLM timeout. + - Everything offline-by-design (DreamDict lookups, spellcheck, gloss, + garden, search, the whole editor) keeps working when the VPN is down — + another argument for MULTIUSER_PLAN Option 3 over an HTTP dictionary + service, which would otherwise add a second cross-machine dependency. + - vLLM and Piper on millenia should bind to the headscale interface only, + never 0.0.0.0 on the LAN-facing side. + - The writing moves onto rented VPS disk. "The writing never leaves the + box" (§8) becomes "the box is a VPS" — at-rest encryption and an + off-VPS backup of `petal.db` (e.g. nightly to millenia over the same + VPN) deserve a line in the deploy phase. +- **OPEN #6a (DreamDict):** Option 3 (import, read-only `dict.db`), agreed — + it's the only option where four languages stay offline and instant, which + §6 and §9 treat as non-negotiable. +- **Phase B provisioning** should set `users.pair_lang` from the operator's + provisioning step or a first-run picker — add the column in the same + migration. (The plan's Phase D "native language becomes a `users` column" + becomes this: one column, the X half of the pair.) +- **localStorage namespacing** matters slightly more than the plan says once a + household mixes pairs: the personal spell dictionary is per-*language* as + well as per-user (a user's en words and pt-PT words must not merge into one + Hunspell overlay). Key by `user + lang`. + +## 9. Privacy & warmth guardrails (the checklist for every item above) + +Everything suggested here passes these; future ideas should too. + +1. **Offline-first, always — and code-first (§6).** Every essential surface + must work with the LLM unreachable and the network unplugged + (DreamDict-as-local-file preserves this; an HTTP dictionary service would + not). The LLM only ever adds depth to something that already works. + Cloud APIs are off the table even when they'd be easier. +2. **The writing never leaves the box.** No telemetry, no "anonymous usage + stats," ever. The growth journal (§5a) is computed locally from local rows. +3. **No scores, no percentages, no red.** Petal already refuses AI-detection + scores and classic red squiggles; the growth journal and false-friend flags + must hold the same line — evidence and gentle phrasing, never grades. +4. **No streaks, no guilt.** The SR scheduler set the precedent (gentle + "again", no wipe). Daily prompts (§5c) are invitations, not obligations. +5. **Both languages of the pair, always visible** — every explanation, tip, + and pet response renders bilingual in (en + X). That's the warmth: being + helped in the language you think in, next to the one you're learning. +6. **The kitten stays asleep.** Every new companion behavior routes through + the existing mood/cooldown engine; 瞌睡猫 keeps mumbling helpful things + without waking up. (A reactive-animation puppy is on the wishlist — low + priority per the user; the `companions.ts` roster + mood engine is already + the drop-in point, richer per-mood Lottie segments are the only new work.) + +## 10. Suggested sequence (interleaved with the multi-user plan's) + +1. Add the `users.pair_lang` column (with the Phase B migration or sooner). +2. Extract the bilingual UI copy into the langpack (zh pack = today's strings + verbatim; pure refactor, no visible change). +3. DreamDict integration per MULTIUSER_PLAN Option 3 (module rename → lexicon + provider → pt-PT/fr wired first, zh compared before converging). +4. pt-PT as the first full second pair: Hunspell pt-PT, Piper pt-PT voices, + pinned-pt-PT prompts, native-speaker copy review, and the both-dictionaries + spellcheck + show-both-gloss behavior from §3a. French follows the same + groove; Spanish gated on DreamDict es data. +5. Learning-loop features (§5) — each is small and independent; growth journal + and garden-planting of collocations first, since they're read-side over + existing data. +6. Code-first layers (§6): the embedded miscollocation list first (same shape + as false friends, drops into the existing collocation family), then + grammar lite as its own suggestion family. Both are per-pair data, so + they slot naturally into the langpacks from step 2. +7. Learner-facing Chinese writing (the zh pair's second direction): spec it as + its own phase (§4) only after the pair model is proven on pt-PT/fr. + +## 11. Questions for the reviewer (1–3 settled 2026-07-26) + +1. ~~§3a's no-detector stance~~ **Settled: yes** — pass if either dictionary + accepts it, show both glosses on collision, no language detector. +2. ~~UI-copy extraction first?~~ **Settled: yes** — the extraction (§2) is a + prerequisite chore, done before pt-PT is wired. +3. ~~Growth journal framing~~ **Settled: build it** with the two framing rules + as hard constraints (growth only, self-comparison only). +4. When (not whether) to build the learner-facing hanzi direction of the zh + pair — after pt-PT/fr, or is it wanted sooner? +5. Spanish: worth asking DreamDict to grow an es dataset now, or park it? +6. Grammar lite (§6): hand-curate the rule pack from ESL teaching materials + (small, fully understood), or mine LanguageTool's open rule corpus for + vetted patterns (bigger head start, needs licensing + quality triage)?