Phase 17: run the migration, and fix the guard that locked out its own backup

Claire's writing — 8 documents, 33 snapshots, 103 suggestions, 3 vocabulary
words and an image — now belongs to her account rather than to the pre-auth
'local' user, and the VPS is canonical. millenia was left running and
untouched as a frozen fallback; it diverges the moment either side is
written to, so it wants retiring rather than syncing.

The plan's stated prerequisite, that she log in once so her subject exists,
turned out to be false. Authentik's hashed_user_id sub is the user's uid,
derived from her id and the instance secret, so it can be read in advance —
which means the data moves first and she signs in to find her writing
already there, instead of to an empty Petal that fills in later.

The fix here is to the liveness guard, and it is the second attempt at it.
PRAGMA locking_mode = EXCLUSIVE goes on holding its lock after being set
back to NORMAL — SQLite only lets go on that connection's next database
access — so against a real WAL database the script locked itself out of its
own VACUUM INTO backup. It passed locally because the test database had
come out of VACUUM INTO and so was never in WAL mode: the fixture didn't
look like production, the same way the stub identity provider's slashless
issuer didn't. The probe now runs on its own connection and closes it.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua

BUILD_PLAN.md

@@ -178,11 +178,14 @@ Option B ratified. `go-oidc` + `x/oauth2`; config fields already existed. The `R
 - **Two bugs deploying caught that the whole test suite could not**, both fatal before the login page ever renders: (1) the code trimmed the issuer's **trailing slash**, and Authentik's issuer has one — OIDC requires a byte-for-byte match, so discovery failed every time while the stub IdP (which advertised a slashless issuer) kept passing. Fixed, and the stub's issuer is now a knob with a regression test that ends in a slash. (2) A provider created through `ak shell` rather than the admin UI comes up with **`grant_types = []`**, which authentik reads as "no grant type is permitted here" and answers with `invalid_request` / *The request is otherwise malformed*. Both are written up in `deploy/README.md` §4.
 - **Allowlist is currently `prosolis@proton.me` only.** That Authentik instance fronts ~40 accounts across several applications, so an empty list was not an option, and guessing which account is hers would either lock her out or let a stranger in. Adding her is one line in `.env` plus a restart. Note that an Authentik account with **no email set** (e.g. `akadmin`) can't match an email-based entry — use its subject id.
 
-### Phase 17 — Migrate the `local` user ✅ script (2026-07-27)
+### Phase 17 — Migrate the `local` user ✅ (2026-07-27) — her writing now lives on her account
 Script, app stopped, backup first (OPEN #4). **The "she logs in once first" dependency turned out not to exist**: authentik's default `hashed_user_id` sub mode makes the subject `User.uid`, which is derived from her user id and the instance secret — stable, and readable before she has ever signed in (`ak shell -c "…User.objects.get(username='claire').uid"`). So the data can move *first*, and she signs in to find her writing already there rather than to an empty Petal that fills in later.
 - [ ] `scripts/migrate_local_user.*`: single transaction, `PRAGMA foreign_keys=OFF`, re-point `documents`/`tags`/`vocab_words` **and `images`** (versions/suggestions follow parents; `images` is new in Phase 16 and carries `user_id` directly — miss it and every pasted picture 404s), delete the empty provisioned row, verify row counts before commit; refuses to run if the app is up or the target has data
 - [x] `scripts/migrate_local_user.py` — dry-run by default, `VACUUM INTO` backup before touching anything, one transaction with `PRAGMA foreign_keys=OFF`, re-points `documents`/`tags`/`vocab_words`/`images`, deletes the old user row, and **verifies every expected row actually moved (and that the source is left owning nothing) before it commits**, rolling back otherwise. Refuses to merge into an account that already owns writing. Runbook in the script header.
 - **The "is the app stopped?" guard needed a second attempt.** `BEGIN EXCLUSIVE` — the obvious check — passes straight through against a *running but idle* Petal, because in WAL mode it only conflicts with another writer. That is exactly the case the guard exists to catch, and it would have failed silently. `PRAGMA locking_mode = EXCLUSIVE` conflicts with any connection at all, since it locks the shared-memory index every WAL reader maps; verified against a live server.
+- [x] **Run for real.** Her 8 documents, 33 snapshots, 103 suggestions, 3 vocabulary words and 1 image moved from `local` onto `5f47d955…` (Claire, `clairew8@pm.me`). Sequence: `-backup` snapshot of millenia's live DB (taken while it kept running — `VACUUM INTO` needs no write lock), shipped to the VPS, installed over the throwaway staging database (kept as `petal.db.staging-*`), **started once so migration `0010` applied**, stopped, migrated, started. Verified over public HTTPS with a short-lived probe session, then removed: `/api/me` is her, 8 documents listed, her image 200s, vocabulary garden and version history intact, search returns hits — and 401 without the cookie.
+- **millenia is a frozen fallback, not a mirror** (user's call: "both, VPS first"). It was left running and completely untouched, still serving the same writing under the pre-auth `local` user. The two diverge the moment anything is written on either, so it wants retiring rather than syncing.
+- **A second bug in the guard, found by running it against production rather than a test file.** `PRAGMA locking_mode = EXCLUSIVE` keeps holding the lock after being set back to `NORMAL` — SQLite only releases it on that connection's next database access — so on a **WAL** database the script locked itself out of its own `VACUUM INTO` backup. It passed locally because the test database had come out of `VACUUM INTO` and so wasn't in WAL mode at all — the same shape of miss as the trailing-slash issuer: the fixture didn't look like production. The probe now runs on its own connection and closes it, and the fix was re-verified against a database that had genuinely been served in WAL mode.
 - **Startup crash averted while sequencing this**: the image backfill claims unowned files for `local`, which stops existing after the migration — a foreign-key error inside `images.New`, which `main.go` treats as fatal. Petal would have entered a crash loop the first time it started on a migrated database. The backfill now skips a missing owner (there is nothing to claim in that case anyway; the migration moves the image rows itself).
 
 ### Phase 18 — Per-user, per-language client state
@@ -235,6 +238,7 @@ Each item independent and small; order within is free (SUGGESTIONS §5–§6).
 - [x] **Phase 14 — companion warmth + bedtime nag + night mode**: more encouraging phrases, a gentle "go to bed" nudge after 11pm, and a calm dark theme + falling stars at night. ✅ (see Phase 14 above)
 
 ## Session log
+- 2026-07-27: **Phase 17 — Claire's writing moved onto her real account** (user: "claire is local user today in Petal. let's make sure to migrate existing data to her account"). `scripts/migrate_local_user.py`: dry-run by default, own `VACUUM INTO` backup, one transaction with foreign keys off, re-points `documents`/`tags`/`vocab_words`/`images`, verifies every expected row moved before committing. **The plan's stated prerequisite — "she logs in once so her sub exists" — turned out to be false**: authentik's `hashed_user_id` sub is `User.uid`, derived from her id and the instance secret, so it is readable in advance and the data could move *first*; she signs in to find her writing already there instead of to an empty Petal. Her 8 documents, 33 snapshots, 103 suggestions, 3 vocabulary words and 1 image now belong to `5f47d955…`, verified end to end over public HTTPS. Per the user's call the VPS is now canonical and millenia was left running and untouched as a frozen fallback (it diverges the moment either is written to — retire it rather than sync it). **Three bugs, each found by a different kind of contact with reality**: (1) the image backfill claims files for `local`, which stops existing after a migration — a foreign-key error inside `images.New`, which `main.go` treats as fatal, so Petal would have crash-looped on first start against a migrated database; (2) `BEGIN EXCLUSIVE` was the wrong liveness check, since in WAL mode it only conflicts with another *writer* and sails past a running-but-idle Petal — exactly the case the guard exists for; (3) the replacement, `PRAGMA locking_mode = EXCLUSIVE`, holds its lock past being reset to `NORMAL`, so on a real WAL database the script locked itself out of its own backup — invisible locally because the test file had come from `VACUUM INTO` and wasn't in WAL mode. Same shape as Phase 16's trailing-slash issuer: the fixture didn't look like production.
 - 2026-07-27: **Phase 16 built — Petal authenticates for itself** (user: "let's continue the build plan"; box access granted mid-session). New `internal/auth` surface on top of the Phase-0 `Resolver` seam: `session.go` (opaque cookie, **SHA-256-at-rest**, 30-day sliding expiry throttled to one write an hour, revoke/revoke-all/prune), `oidc.go` (login/callback/logout with state + nonce + PKCE, **lazy retried discovery** so an IdP outage can't stop Petal booting or invalidate live sessions), `users.go` (provisioning upsert keyed on `sub`, `/api/me`, allowlist). Migration `0010` lands `sessions`, `images` and `users.pair_lang` together. `main.go` picks the resolver from config, so a laptop build is unchanged. **Image ownership** closes the capability-URL hole flagged in the Phase-0 audit — one row per owner keeps dedup, a stranger gets 404 not 403, `Cache-Control` dropped to `private`, and pre-existing files are claimed at startup or they'd all 404. Frontend: a single 401 interceptor, a warm bilingual sign-in overlay over a still-visible editor, and a **draft rescue** to localStorage so an expired session can't cost writing — the auto-save stashes the body it couldn't send and reclaims it after re-login. **Three deliberate deviations from the plan**, all noted above: the allowlist matches emails as well as subject ids (a subject doesn't exist until first login, so a subject-only list is unusable in advance); `SESSION_SECRET` was dropped from config rather than left unused (nothing signs anything — sessions are opaque and server-side); and image rows are keyed `(name, user_id)` rather than owned singly, which is what preserves deduplication. **A real bug caught by writing the round-trip test rather than by reading the code**: the one-shot state/nonce/PKCE cookies were cleared in a `defer`, i.e. after the redirect had already written the header, so the clearing `Set-Cookie` was silently dropped. Verified: full go/tsc/vite/vitest suites, migration `0010` against a `VACUUM INTO` copy of the live millenia DB (counts intact, FTS still matching, image claimed), and a live smoke against the binary in both auth-off and auth-on modes including a hand-inserted session (valid → 200; absent/forged/expired → 401). **Then deployed** (user: "do it! register it!"): provider + application registered in Authentik via `ak shell`, `.env` filled in, image rebuilt, and the **Traefik basic-auth gate removed** — Petal holds its own door now. Deploying immediately found two things no test could: the issuer's **trailing slash is significant** (Authentik's has one, OIDC compares byte-for-byte, and my normalising it away broke discovery while the slashless stub kept passing — now a knob with a regression test), and a provider created through the shell rather than the admin UI comes up with **empty `grant_types`**, which authentik answers with `invalid_request` before the login page renders. Verified over public HTTPS: health 200, `/api/docs` 401 with no basic-auth challenge, `/auth/login` → Authentik with state+nonce+PKCE, following it lands on the real sign-in page. Also swapped the emoji favicon for a **drawn sakura** (`web/public/petal.svg`) that renders in Petal's own rose palette everywhere instead of at each platform's discretion, and doubles as the Authentik app tile (inlined as a data URI, since this authentik doesn't serve `/media`). **The allowlist is `prosolis@proton.me` only** — that IdP fronts ~40 accounts, so empty was not an option and guessing her account would either lock her out or let a stranger in; adding her is one `.env` line and a restart.
 - 2026-07-27: **Phase 15 finished off on the two boxes** (user granted millenia access mid-session: `ssh reala@192.168.1.212`, and parodia is `ssh reala@100.64.0.1` over headscale). **LLM link**: rather than rebinding vLLM as planned, `deploy/vllm-headscale-proxy.service` (socat) adds a listener on `100.64.0.2` only — `vllm-chat.service` is shared with **Gogobee** and **Open WebUI** (whose endpoint lives in its own DB, not env), so a rebind meant three consumer edits and a 35B reload; the forwarder cost nothing and no downtime. Grammar checkpoint from the VPS now returns real suggestions in ~3s. **Backups**: the user pointed out the VPS already has daily provider VM backups *and* an age-encrypted offsite `parodia-backup` job, so Petal was folded into the latter instead of running a parallel cron — and doing so **exposed a real bug in that job's `sqlite_dump` helper**: Python's `iterdump` does not reproduce an FTS5 virtual table, so any restore would have come back with cross-document search silently missing (fixed with a `VACUUM INTO`-based helper, round-trip verified). The **bigger** find: millenia, which holds her actual writing, had **no scheduled backup at all** — now `petal-backup.timer`, age-encrypted with the parodia public recipient and pushed off-box, neither machine able to decrypt it. **Encryption at rest** (user raised it; correctly): VPS data dir is now LUKS2 covering the DB, images *and* the TTS cache; key on-box as a deliberate availability tradeoff, documented for what it does and doesn't stop. Rehearsing a reboot caught two bugs a clean run would have hidden — the plaintext originals were still on the unencrypted root fs *under* the mount, and `systemd-cryptsetup` wasn't installed so crypttab was being ignored entirely and the volume would never have unlocked at boot. Added a `.volume-ok` guard so an unmounted volume fails loudly instead of serving a blank DB. **Millenia hygiene**: Piper had been dead since the Jul 26 reboot — **26,800+ failed restarts**, read-aloud silently degrading to browser Web Speech — because an OS upgrade moved `/usr/bin/python3` 3.13→3.14 and the venv's `site-packages` went invisible; venv recreated (lands Piper 1.6.0, which is what `TTS_PATH` exists for), both voices verified through Petal. Petal itself was running unsupervised at PPID 1 and is now `petal.service` (verified by `kill -9`); the Piper units got `StartLimitIntervalSec`/`Burst` so a broken service enters `failed` instead of looping forever unnoticed. Remaining: an external uptime-kuma probe (needs the UI), a true VPS reboot test (shared public host, user's call), and millenia is still unencrypted at rest.
 - 2026-07-26: **Phase 15 complete — Petal is deployed at https://petal.parodia.dev** (user: "let's start this build plan"; scope confirmed as artifacts **plus** the actual deploy, millenia stays canonical, hostname `petal.parodia.dev`). Stack: `Dockerfile` (node → go → alpine; CGO off, so the runtime layer carries only ffmpeg + tzdata), `docker-compose.yml` behind the host's existing Traefik, and **two Piper sidecars** instead of the planned host systemd units — Piper turned out never to have been installed on the VPS and the account has no lingering session, so containers on an internal network with no published ports are both simpler and tighter. **Three real problems found by deploying rather than by planning:** (1) the image's `petal` user (uid 10001) has no claim on a bind-mounted host directory → SQLite `unable to open database file (14)` and a restart loop; the container now runs as the stack directory's owner (still non-root, and the host account keeps write access the backup script needs); (2) piper-tts **1.6.0 moved synthesis from `POST /` to `POST /synthesize`** with an identical body → every read-aloud 405'd; rather than pin both deployments to one Piper release the path became config (`TTS_PATH`, default `/`, so millenia is untouched); (3) once the instance was live it was **a public, writable, unauthenticated API** — Petal authenticates nobody yet, so Traefik basic auth now holds the door until Phase 16, with `/api/health` exempt on its own higher-priority router. Backups: `db.Backup` via **`VACUUM INTO`** (WAL-coherent, no write lock, single file, refuses to overwrite) behind a `-backup` flag so the nightly job snapshots the running container; `deploy/backup-petal.sh` compresses, pushes to millenia with a size check, prunes both sides; cron at 03:15; restore documented and verified by round-tripping an archive through the binary. Tests: `internal/db/backup_test.go` (WAL capture, seeded user survives, no `-wal`/`-shm` companions, refuses an existing destination, missing source), `internal/tts` path-normalisation + configured-path. go build/vet/test clean. **Acceptance verified over public HTTPS with the LLM link genuinely down**: dictionaries, gloss, word lookup + phonetic, doc create/save, CJK FTS search, md/docx export, vocab capture, read-aloud EN + zh (real mp3, cache hit, 404-fallback for an unconfigured language) — all fine; `/check` → the warm 502 that renders as 小助手在休息; health public, HTTP→HTTPS with a valid cert. **Two items outstanding, both needing millenia access I don't have**: vLLM isn't bound to its headscale interface (so no AI pass works yet), and parodia's ssh key isn't authorized on millenia (so backups are VPS-local only — not yet a real off-box backup). Both have one-command fixes in `deploy/README.md` §3 and §5. Also this session: **DreamDict gained Spanish**, so the es pair is no longer gated — folded into Phases 20/21 and the "Later" bucket. Next: **Phase 16 (auth)** — Authentik already runs on the same VPS.