millenia: nightly encrypted backup, supervision, and hardened Piper units

The canonical instance -- the one with her actual writing -- turned out
to be the least protected thing in the estate:

- No scheduled backup at all; the newest snapshot was a month old. Now
  petal-backup.timer: VACUUM INTO, gzip, age-encrypt with the parodia
  public recipient, push to the VPS over headscale with a size check,
  prune both ends. Persistent=true because the box is not on 24/7.
  Neither machine can decrypt what it holds; the identity is offline.

- Petal ran as a bare ./petal with PPID 1, so a crash or reboot left it
  down until somebody noticed. Now petal.service, verified by kill -9.

- The Piper units retried forever without ever failing: RestartSec=3
  against systemd's default 10s window means the burst limit is never
  reached, which is how a dead service logged 26,800+ restarts over a
  day while read-aloud silently fell back to browser speech.
  StartLimitIntervalSec=300 makes a broken Piper show up in --failed.

backup-petal.sh now handles both deployment shapes (compose exec on the
VPS, local binary on millenia) and encrypts before anything leaves the
host. The VPS no longer uses it -- Petal rides parodia-backup there.

BUILD_PLAN.md

@@ -147,14 +147,18 @@ Decisions behind these are ratified in `MULTIUSER_PLAN.md` (all OPENs settled) a
 - **Aesthetic + bilingual-in-the-pair copy remain acceptance criteria** on every user-visible change.
 - Verify per project convention: go build/vet/test, tsc, vite build, vitest, live smoke on a throwaway DB/port.
 
-### Phase 15 — Deploy plumbing (parodia.dev + headscale) ✅ (2026-07-26, two items outstanding on millenia)
+### Phase 15 — Deploy plumbing (parodia.dev + headscale) ✅ (2026-07-26/27)
 Petal hosted on the parodia.dev VPS; vLLM stays on millenia over headscale. Auth (Phase 16) needs the stable `BASE_URL`/redirect URI this phase creates. **Hostname: `petal.parodia.dev`** (DNS already pointed at the VPS). Runbook: `deploy/README.md`.
 - [x] Dockerfile (multi-stage: `npm run build` → `go build` → alpine runtime) + docker-compose. CGO stays off (modernc SQLite is pure Go), so the runtime layer exists only for **ffmpeg** (read-aloud transcode) and **tzdata** (the bedtime nag + night mode read the local clock). Non-root; `/data` is the single writable mount. **`.dockerignore`** keeps the live DB and a stale local `web/dist` out of the image.
 - [x] Traefik route + HTTPS on `petal.parodia.dev` — labels follow the host's existing convention (external `traefik` network, `web-secure` entrypoint, `default` cert resolver, `compression@file`) plus Petal's own header middleware. No host port is published; Traefik is the only way in. `BASE_URL=https://petal.parodia.dev` set for Phase 16's redirect URI.
-- [x] `LLM_ENDPOINT` → millenia's headscale address (`100.64.0.2:8000`); `LLM_TIMEOUT` **30s → 90s** for the WAN+VPN round trip (the voice/collocation passes send a whole document and the timeout is a hard deadline on `Complete`). ⚠️ **Outstanding, needs millenia access:** vLLM currently refuses connections from the VPS — it isn't bound to the headscale interface. Petal degrades correctly meanwhile (verified). Fix + model-id capture documented in `deploy/README.md` §3.
+- [x] `LLM_ENDPOINT` → millenia's headscale address (`100.64.0.2:8000`); `LLM_TIMEOUT` **30s → 90s** for the WAN+VPN round trip (the voice/collocation passes send a whole document and the timeout is a hard deadline on `Complete`). **Exposed with a forwarder, not a rebind** (`deploy/vllm-headscale-proxy.service`, socat): `vllm-chat.service` is shared — Petal, **Gogobee** and Open WebUI all point at `127.0.0.1:8000`, and Open WebUI keeps its endpoint in its own database rather than in env, so rebinding meant editing three consumers and reloading a 35B AWQ model. The forwarder adds a second listener on `100.64.0.2` only (never `0.0.0.0` — the far end is a public host), zero downtime, zero consumer changes. Model is `qwen3.6-35b`. **Verified end to end: a grammar checkpoint from `petal.parodia.dev` returns real suggestions in ~3s over the VPN.**
 - [x] TTS — **deviation from the plan, deliberate**: Piper was *not* actually installed on parodia, and the `reala` account has no lingering session to keep user systemd units alive. Runs as **two sibling containers** (`piper-en`, `piper-zh`) off one image, models cached in a shared volume, on an internal network with no published ports. pt-PT in Phase 21 is a fourth service, not a new image. **Found + fixed while wiring**: piper-tts 1.6.0 moved synthesis from `POST /` to `POST /synthesize` (identical body); rather than pin both deployments to one release, the path is now config (`TTS_PATH`, default `/` so millenia is untouched).
-- [x] Off-VPS nightly backup + restore — `db.Backup` uses **`VACUUM INTO`**, not a file copy: in WAL mode the newest committed pages may live in `petal.db-wal`, and copying the three files separately can capture a torn mid-checkpoint state. `VACUUM INTO` reads one coherent snapshot including the WAL, takes no write lock (safe against the live app), and emits a single file with no companions; it refuses an existing destination so a failed run can't destroy the last good backup. Driven by a `-backup` flag on the binary, so the nightly job snapshots the *running container*. `deploy/backup-petal.sh` gzips, pushes to millenia over headscale with a post-transfer size check, and prunes both sides (7d local / 30d remote). Cron installed at 03:15. Restore documented + verified. ⚠️ **Outstanding, needs millenia access:** parodia's ssh key isn't authorized on millenia, so `REMOTE_HOST` is empty and backups are VPS-local only — the one command to fix it is in `deploy/README.md` §5.
+- [x] Backups — `db.Backup` uses **`VACUUM INTO`**, not a file copy: in WAL mode the newest committed pages may live in `petal.db-wal`, and copying the three files separately can capture a torn mid-checkpoint state. `VACUUM INTO` reads one coherent snapshot including the WAL, takes no write lock (safe against the live app), and emits a single file with no companions; it refuses an existing destination so a failed run can't destroy the last good backup. Driven by a `-backup` flag on the binary.
+  - **On the VPS: folded into the host's existing `parodia-backup`** (age-encrypted, offsite to S3, 14-day retention, dead-man snitch) rather than a parallel cron — the user pointed out that layer already existed. **Found a real bug while doing it:** that script's `sqlite_dump` helper uses Python `iterdump`, which **does not reproduce an FTS5 virtual table** — it emits `documents_fts` as a raw `sqlite_master` row plus shadow tables, and replaying the result dies with `no such table`. Cross-document search would have been silently missing after any restore. Added a `sqlite_file_dump` helper using `VACUUM INTO` instead; round-trip verified (counts + a live FTS `MATCH`).
+  - **On millenia: `petal-backup.timer`** — the canonical instance had **no scheduled backup at all** (newest snapshot a month old), which mattered far more than the staging one. Nightly 03:20, `Persistent=true` (the box isn't on 24/7), snapshot → gzip → **age-encrypt with the parodia public recipient** → push to the VPS over headscale with a size check → prune both ends. Verified the pushed archive is real age ciphertext and that neither box can decrypt it.
 - [x] Migration decision: **millenia stays canonical** (user's call). The VPS runs an empty staging DB so she moves accounts exactly once, when Phase 16/17 land.
+- [x] **Encryption at rest** (not in the original plan — the user raised it mid-session, correctly). VPS data dir is now a **LUKS2 volume** (`deploy/setup-encrypted-data.sh`), covering `petal.db`, `images/` **and the TTS cache** (synthesized audio of her sentences). LUKS-on-a-file rather than gocryptfs because SQLite in WAL mode needs a shared-memory index mapped consistently across processes and FUSE has a long history of mmap/locking differences. Key on the same box — a deliberate availability tradeoff, documented honestly: it stops a decommissioned disk or a raw block-device read, **not** anyone holding the whole VM image. Canary-verified: a marker written through the app is absent from the raw image and present through the mount. **Two bugs caught by rehearsing a reboot rather than trusting the clean run** — (1) mounting over a directory *hides* its contents rather than removing them, so the first pass left the original plaintext `petal.db` and WAL on the unencrypted root filesystem, invisible under the mount (now shredded pre-mount, with a refusal if the mountpoint won't come up empty); (2) **`systemd-cryptsetup` wasn't installed**, so `/etc/crypttab` was ignored entirely and the volume would never have unlocked at boot. Added a **mount-liveness guard** (`.volume-ok` bind-mounted with `create_host_path: false`) so an unmounted volume is a loud container start failure instead of Petal quietly serving a blank database. ⚠️ A true reboot is untested — the VPS also runs matrix/lemmy/akkoma/gitea/authentik, so that's the user's call. millenia remains unencrypted at rest (LVM, no LUKS).
+- [x] **Supervision** (also not in the original plan). millenia's Petal had been running as a bare `./petal` with **PPID 1** — no unit, no screen session — so a crash or reboot left it silently down; now `petal.service`, verified by `kill -9`. **Piper's silent-failure mode fixed**: with `RestartSec=3` against systemd's default 10s window the burst limit was never reached, so a dead service looped **26,800+ times over a day without entering `failed`**; both units now set `StartLimitIntervalSec=300`/`StartLimitBurst=5`. Still missing: an external probe (uptime-kuma monitors on `/api/health` and `/api/tts` — needs the UI, written up in `deploy/README.md` §7).
 - [x] **Interim edge gate** (not in the original plan; added once the instance was live). Petal authenticates nobody yet — `StaticResolver` hands every request the same `local` user — so on a public host the whole API was open to read/write and image upload. Traefik basic auth holds the door until Phase 16, with `/api/health` exempt on its own higher-priority router. Deleted when OIDC lands.
 - [x] Acceptance — verified over public HTTPS **with the LLM link down** (it genuinely is): Hunspell dictionaries 200, gloss + word lookup (incl. phonetic) 200, doc create/save, FTS search on 春天, md + docx export, vocab capture/list, read-aloud EN + zh (real mp3 via ffmpeg, cache hit on repeat, 404 for an unconfigured language so the client falls back). `POST /check` → the warm 502 that renders as 小助手在休息. `/api/health` public; HTTP 301 → HTTPS with a valid cert.
 
@@ -224,6 +228,7 @@ Each item independent and small; order within is free (SUGGESTIONS §5–§6).
 - [x] **Phase 14 — companion warmth + bedtime nag + night mode**: more encouraging phrases, a gentle "go to bed" nudge after 11pm, and a calm dark theme + falling stars at night. ✅ (see Phase 14 above)
 
 ## Session log
+- 2026-07-27: **Phase 15 finished off on the two boxes** (user granted millenia access mid-session: `ssh reala@192.168.1.212`, and parodia is `ssh reala@100.64.0.1` over headscale). **LLM link**: rather than rebinding vLLM as planned, `deploy/vllm-headscale-proxy.service` (socat) adds a listener on `100.64.0.2` only — `vllm-chat.service` is shared with **Gogobee** and **Open WebUI** (whose endpoint lives in its own DB, not env), so a rebind meant three consumer edits and a 35B reload; the forwarder cost nothing and no downtime. Grammar checkpoint from the VPS now returns real suggestions in ~3s. **Backups**: the user pointed out the VPS already has daily provider VM backups *and* an age-encrypted offsite `parodia-backup` job, so Petal was folded into the latter instead of running a parallel cron — and doing so **exposed a real bug in that job's `sqlite_dump` helper**: Python's `iterdump` does not reproduce an FTS5 virtual table, so any restore would have come back with cross-document search silently missing (fixed with a `VACUUM INTO`-based helper, round-trip verified). The **bigger** find: millenia, which holds her actual writing, had **no scheduled backup at all** — now `petal-backup.timer`, age-encrypted with the parodia public recipient and pushed off-box, neither machine able to decrypt it. **Encryption at rest** (user raised it; correctly): VPS data dir is now LUKS2 covering the DB, images *and* the TTS cache; key on-box as a deliberate availability tradeoff, documented for what it does and doesn't stop. Rehearsing a reboot caught two bugs a clean run would have hidden — the plaintext originals were still on the unencrypted root fs *under* the mount, and `systemd-cryptsetup` wasn't installed so crypttab was being ignored entirely and the volume would never have unlocked at boot. Added a `.volume-ok` guard so an unmounted volume fails loudly instead of serving a blank DB. **Millenia hygiene**: Piper had been dead since the Jul 26 reboot — **26,800+ failed restarts**, read-aloud silently degrading to browser Web Speech — because an OS upgrade moved `/usr/bin/python3` 3.13→3.14 and the venv's `site-packages` went invisible; venv recreated (lands Piper 1.6.0, which is what `TTS_PATH` exists for), both voices verified through Petal. Petal itself was running unsupervised at PPID 1 and is now `petal.service` (verified by `kill -9`); the Piper units got `StartLimitIntervalSec`/`Burst` so a broken service enters `failed` instead of looping forever unnoticed. Remaining: an external uptime-kuma probe (needs the UI), a true VPS reboot test (shared public host, user's call), and millenia is still unencrypted at rest.
 - 2026-07-26: **Phase 15 complete — Petal is deployed at https://petal.parodia.dev** (user: "let's start this build plan"; scope confirmed as artifacts **plus** the actual deploy, millenia stays canonical, hostname `petal.parodia.dev`). Stack: `Dockerfile` (node → go → alpine; CGO off, so the runtime layer carries only ffmpeg + tzdata), `docker-compose.yml` behind the host's existing Traefik, and **two Piper sidecars** instead of the planned host systemd units — Piper turned out never to have been installed on the VPS and the account has no lingering session, so containers on an internal network with no published ports are both simpler and tighter. **Three real problems found by deploying rather than by planning:** (1) the image's `petal` user (uid 10001) has no claim on a bind-mounted host directory → SQLite `unable to open database file (14)` and a restart loop; the container now runs as the stack directory's owner (still non-root, and the host account keeps write access the backup script needs); (2) piper-tts **1.6.0 moved synthesis from `POST /` to `POST /synthesize`** with an identical body → every read-aloud 405'd; rather than pin both deployments to one Piper release the path became config (`TTS_PATH`, default `/`, so millenia is untouched); (3) once the instance was live it was **a public, writable, unauthenticated API** — Petal authenticates nobody yet, so Traefik basic auth now holds the door until Phase 16, with `/api/health` exempt on its own higher-priority router. Backups: `db.Backup` via **`VACUUM INTO`** (WAL-coherent, no write lock, single file, refuses to overwrite) behind a `-backup` flag so the nightly job snapshots the running container; `deploy/backup-petal.sh` compresses, pushes to millenia with a size check, prunes both sides; cron at 03:15; restore documented and verified by round-tripping an archive through the binary. Tests: `internal/db/backup_test.go` (WAL capture, seeded user survives, no `-wal`/`-shm` companions, refuses an existing destination, missing source), `internal/tts` path-normalisation + configured-path. go build/vet/test clean. **Acceptance verified over public HTTPS with the LLM link genuinely down**: dictionaries, gloss, word lookup + phonetic, doc create/save, CJK FTS search, md/docx export, vocab capture, read-aloud EN + zh (real mp3, cache hit, 404-fallback for an unconfigured language) — all fine; `/check` → the warm 502 that renders as 小助手在休息; health public, HTTP→HTTPS with a valid cert. **Two items outstanding, both needing millenia access I don't have**: vLLM isn't bound to its headscale interface (so no AI pass works yet), and parodia's ssh key isn't authorized on millenia (so backups are VPS-local only — not yet a real off-box backup). Both have one-command fixes in `deploy/README.md` §3 and §5. Also this session: **DreamDict gained Spanish**, so the es pair is no longer gated — folded into Phases 20/21 and the "Later" bucket. Next: **Phase 16 (auth)** — Authentik already runs on the same VPS.
 - 2026-07-26: **Product direction + execution plan ratified** (user: "make it so, number one"). New `SUGGESTIONS.md` (product rationale for the language-learning direction): the **pair model** — every user gets one (English + X) pair, X ∈ {zh, pt-PT, fr, maybe es}, bilingual UI in the pair, type in either language, direction inferred (no detector: both-dictionaries spellcheck, show-both gloss on collision); **langpacks** keyed by X; **LLM-minimalism** as a standing principle (LLM is garnish, never a gatekeeper — grammar-lite rule pack + embedded miscollocation list planned as code-first layers). Deployment settled: Petal on the **parodia.dev VPS**, vLLM on millenia over **headscale** (the only cross-VPN dependency; Piper is VPS-local). All `MULTIUSER_PLAN.md` OPENs ratified: in-app OIDC (B), 30-day sliding sessions, allowlist, migration script, image-store fix with auth, DreamDict via package import (Option 3, module rename prereq in the dreamdict repo), zh stays on ECDICT until compared. Everything expanded into **Phases 15–22** above with standing rules (isolation tests same-commit, LLM-minimalism, bilingual aesthetic). Ready for implementation handoff starting at Phase 15.
 - 2026-07-26: **Multi-user groundwork** (user: "let's start preparing Petal for multi-user support"; scope agreed as plumbing-only, aimed at Authentik). New **`internal/auth`** package — context-carried identity (`WithUser`/`UserID`), a `Resolver` seam (`Resolve(*http.Request) (string, error)`), `StaticResolver` for today's single user, and `Middleware` that 401s anything unresolved. `main.go` splits `/api` into a **public group** (`/health`, `/version` — a monitoring probe must not need a session) and an **authenticated group** carrying everything else. All ~35 `db.LocalUserID` call sites across `docs`/`suggestions`/`vocab` now read the caller from the request; helpers that had no request in scope (`fetch`, `ownsDoc`, `ownsTag`, `tagsByDoc`, `fetchVersion`, `passportVersions`, `fetchPending`, vocab `fetch`) take an explicit `userID` param. `UserID` returns `""` rather than panicking when middleware is absent, so a mis-wired route **fails closed** (every query is `WHERE user_id = ?` → matches nothing). **Two real access-control gaps found and fixed while threading**: `setStatus` (accept/dismiss) updated a suggestion by bare id with **no ownership check at all**, and `fetchPending`/`listForDoc` read a document's suggestions by `doc_id` alone — a leak of the quoted source sentences. Both now scope through `documents.user_id`. **A third bug was caught by the new tests, not by the compiler**: `docs.fetch` gained a `userID` parameter but kept binding `db.LocalUserID` in the query — legal Go (unused params compile), silently unscoped, and it would have shipped. New tests: `internal/auth/auth_test.go` (round-trip, absent-context, both 401 paths) and **two-user isolation suites** (`docs/isolation_test.go`, `suggestions/isolation_test.go`) that mount the same routers twice behind two resolvers over one DB and assert a stranger gets 404 on get/update/delete/export/passport/snapshot/version-preview/restore/tag-assign/tag-rename/tag-delete/suggestion-accept/dismiss, sees nothing in list/search/version-list, and leaves the owner's data untouched. go build/vet/test all clean. **Still global, deliberately out of scope** (flagged for the auth phase): the image store is content-addressed with no per-user association or DB row — any authenticated user holding a hash can fetch any image (capability-URL security, needs a table + migration to fix); `export-all` is correctly scoped; frontend `localStorage` keys (`petal.spell.personal`, `petal.companion`, sound/petals prefs) are per-browser, not per-account, so they'd bleed across users sharing a device.