millenia: nightly encrypted backup, supervision, and hardened Piper units

The canonical instance -- the one with her actual writing -- turned out
to be the least protected thing in the estate:

- No scheduled backup at all; the newest snapshot was a month old. Now
  petal-backup.timer: VACUUM INTO, gzip, age-encrypt with the parodia
  public recipient, push to the VPS over headscale with a size check,
  prune both ends. Persistent=true because the box is not on 24/7.
  Neither machine can decrypt what it holds; the identity is offline.

- Petal ran as a bare ./petal with PPID 1, so a crash or reboot left it
  down until somebody noticed. Now petal.service, verified by kill -9.

- The Piper units retried forever without ever failing: RestartSec=3
  against systemd's default 10s window means the burst limit is never
  reached, which is how a dead service logged 26,800+ restarts over a
  day while read-aloud silently fell back to browser speech.
  StartLimitIntervalSec=300 makes a broken Piper show up in --failed.

backup-petal.sh now handles both deployment shapes (compose exec on the
VPS, local binary on millenia) and encrypts before anything leaves the
host. The VPS no longer uses it -- Petal rides parodia-backup there.
This commit is contained in:
prosolis
2026-07-27 06:35:34 -07:00
parent 33e49ddb62
commit 42d857a878
8 changed files with 251 additions and 32 deletions
+8 -3
View File
@@ -147,14 +147,18 @@ Decisions behind these are ratified in `MULTIUSER_PLAN.md` (all OPENs settled) a
- **Aesthetic + bilingual-in-the-pair copy remain acceptance criteria** on every user-visible change. - **Aesthetic + bilingual-in-the-pair copy remain acceptance criteria** on every user-visible change.
- Verify per project convention: go build/vet/test, tsc, vite build, vitest, live smoke on a throwaway DB/port. - Verify per project convention: go build/vet/test, tsc, vite build, vitest, live smoke on a throwaway DB/port.
### Phase 15 — Deploy plumbing (parodia.dev + headscale) ✅ (2026-07-26, two items outstanding on millenia) ### Phase 15 — Deploy plumbing (parodia.dev + headscale) ✅ (2026-07-26/27)
Petal hosted on the parodia.dev VPS; vLLM stays on millenia over headscale. Auth (Phase 16) needs the stable `BASE_URL`/redirect URI this phase creates. **Hostname: `petal.parodia.dev`** (DNS already pointed at the VPS). Runbook: `deploy/README.md`. Petal hosted on the parodia.dev VPS; vLLM stays on millenia over headscale. Auth (Phase 16) needs the stable `BASE_URL`/redirect URI this phase creates. **Hostname: `petal.parodia.dev`** (DNS already pointed at the VPS). Runbook: `deploy/README.md`.
- [x] Dockerfile (multi-stage: `npm run build``go build` → alpine runtime) + docker-compose. CGO stays off (modernc SQLite is pure Go), so the runtime layer exists only for **ffmpeg** (read-aloud transcode) and **tzdata** (the bedtime nag + night mode read the local clock). Non-root; `/data` is the single writable mount. **`.dockerignore`** keeps the live DB and a stale local `web/dist` out of the image. - [x] Dockerfile (multi-stage: `npm run build``go build` → alpine runtime) + docker-compose. CGO stays off (modernc SQLite is pure Go), so the runtime layer exists only for **ffmpeg** (read-aloud transcode) and **tzdata** (the bedtime nag + night mode read the local clock). Non-root; `/data` is the single writable mount. **`.dockerignore`** keeps the live DB and a stale local `web/dist` out of the image.
- [x] Traefik route + HTTPS on `petal.parodia.dev` — labels follow the host's existing convention (external `traefik` network, `web-secure` entrypoint, `default` cert resolver, `compression@file`) plus Petal's own header middleware. No host port is published; Traefik is the only way in. `BASE_URL=https://petal.parodia.dev` set for Phase 16's redirect URI. - [x] Traefik route + HTTPS on `petal.parodia.dev` — labels follow the host's existing convention (external `traefik` network, `web-secure` entrypoint, `default` cert resolver, `compression@file`) plus Petal's own header middleware. No host port is published; Traefik is the only way in. `BASE_URL=https://petal.parodia.dev` set for Phase 16's redirect URI.
- [x] `LLM_ENDPOINT` → millenia's headscale address (`100.64.0.2:8000`); `LLM_TIMEOUT` **30s → 90s** for the WAN+VPN round trip (the voice/collocation passes send a whole document and the timeout is a hard deadline on `Complete`). ⚠️ **Outstanding, needs millenia access:** vLLM currently refuses connections from the VPS — it isn't bound to the headscale interface. Petal degrades correctly meanwhile (verified). Fix + model-id capture documented in `deploy/README.md` §3. - [x] `LLM_ENDPOINT` → millenia's headscale address (`100.64.0.2:8000`); `LLM_TIMEOUT` **30s → 90s** for the WAN+VPN round trip (the voice/collocation passes send a whole document and the timeout is a hard deadline on `Complete`). **Exposed with a forwarder, not a rebind** (`deploy/vllm-headscale-proxy.service`, socat): `vllm-chat.service` is shared — Petal, **Gogobee** and Open WebUI all point at `127.0.0.1:8000`, and Open WebUI keeps its endpoint in its own database rather than in env, so rebinding meant editing three consumers and reloading a 35B AWQ model. The forwarder adds a second listener on `100.64.0.2` only (never `0.0.0.0` — the far end is a public host), zero downtime, zero consumer changes. Model is `qwen3.6-35b`. **Verified end to end: a grammar checkpoint from `petal.parodia.dev` returns real suggestions in ~3s over the VPN.**
- [x] TTS — **deviation from the plan, deliberate**: Piper was *not* actually installed on parodia, and the `reala` account has no lingering session to keep user systemd units alive. Runs as **two sibling containers** (`piper-en`, `piper-zh`) off one image, models cached in a shared volume, on an internal network with no published ports. pt-PT in Phase 21 is a fourth service, not a new image. **Found + fixed while wiring**: piper-tts 1.6.0 moved synthesis from `POST /` to `POST /synthesize` (identical body); rather than pin both deployments to one release, the path is now config (`TTS_PATH`, default `/` so millenia is untouched). - [x] TTS — **deviation from the plan, deliberate**: Piper was *not* actually installed on parodia, and the `reala` account has no lingering session to keep user systemd units alive. Runs as **two sibling containers** (`piper-en`, `piper-zh`) off one image, models cached in a shared volume, on an internal network with no published ports. pt-PT in Phase 21 is a fourth service, not a new image. **Found + fixed while wiring**: piper-tts 1.6.0 moved synthesis from `POST /` to `POST /synthesize` (identical body); rather than pin both deployments to one release, the path is now config (`TTS_PATH`, default `/` so millenia is untouched).
- [x] Off-VPS nightly backup + restore`db.Backup` uses **`VACUUM INTO`**, not a file copy: in WAL mode the newest committed pages may live in `petal.db-wal`, and copying the three files separately can capture a torn mid-checkpoint state. `VACUUM INTO` reads one coherent snapshot including the WAL, takes no write lock (safe against the live app), and emits a single file with no companions; it refuses an existing destination so a failed run can't destroy the last good backup. Driven by a `-backup` flag on the binary, so the nightly job snapshots the *running container*. `deploy/backup-petal.sh` gzips, pushes to millenia over headscale with a post-transfer size check, and prunes both sides (7d local / 30d remote). Cron installed at 03:15. Restore documented + verified. ⚠️ **Outstanding, needs millenia access:** parodia's ssh key isn't authorized on millenia, so `REMOTE_HOST` is empty and backups are VPS-local only — the one command to fix it is in `deploy/README.md` §5. - [x] Backups`db.Backup` uses **`VACUUM INTO`**, not a file copy: in WAL mode the newest committed pages may live in `petal.db-wal`, and copying the three files separately can capture a torn mid-checkpoint state. `VACUUM INTO` reads one coherent snapshot including the WAL, takes no write lock (safe against the live app), and emits a single file with no companions; it refuses an existing destination so a failed run can't destroy the last good backup. Driven by a `-backup` flag on the binary.
- **On the VPS: folded into the host's existing `parodia-backup`** (age-encrypted, offsite to S3, 14-day retention, dead-man snitch) rather than a parallel cron — the user pointed out that layer already existed. **Found a real bug while doing it:** that script's `sqlite_dump` helper uses Python `iterdump`, which **does not reproduce an FTS5 virtual table** — it emits `documents_fts` as a raw `sqlite_master` row plus shadow tables, and replaying the result dies with `no such table`. Cross-document search would have been silently missing after any restore. Added a `sqlite_file_dump` helper using `VACUUM INTO` instead; round-trip verified (counts + a live FTS `MATCH`).
- **On millenia: `petal-backup.timer`** — the canonical instance had **no scheduled backup at all** (newest snapshot a month old), which mattered far more than the staging one. Nightly 03:20, `Persistent=true` (the box isn't on 24/7), snapshot → gzip → **age-encrypt with the parodia public recipient** → push to the VPS over headscale with a size check → prune both ends. Verified the pushed archive is real age ciphertext and that neither box can decrypt it.
- [x] Migration decision: **millenia stays canonical** (user's call). The VPS runs an empty staging DB so she moves accounts exactly once, when Phase 16/17 land. - [x] Migration decision: **millenia stays canonical** (user's call). The VPS runs an empty staging DB so she moves accounts exactly once, when Phase 16/17 land.
- [x] **Encryption at rest** (not in the original plan — the user raised it mid-session, correctly). VPS data dir is now a **LUKS2 volume** (`deploy/setup-encrypted-data.sh`), covering `petal.db`, `images/` **and the TTS cache** (synthesized audio of her sentences). LUKS-on-a-file rather than gocryptfs because SQLite in WAL mode needs a shared-memory index mapped consistently across processes and FUSE has a long history of mmap/locking differences. Key on the same box — a deliberate availability tradeoff, documented honestly: it stops a decommissioned disk or a raw block-device read, **not** anyone holding the whole VM image. Canary-verified: a marker written through the app is absent from the raw image and present through the mount. **Two bugs caught by rehearsing a reboot rather than trusting the clean run** — (1) mounting over a directory *hides* its contents rather than removing them, so the first pass left the original plaintext `petal.db` and WAL on the unencrypted root filesystem, invisible under the mount (now shredded pre-mount, with a refusal if the mountpoint won't come up empty); (2) **`systemd-cryptsetup` wasn't installed**, so `/etc/crypttab` was ignored entirely and the volume would never have unlocked at boot. Added a **mount-liveness guard** (`.volume-ok` bind-mounted with `create_host_path: false`) so an unmounted volume is a loud container start failure instead of Petal quietly serving a blank database. ⚠️ A true reboot is untested — the VPS also runs matrix/lemmy/akkoma/gitea/authentik, so that's the user's call. millenia remains unencrypted at rest (LVM, no LUKS).
- [x] **Supervision** (also not in the original plan). millenia's Petal had been running as a bare `./petal` with **PPID 1** — no unit, no screen session — so a crash or reboot left it silently down; now `petal.service`, verified by `kill -9`. **Piper's silent-failure mode fixed**: with `RestartSec=3` against systemd's default 10s window the burst limit was never reached, so a dead service looped **26,800+ times over a day without entering `failed`**; both units now set `StartLimitIntervalSec=300`/`StartLimitBurst=5`. Still missing: an external probe (uptime-kuma monitors on `/api/health` and `/api/tts` — needs the UI, written up in `deploy/README.md` §7).
- [x] **Interim edge gate** (not in the original plan; added once the instance was live). Petal authenticates nobody yet — `StaticResolver` hands every request the same `local` user — so on a public host the whole API was open to read/write and image upload. Traefik basic auth holds the door until Phase 16, with `/api/health` exempt on its own higher-priority router. Deleted when OIDC lands. - [x] **Interim edge gate** (not in the original plan; added once the instance was live). Petal authenticates nobody yet — `StaticResolver` hands every request the same `local` user — so on a public host the whole API was open to read/write and image upload. Traefik basic auth holds the door until Phase 16, with `/api/health` exempt on its own higher-priority router. Deleted when OIDC lands.
- [x] Acceptance — verified over public HTTPS **with the LLM link down** (it genuinely is): Hunspell dictionaries 200, gloss + word lookup (incl. phonetic) 200, doc create/save, FTS search on 春天, md + docx export, vocab capture/list, read-aloud EN + zh (real mp3 via ffmpeg, cache hit on repeat, 404 for an unconfigured language so the client falls back). `POST /check` → the warm 502 that renders as 小助手在休息. `/api/health` public; HTTP 301 → HTTPS with a valid cert. - [x] Acceptance — verified over public HTTPS **with the LLM link down** (it genuinely is): Hunspell dictionaries 200, gloss + word lookup (incl. phonetic) 200, doc create/save, FTS search on 春天, md + docx export, vocab capture/list, read-aloud EN + zh (real mp3 via ffmpeg, cache hit on repeat, 404 for an unconfigured language so the client falls back). `POST /check` → the warm 502 that renders as 小助手在休息. `/api/health` public; HTTP 301 → HTTPS with a valid cert.
@@ -224,6 +228,7 @@ Each item independent and small; order within is free (SUGGESTIONS §5–§6).
- [x] **Phase 14 — companion warmth + bedtime nag + night mode**: more encouraging phrases, a gentle "go to bed" nudge after 11pm, and a calm dark theme + falling stars at night. ✅ (see Phase 14 above) - [x] **Phase 14 — companion warmth + bedtime nag + night mode**: more encouraging phrases, a gentle "go to bed" nudge after 11pm, and a calm dark theme + falling stars at night. ✅ (see Phase 14 above)
## Session log ## Session log
- 2026-07-27: **Phase 15 finished off on the two boxes** (user granted millenia access mid-session: `ssh reala@192.168.1.212`, and parodia is `ssh reala@100.64.0.1` over headscale). **LLM link**: rather than rebinding vLLM as planned, `deploy/vllm-headscale-proxy.service` (socat) adds a listener on `100.64.0.2` only — `vllm-chat.service` is shared with **Gogobee** and **Open WebUI** (whose endpoint lives in its own DB, not env), so a rebind meant three consumer edits and a 35B reload; the forwarder cost nothing and no downtime. Grammar checkpoint from the VPS now returns real suggestions in ~3s. **Backups**: the user pointed out the VPS already has daily provider VM backups *and* an age-encrypted offsite `parodia-backup` job, so Petal was folded into the latter instead of running a parallel cron — and doing so **exposed a real bug in that job's `sqlite_dump` helper**: Python's `iterdump` does not reproduce an FTS5 virtual table, so any restore would have come back with cross-document search silently missing (fixed with a `VACUUM INTO`-based helper, round-trip verified). The **bigger** find: millenia, which holds her actual writing, had **no scheduled backup at all** — now `petal-backup.timer`, age-encrypted with the parodia public recipient and pushed off-box, neither machine able to decrypt it. **Encryption at rest** (user raised it; correctly): VPS data dir is now LUKS2 covering the DB, images *and* the TTS cache; key on-box as a deliberate availability tradeoff, documented for what it does and doesn't stop. Rehearsing a reboot caught two bugs a clean run would have hidden — the plaintext originals were still on the unencrypted root fs *under* the mount, and `systemd-cryptsetup` wasn't installed so crypttab was being ignored entirely and the volume would never have unlocked at boot. Added a `.volume-ok` guard so an unmounted volume fails loudly instead of serving a blank DB. **Millenia hygiene**: Piper had been dead since the Jul 26 reboot — **26,800+ failed restarts**, read-aloud silently degrading to browser Web Speech — because an OS upgrade moved `/usr/bin/python3` 3.13→3.14 and the venv's `site-packages` went invisible; venv recreated (lands Piper 1.6.0, which is what `TTS_PATH` exists for), both voices verified through Petal. Petal itself was running unsupervised at PPID 1 and is now `petal.service` (verified by `kill -9`); the Piper units got `StartLimitIntervalSec`/`Burst` so a broken service enters `failed` instead of looping forever unnoticed. Remaining: an external uptime-kuma probe (needs the UI), a true VPS reboot test (shared public host, user's call), and millenia is still unencrypted at rest.
- 2026-07-26: **Phase 15 complete — Petal is deployed at https://petal.parodia.dev** (user: "let's start this build plan"; scope confirmed as artifacts **plus** the actual deploy, millenia stays canonical, hostname `petal.parodia.dev`). Stack: `Dockerfile` (node → go → alpine; CGO off, so the runtime layer carries only ffmpeg + tzdata), `docker-compose.yml` behind the host's existing Traefik, and **two Piper sidecars** instead of the planned host systemd units — Piper turned out never to have been installed on the VPS and the account has no lingering session, so containers on an internal network with no published ports are both simpler and tighter. **Three real problems found by deploying rather than by planning:** (1) the image's `petal` user (uid 10001) has no claim on a bind-mounted host directory → SQLite `unable to open database file (14)` and a restart loop; the container now runs as the stack directory's owner (still non-root, and the host account keeps write access the backup script needs); (2) piper-tts **1.6.0 moved synthesis from `POST /` to `POST /synthesize`** with an identical body → every read-aloud 405'd; rather than pin both deployments to one Piper release the path became config (`TTS_PATH`, default `/`, so millenia is untouched); (3) once the instance was live it was **a public, writable, unauthenticated API** — Petal authenticates nobody yet, so Traefik basic auth now holds the door until Phase 16, with `/api/health` exempt on its own higher-priority router. Backups: `db.Backup` via **`VACUUM INTO`** (WAL-coherent, no write lock, single file, refuses to overwrite) behind a `-backup` flag so the nightly job snapshots the running container; `deploy/backup-petal.sh` compresses, pushes to millenia with a size check, prunes both sides; cron at 03:15; restore documented and verified by round-tripping an archive through the binary. Tests: `internal/db/backup_test.go` (WAL capture, seeded user survives, no `-wal`/`-shm` companions, refuses an existing destination, missing source), `internal/tts` path-normalisation + configured-path. go build/vet/test clean. **Acceptance verified over public HTTPS with the LLM link genuinely down**: dictionaries, gloss, word lookup + phonetic, doc create/save, CJK FTS search, md/docx export, vocab capture, read-aloud EN + zh (real mp3, cache hit, 404-fallback for an unconfigured language) — all fine; `/check` → the warm 502 that renders as 小助手在休息; health public, HTTP→HTTPS with a valid cert. **Two items outstanding, both needing millenia access I don't have**: vLLM isn't bound to its headscale interface (so no AI pass works yet), and parodia's ssh key isn't authorized on millenia (so backups are VPS-local only — not yet a real off-box backup). Both have one-command fixes in `deploy/README.md` §3 and §5. Also this session: **DreamDict gained Spanish**, so the es pair is no longer gated — folded into Phases 20/21 and the "Later" bucket. Next: **Phase 16 (auth)** — Authentik already runs on the same VPS. - 2026-07-26: **Phase 15 complete — Petal is deployed at https://petal.parodia.dev** (user: "let's start this build plan"; scope confirmed as artifacts **plus** the actual deploy, millenia stays canonical, hostname `petal.parodia.dev`). Stack: `Dockerfile` (node → go → alpine; CGO off, so the runtime layer carries only ffmpeg + tzdata), `docker-compose.yml` behind the host's existing Traefik, and **two Piper sidecars** instead of the planned host systemd units — Piper turned out never to have been installed on the VPS and the account has no lingering session, so containers on an internal network with no published ports are both simpler and tighter. **Three real problems found by deploying rather than by planning:** (1) the image's `petal` user (uid 10001) has no claim on a bind-mounted host directory → SQLite `unable to open database file (14)` and a restart loop; the container now runs as the stack directory's owner (still non-root, and the host account keeps write access the backup script needs); (2) piper-tts **1.6.0 moved synthesis from `POST /` to `POST /synthesize`** with an identical body → every read-aloud 405'd; rather than pin both deployments to one Piper release the path became config (`TTS_PATH`, default `/`, so millenia is untouched); (3) once the instance was live it was **a public, writable, unauthenticated API** — Petal authenticates nobody yet, so Traefik basic auth now holds the door until Phase 16, with `/api/health` exempt on its own higher-priority router. Backups: `db.Backup` via **`VACUUM INTO`** (WAL-coherent, no write lock, single file, refuses to overwrite) behind a `-backup` flag so the nightly job snapshots the running container; `deploy/backup-petal.sh` compresses, pushes to millenia with a size check, prunes both sides; cron at 03:15; restore documented and verified by round-tripping an archive through the binary. Tests: `internal/db/backup_test.go` (WAL capture, seeded user survives, no `-wal`/`-shm` companions, refuses an existing destination, missing source), `internal/tts` path-normalisation + configured-path. go build/vet/test clean. **Acceptance verified over public HTTPS with the LLM link genuinely down**: dictionaries, gloss, word lookup + phonetic, doc create/save, CJK FTS search, md/docx export, vocab capture, read-aloud EN + zh (real mp3, cache hit, 404-fallback for an unconfigured language) — all fine; `/check` → the warm 502 that renders as 小助手在休息; health public, HTTP→HTTPS with a valid cert. **Two items outstanding, both needing millenia access I don't have**: vLLM isn't bound to its headscale interface (so no AI pass works yet), and parodia's ssh key isn't authorized on millenia (so backups are VPS-local only — not yet a real off-box backup). Both have one-command fixes in `deploy/README.md` §3 and §5. Also this session: **DreamDict gained Spanish**, so the es pair is no longer gated — folded into Phases 20/21 and the "Later" bucket. Next: **Phase 16 (auth)** — Authentik already runs on the same VPS.
- 2026-07-26: **Product direction + execution plan ratified** (user: "make it so, number one"). New `SUGGESTIONS.md` (product rationale for the language-learning direction): the **pair model** — every user gets one (English + X) pair, X ∈ {zh, pt-PT, fr, maybe es}, bilingual UI in the pair, type in either language, direction inferred (no detector: both-dictionaries spellcheck, show-both gloss on collision); **langpacks** keyed by X; **LLM-minimalism** as a standing principle (LLM is garnish, never a gatekeeper — grammar-lite rule pack + embedded miscollocation list planned as code-first layers). Deployment settled: Petal on the **parodia.dev VPS**, vLLM on millenia over **headscale** (the only cross-VPN dependency; Piper is VPS-local). All `MULTIUSER_PLAN.md` OPENs ratified: in-app OIDC (B), 30-day sliding sessions, allowlist, migration script, image-store fix with auth, DreamDict via package import (Option 3, module rename prereq in the dreamdict repo), zh stays on ECDICT until compared. Everything expanded into **Phases 1522** above with standing rules (isolation tests same-commit, LLM-minimalism, bilingual aesthetic). Ready for implementation handoff starting at Phase 15. - 2026-07-26: **Product direction + execution plan ratified** (user: "make it so, number one"). New `SUGGESTIONS.md` (product rationale for the language-learning direction): the **pair model** — every user gets one (English + X) pair, X ∈ {zh, pt-PT, fr, maybe es}, bilingual UI in the pair, type in either language, direction inferred (no detector: both-dictionaries spellcheck, show-both gloss on collision); **langpacks** keyed by X; **LLM-minimalism** as a standing principle (LLM is garnish, never a gatekeeper — grammar-lite rule pack + embedded miscollocation list planned as code-first layers). Deployment settled: Petal on the **parodia.dev VPS**, vLLM on millenia over **headscale** (the only cross-VPN dependency; Piper is VPS-local). All `MULTIUSER_PLAN.md` OPENs ratified: in-app OIDC (B), 30-day sliding sessions, allowlist, migration script, image-store fix with auth, DreamDict via package import (Option 3, module rename prereq in the dreamdict repo), zh stays on ECDICT until compared. Everything expanded into **Phases 1522** above with standing rules (isolation tests same-commit, LLM-minimalism, bilingual aesthetic). Ready for implementation handoff starting at Phase 15.
- 2026-07-26: **Multi-user groundwork** (user: "let's start preparing Petal for multi-user support"; scope agreed as plumbing-only, aimed at Authentik). New **`internal/auth`** package — context-carried identity (`WithUser`/`UserID`), a `Resolver` seam (`Resolve(*http.Request) (string, error)`), `StaticResolver` for today's single user, and `Middleware` that 401s anything unresolved. `main.go` splits `/api` into a **public group** (`/health`, `/version` — a monitoring probe must not need a session) and an **authenticated group** carrying everything else. All ~35 `db.LocalUserID` call sites across `docs`/`suggestions`/`vocab` now read the caller from the request; helpers that had no request in scope (`fetch`, `ownsDoc`, `ownsTag`, `tagsByDoc`, `fetchVersion`, `passportVersions`, `fetchPending`, vocab `fetch`) take an explicit `userID` param. `UserID` returns `""` rather than panicking when middleware is absent, so a mis-wired route **fails closed** (every query is `WHERE user_id = ?` → matches nothing). **Two real access-control gaps found and fixed while threading**: `setStatus` (accept/dismiss) updated a suggestion by bare id with **no ownership check at all**, and `fetchPending`/`listForDoc` read a document's suggestions by `doc_id` alone — a leak of the quoted source sentences. Both now scope through `documents.user_id`. **A third bug was caught by the new tests, not by the compiler**: `docs.fetch` gained a `userID` parameter but kept binding `db.LocalUserID` in the query — legal Go (unused params compile), silently unscoped, and it would have shipped. New tests: `internal/auth/auth_test.go` (round-trip, absent-context, both 401 paths) and **two-user isolation suites** (`docs/isolation_test.go`, `suggestions/isolation_test.go`) that mount the same routers twice behind two resolvers over one DB and assert a stranger gets 404 on get/update/delete/export/passport/snapshot/version-preview/restore/tag-assign/tag-rename/tag-delete/suggestion-accept/dismiss, sees nothing in list/search/version-list, and leaves the owner's data untouched. go build/vet/test all clean. **Still global, deliberately out of scope** (flagged for the auth phase): the image store is content-addressed with no per-user association or DB row — any authenticated user holding a hash can fetch any image (capability-URL security, needs a table + migration to fix); `export-all` is correctly scoped; frontend `localStorage` keys (`petal.spell.personal`, `petal.companion`, sound/petals prefs) are per-browser, not per-account, so they'd bleed across users sharing a device. - 2026-07-26: **Multi-user groundwork** (user: "let's start preparing Petal for multi-user support"; scope agreed as plumbing-only, aimed at Authentik). New **`internal/auth`** package — context-carried identity (`WithUser`/`UserID`), a `Resolver` seam (`Resolve(*http.Request) (string, error)`), `StaticResolver` for today's single user, and `Middleware` that 401s anything unresolved. `main.go` splits `/api` into a **public group** (`/health`, `/version` — a monitoring probe must not need a session) and an **authenticated group** carrying everything else. All ~35 `db.LocalUserID` call sites across `docs`/`suggestions`/`vocab` now read the caller from the request; helpers that had no request in scope (`fetch`, `ownsDoc`, `ownsTag`, `tagsByDoc`, `fetchVersion`, `passportVersions`, `fetchPending`, vocab `fetch`) take an explicit `userID` param. `UserID` returns `""` rather than panicking when middleware is absent, so a mis-wired route **fails closed** (every query is `WHERE user_id = ?` → matches nothing). **Two real access-control gaps found and fixed while threading**: `setStatus` (accept/dismiss) updated a suggestion by bare id with **no ownership check at all**, and `fetchPending`/`listForDoc` read a document's suggestions by `doc_id` alone — a leak of the quoted source sentences. Both now scope through `documents.user_id`. **A third bug was caught by the new tests, not by the compiler**: `docs.fetch` gained a `userID` parameter but kept binding `db.LocalUserID` in the query — legal Go (unused params compile), silently unscoped, and it would have shipped. New tests: `internal/auth/auth_test.go` (round-trip, absent-context, both 401 paths) and **two-user isolation suites** (`docs/isolation_test.go`, `suggestions/isolation_test.go`) that mount the same routers twice behind two resolvers over one DB and assert a stranger gets 404 on get/update/delete/export/passport/snapshot/version-preview/restore/tag-assign/tag-rename/tag-delete/suggestion-accept/dismiss, sees nothing in list/search/version-list, and leaves the owner's data untouched. go build/vet/test all clean. **Still global, deliberately out of scope** (flagged for the auth phase): the image store is content-addressed with no per-user association or DB row — any authenticated user holding a hash can fetch any image (capability-URL security, needs a table + migration to fix); `export-all` is correctly scoped; frontend `localStorage` keys (`petal.spell.personal`, `petal.companion`, sound/petals prefs) are per-browser, not per-account, so they'd bleed across users sharing a device.
+121 -14
View File
@@ -195,18 +195,23 @@ docker run --rm -v /tmp/restore-check:/data --user "$(id -u):$(id -g)" \
--entrypoint sh petal:local -c '/app/petal -backup /data/verify.db' --entrypoint sh petal:local -c '/app/petal -backup /data/verify.db'
``` ```
### ⚠️ millenia — the canonical instance — has no automated backup ### On millenia — `petal-backup.timer`
**This is the real gap.** Her actual writing lives on millenia, and as of Until 2026-07-27 her actual writing had **no scheduled backup at all**; the
2026-07-27 nothing backs it up on a schedule: `~/petal/backups` holds only newest snapshot was a month old. It now runs nightly at 03:20
ad-hoc pre-deploy snapshots, the newest a month old. The empty VPS staging (`Persistent=true`, because the box isn't on 24/7 and a missed window would
database is currently better protected than the live one. otherwise be skipped silently):
`deploy/backup-petal.sh` is written for exactly this and is not yet installed ```bash
there — it snapshots via `petal -backup` (`VACUUM INTO`, safe against the live sudo install -m 0644 deploy/petal-backup.service deploy/petal-backup.timer /etc/systemd/system/
app), compresses, pushes off-box with a post-transfer size check, and prunes sudo systemctl daemon-reload && sudo systemctl enable --now petal-backup.timer
both ends. What it does **not** yet do is encrypt at rest, which is being sudo systemctl start petal-backup.service # prove it before trusting it
decided separately. ```
`deploy/backup-petal.sh` snapshots via `petal -backup`, gzips, **age-encrypts
with the parodia public recipient**, pushes to the VPS over headscale with a
post-transfer size check, and prunes both ends. The private identity is offline,
so neither millenia nor the VPS can decrypt what it is holding — verified.
A manual snapshot any time, no tooling required: A manual snapshot any time, no tooling required:
@@ -214,11 +219,113 @@ A manual snapshot any time, no tooling required:
cd ~/petal && ./petal -backup ~/petal/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).db cd ~/petal && ./petal -backup ~/petal/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).db
``` ```
### ⚠️ millenia's Petal is unsupervised Restore is a copy — stop Petal, drop the file in as `data/petal.db`, remove any
stale `-wal`/`-shm`, start.
It runs as a bare `./petal` with PPID 1 — no systemd unit, no screen session. A ---
crash or reboot leaves it down until someone notices. `deploy/piper.service` is
the pattern to copy if you want it supervised. ## 6. Encryption at rest
### VPS — `/home/reala/petal/data` is a LUKS volume
`deploy/setup-encrypted-data.sh` puts the data directory on LUKS2 over a sparse
file at `/var/lib/petal-crypt.img`. That covers `petal.db`, uploaded `images/`,
**and the TTS cache** — which is synthesized audio of her sentences and is easy
to forget.
LUKS-on-a-file rather than gocryptfs because Petal is SQLite in WAL mode: WAL
needs a shared-memory index (`-shm`) mapped consistently across processes, and
FUSE has a long history of subtle mmap/locking differences. A block device with
ext4 behaves exactly like a disk to SQLite, which is the only guarantee worth
having under a database.
**What it protects, honestly.** The key lives at `/etc/petal/dataset.key` on the
same host so the volume auto-unlocks at boot. That is a deliberate availability
tradeoff:
| | |
|---|---|
| protects against | a decommissioned or resold disk; reading the raw block device; casual browsing of a filesystem snapshot that excludes `/etc` |
| does **not** protect against | anyone holding the whole VM image — they get the keyfile with the ciphertext; or anything at all while the host is running and mounted |
Real protection from a provider-side snapshot needs the key off-box (fetched
over the VPN at boot). Considered, not chosen.
Two things this setup got wrong the first time, both caught by rehearsing a
reboot rather than trusting a clean run — worth knowing if you rebuild it:
- **Mounting over a directory hides its contents, it does not remove them.** The
first pass left the original plaintext `petal.db` and WAL sitting on the
unencrypted root filesystem, invisible under the mount. The script now shreds
the originals before mounting and refuses to continue if the mountpoint will
not come up empty.
- **`systemd-cryptsetup` was not installed**, so `/etc/crypttab` was ignored
entirely and the volume would never have unlocked at boot. The script now
refuses to run without the generator present.
Check it any time:
```bash
sudo ./deploy/setup-encrypted-data.sh --status
```
### The mount-liveness guard
The mountpoint directory exists whether or not the volume is mounted, so a boot
where the unlock failed would start Petal against an empty unencrypted
directory and quietly serve a blank database — the failure that looks like data
loss. `data/.volume-ok` lives on the encrypted filesystem and is bind-mounted
with `create_host_path: false`, turning that into a loud container start
failure:
```
Error response from daemon: invalid mount config for type "bind":
bind source path does not exist: /home/reala/petal/data/.volume-ok
```
Verified by unmounting and attempting a start.
**A true reboot has not been tested** — the VPS also runs matrix, lemmy, akkoma,
gitea and authentik, so rebooting it is your call. The boot path was rehearsed
through `local-fs.target`, which pulls the mount, which pulls the unlock.
### millenia is not encrypted at rest
LVM, no LUKS. Her canonical writing sits in plaintext on the home box. Backups
leaving it are age-encrypted; the disk itself is not.
---
## 7. Supervision and monitoring
Petal on millenia ran for months as a bare `./petal` with PPID 1 — no unit, no
screen session — so a crash or reboot left it down until someone noticed. It is
now `petal.service`:
```bash
sudo install -m 0644 deploy/petal.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now petal.service
```
Verified by `kill -9`-ing it and watching systemd bring it back.
**Piper's silent-failure mode is fixed.** Both units now set
`StartLimitIntervalSec=300` / `StartLimitBurst=5`. With `RestartSec=3` and
systemd's default 10-second window, only ~3 restarts ever landed inside it, so
the burst limit was never reached and a dead service looped **26,800+ times over
a day without ever entering `failed`**. A genuinely broken Piper now shows up in
`systemctl --user --failed`.
### Still to do — an external probe
Nothing yet watches millenia from outside. uptime-kuma already runs on the VPS
and can reach millenia over headscale, so the missing piece is two monitors
(they need the uptime-kuma UI, hence not scripted here):
- `http://100.64.0.2:8088/api/health` — Petal itself
- a POST to `http://100.64.0.2:8088/api/tts` — catches a dead Piper, which
`/api/health` will not, because read-aloud degrades silently to browser
speech
--- ---
+47 -15
View File
@@ -1,15 +1,17 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Nightly off-VPS backup of Petal's database. # Nightly off-box backup of Petal's database.
# #
# ./backup-petal.sh # snapshot, compress, push off-box, prune # ./backup-petal.sh # snapshot, compress, encrypt, push, prune
# ./backup-petal.sh --local-only # snapshot + prune, skip the remote push # ./backup-petal.sh --local-only # snapshot + prune, skip the remote push
# #
# Run it from cron on the VPS (see deploy/README.md). The snapshot itself goes # Used on millenia, driven by petal-backup.timer (see deploy/README.md). The
# through `petal -backup`, which uses SQLite's VACUUM INTO: one coherent file # VPS does not use this script -- Petal rides parodia-backup there.
# including anything still in the WAL, taken without a write lock, so it is #
# safe against the live running app. That is why this script never touches # The snapshot goes through `petal -backup`, which uses SQLite's VACUUM INTO:
# petal.db / -wal / -shm directly — copying those three separately can capture # one coherent file including anything still in the WAL, taken without a write
# a torn mid-checkpoint state. # lock, so it is safe against the live running app. That is why this script
# never touches petal.db / -wal / -shm directly — copying those three
# separately can capture a torn mid-checkpoint state.
# #
# Everything below is overridable from the environment. # Everything below is overridable from the environment.
set -euo pipefail set -euo pipefail
@@ -26,6 +28,9 @@ REMOTE_DIR="${REMOTE_DIR:-petal-backups}"
# Retention, in days, on each side. # Retention, in days, on each side.
KEEP_LOCAL_DAYS="${KEEP_LOCAL_DAYS:-7}" KEEP_LOCAL_DAYS="${KEEP_LOCAL_DAYS:-7}"
KEEP_REMOTE_DAYS="${KEEP_REMOTE_DAYS:-30}" KEEP_REMOTE_DAYS="${KEEP_REMOTE_DAYS:-30}"
# age public recipient. Set it and every archive is encrypted before it leaves
# (and at rest locally too); leave it empty and the script says so loudly.
AGE_RECIPIENT="${AGE_RECIPIENT:-}"
local_only=0 local_only=0
[ "${1:-}" = "--local-only" ] && local_only=1 [ "${1:-}" = "--local-only" ] && local_only=1
@@ -35,17 +40,44 @@ name="petal-${stamp}.db"
cd "$STACK_DIR" cd "$STACK_DIR"
echo ">> snapshotting to data/backups/${name}" mkdir -p "$LOCAL_DIR"
# The container writes to its own /data mount; ./data/backups is the same
# directory seen from the host.
docker compose exec -T petal /app/petal -backup "/data/backups/${name}"
snapshot="${LOCAL_DIR}/${name}" snapshot="${LOCAL_DIR}/${name}"
# Two deployment shapes: the VPS runs the compose stack, millenia runs a bare
# binary. Either way the snapshot goes through `petal -backup` (VACUUM INTO),
# which is safe against the live process, so neither has to stop writing.
if [ -f "$STACK_DIR/docker-compose.yml" ] && docker compose ps --status running 2>/dev/null | grep -q petal; then
echo ">> snapshotting via the running container -> data/backups/${name}"
# ./data/backups on the host is the container's /data/backups.
docker compose exec -T petal /app/petal -backup "/data/backups/${name}"
elif [ -x "$STACK_DIR/petal" ]; then
echo ">> snapshotting via the local binary -> ${snapshot}"
# DATABASE_PATH must match the running instance; start.sh is the source of
# truth for it, so read it from there rather than guessing.
DB_PATH="$(sed -n 's/^export DATABASE_PATH=//p' "$STACK_DIR/start.sh" 2>/dev/null | tail -1)"
DATABASE_PATH="${DB_PATH:-$STACK_DIR/data/petal.db}" "$STACK_DIR/petal" -backup "$snapshot"
else
echo "no way to snapshot: neither a running petal container nor $STACK_DIR/petal" >&2
exit 1
fi
[ -s "$snapshot" ] || { echo "snapshot missing or empty: $snapshot" >&2; exit 1; } [ -s "$snapshot" ] || { echo "snapshot missing or empty: $snapshot" >&2; exit 1; }
echo ">> compressing" echo ">> compressing"
gzip -9 "$snapshot" gzip -9 "$snapshot"
archive="${snapshot}.gz" archive="${snapshot}.gz"
# Encrypt with age when a recipient is configured. The recipient is a PUBLIC
# key -- this host can write backups it cannot itself decrypt, and the private
# identity stays offline. Same custody model as parodia-backup. Without this,
# an off-box copy is just her writing sitting in plaintext on another machine.
if [ -n "$AGE_RECIPIENT" ]; then
age -r "$AGE_RECIPIENT" -o "${archive}.age" "$archive"
shred -uz "$archive" 2>/dev/null || rm -f "$archive"
archive="${archive}.age"
else
echo " (AGE_RECIPIENT unset: this backup is NOT encrypted)" >&2
fi
echo " $(du -h "$archive" | cut -f1) ${archive}" echo " $(du -h "$archive" | cut -f1) ${archive}"
if [ "$local_only" -eq 0 ] && [ -n "$REMOTE_HOST" ]; then if [ "$local_only" -eq 0 ] && [ -n "$REMOTE_HOST" ]; then
@@ -68,7 +100,7 @@ if [ "$local_only" -eq 0 ] && [ -n "$REMOTE_HOST" ]; then
echo ">> pruning remote copies older than ${KEEP_REMOTE_DAYS} days" echo ">> pruning remote copies older than ${KEEP_REMOTE_DAYS} days"
ssh -o BatchMode=yes "$target" \ ssh -o BatchMode=yes "$target" \
"find '${REMOTE_DIR}' -name 'petal-*.db.gz' -type f -mtime +${KEEP_REMOTE_DAYS} -delete" "find '${REMOTE_DIR}' \\( -name 'petal-*.db.gz' -o -name 'petal-*.db.gz.age' \\) -type f -mtime +${KEEP_REMOTE_DAYS} -delete"
elif [ "$local_only" -eq 1 ]; then elif [ "$local_only" -eq 1 ]; then
echo ">> --local-only: skipping the remote push" echo ">> --local-only: skipping the remote push"
else else
@@ -76,6 +108,6 @@ else
fi fi
echo ">> pruning local copies older than ${KEEP_LOCAL_DAYS} days" echo ">> pruning local copies older than ${KEEP_LOCAL_DAYS} days"
find "$LOCAL_DIR" -name 'petal-*.db.gz' -type f -mtime "+${KEEP_LOCAL_DAYS}" -delete find "$LOCAL_DIR" \( -name 'petal-*.db.gz' -o -name 'petal-*.db.gz.age' \) -type f -mtime "+${KEEP_LOCAL_DAYS}" -delete
echo ">> done" echo ">> done"
+23
View File
@@ -0,0 +1,23 @@
[Unit]
Description=Nightly backup of Petal's database (millenia)
Documentation=file:///home/reala/petal/deploy/README.md
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=reala
WorkingDirectory=/home/reala/petal
# Encrypted with the parodia age recipient before it leaves the box, then
# pushed to the VPS over headscale. The recipient is a public key and the
# private identity is offline, so neither millenia nor the VPS can decrypt what
# they are holding. Replaces nothing -- before this there was no scheduled
# backup of her writing at all; the newest snapshot on 2026-07-27 was a month
# old.
Environment=AGE_RECIPIENT=age19n4k55m9d50xew5vj2ehmcsf3wuj7fhgmfpckadpvcya4032q9dqrt4yjw
Environment=REMOTE_USER=reala
Environment=REMOTE_HOST=100.64.0.1
Environment=REMOTE_DIR=petal-backups-millenia
ExecStart=/home/reala/petal/deploy/backup-petal.sh
Nice=10
IOSchedulingClass=idle
+12
View File
@@ -0,0 +1,12 @@
[Unit]
Description=Nightly Petal database backup (millenia)
[Timer]
OnCalendar=*-*-* 03:20:00
# The box is not on 24/7; without this a missed window would just be skipped
# and the backup would silently never run.
Persistent=true
RandomizedDelaySec=300
[Install]
WantedBy=timers.target
+24
View File
@@ -0,0 +1,24 @@
[Unit]
Description=Petal writing editor (millenia)
# Petal ran unsupervised for a long time -- a bare ./petal with PPID 1, no unit
# and no screen session -- so a crash or a reboot left it silently down until
# somebody noticed. It also wants vLLM up first, though it degrades warmly if
# the model is unreachable, so this is Wants and not Requires.
After=network-online.target vllm-chat.service
Wants=network-online.target vllm-chat.service
[Service]
Type=simple
User=reala
WorkingDirectory=/home/reala/petal
# start.sh carries the environment (ports, LLM endpoint, Piper endpoints and
# TTS_PATH) and execs the binary, so the service supervises Petal itself rather
# than a shell wrapper.
ExecStart=/home/reala/petal/start.sh
Restart=on-failure
RestartSec=5
StandardOutput=append:/home/reala/petal/petal.log
StandardError=append:/home/reala/petal/petal.log
[Install]
WantedBy=multi-user.target
+8
View File
@@ -2,6 +2,14 @@
Description=Piper TTS HTTP server — Chinese voice (read-aloud backend for petal) Description=Piper TTS HTTP server — Chinese voice (read-aloud backend for petal)
After=network-online.target After=network-online.target
Wants=network-online.target Wants=network-online.target
# Give up loudly instead of retrying forever. This service once failed 26,800+
# times over a day without anyone noticing: RestartSec=3 means only ~3 restarts
# land inside systemd's default 10s StartLimitIntervalSec, so the default burst
# of 5 was never reached and the unit never entered `failed`. Widening the
# window to 5 minutes makes a genuinely broken Piper show up in
# `systemctl --user --failed` while still riding out transient blips.
StartLimitIntervalSec=300
StartLimitBurst=5
[Service] [Service]
Type=simple Type=simple
+8
View File
@@ -2,6 +2,14 @@
Description=Piper TTS HTTP server (read-aloud backend for petal) Description=Piper TTS HTTP server (read-aloud backend for petal)
After=network-online.target After=network-online.target
Wants=network-online.target Wants=network-online.target
# Give up loudly instead of retrying forever. This service once failed 26,800+
# times over a day without anyone noticing: RestartSec=3 means only ~3 restarts
# land inside systemd's default 10s StartLimitIntervalSec, so the default burst
# of 5 was never reached and the unit never entered `failed`. Widening the
# window to 5 minutes makes a genuinely broken Piper show up in
# `systemctl --user --failed` while still riding out transient blips.
StartLimitIntervalSec=300
StartLimitBurst=5
[Service] [Service]
Type=simple Type=simple