millenia: nightly encrypted backup, supervision, and hardened Piper units

The canonical instance -- the one with her actual writing -- turned out
to be the least protected thing in the estate:

- No scheduled backup at all; the newest snapshot was a month old. Now
  petal-backup.timer: VACUUM INTO, gzip, age-encrypt with the parodia
  public recipient, push to the VPS over headscale with a size check,
  prune both ends. Persistent=true because the box is not on 24/7.
  Neither machine can decrypt what it holds; the identity is offline.

- Petal ran as a bare ./petal with PPID 1, so a crash or reboot left it
  down until somebody noticed. Now petal.service, verified by kill -9.

- The Piper units retried forever without ever failing: RestartSec=3
  against systemd's default 10s window means the burst limit is never
  reached, which is how a dead service logged 26,800+ restarts over a
  day while read-aloud silently fell back to browser speech.
  StartLimitIntervalSec=300 makes a broken Piper show up in --failed.

backup-petal.sh now handles both deployment shapes (compose exec on the
VPS, local binary on millenia) and encrypts before anything leaves the
host. The VPS no longer uses it -- Petal rides parodia-backup there.

deploy/README.md

@@ -195,18 +195,23 @@ docker run --rm -v /tmp/restore-check:/data --user "$(id -u):$(id -g)" \
   --entrypoint sh petal:local -c '/app/petal -backup /data/verify.db'
 ```
 
-### ⚠️ millenia — the canonical instance — has no automated backup
+### On millenia — `petal-backup.timer`
 
-**This is the real gap.** Her actual writing lives on millenia, and as of
-2026-07-27 nothing backs it up on a schedule: `~/petal/backups` holds only
-ad-hoc pre-deploy snapshots, the newest a month old. The empty VPS staging
-database is currently better protected than the live one.
+Until 2026-07-27 her actual writing had **no scheduled backup at all**; the
+newest snapshot was a month old. It now runs nightly at 03:20
+(`Persistent=true`, because the box isn't on 24/7 and a missed window would
+otherwise be skipped silently):
 
-`deploy/backup-petal.sh` is written for exactly this and is not yet installed
-there — it snapshots via `petal -backup` (`VACUUM INTO`, safe against the live
-app), compresses, pushes off-box with a post-transfer size check, and prunes
-both ends. What it does **not** yet do is encrypt at rest, which is being
-decided separately.
+```bash
+sudo install -m 0644 deploy/petal-backup.service deploy/petal-backup.timer /etc/systemd/system/
+sudo systemctl daemon-reload && sudo systemctl enable --now petal-backup.timer
+sudo systemctl start petal-backup.service     # prove it before trusting it
+```
+
+`deploy/backup-petal.sh` snapshots via `petal -backup`, gzips, **age-encrypts
+with the parodia public recipient**, pushes to the VPS over headscale with a
+post-transfer size check, and prunes both ends. The private identity is offline,
+so neither millenia nor the VPS can decrypt what it is holding — verified.
 
 A manual snapshot any time, no tooling required:
 
@@ -214,11 +219,113 @@ A manual snapshot any time, no tooling required:
 cd ~/petal && ./petal -backup ~/petal/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).db
 ```
 
-### ⚠️ millenia's Petal is unsupervised
+Restore is a copy — stop Petal, drop the file in as `data/petal.db`, remove any
+stale `-wal`/`-shm`, start.
 
-It runs as a bare `./petal` with PPID 1 — no systemd unit, no screen session. A
-crash or reboot leaves it down until someone notices. `deploy/piper.service` is
-the pattern to copy if you want it supervised.
+---
+
+## 6. Encryption at rest
+
+### VPS — `/home/reala/petal/data` is a LUKS volume
+
+`deploy/setup-encrypted-data.sh` puts the data directory on LUKS2 over a sparse
+file at `/var/lib/petal-crypt.img`. That covers `petal.db`, uploaded `images/`,
+**and the TTS cache** — which is synthesized audio of her sentences and is easy
+to forget.
+
+LUKS-on-a-file rather than gocryptfs because Petal is SQLite in WAL mode: WAL
+needs a shared-memory index (`-shm`) mapped consistently across processes, and
+FUSE has a long history of subtle mmap/locking differences. A block device with
+ext4 behaves exactly like a disk to SQLite, which is the only guarantee worth
+having under a database.
+
+**What it protects, honestly.** The key lives at `/etc/petal/dataset.key` on the
+same host so the volume auto-unlocks at boot. That is a deliberate availability
+tradeoff:
+
+| | |
+|---|---|
+| protects against | a decommissioned or resold disk; reading the raw block device; casual browsing of a filesystem snapshot that excludes `/etc` |
+| does **not** protect against | anyone holding the whole VM image — they get the keyfile with the ciphertext; or anything at all while the host is running and mounted |
+
+Real protection from a provider-side snapshot needs the key off-box (fetched
+over the VPN at boot). Considered, not chosen.
+
+Two things this setup got wrong the first time, both caught by rehearsing a
+reboot rather than trusting a clean run — worth knowing if you rebuild it:
+
+- **Mounting over a directory hides its contents, it does not remove them.** The
+  first pass left the original plaintext `petal.db` and WAL sitting on the
+  unencrypted root filesystem, invisible under the mount. The script now shreds
+  the originals before mounting and refuses to continue if the mountpoint will
+  not come up empty.
+- **`systemd-cryptsetup` was not installed**, so `/etc/crypttab` was ignored
+  entirely and the volume would never have unlocked at boot. The script now
+  refuses to run without the generator present.
+
+Check it any time:
+
+```bash
+sudo ./deploy/setup-encrypted-data.sh --status
+```
+
+### The mount-liveness guard
+
+The mountpoint directory exists whether or not the volume is mounted, so a boot
+where the unlock failed would start Petal against an empty unencrypted
+directory and quietly serve a blank database — the failure that looks like data
+loss. `data/.volume-ok` lives on the encrypted filesystem and is bind-mounted
+with `create_host_path: false`, turning that into a loud container start
+failure:
+
+```
+Error response from daemon: invalid mount config for type "bind":
+bind source path does not exist: /home/reala/petal/data/.volume-ok
+```
+
+Verified by unmounting and attempting a start.
+
+**A true reboot has not been tested** — the VPS also runs matrix, lemmy, akkoma,
+gitea and authentik, so rebooting it is your call. The boot path was rehearsed
+through `local-fs.target`, which pulls the mount, which pulls the unlock.
+
+### millenia is not encrypted at rest
+
+LVM, no LUKS. Her canonical writing sits in plaintext on the home box. Backups
+leaving it are age-encrypted; the disk itself is not.
+
+---
+
+## 7. Supervision and monitoring
+
+Petal on millenia ran for months as a bare `./petal` with PPID 1 — no unit, no
+screen session — so a crash or reboot left it down until someone noticed. It is
+now `petal.service`:
+
+```bash
+sudo install -m 0644 deploy/petal.service /etc/systemd/system/
+sudo systemctl daemon-reload && sudo systemctl enable --now petal.service
+```
+
+Verified by `kill -9`-ing it and watching systemd bring it back.
+
+**Piper's silent-failure mode is fixed.** Both units now set
+`StartLimitIntervalSec=300` / `StartLimitBurst=5`. With `RestartSec=3` and
+systemd's default 10-second window, only ~3 restarts ever landed inside it, so
+the burst limit was never reached and a dead service looped **26,800+ times over
+a day without ever entering `failed`**. A genuinely broken Piper now shows up in
+`systemctl --user --failed`.
+
+### Still to do — an external probe
+
+Nothing yet watches millenia from outside. uptime-kuma already runs on the VPS
+and can reach millenia over headscale, so the missing piece is two monitors
+(they need the uptime-kuma UI, hence not scripted here):
+
+- `http://100.64.0.2:8088/api/health` — Petal itself
+- a POST to `http://100.64.0.2:8088/api/tts` — catches a dead Piper, which
+  `/api/health` will not, because read-aloud degrades silently to browser
+  speech
 
 ---