millenia: nightly encrypted backup, supervision, and hardened Piper units

The canonical instance -- the one with her actual writing -- turned out
to be the least protected thing in the estate:

- No scheduled backup at all; the newest snapshot was a month old. Now
  petal-backup.timer: VACUUM INTO, gzip, age-encrypt with the parodia
  public recipient, push to the VPS over headscale with a size check,
  prune both ends. Persistent=true because the box is not on 24/7.
  Neither machine can decrypt what it holds; the identity is offline.

- Petal ran as a bare ./petal with PPID 1, so a crash or reboot left it
  down until somebody noticed. Now petal.service, verified by kill -9.

- The Piper units retried forever without ever failing: RestartSec=3
  against systemd's default 10s window means the burst limit is never
  reached, which is how a dead service logged 26,800+ restarts over a
  day while read-aloud silently fell back to browser speech.
  StartLimitIntervalSec=300 makes a broken Piper show up in --failed.

backup-petal.sh now handles both deployment shapes (compose exec on the
VPS, local binary on millenia) and encrypts before anything leaves the
host. The VPS no longer uses it -- Petal rides parodia-backup there.

deploy/README.md

@@ -195,18 +195,23 @@ docker run --rm -v /tmp/restore-check:/data --user "$(id -u):$(id -g)" \
   --entrypoint sh petal:local -c '/app/petal -backup /data/verify.db'
 ```
 
-### ⚠️ millenia — the canonical instance — has no automated backup
+### On millenia — `petal-backup.timer`
 
-**This is the real gap.** Her actual writing lives on millenia, and as of
-2026-07-27 nothing backs it up on a schedule: `~/petal/backups` holds only
-ad-hoc pre-deploy snapshots, the newest a month old. The empty VPS staging
-database is currently better protected than the live one.
+Until 2026-07-27 her actual writing had **no scheduled backup at all**; the
+newest snapshot was a month old. It now runs nightly at 03:20
+(`Persistent=true`, because the box isn't on 24/7 and a missed window would
+otherwise be skipped silently):
 
-`deploy/backup-petal.sh` is written for exactly this and is not yet installed
-there — it snapshots via `petal -backup` (`VACUUM INTO`, safe against the live
-app), compresses, pushes off-box with a post-transfer size check, and prunes
-both ends. What it does **not** yet do is encrypt at rest, which is being
-decided separately.
+```bash
+sudo install -m 0644 deploy/petal-backup.service deploy/petal-backup.timer /etc/systemd/system/
+sudo systemctl daemon-reload && sudo systemctl enable --now petal-backup.timer
+sudo systemctl start petal-backup.service     # prove it before trusting it
+```
+
+`deploy/backup-petal.sh` snapshots via `petal -backup`, gzips, **age-encrypts
+with the parodia public recipient**, pushes to the VPS over headscale with a
+post-transfer size check, and prunes both ends. The private identity is offline,
+so neither millenia nor the VPS can decrypt what it is holding — verified.
 
 A manual snapshot any time, no tooling required:
 
@@ -214,11 +219,113 @@ A manual snapshot any time, no tooling required:
 cd ~/petal && ./petal -backup ~/petal/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).db
 ```
 
-### ⚠️ millenia's Petal is unsupervised
+Restore is a copy — stop Petal, drop the file in as `data/petal.db`, remove any
+stale `-wal`/`-shm`, start.
 
-It runs as a bare `./petal` with PPID 1 — no systemd unit, no screen session. A
-crash or reboot leaves it down until someone notices. `deploy/piper.service` is
-the pattern to copy if you want it supervised.
+---
+
+## 6. Encryption at rest
+
+### VPS — `/home/reala/petal/data` is a LUKS volume
+
+`deploy/setup-encrypted-data.sh` puts the data directory on LUKS2 over a sparse
+file at `/var/lib/petal-crypt.img`. That covers `petal.db`, uploaded `images/`,
+**and the TTS cache** — which is synthesized audio of her sentences and is easy
+to forget.
+
+LUKS-on-a-file rather than gocryptfs because Petal is SQLite in WAL mode: WAL
+needs a shared-memory index (`-shm`) mapped consistently across processes, and
+FUSE has a long history of subtle mmap/locking differences. A block device with
+ext4 behaves exactly like a disk to SQLite, which is the only guarantee worth
+having under a database.
+
+**What it protects, honestly.** The key lives at `/etc/petal/dataset.key` on the
+same host so the volume auto-unlocks at boot. That is a deliberate availability
+tradeoff:
+
+| | |
+|---|---|
+| protects against | a decommissioned or resold disk; reading the raw block device; casual browsing of a filesystem snapshot that excludes `/etc` |
+| does **not** protect against | anyone holding the whole VM image — they get the keyfile with the ciphertext; or anything at all while the host is running and mounted |
+
+Real protection from a provider-side snapshot needs the key off-box (fetched
+over the VPN at boot). Considered, not chosen.
+
+Two things this setup got wrong the first time, both caught by rehearsing a
+reboot rather than trusting a clean run — worth knowing if you rebuild it:
+
+- **Mounting over a directory hides its contents, it does not remove them.** The
+  first pass left the original plaintext `petal.db` and WAL sitting on the
+  unencrypted root filesystem, invisible under the mount. The script now shreds
+  the originals before mounting and refuses to continue if the mountpoint will
+  not come up empty.
+- **`systemd-cryptsetup` was not installed**, so `/etc/crypttab` was ignored
+  entirely and the volume would never have unlocked at boot. The script now
+  refuses to run without the generator present.
+
+Check it any time:
+
+```bash
+sudo ./deploy/setup-encrypted-data.sh --status
+```
+
+### The mount-liveness guard
+
+The mountpoint directory exists whether or not the volume is mounted, so a boot
+where the unlock failed would start Petal against an empty unencrypted
+directory and quietly serve a blank database — the failure that looks like data
+loss. `data/.volume-ok` lives on the encrypted filesystem and is bind-mounted
+with `create_host_path: false`, turning that into a loud container start
+failure:
+
+```
+Error response from daemon: invalid mount config for type "bind":
+bind source path does not exist: /home/reala/petal/data/.volume-ok
+```
+
+Verified by unmounting and attempting a start.
+
+**A true reboot has not been tested** — the VPS also runs matrix, lemmy, akkoma,
+gitea and authentik, so rebooting it is your call. The boot path was rehearsed
+through `local-fs.target`, which pulls the mount, which pulls the unlock.
+
+### millenia is not encrypted at rest
+
+LVM, no LUKS. Her canonical writing sits in plaintext on the home box. Backups
+leaving it are age-encrypted; the disk itself is not.
+
+---
+
+## 7. Supervision and monitoring
+
+Petal on millenia ran for months as a bare `./petal` with PPID 1 — no unit, no
+screen session — so a crash or reboot left it down until someone noticed. It is
+now `petal.service`:
+
+```bash
+sudo install -m 0644 deploy/petal.service /etc/systemd/system/
+sudo systemctl daemon-reload && sudo systemctl enable --now petal.service
+```
+
+Verified by `kill -9`-ing it and watching systemd bring it back.
+
+**Piper's silent-failure mode is fixed.** Both units now set
+`StartLimitIntervalSec=300` / `StartLimitBurst=5`. With `RestartSec=3` and
+systemd's default 10-second window, only ~3 restarts ever landed inside it, so
+the burst limit was never reached and a dead service looped **26,800+ times over
+a day without ever entering `failed`**. A genuinely broken Piper now shows up in
+`systemctl --user --failed`.
+
+### Still to do — an external probe
+
+Nothing yet watches millenia from outside. uptime-kuma already runs on the VPS
+and can reach millenia over headscale, so the missing piece is two monitors
+(they need the uptime-kuma UI, hence not scripted here):
+
+- `http://100.64.0.2:8088/api/health` — Petal itself
+- a POST to `http://100.64.0.2:8088/api/tts` — catches a dead Piper, which
+  `/api/health` will not, because read-aloud degrades silently to browser
+  speech
 
 ---
 

deploy/backup-petal.sh

@@ -1,15 +1,17 @@
 #!/usr/bin/env bash
-# Nightly off-VPS backup of Petal's database.
+# Nightly off-box backup of Petal's database.
 #
-#   ./backup-petal.sh              # snapshot, compress, push off-box, prune
+#   ./backup-petal.sh              # snapshot, compress, encrypt, push, prune
 #   ./backup-petal.sh --local-only # snapshot + prune, skip the remote push
 #
-# Run it from cron on the VPS (see deploy/README.md). The snapshot itself goes
-# through `petal -backup`, which uses SQLite's VACUUM INTO: one coherent file
-# including anything still in the WAL, taken without a write lock, so it is
-# safe against the live running app. That is why this script never touches
-# petal.db / -wal / -shm directly — copying those three separately can capture
-# a torn mid-checkpoint state.
+# Used on millenia, driven by petal-backup.timer (see deploy/README.md). The
+# VPS does not use this script -- Petal rides parodia-backup there.
+#
+# The snapshot goes through `petal -backup`, which uses SQLite's VACUUM INTO:
+# one coherent file including anything still in the WAL, taken without a write
+# lock, so it is safe against the live running app. That is why this script
+# never touches petal.db / -wal / -shm directly — copying those three
+# separately can capture a torn mid-checkpoint state.
 #
 # Everything below is overridable from the environment.
 set -euo pipefail
@@ -26,6 +28,9 @@ REMOTE_DIR="${REMOTE_DIR:-petal-backups}"
 # Retention, in days, on each side.
 KEEP_LOCAL_DAYS="${KEEP_LOCAL_DAYS:-7}"
 KEEP_REMOTE_DAYS="${KEEP_REMOTE_DAYS:-30}"
+# age public recipient. Set it and every archive is encrypted before it leaves
+# (and at rest locally too); leave it empty and the script says so loudly.
+AGE_RECIPIENT="${AGE_RECIPIENT:-}"
 
 local_only=0
 [ "${1:-}" = "--local-only" ] && local_only=1
@@ -35,17 +40,44 @@ name="petal-${stamp}.db"
 
 cd "$STACK_DIR"
 
-echo ">> snapshotting to data/backups/${name}"
-# The container writes to its own /data mount; ./data/backups is the same
-# directory seen from the host.
-docker compose exec -T petal /app/petal -backup "/data/backups/${name}"
-
+mkdir -p "$LOCAL_DIR"
 snapshot="${LOCAL_DIR}/${name}"
+
+# Two deployment shapes: the VPS runs the compose stack, millenia runs a bare
+# binary. Either way the snapshot goes through `petal -backup` (VACUUM INTO),
+# which is safe against the live process, so neither has to stop writing.
+if [ -f "$STACK_DIR/docker-compose.yml" ] && docker compose ps --status running 2>/dev/null | grep -q petal; then
+  echo ">> snapshotting via the running container -> data/backups/${name}"
+  # ./data/backups on the host is the container's /data/backups.
+  docker compose exec -T petal /app/petal -backup "/data/backups/${name}"
+elif [ -x "$STACK_DIR/petal" ]; then
+  echo ">> snapshotting via the local binary -> ${snapshot}"
+  # DATABASE_PATH must match the running instance; start.sh is the source of
+  # truth for it, so read it from there rather than guessing.
+  DB_PATH="$(sed -n 's/^export DATABASE_PATH=//p' "$STACK_DIR/start.sh" 2>/dev/null | tail -1)"
+  DATABASE_PATH="${DB_PATH:-$STACK_DIR/data/petal.db}" "$STACK_DIR/petal" -backup "$snapshot"
+else
+  echo "no way to snapshot: neither a running petal container nor $STACK_DIR/petal" >&2
+  exit 1
+fi
+
 [ -s "$snapshot" ] || { echo "snapshot missing or empty: $snapshot" >&2; exit 1; }
 
 echo ">> compressing"
 gzip -9 "$snapshot"
 archive="${snapshot}.gz"
+
+# Encrypt with age when a recipient is configured. The recipient is a PUBLIC
+# key -- this host can write backups it cannot itself decrypt, and the private
+# identity stays offline. Same custody model as parodia-backup. Without this,
+# an off-box copy is just her writing sitting in plaintext on another machine.
+if [ -n "$AGE_RECIPIENT" ]; then
+  age -r "$AGE_RECIPIENT" -o "${archive}.age" "$archive"
+  shred -uz "$archive" 2>/dev/null || rm -f "$archive"
+  archive="${archive}.age"
+else
+  echo "   (AGE_RECIPIENT unset: this backup is NOT encrypted)" >&2
+fi
 echo "   $(du -h "$archive" | cut -f1)  ${archive}"
 
 if [ "$local_only" -eq 0 ] && [ -n "$REMOTE_HOST" ]; then
@@ -68,7 +100,7 @@ if [ "$local_only" -eq 0 ] && [ -n "$REMOTE_HOST" ]; then
 
   echo ">> pruning remote copies older than ${KEEP_REMOTE_DAYS} days"
   ssh -o BatchMode=yes "$target" \
-    "find '${REMOTE_DIR}' -name 'petal-*.db.gz' -type f -mtime +${KEEP_REMOTE_DAYS} -delete"
+    "find '${REMOTE_DIR}' \\( -name 'petal-*.db.gz' -o -name 'petal-*.db.gz.age' \\) -type f -mtime +${KEEP_REMOTE_DAYS} -delete"
 elif [ "$local_only" -eq 1 ]; then
   echo ">> --local-only: skipping the remote push"
 else
@@ -76,6 +108,6 @@ else
 fi
 
 echo ">> pruning local copies older than ${KEEP_LOCAL_DAYS} days"
-find "$LOCAL_DIR" -name 'petal-*.db.gz' -type f -mtime "+${KEEP_LOCAL_DAYS}" -delete
+find "$LOCAL_DIR" \( -name 'petal-*.db.gz' -o -name 'petal-*.db.gz.age' \) -type f -mtime "+${KEEP_LOCAL_DAYS}" -delete
 
 echo ">> done"

deploy/petal-backup.service

@@ -0,0 +1,23 @@
+[Unit]
+Description=Nightly backup of Petal's database (millenia)
+Documentation=file:///home/reala/petal/deploy/README.md
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+User=reala
+WorkingDirectory=/home/reala/petal
+# Encrypted with the parodia age recipient before it leaves the box, then
+# pushed to the VPS over headscale. The recipient is a public key and the
+# private identity is offline, so neither millenia nor the VPS can decrypt what
+# they are holding. Replaces nothing -- before this there was no scheduled
+# backup of her writing at all; the newest snapshot on 2026-07-27 was a month
+# old.
+Environment=AGE_RECIPIENT=age19n4k55m9d50xew5vj2ehmcsf3wuj7fhgmfpckadpvcya4032q9dqrt4yjw
+Environment=REMOTE_USER=reala
+Environment=REMOTE_HOST=100.64.0.1
+Environment=REMOTE_DIR=petal-backups-millenia
+ExecStart=/home/reala/petal/deploy/backup-petal.sh
+Nice=10
+IOSchedulingClass=idle

deploy/petal-backup.timer

@@ -0,0 +1,12 @@
+[Unit]
+Description=Nightly Petal database backup (millenia)
+
+[Timer]
+OnCalendar=*-*-* 03:20:00
+# The box is not on 24/7; without this a missed window would just be skipped
+# and the backup would silently never run.
+Persistent=true
+RandomizedDelaySec=300
+
+[Install]
+WantedBy=timers.target

deploy/petal.service

@@ -0,0 +1,24 @@
+[Unit]
+Description=Petal writing editor (millenia)
+# Petal ran unsupervised for a long time -- a bare ./petal with PPID 1, no unit
+# and no screen session -- so a crash or a reboot left it silently down until
+# somebody noticed. It also wants vLLM up first, though it degrades warmly if
+# the model is unreachable, so this is Wants and not Requires.
+After=network-online.target vllm-chat.service
+Wants=network-online.target vllm-chat.service
+
+[Service]
+Type=simple
+User=reala
+WorkingDirectory=/home/reala/petal
+# start.sh carries the environment (ports, LLM endpoint, Piper endpoints and
+# TTS_PATH) and execs the binary, so the service supervises Petal itself rather
+# than a shell wrapper.
+ExecStart=/home/reala/petal/start.sh
+Restart=on-failure
+RestartSec=5
+StandardOutput=append:/home/reala/petal/petal.log
+StandardError=append:/home/reala/petal/petal.log
+
+[Install]
+WantedBy=multi-user.target

deploy/piper-zh.service

@@ -2,6 +2,14 @@
 Description=Piper TTS HTTP server — Chinese voice (read-aloud backend for petal)
 After=network-online.target
 Wants=network-online.target
+# Give up loudly instead of retrying forever. This service once failed 26,800+
+# times over a day without anyone noticing: RestartSec=3 means only ~3 restarts
+# land inside systemd's default 10s StartLimitIntervalSec, so the default burst
+# of 5 was never reached and the unit never entered `failed`. Widening the
+# window to 5 minutes makes a genuinely broken Piper show up in
+# `systemctl --user --failed` while still riding out transient blips.
+StartLimitIntervalSec=300
+StartLimitBurst=5
 
 [Service]
 Type=simple

deploy/piper.service

@@ -2,6 +2,14 @@
 Description=Piper TTS HTTP server (read-aloud backend for petal)
 After=network-online.target
 Wants=network-online.target
+# Give up loudly instead of retrying forever. This service once failed 26,800+
+# times over a day without anyone noticing: RestartSec=3 means only ~3 restarts
+# land inside systemd's default 10s StartLimitIntervalSec, so the default burst
+# of 5 was never reached and the unit never entered `failed`. Widening the
+# window to 5 minutes makes a genuinely broken Piper show up in
+# `systemctl --user --failed` while still riding out transient blips.
+StartLimitIntervalSec=300
+StartLimitBurst=5
 
 [Service]
 Type=simple