millenia: nightly encrypted backup, supervision, and hardened Piper units

The canonical instance -- the one with her actual writing -- turned out
to be the least protected thing in the estate:

- No scheduled backup at all; the newest snapshot was a month old. Now
  petal-backup.timer: VACUUM INTO, gzip, age-encrypt with the parodia
  public recipient, push to the VPS over headscale with a size check,
  prune both ends. Persistent=true because the box is not on 24/7.
  Neither machine can decrypt what it holds; the identity is offline.

- Petal ran as a bare ./petal with PPID 1, so a crash or reboot left it
  down until somebody noticed. Now petal.service, verified by kill -9.

- The Piper units retried forever without ever failing: RestartSec=3
  against systemd's default 10s window means the burst limit is never
  reached, which is how a dead service logged 26,800+ restarts over a
  day while read-aloud silently fell back to browser speech.
  StartLimitIntervalSec=300 makes a broken Piper show up in --failed.

backup-petal.sh now handles both deployment shapes (compose exec on the
VPS, local binary on millenia) and encrypts before anything leaves the
host. The VPS no longer uses it -- Petal rides parodia-backup there.
This commit is contained in:
prosolis
2026-07-27 06:35:34 -07:00
parent 33e49ddb62
commit 42d857a878
8 changed files with 251 additions and 32 deletions
+47 -15
View File
@@ -1,15 +1,17 @@
#!/usr/bin/env bash
# Nightly off-VPS backup of Petal's database.
# Nightly off-box backup of Petal's database.
#
# ./backup-petal.sh # snapshot, compress, push off-box, prune
# ./backup-petal.sh # snapshot, compress, encrypt, push, prune
# ./backup-petal.sh --local-only # snapshot + prune, skip the remote push
#
# Run it from cron on the VPS (see deploy/README.md). The snapshot itself goes
# through `petal -backup`, which uses SQLite's VACUUM INTO: one coherent file
# including anything still in the WAL, taken without a write lock, so it is
# safe against the live running app. That is why this script never touches
# petal.db / -wal / -shm directly — copying those three separately can capture
# a torn mid-checkpoint state.
# Used on millenia, driven by petal-backup.timer (see deploy/README.md). The
# VPS does not use this script -- Petal rides parodia-backup there.
#
# The snapshot goes through `petal -backup`, which uses SQLite's VACUUM INTO:
# one coherent file including anything still in the WAL, taken without a write
# lock, so it is safe against the live running app. That is why this script
# never touches petal.db / -wal / -shm directly — copying those three
# separately can capture a torn mid-checkpoint state.
#
# Everything below is overridable from the environment.
set -euo pipefail
@@ -26,6 +28,9 @@ REMOTE_DIR="${REMOTE_DIR:-petal-backups}"
# Retention, in days, on each side.
KEEP_LOCAL_DAYS="${KEEP_LOCAL_DAYS:-7}"
KEEP_REMOTE_DAYS="${KEEP_REMOTE_DAYS:-30}"
# age public recipient. Set it and every archive is encrypted before it leaves
# (and at rest locally too); leave it empty and the script says so loudly.
AGE_RECIPIENT="${AGE_RECIPIENT:-}"
local_only=0
[ "${1:-}" = "--local-only" ] && local_only=1
@@ -35,17 +40,44 @@ name="petal-${stamp}.db"
cd "$STACK_DIR"
echo ">> snapshotting to data/backups/${name}"
# The container writes to its own /data mount; ./data/backups is the same
# directory seen from the host.
docker compose exec -T petal /app/petal -backup "/data/backups/${name}"
mkdir -p "$LOCAL_DIR"
snapshot="${LOCAL_DIR}/${name}"
# Two deployment shapes: the VPS runs the compose stack, millenia runs a bare
# binary. Either way the snapshot goes through `petal -backup` (VACUUM INTO),
# which is safe against the live process, so neither has to stop writing.
if [ -f "$STACK_DIR/docker-compose.yml" ] && docker compose ps --status running 2>/dev/null | grep -q petal; then
echo ">> snapshotting via the running container -> data/backups/${name}"
# ./data/backups on the host is the container's /data/backups.
docker compose exec -T petal /app/petal -backup "/data/backups/${name}"
elif [ -x "$STACK_DIR/petal" ]; then
echo ">> snapshotting via the local binary -> ${snapshot}"
# DATABASE_PATH must match the running instance; start.sh is the source of
# truth for it, so read it from there rather than guessing.
DB_PATH="$(sed -n 's/^export DATABASE_PATH=//p' "$STACK_DIR/start.sh" 2>/dev/null | tail -1)"
DATABASE_PATH="${DB_PATH:-$STACK_DIR/data/petal.db}" "$STACK_DIR/petal" -backup "$snapshot"
else
echo "no way to snapshot: neither a running petal container nor $STACK_DIR/petal" >&2
exit 1
fi
[ -s "$snapshot" ] || { echo "snapshot missing or empty: $snapshot" >&2; exit 1; }
echo ">> compressing"
gzip -9 "$snapshot"
archive="${snapshot}.gz"
# Encrypt with age when a recipient is configured. The recipient is a PUBLIC
# key -- this host can write backups it cannot itself decrypt, and the private
# identity stays offline. Same custody model as parodia-backup. Without this,
# an off-box copy is just her writing sitting in plaintext on another machine.
if [ -n "$AGE_RECIPIENT" ]; then
age -r "$AGE_RECIPIENT" -o "${archive}.age" "$archive"
shred -uz "$archive" 2>/dev/null || rm -f "$archive"
archive="${archive}.age"
else
echo " (AGE_RECIPIENT unset: this backup is NOT encrypted)" >&2
fi
echo " $(du -h "$archive" | cut -f1) ${archive}"
if [ "$local_only" -eq 0 ] && [ -n "$REMOTE_HOST" ]; then
@@ -68,7 +100,7 @@ if [ "$local_only" -eq 0 ] && [ -n "$REMOTE_HOST" ]; then
echo ">> pruning remote copies older than ${KEEP_REMOTE_DAYS} days"
ssh -o BatchMode=yes "$target" \
"find '${REMOTE_DIR}' -name 'petal-*.db.gz' -type f -mtime +${KEEP_REMOTE_DAYS} -delete"
"find '${REMOTE_DIR}' \\( -name 'petal-*.db.gz' -o -name 'petal-*.db.gz.age' \\) -type f -mtime +${KEEP_REMOTE_DAYS} -delete"
elif [ "$local_only" -eq 1 ]; then
echo ">> --local-only: skipping the remote push"
else
@@ -76,6 +108,6 @@ else
fi
echo ">> pruning local copies older than ${KEEP_LOCAL_DAYS} days"
find "$LOCAL_DIR" -name 'petal-*.db.gz' -type f -mtime "+${KEEP_LOCAL_DAYS}" -delete
find "$LOCAL_DIR" \( -name 'petal-*.db.gz' -o -name 'petal-*.db.gz.age' \) -type f -mtime "+${KEEP_LOCAL_DAYS}" -delete
echo ">> done"