Reach vLLM over headscale with a forwarder, and fix millenia's Piper

Rebinding vllm-chat was the expensive option: Petal, Gogobee and Open
WebUI all point at 127.0.0.1:8000, and Open WebUI keeps its endpoint in
its own database rather than in env, so moving the bind address meant
editing three consumers and reloading a 35B AWQ model. The socat unit
adds a second listener on the headscale address instead -- local callers
untouched, no downtime, and the only new exposure is on the VPN. Bound
to 100.64.0.2 specifically, never 0.0.0.0: the far end is a public host.

Verified: a grammar checkpoint from petal.parodia.dev returns real
suggestions in ~3s over the VPN.

Also documents two things found on millenia that were invisible from
outside it:

- Piper had been dead since the Jul 26 reboot, 26,800+ failed restarts,
  with read-aloud silently falling back to browser Web Speech. An OS
  upgrade moved /usr/bin/python3 from 3.13 to 3.14, and venv/bin/python3
  is a symlink to the system interpreter, so lib/python3.13/site-packages
  went invisible -- sys.path had no site-packages at all. Recreating the
  venv lands Piper 1.6.0, which is what TTS_PATH exists for.
- The canonical instance has no automated backup (newest snapshot a
  month old) and runs unsupervised with PPID 1. Both written down; the
  backup one is pending the encryption-at-rest decision.

Backups on the VPS now ride parodia-backup (age-encrypted, offsite, S3),
using VACUUM INTO rather than that script's iterdump helper -- iterdump
does not reproduce an FTS5 virtual table, so a restore would have come
back with cross-document search silently missing.

deploy/README.md

@@ -96,19 +96,33 @@ the link down, and the status bar shows the warm
 voice and collocation passes send a whole document, the timeout is a hard
 deadline on the completion call, and a WAN+VPN round trip eats the margin.
 
-### Outstanding — vLLM is not yet listening on headscale
+### How the link is exposed — a forwarder, not a rebind
 
-As of the Phase 15 deploy, `100.64.0.2:8000` refuses connections from the VPS:
+vLLM stays bound to `127.0.0.1:8000`. `vllm-headscale-proxy.service` (a socat
-vLLM is bound to loopback or the LAN interface, not the headscale one. Petal
+unit, in this directory) adds a second listener on `100.64.0.2:8000` that
-degrades correctly (verified — `POST /api/docs/{id}/check` returns the warm
+forwards to it.
-502), but no AI pass will work until this is fixed on millenia:
 
-- start vLLM with `--host 100.64.0.2` — the headscale address specifically,
+The plan originally said to rebind vLLM itself. That turned out to be the
-  **not** `0.0.0.0`; the other end of this link is a public host
+expensive option: `vllm-chat.service` is **shared** — Petal, Gogobee and Open
-- confirm with `ss -lntp | grep 8000` that it binds only that interface
+WebUI all point at `127.0.0.1:8000`, and Open WebUI stores its endpoint in its
-- from the VPS: `curl -s http://100.64.0.2:8000/v1/models`
+own database rather than in env — so moving the bind address would mean editing
-- put the model id it reports into `LLM_MODEL` / `LLM_CHAT_MODEL` in `.env`,
+three consumers and reloading a 35B AWQ model, minutes of downtime for all of
-  then `docker compose up -d`
+them. The forwarder adds a door instead of moving one: local callers are
+untouched, and the only new exposure is on the VPN interface.
+
+It binds `100.64.0.2` specifically, **never** `0.0.0.0`: the far end of this
+link is a public host, and the LAN has no business seeing an unauthenticated
+inference endpoint.
+
+```bash
+sudo install -m 0644 deploy/vllm-headscale-proxy.service /etc/systemd/system/
+sudo systemctl daemon-reload && sudo systemctl enable --now vllm-headscale-proxy
+ss -lntp | grep 8000          # expect BOTH 127.0.0.1:8000 and 100.64.0.2:8000
+```
+
+The model id (`qwen3.6-35b`) goes into `LLM_MODEL` / `LLM_CHAT_MODEL` on the
+VPS. Verified end to end: a grammar checkpoint from `petal.parodia.dev` returns
+real suggestions in ~3s over the VPN.
 
 ---
 
@@ -134,64 +148,77 @@ When Phase 16 lands, delete the `petal-auth` middleware label, the
 
 ## 5. Backups
 
-`deploy/backup-petal.sh` runs nightly from cron at 03:15:
+### On the VPS — folded into `parodia-backup`
+
+Petal rides the host's existing offsite job (`/usr/local/bin/parodia-backup`,
+`parodia-backup.timer`, nightly ~03:40): age-encrypted to S3, 14-day retention,
+dead-man snitch. The host holds only the age *public* recipient, so it writes
+backups it cannot itself decrypt.
 
 ```
-15 3 * * * REMOTE_HOST= /bin/bash $HOME/petal/deploy/backup-petal.sh >> $HOME/petal/data/backups/backup.log 2>&1
+push petal.db.age       sqlite_file_dump /home/reala/petal/data/petal.db
 ```
 
-The snapshot goes through `petal -backup`, which uses SQLite's `VACUUM INTO`.
+`/home/reala/petal/.env` is in the same job's secrets tarball — it carries the
-That matters: Petal runs in WAL mode, so the newest committed pages may live in
+interim basic-auth hash and, from Phase 16, the OIDC client secret.
-`petal.db-wal` rather than `petal.db`, and copying the three files separately can
-capture a torn mid-checkpoint state. `VACUUM INTO` reads one coherent snapshot
-including the WAL, takes no write lock (so it is safe against the live app), and
-emits a single file with no `-wal`/`-shm` companions. It refuses an existing
-destination, so a failed run cannot destroy the last good backup.
 
-The script then gzips, pushes off-box, verifies the transferred size, and prunes
+**Why `sqlite_file_dump` and not the script's existing `sqlite_dump`:** that
-both sides (7 days local, 30 days remote).
+helper uses Python's `iterdump`, which **does not reproduce an FTS5 virtual
+table**. It emits `documents_fts` as a raw `sqlite_master` row plus its shadow
+tables, and replaying the result dies with `no such table: documents_fts` —
+verified by round-tripping a real dump on 2026-07-27. Petal's cross-document
+search would have been silently missing after any restore. `sqlite_file_dump`
+runs `VACUUM INTO` instead: a genuine database file, virtual tables intact, WAL
+folded in, no write lock. Restore is a copy rather than a replay.
 
-### Outstanding — the off-VPS push is not yet enabled
+> If `apply.db` ever gains a virtual table, it needs the same treatment.
 
-`REMOTE_HOST` is empty in the cron line, so backups are currently **local to the
+### Restore (VPS)
-VPS only** — which is not a backup in the sense that matters. parodia's ssh key
-is not authorized on millenia. To enable it, on millenia:
-
-```bash
-echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICGL6fS7IKCs8xFxUoH/bI/0kq4AzW05bcfV2XHoEXYM ditto-deploy@parodia-box' >> ~/.ssh/authorized_keys
-```
-
-then on the VPS, prove it works by hand before touching cron:
-
-```bash
-cd ~/petal && REMOTE_USER=<millenia-user> ./deploy/backup-petal.sh
-```
-
-and set `REMOTE_USER=<millenia-user> REMOTE_HOST=100.64.0.2` in the cron line.
-
-### Restore
 
 ```bash
+age -d -i <offline-identity> petal.db.age > /tmp/petal.db   # from S3
 cd ~/petal
 docker compose stop petal                       # stop writers first
 mv data/petal.db data/petal.db.before-restore   # keep the current state
 rm -f data/petal.db-wal data/petal.db-shm       # a stale WAL against a new file
-gunzip -c data/backups/petal-<stamp>.db.gz > data/petal.db
+cp /tmp/petal.db data/petal.db
 docker compose start petal
 docker compose logs petal --tail 5              # expect "database ready"
 ```
 
-To sanity-check an archive before committing to it, restore it into a scratch
+To sanity-check an archive before committing to it, have Petal open it in a
-directory and have Petal open it:
+scratch directory — a clean exit means it reads end to end:
 
 ```bash
-mkdir -p /tmp/restore-check
+mkdir -p /tmp/restore-check && cp /tmp/petal.db /tmp/restore-check/petal.db
-gunzip -c data/backups/petal-<stamp>.db.gz > /tmp/restore-check/petal.db
 docker run --rm -v /tmp/restore-check:/data --user "$(id -u):$(id -g)" \
   --entrypoint sh petal:local -c '/app/petal -backup /data/verify.db'
 ```
 
-A clean exit means the file opens and reads end to end.
+### ⚠️ millenia — the canonical instance — has no automated backup
+
+**This is the real gap.** Her actual writing lives on millenia, and as of
+2026-07-27 nothing backs it up on a schedule: `~/petal/backups` holds only
+ad-hoc pre-deploy snapshots, the newest a month old. The empty VPS staging
+database is currently better protected than the live one.
+
+`deploy/backup-petal.sh` is written for exactly this and is not yet installed
+there — it snapshots via `petal -backup` (`VACUUM INTO`, safe against the live
+app), compresses, pushes off-box with a post-transfer size check, and prunes
+both ends. What it does **not** yet do is encrypt at rest, which is being
+decided separately.
+
+A manual snapshot any time, no tooling required:
+
+```bash
+cd ~/petal && ./petal -backup ~/petal/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).db
+```
+
+### ⚠️ millenia's Petal is unsupervised
+
+It runs as a bare `./petal` with PPID 1 — no systemd unit, no screen session. A
+crash or reboot leaves it down until someone notices. `deploy/piper.service` is
+the pattern to copy if you want it supervised.
 
 ---
 
@@ -226,9 +253,38 @@ plus an env pair, no code change.
 
 **Piper version note:** piper-tts moved synthesis from `POST /` to
 `POST /synthesize` in 1.6.0, with an identical request body. `TTS_PATH` selects
-which — it defaults to `/` for millenia's older server, and the VPS compose sets
+which — it defaults to `/`, and both the VPS compose and millenia's `start.sh`
-`/synthesize` for the 1.6.0 sidecars. If read-aloud starts returning 502 after a
+now set `/synthesize`. If read-aloud starts returning 502 after a Piper upgrade,
-Piper upgrade on millenia, that flag is the fix.
+that flag is the fix.
+
+**The venv is fragile across Python upgrades.** On 2026-07-27 millenia's Piper
+was found dead with **26,800+ failed restarts**, silently since the Jul 26
+reboot — read-aloud had been falling back to browser Web Speech the whole time.
+Root cause: an OS upgrade moved `/usr/bin/python3` from 3.13 to 3.14, and
+`venv/bin/python3` is a *symlink to the system interpreter*, so the venv's
+`lib/python3.13/site-packages` became invisible — `sys.path` contained no
+site-packages at all. The failure surfaced as the misleading
+`No module named piper.http_server` even though `http_server.py` was sitting
+right there on disk.
+
+Fix (what was done — recreating the venv, not repairing it):
+
+```bash
+systemctl --user stop piper.service piper-zh.service
+mv ~/piper/venv ~/piper/venv.broken-py313
+python3 -m venv ~/piper/venv
+~/piper/venv/bin/pip install "piper-tts[http]"
+~/piper/venv/bin/python -c 'import piper.http_server'   # must not raise
+systemctl --user start piper.service piper-zh.service
+```
+
+That reinstall lands 1.6.0, so it must be paired with `TTS_PATH=/synthesize` in
+`start.sh` and a binary new enough to read that variable. Voices in
+`~/piper/voices` survive and do not need re-downloading.
+
+Worth knowing: `Restart=on-failure` will retry forever without ever alerting.
+Neither service reports its health anywhere, which is why this went unnoticed
+for a day. A `/api/tts` probe in uptime-kuma would have caught it.
 
 Verify end to end (through Petal, including the ffmpeg transcode):
 

deploy/vllm-headscale-proxy.service

@@ -0,0 +1,40 @@
+[Unit]
+Description=Expose millenia's vLLM chat server on the headscale interface only
+# Why a forwarder instead of just rebinding vLLM: vllm-chat.service is shared.
+# Petal, Gogobee and Open WebUI all talk to 127.0.0.1:8000, and Open WebUI keeps
+# its endpoint in its own database rather than in env, so moving vLLM's bind
+# address would mean editing three consumers and reloading a 35B AWQ model
+# (minutes of downtime for all of them). This adds a second door instead: local
+# callers keep loopback untouched, and only the headscale address gains a
+# listener. Nothing about vllm-chat changes.
+#
+# Deliberately NOT 0.0.0.0 — this reaches a public VPS over the VPN, and the
+# LAN has no business seeing an unauthenticated inference endpoint.
+After=network-online.target tailscaled.service vllm-chat.service
+Wants=network-online.target
+BindsTo=vllm-chat.service
+
+[Service]
+Type=simple
+# fork: one child per connection, so a single client can't block the others.
+# reuseaddr: survive a restart while sockets are still in TIME_WAIT.
+# The bind address is millenia's headscale IP; if tailscaled hasn't brought the
+# interface up yet the bind fails and Restart retries until it has.
+ExecStart=/usr/bin/socat -d TCP-LISTEN:8000,bind=100.64.0.2,fork,reuseaddr TCP:127.0.0.1:8000
+Restart=always
+RestartSec=5
+# Long generations hold a connection open; don't let systemd reap a healthy one.
+TimeoutStopSec=10
+
+# The process only shuttles bytes between two sockets — give it nothing else.
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=strict
+ProtectHome=true
+ProtectKernelTunables=true
+ProtectControlGroups=true
+RestrictAddressFamilies=AF_INET AF_INET6
+DynamicUser=true
+
+[Install]
+WantedBy=multi-user.target