From 623bd02b9caae5ac4a55afa8d3f7a3cb62ad31ee Mon Sep 17 00:00:00 2001 From: prosolis <5590409+prosolis@users.noreply.github.com> Date: Mon, 27 Jul 2026 06:15:00 -0700 Subject: [PATCH] Reach vLLM over headscale with a forwarder, and fix millenia's Piper Rebinding vllm-chat was the expensive option: Petal, Gogobee and Open WebUI all point at 127.0.0.1:8000, and Open WebUI keeps its endpoint in its own database rather than in env, so moving the bind address meant editing three consumers and reloading a 35B AWQ model. The socat unit adds a second listener on the headscale address instead -- local callers untouched, no downtime, and the only new exposure is on the VPN. Bound to 100.64.0.2 specifically, never 0.0.0.0: the far end is a public host. Verified: a grammar checkpoint from petal.parodia.dev returns real suggestions in ~3s over the VPN. Also documents two things found on millenia that were invisible from outside it: - Piper had been dead since the Jul 26 reboot, 26,800+ failed restarts, with read-aloud silently falling back to browser Web Speech. An OS upgrade moved /usr/bin/python3 from 3.13 to 3.14, and venv/bin/python3 is a symlink to the system interpreter, so lib/python3.13/site-packages went invisible -- sys.path had no site-packages at all. Recreating the venv lands Piper 1.6.0, which is what TTS_PATH exists for. - The canonical instance has no automated backup (newest snapshot a month old) and runs unsupervised with PPID 1. Both written down; the backup one is pending the encryption-at-rest decision. Backups on the VPS now ride parodia-backup (age-encrypted, offsite, S3), using VACUUM INTO rather than that script's iterdump helper -- iterdump does not reproduce an FTS5 virtual table, so a restore would have come back with cross-document search silently missing. --- deploy/README.md | 154 +++++++++++++++++++--------- deploy/vllm-headscale-proxy.service | 40 ++++++++ 2 files changed, 145 insertions(+), 49 deletions(-) create mode 100644 deploy/vllm-headscale-proxy.service diff --git a/deploy/README.md b/deploy/README.md index 3688957..88e6bb0 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -96,19 +96,33 @@ the link down, and the status bar shows the warm voice and collocation passes send a whole document, the timeout is a hard deadline on the completion call, and a WAN+VPN round trip eats the margin. -### Outstanding — vLLM is not yet listening on headscale +### How the link is exposed — a forwarder, not a rebind -As of the Phase 15 deploy, `100.64.0.2:8000` refuses connections from the VPS: -vLLM is bound to loopback or the LAN interface, not the headscale one. Petal -degrades correctly (verified — `POST /api/docs/{id}/check` returns the warm -502), but no AI pass will work until this is fixed on millenia: +vLLM stays bound to `127.0.0.1:8000`. `vllm-headscale-proxy.service` (a socat +unit, in this directory) adds a second listener on `100.64.0.2:8000` that +forwards to it. -- start vLLM with `--host 100.64.0.2` — the headscale address specifically, - **not** `0.0.0.0`; the other end of this link is a public host -- confirm with `ss -lntp | grep 8000` that it binds only that interface -- from the VPS: `curl -s http://100.64.0.2:8000/v1/models` -- put the model id it reports into `LLM_MODEL` / `LLM_CHAT_MODEL` in `.env`, - then `docker compose up -d` +The plan originally said to rebind vLLM itself. That turned out to be the +expensive option: `vllm-chat.service` is **shared** — Petal, Gogobee and Open +WebUI all point at `127.0.0.1:8000`, and Open WebUI stores its endpoint in its +own database rather than in env — so moving the bind address would mean editing +three consumers and reloading a 35B AWQ model, minutes of downtime for all of +them. The forwarder adds a door instead of moving one: local callers are +untouched, and the only new exposure is on the VPN interface. + +It binds `100.64.0.2` specifically, **never** `0.0.0.0`: the far end of this +link is a public host, and the LAN has no business seeing an unauthenticated +inference endpoint. + +```bash +sudo install -m 0644 deploy/vllm-headscale-proxy.service /etc/systemd/system/ +sudo systemctl daemon-reload && sudo systemctl enable --now vllm-headscale-proxy +ss -lntp | grep 8000 # expect BOTH 127.0.0.1:8000 and 100.64.0.2:8000 +``` + +The model id (`qwen3.6-35b`) goes into `LLM_MODEL` / `LLM_CHAT_MODEL` on the +VPS. Verified end to end: a grammar checkpoint from `petal.parodia.dev` returns +real suggestions in ~3s over the VPN. --- @@ -134,64 +148,77 @@ When Phase 16 lands, delete the `petal-auth` middleware label, the ## 5. Backups -`deploy/backup-petal.sh` runs nightly from cron at 03:15: +### On the VPS — folded into `parodia-backup` + +Petal rides the host's existing offsite job (`/usr/local/bin/parodia-backup`, +`parodia-backup.timer`, nightly ~03:40): age-encrypted to S3, 14-day retention, +dead-man snitch. The host holds only the age *public* recipient, so it writes +backups it cannot itself decrypt. ``` -15 3 * * * REMOTE_HOST= /bin/bash $HOME/petal/deploy/backup-petal.sh >> $HOME/petal/data/backups/backup.log 2>&1 +push petal.db.age sqlite_file_dump /home/reala/petal/data/petal.db ``` -The snapshot goes through `petal -backup`, which uses SQLite's `VACUUM INTO`. -That matters: Petal runs in WAL mode, so the newest committed pages may live in -`petal.db-wal` rather than `petal.db`, and copying the three files separately can -capture a torn mid-checkpoint state. `VACUUM INTO` reads one coherent snapshot -including the WAL, takes no write lock (so it is safe against the live app), and -emits a single file with no `-wal`/`-shm` companions. It refuses an existing -destination, so a failed run cannot destroy the last good backup. +`/home/reala/petal/.env` is in the same job's secrets tarball — it carries the +interim basic-auth hash and, from Phase 16, the OIDC client secret. -The script then gzips, pushes off-box, verifies the transferred size, and prunes -both sides (7 days local, 30 days remote). +**Why `sqlite_file_dump` and not the script's existing `sqlite_dump`:** that +helper uses Python's `iterdump`, which **does not reproduce an FTS5 virtual +table**. It emits `documents_fts` as a raw `sqlite_master` row plus its shadow +tables, and replaying the result dies with `no such table: documents_fts` — +verified by round-tripping a real dump on 2026-07-27. Petal's cross-document +search would have been silently missing after any restore. `sqlite_file_dump` +runs `VACUUM INTO` instead: a genuine database file, virtual tables intact, WAL +folded in, no write lock. Restore is a copy rather than a replay. -### Outstanding — the off-VPS push is not yet enabled +> If `apply.db` ever gains a virtual table, it needs the same treatment. -`REMOTE_HOST` is empty in the cron line, so backups are currently **local to the -VPS only** — which is not a backup in the sense that matters. parodia's ssh key -is not authorized on millenia. To enable it, on millenia: - -```bash -echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICGL6fS7IKCs8xFxUoH/bI/0kq4AzW05bcfV2XHoEXYM ditto-deploy@parodia-box' >> ~/.ssh/authorized_keys -``` - -then on the VPS, prove it works by hand before touching cron: - -```bash -cd ~/petal && REMOTE_USER= ./deploy/backup-petal.sh -``` - -and set `REMOTE_USER= REMOTE_HOST=100.64.0.2` in the cron line. - -### Restore +### Restore (VPS) ```bash +age -d -i petal.db.age > /tmp/petal.db # from S3 cd ~/petal docker compose stop petal # stop writers first mv data/petal.db data/petal.db.before-restore # keep the current state rm -f data/petal.db-wal data/petal.db-shm # a stale WAL against a new file -gunzip -c data/backups/petal-.db.gz > data/petal.db +cp /tmp/petal.db data/petal.db docker compose start petal docker compose logs petal --tail 5 # expect "database ready" ``` -To sanity-check an archive before committing to it, restore it into a scratch -directory and have Petal open it: +To sanity-check an archive before committing to it, have Petal open it in a +scratch directory — a clean exit means it reads end to end: ```bash -mkdir -p /tmp/restore-check -gunzip -c data/backups/petal-.db.gz > /tmp/restore-check/petal.db +mkdir -p /tmp/restore-check && cp /tmp/petal.db /tmp/restore-check/petal.db docker run --rm -v /tmp/restore-check:/data --user "$(id -u):$(id -g)" \ --entrypoint sh petal:local -c '/app/petal -backup /data/verify.db' ``` -A clean exit means the file opens and reads end to end. +### ⚠️ millenia — the canonical instance — has no automated backup + +**This is the real gap.** Her actual writing lives on millenia, and as of +2026-07-27 nothing backs it up on a schedule: `~/petal/backups` holds only +ad-hoc pre-deploy snapshots, the newest a month old. The empty VPS staging +database is currently better protected than the live one. + +`deploy/backup-petal.sh` is written for exactly this and is not yet installed +there — it snapshots via `petal -backup` (`VACUUM INTO`, safe against the live +app), compresses, pushes off-box with a post-transfer size check, and prunes +both ends. What it does **not** yet do is encrypt at rest, which is being +decided separately. + +A manual snapshot any time, no tooling required: + +```bash +cd ~/petal && ./petal -backup ~/petal/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).db +``` + +### ⚠️ millenia's Petal is unsupervised + +It runs as a bare `./petal` with PPID 1 — no systemd unit, no screen session. A +crash or reboot leaves it down until someone notices. `deploy/piper.service` is +the pattern to copy if you want it supervised. --- @@ -226,9 +253,38 @@ plus an env pair, no code change. **Piper version note:** piper-tts moved synthesis from `POST /` to `POST /synthesize` in 1.6.0, with an identical request body. `TTS_PATH` selects -which — it defaults to `/` for millenia's older server, and the VPS compose sets -`/synthesize` for the 1.6.0 sidecars. If read-aloud starts returning 502 after a -Piper upgrade on millenia, that flag is the fix. +which — it defaults to `/`, and both the VPS compose and millenia's `start.sh` +now set `/synthesize`. If read-aloud starts returning 502 after a Piper upgrade, +that flag is the fix. + +**The venv is fragile across Python upgrades.** On 2026-07-27 millenia's Piper +was found dead with **26,800+ failed restarts**, silently since the Jul 26 +reboot — read-aloud had been falling back to browser Web Speech the whole time. +Root cause: an OS upgrade moved `/usr/bin/python3` from 3.13 to 3.14, and +`venv/bin/python3` is a *symlink to the system interpreter*, so the venv's +`lib/python3.13/site-packages` became invisible — `sys.path` contained no +site-packages at all. The failure surfaced as the misleading +`No module named piper.http_server` even though `http_server.py` was sitting +right there on disk. + +Fix (what was done — recreating the venv, not repairing it): + +```bash +systemctl --user stop piper.service piper-zh.service +mv ~/piper/venv ~/piper/venv.broken-py313 +python3 -m venv ~/piper/venv +~/piper/venv/bin/pip install "piper-tts[http]" +~/piper/venv/bin/python -c 'import piper.http_server' # must not raise +systemctl --user start piper.service piper-zh.service +``` + +That reinstall lands 1.6.0, so it must be paired with `TTS_PATH=/synthesize` in +`start.sh` and a binary new enough to read that variable. Voices in +`~/piper/voices` survive and do not need re-downloading. + +Worth knowing: `Restart=on-failure` will retry forever without ever alerting. +Neither service reports its health anywhere, which is why this went unnoticed +for a day. A `/api/tts` probe in uptime-kuma would have caught it. Verify end to end (through Petal, including the ffmpeg transcode): diff --git a/deploy/vllm-headscale-proxy.service b/deploy/vllm-headscale-proxy.service new file mode 100644 index 0000000..9ab0f4c --- /dev/null +++ b/deploy/vllm-headscale-proxy.service @@ -0,0 +1,40 @@ +[Unit] +Description=Expose millenia's vLLM chat server on the headscale interface only +# Why a forwarder instead of just rebinding vLLM: vllm-chat.service is shared. +# Petal, Gogobee and Open WebUI all talk to 127.0.0.1:8000, and Open WebUI keeps +# its endpoint in its own database rather than in env, so moving vLLM's bind +# address would mean editing three consumers and reloading a 35B AWQ model +# (minutes of downtime for all of them). This adds a second door instead: local +# callers keep loopback untouched, and only the headscale address gains a +# listener. Nothing about vllm-chat changes. +# +# Deliberately NOT 0.0.0.0 — this reaches a public VPS over the VPN, and the +# LAN has no business seeing an unauthenticated inference endpoint. +After=network-online.target tailscaled.service vllm-chat.service +Wants=network-online.target +BindsTo=vllm-chat.service + +[Service] +Type=simple +# fork: one child per connection, so a single client can't block the others. +# reuseaddr: survive a restart while sockets are still in TIME_WAIT. +# The bind address is millenia's headscale IP; if tailscaled hasn't brought the +# interface up yet the bind fails and Restart retries until it has. +ExecStart=/usr/bin/socat -d TCP-LISTEN:8000,bind=100.64.0.2,fork,reuseaddr TCP:127.0.0.1:8000 +Restart=always +RestartSec=5 +# Long generations hold a connection open; don't let systemd reap a healthy one. +TimeoutStopSec=10 + +# The process only shuttles bytes between two sockets — give it nothing else. +NoNewPrivileges=true +PrivateTmp=true +ProtectSystem=strict +ProtectHome=true +ProtectKernelTunables=true +ProtectControlGroups=true +RestrictAddressFamilies=AF_INET AF_INET6 +DynamicUser=true + +[Install] +WantedBy=multi-user.target