drwily/petal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reach vLLM over headscale with a forwarder, and fix millenia's Piper

Browse Source 
Rebinding vllm-chat was the expensive option: Petal, Gogobee and Open
WebUI all point at 127.0.0.1:8000, and Open WebUI keeps its endpoint in
its own database rather than in env, so moving the bind address meant
editing three consumers and reloading a 35B AWQ model. The socat unit
adds a second listener on the headscale address instead -- local callers
untouched, no downtime, and the only new exposure is on the VPN. Bound
to 100.64.0.2 specifically, never 0.0.0.0: the far end is a public host.

Verified: a grammar checkpoint from petal.parodia.dev returns real
suggestions in ~3s over the VPN.

Also documents two things found on millenia that were invisible from
outside it:

- Piper had been dead since the Jul 26 reboot, 26,800+ failed restarts,
  with read-aloud silently falling back to browser Web Speech. An OS
  upgrade moved /usr/bin/python3 from 3.13 to 3.14, and venv/bin/python3
  is a symlink to the system interpreter, so lib/python3.13/site-packages
  went invisible -- sys.path had no site-packages at all. Recreating the
  venv lands Piper 1.6.0, which is what TTS_PATH exists for.
- The canonical instance has no automated backup (newest snapshot a
  month old) and runs unsupervised with PPID 1. Both written down; the
  backup one is pending the encryption-at-rest decision.

Backups on the VPS now ride parodia-backup (age-encrypted, offsite, S3),
using VACUUM INTO rather than that script's iterdump helper -- iterdump
does not reproduce an FTS5 virtual table, so a restore would have come
back with cross-document search silently missing.
This commit is contained in:
prosolis
2026-07-27 06:15:00 -07:00
parent d01a0f1f0a
commit 623bd02b9c
2 changed files with 145 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/README.md
+105 -49
View File
@@ -96,19 +96,33 @@ the link down, and the status bar shows the warm
voice and collocation passes send a whole document, the timeout is a hard
deadline on the completion call, and a WAN+VPN round trip eats the margin.
### Outstanding — vLLM is not yet listening on headscale
### How the link is exposed — a forwarder, not a rebind
As of the Phase 15 deploy, `100.64.0.2:8000` refuses connections from the VPS:
vLLM is bound to loopback or the LAN interface, not the headscale one. Petal
degrades correctly (verified — `POST /api/docs/{id}/check` returns the warm
502), but no AI pass will work until this is fixed on millenia:
vLLM stays bound to `127.0.0.1:8000`. `vllm-headscale-proxy.service` (a socat
unit, in this directory) adds a second listener on `100.64.0.2:8000` that
forwards to it.
- start vLLM with `--host 100.64.0.2` — the headscale address specifically,
**not** `0.0.0.0`; the other end of this link is a public host
- confirm with `ss -lntp | grep 8000` that it binds only that interface
- from the VPS: `curl -s http://100.64.0.2:8000/v1/models`
- put the model id it reports into `LLM_MODEL` / `LLM_CHAT_MODEL` in `.env`,
then `docker compose up -d`
The plan originally said to rebind vLLM itself. That turned out to be the
expensive option: `vllm-chat.service` is **shared** — Petal, Gogobee and Open
WebUI all point at `127.0.0.1:8000`, and Open WebUI stores its endpoint in its
own database rather than in env — so moving the bind address would mean editing
three consumers and reloading a 35B AWQ model, minutes of downtime for all of
them. The forwarder adds a door instead of moving one: local callers are
untouched, and the only new exposure is on the VPN interface.
It binds `100.64.0.2` specifically, **never** `0.0.0.0`: the far end of this
link is a public host, and the LAN has no business seeing an unauthenticated
inference endpoint.
```bash
sudo install -m 0644 deploy/vllm-headscale-proxy.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now vllm-headscale-proxy
ss -lntp | grep 8000 # expect BOTH 127.0.0.1:8000 and 100.64.0.2:8000
```
The model id (`qwen3.6-35b`) goes into `LLM_MODEL` / `LLM_CHAT_MODEL` on the
VPS. Verified end to end: a grammar checkpoint from `petal.parodia.dev` returns
real suggestions in ~3s over the VPN.
---
@@ -134,64 +148,77 @@ When Phase 16 lands, delete the `petal-auth` middleware label, the
## 5. Backups
`deploy/backup-petal.sh` runs nightly from cron at 03:15:
### On the VPS — folded into `parodia-backup`
Petal rides the host's existing offsite job (`/usr/local/bin/parodia-backup`,
`parodia-backup.timer`, nightly ~03:40): age-encrypted to S3, 14-day retention,
dead-man snitch. The host holds only the age *public* recipient, so it writes
backups it cannot itself decrypt.
```
15 3 * * * REMOTE_HOST= /bin/bash $HOME/petal/deploy/backup-petal.sh >> $HOME/petal/data/backups/backup.log 2>&1
push petal.db.age sqlite_file_dump /home/reala/petal/data/petal.db
```
The snapshot goes through `petal -backup`, which uses SQLite's `VACUUM INTO`.
That matters: Petal runs in WAL mode, so the newest committed pages may live in
`petal.db-wal` rather than `petal.db`, and copying the three files separately can
capture a torn mid-checkpoint state. `VACUUM INTO` reads one coherent snapshot
including the WAL, takes no write lock (so it is safe against the live app), and
emits a single file with no `-wal`/`-shm` companions. It refuses an existing
destination, so a failed run cannot destroy the last good backup.
`/home/reala/petal/.env` is in the same job's secrets tarball — it carries the
interim basic-auth hash and, from Phase 16, the OIDC client secret.
The script then gzips, pushes off-box, verifies the transferred size, and prunes
both sides (7 days local, 30 days remote).
**Why `sqlite_file_dump` and not the script's existing `sqlite_dump`:** that
helper uses Python's `iterdump`, which **does not reproduce an FTS5 virtual
table**. It emits `documents_fts` as a raw `sqlite_master` row plus its shadow
tables, and replaying the result dies with `no such table: documents_fts` —
verified by round-tripping a real dump on 2026-07-27. Petal's cross-document
search would have been silently missing after any restore. `sqlite_file_dump`
runs `VACUUM INTO` instead: a genuine database file, virtual tables intact, WAL
folded in, no write lock. Restore is a copy rather than a replay.
### Outstanding — the off-VPS push is not yet enabled
> If `apply.db` ever gains a virtual table, it needs the same treatment.
`REMOTE_HOST` is empty in the cron line, so backups are currently **local to the
VPS only** — which is not a backup in the sense that matters. parodia's ssh key
is not authorized on millenia. To enable it, on millenia:
```bash
echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICGL6fS7IKCs8xFxUoH/bI/0kq4AzW05bcfV2XHoEXYM ditto-deploy@parodia-box' >> ~/.ssh/authorized_keys
```
then on the VPS, prove it works by hand before touching cron:
```bash
cd ~/petal && REMOTE_USER=<millenia-user> ./deploy/backup-petal.sh
```
and set `REMOTE_USER=<millenia-user> REMOTE_HOST=100.64.0.2` in the cron line.
### Restore
### Restore (VPS)
```bash
age -d -i <offline-identity> petal.db.age > /tmp/petal.db # from S3
cd ~/petal
docker compose stop petal # stop writers first
mv data/petal.db data/petal.db.before-restore # keep the current state
rm -f data/petal.db-wal data/petal.db-shm # a stale WAL against a new file
gunzip -c data/backups/petal-<stamp>.db.gz > data/petal.db
cp /tmp/petal.db data/petal.db
docker compose start petal
docker compose logs petal --tail 5 # expect "database ready"
```
To sanity-check an archive before committing to it, restore it into a scratch
directory and have Petal open it:
To sanity-check an archive before committing to it, have Petal open it in a
scratch directory — a clean exit means it reads end to end:
```bash
mkdir -p /tmp/restore-check
gunzip -c data/backups/petal-<stamp>.db.gz > /tmp/restore-check/petal.db
mkdir -p /tmp/restore-check && cp /tmp/petal.db /tmp/restore-check/petal.db
docker run --rm -v /tmp/restore-check:/data --user "$(id -u):$(id -g)" \
--entrypoint sh petal:local -c '/app/petal -backup /data/verify.db'
```
A clean exit means the file opens and reads end to end.
### ⚠️ millenia — the canonical instance — has no automated backup
**This is the real gap.** Her actual writing lives on millenia, and as of
2026-07-27 nothing backs it up on a schedule: `~/petal/backups` holds only
ad-hoc pre-deploy snapshots, the newest a month old. The empty VPS staging
database is currently better protected than the live one.
`deploy/backup-petal.sh` is written for exactly this and is not yet installed
there — it snapshots via `petal -backup` (`VACUUM INTO`, safe against the live
app), compresses, pushes off-box with a post-transfer size check, and prunes
both ends. What it does **not** yet do is encrypt at rest, which is being
decided separately.
A manual snapshot any time, no tooling required:
```bash
cd ~/petal && ./petal -backup ~/petal/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).db
```
### ⚠️ millenia's Petal is unsupervised
It runs as a bare `./petal` with PPID 1 — no systemd unit, no screen session. A
crash or reboot leaves it down until someone notices. `deploy/piper.service` is
the pattern to copy if you want it supervised.
---
@@ -226,9 +253,38 @@ plus an env pair, no code change.
**Piper version note:** piper-tts moved synthesis from `POST /` to
`POST /synthesize` in 1.6.0, with an identical request body. `TTS_PATH` selects
which — it defaults to `/` for millenia's older server, and the VPS compose sets
`/synthesize` for the 1.6.0 sidecars. If read-aloud starts returning 502 after a
Piper upgrade on millenia, that flag is the fix.
which — it defaults to `/`, and both the VPS compose and millenia's `start.sh`
now set `/synthesize`. If read-aloud starts returning 502 after a Piper upgrade,
that flag is the fix.
**The venv is fragile across Python upgrades.** On 2026-07-27 millenia's Piper
was found dead with **26,800+ failed restarts**, silently since the Jul 26
reboot — read-aloud had been falling back to browser Web Speech the whole time.
Root cause: an OS upgrade moved `/usr/bin/python3` from 3.13 to 3.14, and
`venv/bin/python3` is a *symlink to the system interpreter*, so the venv's
`lib/python3.13/site-packages` became invisible — `sys.path` contained no
site-packages at all. The failure surfaced as the misleading
`No module named piper.http_server` even though `http_server.py` was sitting
right there on disk.
Fix (what was done — recreating the venv, not repairing it):
```bash
systemctl --user stop piper.service piper-zh.service
mv ~/piper/venv ~/piper/venv.broken-py313
python3 -m venv ~/piper/venv
~/piper/venv/bin/pip install "piper-tts[http]"
~/piper/venv/bin/python -c 'import piper.http_server' # must not raise
systemctl --user start piper.service piper-zh.service
```
That reinstall lands 1.6.0, so it must be paired with `TTS_PATH=/synthesize` in
`start.sh` and a binary new enough to read that variable. Voices in
`~/piper/voices` survive and do not need re-downloading.
Worth knowing: `Restart=on-failure` will retry forever without ever alerting.
Neither service reports its health anywhere, which is why this went unnoticed
for a day. A `/api/tts` probe in uptime-kuma would have caught it.
Verify end to end (through Petal, including the ffmpeg transcode):
deploy/vllm-headscale-proxy.service
+40
View File
@@ -0,0 +1,40 @@
[Unit]
Description=Expose millenia's vLLM chat server on the headscale interface only
# Why a forwarder instead of just rebinding vLLM: vllm-chat.service is shared.
# Petal, Gogobee and Open WebUI all talk to 127.0.0.1:8000, and Open WebUI keeps
# its endpoint in its own database rather than in env, so moving vLLM's bind
# address would mean editing three consumers and reloading a 35B AWQ model
# (minutes of downtime for all of them). This adds a second door instead: local
# callers keep loopback untouched, and only the headscale address gains a
# listener. Nothing about vllm-chat changes.
#
# Deliberately NOT 0.0.0.0 — this reaches a public VPS over the VPN, and the
# LAN has no business seeing an unauthenticated inference endpoint.
After=network-online.target tailscaled.service vllm-chat.service
Wants=network-online.target
BindsTo=vllm-chat.service
[Service]
Type=simple
# fork: one child per connection, so a single client can't block the others.
# reuseaddr: survive a restart while sockets are still in TIME_WAIT.
# The bind address is millenia's headscale IP; if tailscaled hasn't brought the
# interface up yet the bind fails and Restart retries until it has.
ExecStart=/usr/bin/socat -d TCP-LISTEN:8000,bind=100.64.0.2,fork,reuseaddr TCP:127.0.0.1:8000
Restart=always
RestartSec=5
# Long generations hold a connection open; don't let systemd reap a healthy one.
TimeoutStopSec=10
# The process only shuttles bytes between two sockets — give it nothing else.
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6
DynamicUser=true
[Install]
WantedBy=multi-user.target