petal/deploy/README.md

# Deploying Petal

Two deployments exist right now:

| | host | shape | status |
|---|---|---|---|
| **parodia** | `petal.parodia.dev` / `100.64.0.1` | docker compose behind the host's Traefik | **canonical** since 2026-07-27 — she signs in here |
| **millenia** | `192.168.1.212` / `100.64.0.2` | `petal.service`, bare binary on `:8088`, Piper as user systemd units | frozen fallback: a copy of her writing as it stood at the move, still owned by the pre-auth `local` user |

Her writing moved to the VPS when sign-in landed (Phase 16/17): it is the
instance that actually authenticates, its data directory is LUKS-encrypted, and
it is reachable from anywhere. millenia was left running and untouched as a
fallback — but the two diverge the moment anything is written on either, so it
should be retired rather than kept in step. Everything below is the VPS side;
the millenia Piper notes are kept in the appendix because that instance still
runs them.

---

## 1. The VPS stack

`docker-compose.yml` at the repo root brings up three containers:

- **petal** — the single Go binary with the frontend embedded. Publishes no host
  port; Traefik is the only way in.
- **piper-en** / **piper-zh** — read-aloud. Each Piper HTTP server loads exactly
  one voice, so English and Chinese are separate containers off one image, with
  the models cached in a shared volume. They sit on an internal network with no
  published ports, so only Petal can reach them. Adding pt-PT in Phase 21 is a
  fourth service, not a new image.

They run as containers rather than the host systemd units millenia uses because
Piper was never actually installed on the VPS, and the `reala` account has no
lingering session to keep user units alive across logout.

### Prerequisites on the host

- Docker with the compose plugin, and the existing external `traefik` network
- A DNS A record for the hostname pointing at the VPS (`petal.parodia.dev` is
  already in place)

### First deploy

```bash
ssh reala@100.64.0.1
git clone https://gitea.parodia.dev/drwily/petal.git ~/petal
cd ~/petal
cp deploy/petal.env.example .env
```

Then edit `.env`:

- `PETAL_UID` / `PETAL_GID` — `id -u` / `id -g` for this account. `./data` is a
  bind mount, so the image's own `petal` user has no claim on it; a mismatch
  shows up as `unable to open database file (14)` and a restart loop.
- `AUTHENTIK_URL` / `AUTHENTIK_CLIENT_ID` / `AUTHENTIK_CLIENT_SECRET` /
  `PETAL_ALLOWED_SUBS` — sign-in, see §4. Without them Petal runs as the single
  `local` user and must not be exposed.
- `LLM_MODEL` / `LLM_CHAT_MODEL` — see §3.

```bash
mkdir -p data/backups
docker compose up -d --build
docker compose ps        # all three healthy
```

### Updating

```bash
cd ~/petal && git pull && docker compose up -d --build
```

The frontend is embedded in the binary, so a rebuild is the whole deploy. The
client polls `/api/version` (a hash of the built `index.html`) and offers a
refresh when it changes.

---

## 2. What Traefik does

Labels follow the convention the other services on this box use: the external
`traefik` network, the `web-secure` entrypoint, the `default` cert resolver and
`compression@file`. Petal adds its own response-header middleware
(`frame-ancestors 'self'`, HSTS, nosniff, `Referrer-Policy: same-origin`).

There is no auth middleware at the edge: Petal does its own (§4). `/api/health`
and `/api/version` sit outside Petal's own auth for the same reason they always
did — a monitoring probe must not need a session, and neither carries user
data.

---

## 3. The LLM link over headscale

The vLLM backend stays on millenia and is reached over headscale
(`100.64.0.2`). **This is the only cross-VPN dependency**, and by the
LLM-minimalism principle it never gates essential functionality — spell check,
gloss, vocabulary garden, search, export and read-aloud all keep working with
the link down, and the status bar shows the warm
`🌙 小助手在休息 · Petal's helper is resting · 文字已保存`.

`LLM_TIMEOUT` is raised from the local-network default of 30s to **90s**: the
voice and collocation passes send a whole document, the timeout is a hard
deadline on the completion call, and a WAN+VPN round trip eats the margin.

### How the link is exposed — a forwarder, not a rebind

vLLM stays bound to `127.0.0.1:8000`. `vllm-headscale-proxy.service` (a socat
unit, in this directory) adds a second listener on `100.64.0.2:8000` that
forwards to it.

The plan originally said to rebind vLLM itself. That turned out to be the
expensive option: `vllm-chat.service` is **shared** — Petal, Gogobee and Open
WebUI all point at `127.0.0.1:8000`, and Open WebUI stores its endpoint in its
own database rather than in env — so moving the bind address would mean editing
three consumers and reloading a 35B AWQ model, minutes of downtime for all of
them. The forwarder adds a door instead of moving one: local callers are
untouched, and the only new exposure is on the VPN interface.

It binds `100.64.0.2` specifically, **never** `0.0.0.0`: the far end of this
link is a public host, and the LAN has no business seeing an unauthenticated
inference endpoint.

```bash
sudo install -m 0644 deploy/vllm-headscale-proxy.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now vllm-headscale-proxy
ss -lntp | grep 8000          # expect BOTH 127.0.0.1:8000 and 100.64.0.2:8000
```

The model id (`qwen3.6-35b`) goes into `LLM_MODEL` / `LLM_CHAT_MODEL` on the
VPS. Verified end to end: a grammar checkpoint from `petal.parodia.dev` returns
real suggestions in ~3s over the VPN.

---

## 4. Sign-in (Authentik OIDC)

Petal is an OIDC client in its own right: it runs the login itself rather than
trusting a header from the proxy. Nothing about the container has to be
unreachable for that to be safe.

Login turns on only when `AUTHENTIK_URL`, `AUTHENTIK_CLIENT_ID` and
`AUTHENTIK_CLIENT_SECRET` are all set. With any of them missing Petal falls back
to the single hardcoded `local` user — which is what local development wants,
and what every deployment did before this landed. A host serving the public
must have them set.

### Register Petal in Authentik

In the Authentik admin UI (**Applications → Providers → Create → OAuth2/OpenID
Provider**):

| Field | Value |
| --- | --- |
| Client type | Confidential |
| Redirect URI | `https://petal.parodia.dev/auth/callback` (strict) |
| Scopes | `openid`, `profile`, `email` |
| Signing key | any (Petal fetches the JWKS from discovery) |

Then create an **Application** bound to that provider, and copy the client id,
the client secret, and the provider's **OpenID Configuration Issuer** (it looks
like `https://auth.parodia.dev/application/o/petal/` — the issuer, not the
`.well-known` URL; Petal appends that itself).

Put them in `.env`:

```
AUTHENTIK_URL=https://auth.parodia.dev/application/o/petal/
AUTHENTIK_CLIENT_ID=…
AUTHENTIK_CLIENT_SECRET=…
PETAL_ALLOWED_SUBS=her@example.com,me@example.com
```

`PETAL_ALLOWED_SUBS` is the guest list: comma-separated OIDC subject ids and/or
email addresses. Authentik fronts several applications on this host, and being a
valid user there does not mean being a user here. Leaving it empty lets in
everyone Authentik authenticates. Emails are accepted alongside subject ids
precisely so the list can be written *before* anyone has logged in — a subject
is an opaque uuid that doesn't exist until first sign-in.

A valid login that isn't on the list gets a warm bilingual "this Petal isn't
yours to write in" page, and no account is provisioned.

### Checking it

```bash
curl -si https://petal.parodia.dev/api/docs | head -1     # 401 without a session
curl -si https://petal.parodia.dev/auth/login | grep -i location   # → Authentik
docker compose logs petal | grep '^.*auth:'               # issuer + redirect at boot
```

The startup log prints the redirect URI it will use; if Authentik rejects the
login with a redirect-uri mismatch, compare that line against what's registered.

Two things bit this deployment, both worth checking first if a login dies early:

- **The issuer's trailing slash is significant.** Authentik's is
  `…/application/o/petal/`, OIDC requires the discovered issuer to match the
  configured one byte-for-byte, and normalising the slash away makes discovery
  fail with `did not match the issuer URL returned by provider`.
- **A provider created through the API or `ak shell` has an empty
  `grant_types`**, which authentik reads as "no grant type is permitted here"
  and answers with `invalid_request` / *The request is otherwise malformed*
  before the login page ever appears. The admin UI fills the list in for you;
  scripted creation must set it (`authorization_code`, `refresh_token`).

Discovery is lazy and retried, so an Authentik outage blocks *new* logins but
leaves existing sessions working — those only need Petal's own database.

### Sessions

Opaque token in a `petal_session` cookie (`HttpOnly`, `SameSite=Lax`, `Secure`
on https); the `sessions` table stores only its SHA-256, so a database copy
yields nothing usable. Thirty-day sliding expiry — every request pushes it out,
throttled to one write an hour. `/auth/logout` deletes the row, not just the
cookie. Expired rows are pruned at startup.

To sign someone out everywhere immediately:

```bash
docker compose exec petal sh -c \
  "sqlite3 /data/petal.db \"DELETE FROM sessions WHERE user_id = '<sub>'\""
```

### The edge gate is gone

Until Phase 16 there was a Traefik basic-auth middleware in front of everything,
because Petal authenticated nobody and a public hostname was a public API. It
was removed when OIDC went live on 2026-07-27, together with the separate
unauthenticated `/api/health` router that existed only to escape it: every `/api`
route now answers 401 without a session, and the only thing an anonymous visitor
gets is the app shell and a redirect to sign in.

If you ever run this stack *without* `AUTHENTIK_*` configured — Petal then falls
back to the single `local` user — put the gate back before pointing DNS at it:

```yaml
traefik.http.routers.petal.middlewares: compression@file,petal-headers,petal-auth
traefik.http.middlewares.petal-auth.basicauth.users: ${PETAL_BASIC_AUTH:?}
```

with `htpasswd -nbB petal 'your-password'` in `.env` as `PETAL_BASIC_AUTH`.

---

## 4a. Moving an account (`scripts/migrate_local_user.py`)

Petal ran as one hardcoded user (`users.id = 'local'`) before sign-in existed.
Moving that writing onto a real account is a deliberate, one-off operation:

```bash
docker compose stop petal
python3 scripts/migrate_local_user.py data/petal.db --to <oidc-sub>     # dry run
python3 scripts/migrate_local_user.py data/petal.db --to <oidc-sub> \
    --email her@example.com --name "Her Name" --apply
docker compose up -d petal
```

The subject id is **knowable before she has ever logged in**. With authentik's
default `hashed_user_id` sub mode it is the user's `uid`:

```bash
docker exec authentik-server-1 ak shell -c \
  "from authentik.core.models import User; print(User.objects.get(username='claire').uid)"
```

so the data can move first and she signs in to find it already there.

**Start Petal once on the incoming database before migrating.** A database
carried over from another instance may be a schema behind, and the app applies
migrations at startup; the script moves rows and does not touch the schema.

The script is dry-run by default, takes its own `VACUUM INTO` backup, runs as
one transaction with foreign keys off, and verifies the row counts before it
commits. It refuses to run while anything else has the database open, and
refuses to merge into an account that already owns writing.

**`local` comes back, and that's expected.** `db.Open` seeds that row on every
startup, so it reappears the moment Petal restarts after a migration. It owns
nothing — the writing is on the real account — and it is only ever resolved to
by `StaticResolver`, which a deployment with `AUTHENTIK_*` set never uses. Check
`SELECT COUNT(*) FROM documents WHERE user_id = 'local'` if you want to be sure
a migration took; the presence of the row itself says nothing.

---

## 4b. The dictionary (`dict.db`)

Word lookups for the French, European Portuguese and Spanish pairs come from
[DreamDict](https://github.com/prosolis/dreamdict)'s built database, which Petal
opens **read-only** beside `petal.db`. Petal imports DreamDict's `dictionary`
package directly — there is no DreamDict service to run and nothing to reach
over the VPN, which matters because a hover gloss must answer in milliseconds.

`dict.db` is **optional**. With no file at `DICT_PATH` Petal logs

```
dictionary: no dict.db at /data/dict.db — English/Chinese only
```

and serves lookups from the datasets compiled into the binary. The Chinese pair
is unaffected either way — it stays on ECDICT (see below) — and a non-Chinese
writer still gets English definitions, synonyms and pronunciation, losing only
the translation. **A dictionary that failed to deploy costs the gloss, not the
popover.** A file that is present but was never imported is a different matter
and is logged as an error.

### Installing it

The database is built by DreamDict's own import CLI from ~6 GB of source data;
it is not built on the VPS. Copy the built file into the data volume:

```bash
# on the machine holding a built dict.db (millenia: ~/dreamdict/data/dict.db)
scp ~/dreamdict/data/dict.db reala@100.64.0.1:/home/reala/petal/data/dict.db
# on parodia
chown "$(id -u):$(id -g)" /home/reala/petal/data/dict.db
docker compose restart petal   # the handle is opened once, at startup
```

Expect ~450 MB. It sits inside the LUKS volume with everything else (§6). The
backups name `petal.db` explicitly rather than sweeping the data directory
(§5), so `dict.db` stays out of them — which is the right outcome and worth
keeping: it is rebuildable from public data and would otherwise dominate every
nightly snapshot. Petal never writes to it.

### Why Chinese doesn't use it

The zh pair stays on the embedded ECDICT gloss, deliberately. Measured on the
deployed database, DreamDict reaches a Chinese gloss for 53% of the 2,000
commonest English words; ECDICT covers essentially all of them and is in daily
use by a real writer. `lexicon.Set.For` is where that decision lives — one
`switch`, changed the day a comparison on her actual lookups says otherwise.

For pt-PT and French the same measurement reads 62%, which is why they use
DreamDict: there is no alternative source for them at all.

### Rebuilding it

Rebuilt 2026-07-27 to add Spanish (the previous file predated DreamDict's
Spanish support). The recipe, since it will be needed again:

```bash
# on millenia, from a clean checkout of dreamdict main
./scripts/download-dict-data.sh ~/dreamdict/data     # idempotent; skips what's there
go run ./cmd/dictimport --data ~/dreamdict/data --db ./dict.db --clean
```

~6 minutes on 32 cores; the data directory is ~7 GB and mostly already
downloaded. **Build to a new path, never over a file in use** — then verify by
hash on both ends before swapping.

Two things worth knowing before trusting a rebuild:

- The SUBTLEX-US download fails (the source moved behind a manual export). It
  does not matter: the loader falls back to `SUBTLEX-US.txt`, which is present,
  and English "frequency" is mostly SCOWL's commonness bucket anyway —
  1000/800/600/…/50, refined by SUBTLEX for only ~1,600 words. That is why the
  word-difficulty chip reads `difficulty`, not `frequency`.
- Check the *other* languages' counts are unchanged before shipping. The 2026-07
  rebuild came out byte-identical for en/fr/pt-PT/zh, which is what says it
  added a language rather than quietly shifting the rest.

Gloss coverage of the 2,000 commonest English words, after the rebuild:
**es 68.6%**, fr 63.1%, pt-PT 62.1%, zh 53.2%. The startup line reports actual
per-language row counts, so a database missing a language says so.

---

## 5. Backups

### On the VPS — folded into `parodia-backup`

Petal rides the host's existing offsite job (`/usr/local/bin/parodia-backup`,
`parodia-backup.timer`, nightly ~03:40): age-encrypted to S3, 14-day retention,
dead-man snitch. The host holds only the age *public* recipient, so it writes
backups it cannot itself decrypt.

```
push petal.db.age       sqlite_file_dump /home/reala/petal/data/petal.db
```

`/home/reala/petal/.env` is in the same job's secrets tarball — it carries the
interim basic-auth hash and, from Phase 16, the OIDC client secret.

**Why `sqlite_file_dump` and not the script's existing `sqlite_dump`:** that
helper uses Python's `iterdump`, which **does not reproduce an FTS5 virtual
table**. It emits `documents_fts` as a raw `sqlite_master` row plus its shadow
tables, and replaying the result dies with `no such table: documents_fts` —
verified by round-tripping a real dump on 2026-07-27. Petal's cross-document
search would have been silently missing after any restore. `sqlite_file_dump`
runs `VACUUM INTO` instead: a genuine database file, virtual tables intact, WAL
folded in, no write lock. Restore is a copy rather than a replay.

> If `apply.db` ever gains a virtual table, it needs the same treatment.

### Restore (VPS)

```bash
age -d -i <offline-identity> petal.db.age > /tmp/petal.db   # from S3
cd ~/petal
docker compose stop petal                       # stop writers first
mv data/petal.db data/petal.db.before-restore   # keep the current state
rm -f data/petal.db-wal data/petal.db-shm       # a stale WAL against a new file
cp /tmp/petal.db data/petal.db
docker compose start petal
docker compose logs petal --tail 5              # expect "database ready"
```

To sanity-check an archive before committing to it, have Petal open it in a
scratch directory — a clean exit means it reads end to end:

```bash
mkdir -p /tmp/restore-check && cp /tmp/petal.db /tmp/restore-check/petal.db
docker run --rm -v /tmp/restore-check:/data --user "$(id -u):$(id -g)" \
  --entrypoint sh petal:local -c '/app/petal -backup /data/verify.db'
```

### On millenia — `petal-backup.timer`

Until 2026-07-27 her actual writing had **no scheduled backup at all**; the
newest snapshot was a month old. It now runs nightly at 03:20
(`Persistent=true`, because the box isn't on 24/7 and a missed window would
otherwise be skipped silently):

```bash
sudo install -m 0644 deploy/petal-backup.service deploy/petal-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now petal-backup.timer
sudo systemctl start petal-backup.service     # prove it before trusting it
```

`deploy/backup-petal.sh` snapshots via `petal -backup`, gzips, **age-encrypts
with the parodia public recipient**, pushes to the VPS over headscale with a
post-transfer size check, and prunes both ends. The private identity is offline,
so neither millenia nor the VPS can decrypt what it is holding — verified.

A manual snapshot any time, no tooling required:

```bash
cd ~/petal && ./petal -backup ~/petal/backups/manual-$(date -u +%Y%m%dT%H%M%SZ).db
```

Restore is a copy — stop Petal, drop the file in as `data/petal.db`, remove any
stale `-wal`/`-shm`, start.

---

## 6. Encryption at rest

### VPS — `/home/reala/petal/data` is a LUKS volume

`deploy/setup-encrypted-data.sh` puts the data directory on LUKS2 over a sparse
file at `/var/lib/petal-crypt.img`. That covers `petal.db`, uploaded `images/`,
**and the TTS cache** — which is synthesized audio of her sentences and is easy
to forget.

LUKS-on-a-file rather than gocryptfs because Petal is SQLite in WAL mode: WAL
needs a shared-memory index (`-shm`) mapped consistently across processes, and
FUSE has a long history of subtle mmap/locking differences. A block device with
ext4 behaves exactly like a disk to SQLite, which is the only guarantee worth
having under a database.

**What it protects, honestly.** The key lives at `/etc/petal/dataset.key` on the
same host so the volume auto-unlocks at boot. That is a deliberate availability
tradeoff:

| | |
|---|---|
| protects against | a decommissioned or resold disk; reading the raw block device; casual browsing of a filesystem snapshot that excludes `/etc` |
| does **not** protect against | anyone holding the whole VM image — they get the keyfile with the ciphertext; or anything at all while the host is running and mounted |

Real protection from a provider-side snapshot needs the key off-box (fetched
over the VPN at boot). Considered, not chosen.

Two things this setup got wrong the first time, both caught by rehearsing a
reboot rather than trusting a clean run — worth knowing if you rebuild it:

- **Mounting over a directory hides its contents, it does not remove them.** The
  first pass left the original plaintext `petal.db` and WAL sitting on the
  unencrypted root filesystem, invisible under the mount. The script now shreds
  the originals before mounting and refuses to continue if the mountpoint will
  not come up empty.
- **`systemd-cryptsetup` was not installed**, so `/etc/crypttab` was ignored
  entirely and the volume would never have unlocked at boot. The script now
  refuses to run without the generator present.

Check it any time:

```bash
sudo ./deploy/setup-encrypted-data.sh --status
```

### The mount-liveness guard

The mountpoint directory exists whether or not the volume is mounted, so a boot
where the unlock failed would start Petal against an empty unencrypted
directory and quietly serve a blank database — the failure that looks like data
loss. `data/.volume-ok` lives on the encrypted filesystem and is bind-mounted
with `create_host_path: false`, turning that into a loud container start
failure:

```
Error response from daemon: invalid mount config for type "bind":
bind source path does not exist: /home/reala/petal/data/.volume-ok
```

Verified by unmounting and attempting a start.

**A true reboot has not been tested** — the VPS also runs matrix, lemmy, akkoma,
gitea and authentik, so rebooting it is your call. The boot path was rehearsed
through `local-fs.target`, which pulls the mount, which pulls the unlock.

### millenia is not encrypted at rest

LVM, no LUKS. Her canonical writing sits in plaintext on the home box. Backups
leaving it are age-encrypted; the disk itself is not.

---

## 7. Supervision and monitoring

Petal on millenia ran for months as a bare `./petal` with PPID 1 — no unit, no
screen session — so a crash or reboot left it down until someone noticed. It is
now `petal.service`:

```bash
sudo install -m 0644 deploy/petal.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now petal.service
```

Verified by `kill -9`-ing it and watching systemd bring it back.

**Piper's silent-failure mode is fixed.** Both units now set
`StartLimitIntervalSec=300` / `StartLimitBurst=5`. With `RestartSec=3` and
systemd's default 10-second window, only ~3 restarts ever landed inside it, so
the burst limit was never reached and a dead service looped **26,800+ times over
a day without ever entering `failed`**. A genuinely broken Piper now shows up in
`systemctl --user --failed`.

### Still to do — an external probe

Nothing yet watches millenia from outside. uptime-kuma already runs on the VPS
and can reach millenia over headscale, so the missing piece is two monitors
(they need the uptime-kuma UI, hence not scripted here):

- `http://100.64.0.2:8088/api/health` — Petal itself
- a POST to `http://100.64.0.2:8088/api/tts` — catches a dead Piper, which
  `/api/health` will not, because read-aloud degrades silently to browser
  speech

---

## Appendix — Piper on millenia (user systemd units)

millenia still runs Piper as user services; these are the original notes.

```bash
scp deploy/piper.service deploy/setup-piper.sh 192.168.1.212:/tmp/
ssh 192.168.1.212 'cd /tmp && sudo ./setup-piper.sh'
```

`setup-piper.sh` creates `~/piper/venv`, installs `piper-tts[http]`, downloads
`en_US-amy-medium` into `~/piper/voices`, installs and enables `piper.service`
(loopback `:5005`), and smoke-tests it. Idempotent. Check it with
`systemctl status piper` / `journalctl -u piper -f`.

Chinese runs as a second instance (`piper-zh.service`, `:5006`,
`zh_CN-huayan-medium`):

```bash
scp deploy/piper-zh.service 192.168.1.212:~/.config/systemd/user/
ssh 192.168.1.212 'export XDG_RUNTIME_DIR=/run/user/$(id -u)
  ~/piper/venv/bin/python -m piper.download_voices zh_CN-huayan-medium --data-dir ~/piper/voices
  systemctl --user daemon-reload && systemctl --user enable --now piper-zh.service'
```

Petal's env then carries `TTS_ENDPOINT=http://127.0.0.1:5005`,
`TTS_ENDPOINT_ZH=http://127.0.0.1:5006` and the matching voice ids. The handler
maps language → instance from config, so another language is another instance
plus an env pair, no code change.

**Piper version note:** piper-tts moved synthesis from `POST /` to
`POST /synthesize` in 1.6.0, with an identical request body. `TTS_PATH` selects
which — it defaults to `/`, and both the VPS compose and millenia's `start.sh`
now set `/synthesize`. If read-aloud starts returning 502 after a Piper upgrade,
that flag is the fix.

**The venv is fragile across Python upgrades.** On 2026-07-27 millenia's Piper
was found dead with **26,800+ failed restarts**, silently since the Jul 26
reboot — read-aloud had been falling back to browser Web Speech the whole time.
Root cause: an OS upgrade moved `/usr/bin/python3` from 3.13 to 3.14, and
`venv/bin/python3` is a *symlink to the system interpreter*, so the venv's
`lib/python3.13/site-packages` became invisible — `sys.path` contained no
site-packages at all. The failure surfaced as the misleading
`No module named piper.http_server` even though `http_server.py` was sitting
right there on disk.

Fix (what was done — recreating the venv, not repairing it):

```bash
systemctl --user stop piper.service piper-zh.service
mv ~/piper/venv ~/piper/venv.broken-py313
python3 -m venv ~/piper/venv
~/piper/venv/bin/pip install "piper-tts[http]"
~/piper/venv/bin/python -c 'import piper.http_server'   # must not raise
systemctl --user start piper.service piper-zh.service
```

That reinstall lands 1.6.0, so it must be paired with `TTS_PATH=/synthesize` in
`start.sh` and a binary new enough to read that variable. Voices in
`~/piper/voices` survive and do not need re-downloading.

Worth knowing: `Restart=on-failure` will retry forever without ever alerting.
Neither service reports its health anywhere, which is why this went unnoticed
for a day. A `/api/tts` probe in uptime-kuma would have caught it.

Verify end to end (through Petal, including the ffmpeg transcode):

```bash
curl -sf -X POST localhost:8088/api/tts \
  -H 'Content-Type: application/json' \
  -d '{"text":"hello there","lang":"en-US"}' -o /tmp/petal-tts.mp3 \
  && file /tmp/petal-tts.mp3   # expect: MPEG ADTS, layer III
```

A second identical call is served from the cache; a language with no configured
instance returns 404 so the client falls back to Web Speech.