Phase 20: the dictionary stops being English and Chinese only

Word lookups now come from DreamDict's dict.db for every pair but Chinese —
opened read-only beside petal.db, no service, nothing over the VPN, because a
hover gloss has to answer in milliseconds.

`Provider` is the two questions the popover and the tooltip already asked, so
the embedded *Lexicon satisfies it with no changes at all; Set.For(lang) is the
single place the choice between them is made. The prerequisite in the dreamdict
repo turned out to be two things, not one: the module path was unfetchable
*and* the query layer sat in internal/, which no other module may import
whatever the module is called. Both fixed upstream.

The plan's central assumption did not survive the data. It mapped
Gloss ← Translate(word, "en", L1) one-to-one; against the real 452 MB database
that table answers for 17% of the 2,000 commonest English words into pt-PT.
Wiktionary's translation sections are thin in that direction — "ephemeral",
"think" and "quickly" have no en→pt-PT row at all. Shared WordNet synsets
answer for 61%, so DreamDict gained Equivalents() and Petal glosses through it.
Ordering those was wrong in an instructive way too: sorting by frequency
glosses "think" as lembrar, "remember", because lembrar is the commoner
Portuguese word even though pensar shares six of think's synsets to lembrar's
one. Counting sense agreement first asks the right question.

The same measurement is why zh stays on ECDICT: DreamDict reaches a Chinese
gloss for 53% of those words, ECDICT for nearly all of them. The plan said
converge only if quality holds. It didn't, so nothing converged.

Two decisions about failure worth keeping. A missing dict.db is not an error —
a laptop checkout has never had one — but a present-and-never-imported one is,
because that is a half-finished deploy. And a pt-PT writer with no dictionary
falls back to the embedded datasets with the gloss suppressed, keeping
definitions, synonyms and phonetics rather than blanking the popover: an empty
field reads as "not found", the wrong language reads as broken.

The new fields surface as an etymology line and a three-band chip. Three, not
five: the difficulty score separates "everyday" from "you'll have to explain
this" but cannot rank obfuscate against serendipity, and a finer scale would be
a confident-looking lie. An unscored word gets no chip.

Writing the tests found two bugs first — trimEtymology sliced by byte, which
would have emitted invalid UTF-8 for exactly the Greek and Latin etymologies
the feature exists for, and its ellipsis path overran its own cap.

go build/vet/test, tsc, vite, vitest 96/96 clean; live smoke against the real
dict.db with one instance flipped from zh to pt-PT mid-run.

Not deployed: go.mod still replaces github.com/prosolis/dreamdict with
../dreamdict, so the Docker build needs the two upstream commits pushed and the
replace dropped. The deployed dict.db also predates DreamDict's Spanish data.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
This commit is contained in:
prosolis
2026-07-27 09:38:50 -07:00
parent 336cae93e0
commit 97e9c269ec
21 changed files with 1262 additions and 60 deletions
+58
View File
@@ -284,6 +284,64 @@ a migration took; the presence of the row itself says nothing.
---
## 4b. The dictionary (`dict.db`)
Word lookups for the French, European Portuguese and Spanish pairs come from
[DreamDict](https://github.com/prosolis/dreamdict)'s built database, which Petal
opens **read-only** beside `petal.db`. Petal imports DreamDict's `dictionary`
package directly — there is no DreamDict service to run and nothing to reach
over the VPN, which matters because a hover gloss must answer in milliseconds.
`dict.db` is **optional**. With no file at `DICT_PATH` Petal logs
```
dictionary: no dict.db at /data/dict.db — English/Chinese only
```
and serves lookups from the datasets compiled into the binary. The Chinese pair
is unaffected either way — it stays on ECDICT (see below) — and a non-Chinese
writer still gets English definitions, synonyms and pronunciation, losing only
the translation. **A dictionary that failed to deploy costs the gloss, not the
popover.** A file that is present but was never imported is a different matter
and is logged as an error.
### Installing it
The database is built by DreamDict's own import CLI from ~6 GB of source data;
it is not built on the VPS. Copy the built file into the data volume:
```bash
# on the machine holding a built dict.db (millenia: ~/dreamdict/data/dict.db)
scp ~/dreamdict/data/dict.db reala@100.64.0.1:/home/reala/petal/data/dict.db
# on parodia
chown "$(id -u):$(id -g)" /home/reala/petal/data/dict.db
docker compose restart petal # the handle is opened once, at startup
```
Expect ~450 MB. It sits inside the LUKS volume with everything else (§6). The
backups name `petal.db` explicitly rather than sweeping the data directory
(§5), so `dict.db` stays out of them — which is the right outcome and worth
keeping: it is rebuildable from public data and would otherwise dominate every
nightly snapshot. Petal never writes to it.
### Why Chinese doesn't use it
The zh pair stays on the embedded ECDICT gloss, deliberately. Measured on the
deployed database, DreamDict reaches a Chinese gloss for 53% of the 2,000
commonest English words; ECDICT covers essentially all of them and is in daily
use by a real writer. `lexicon.Set.For` is where that decision lives — one
`switch`, changed the day a comparison on her actual lookups says otherwise.
For pt-PT and French the same measurement reads 62%, which is why they use
DreamDict: there is no alternative source for them at all.
**The deployed `dict.db` is from 2026-04-04 and has no Spanish data**, because
DreamDict grew Spanish support after it was built. A Spanish-pair writer gets
empty glosses (and English definitions) until it is rebuilt — do that before
the Spanish pair ships.
---
## 5. Backups
### On the VPS — folded into `parodia-backup`