Phase 20: the dictionary stops being English and Chinese only
Word lookups now come from DreamDict's dict.db for every pair but Chinese — opened read-only beside petal.db, no service, nothing over the VPN, because a hover gloss has to answer in milliseconds. `Provider` is the two questions the popover and the tooltip already asked, so the embedded *Lexicon satisfies it with no changes at all; Set.For(lang) is the single place the choice between them is made. The prerequisite in the dreamdict repo turned out to be two things, not one: the module path was unfetchable *and* the query layer sat in internal/, which no other module may import whatever the module is called. Both fixed upstream. The plan's central assumption did not survive the data. It mapped Gloss ← Translate(word, "en", L1) one-to-one; against the real 452 MB database that table answers for 17% of the 2,000 commonest English words into pt-PT. Wiktionary's translation sections are thin in that direction — "ephemeral", "think" and "quickly" have no en→pt-PT row at all. Shared WordNet synsets answer for 61%, so DreamDict gained Equivalents() and Petal glosses through it. Ordering those was wrong in an instructive way too: sorting by frequency glosses "think" as lembrar, "remember", because lembrar is the commoner Portuguese word even though pensar shares six of think's synsets to lembrar's one. Counting sense agreement first asks the right question. The same measurement is why zh stays on ECDICT: DreamDict reaches a Chinese gloss for 53% of those words, ECDICT for nearly all of them. The plan said converge only if quality holds. It didn't, so nothing converged. Two decisions about failure worth keeping. A missing dict.db is not an error — a laptop checkout has never had one — but a present-and-never-imported one is, because that is a half-finished deploy. And a pt-PT writer with no dictionary falls back to the embedded datasets with the gloss suppressed, keeping definitions, synonyms and phonetics rather than blanking the popover: an empty field reads as "not found", the wrong language reads as broken. The new fields surface as an etymology line and a three-band chip. Three, not five: the difficulty score separates "everyday" from "you'll have to explain this" but cannot rank obfuscate against serendipity, and a finer scale would be a confident-looking lie. An unscored word gets no chip. Writing the tests found two bugs first — trimEtymology sliced by byte, which would have emitted invalid UTF-8 for exactly the Greek and Latin etymologies the feature exists for, and its ellipsis path overran its own cap. go build/vet/test, tsc, vite, vitest 96/96 clean; live smoke against the real dict.db with one instance flipped from zh to pt-PT mid-run. Not deployed: go.mod still replaces github.com/prosolis/dreamdict with ../dreamdict, so the Docker build needs the two upstream commits pushed and the replace dropped. The deployed dict.db also predates DreamDict's Spanish data. Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
This commit is contained in:
prosolis
@@ -10,6 +10,12 @@ DATABASE_PATH=./data/petal.db
|
||||
# On-disk store for images pasted/dropped/inserted in the editor
|
||||
IMAGE_DIR=./data/images
|
||||
|
||||
# DreamDict's built dictionary (French, European Portuguese, Spanish, Mandarin),
|
||||
# opened read-only beside petal.db. Optional: with no file here, word lookups use
|
||||
# the embedded English/Chinese datasets, which is how a laptop checkout runs.
|
||||
# Build one with `go run ./cmd/dictimport` in the dreamdict repo.
|
||||
DICT_PATH=./data/dict.db
|
||||
|
||||
# LLM
|
||||
LLM_BACKEND=vllm # vllm | ollama
|
||||
LLM_ENDPOINT=http://localhost:8000 # vLLM :8000, Ollama :11434
|
||||
|
||||
+17
-7
@@ -211,13 +211,22 @@ Pure refactor, zero visible change; prerequisite for every new pair (SUGGESTIONS
|
||||
- Verified: go build/vet/test, tsc, vite build, vitest 90/90 clean; live smoke on a throwaway DB (:8074) — doc create/save, search, md export, spell add, warm 502 from the collocation pass with the LLM down; the built bundle still carries the zh strings.
|
||||
- ⚠️ **Not yet deployed.** Pure refactor with no migration, so the deploy is a rebuild; still unshipped at the end of the session.
|
||||
|
||||
### Phase 20 — DreamDict as a lexicon provider
|
||||
Option 3 ratified (import package, read-only `dict.db`). **Prerequisite in the dreamdict repo:** rename its module path (or add a `replace` for dev).
|
||||
- [ ] Provider seam behind the existing `lexicon` interface; DreamDict provider opens `dict.db` read-only (modernc driver, second handle beside `petal.db`); graceful "no data" when the file is absent
|
||||
- [ ] pt-PT + fr + **es** wired to DreamDict (nothing to regress); **zh stays on ECDICT** until compared on real lookups from her documents — converge only if quality holds
|
||||
- **es is no longer gated** (2026-07-26): DreamDict grew Spanish support, so the "maybe es" in the pair model is now a real option and the provider seam should cover it from day one — it costs nothing here and saves re-opening the package later.
|
||||
- [ ] Surface the new fields where cheap: frequency/difficulty chip in WordCard; etymology line (cognate hook for en-natives)
|
||||
- [ ] Deploy: `dict.db` ships in the data dir alongside `petal.db`
|
||||
### Phase 20 — DreamDict as a lexicon provider ✅ (2026-07-27)
|
||||
Option 3 ratified (import package, read-only `dict.db`).
|
||||
- [x] **The prerequisite was bigger than the plan thought.** Renaming the module was necessary but not sufficient: DreamDict's query layer lived in `internal/dictionary`, which no other module may import whatever the module is called. Both fixed upstream in one commit — `module github.com/prosolis/dreamdict`, `internal/dictionary` → `dictionary`, with a package comment saying why reading a built database is public API while building one stays internal. `internal/loader` is untouched, and DreamDict's own tests pass unchanged.
|
||||
- [x] **Provider seam** (`internal/lexicon/provider.go`): a `Provider` is the two questions the popover and the tooltip have always asked (`Lookup`, `Gloss`), which the embedded `*Lexicon` already satisfied unmodified. `Set.For(lang)` is the single place the choice is made. `OpenDreamDict` opens `dict.db` read-only beside `petal.db`; **a missing file returns `(nil, nil)`, not an error** — a laptop checkout has never had one — while a file that is *present but unimported* does error, because that one is somebody's half-finished deploy.
|
||||
- [x] **The absent-dictionary case degrades better than "no data".** A pt-PT writer with no `dict.db` falls back to the embedded datasets **with the gloss suppressed** (`glossless`), so she keeps English definitions, synonyms and phonetics — all compiled into the binary and all correct for her — and loses only the translation. Handing her the Chinese gloss would be worse than handing her nothing: empty reads as "not found", wrong-language reads as Petal being broken.
|
||||
- [x] pt-PT + fr + **es** wired to DreamDict; **zh stays on ECDICT**, and the routing test is the guard on that decision. The comparison the plan asked for was run against the real 452 MB database: DreamDict reaches a Chinese gloss for **53%** of the 2,000 commonest English words, against ECDICT's essentially total coverage of them. Quality did not hold, so nothing converged. es routes to DreamDict from day one and simply finds no rows in the April build — which is the same code path as any unglossed word.
|
||||
- [x] **The plan's central assumption was wrong, and measuring it is what found that.** `Gloss ← Translate(word, "en", L1)` was mapped 1:1 in `MULTIUSER_PLAN.md`; against real data that table answers for **17%** of common English words into pt-PT (16% into fr). Wiktionary's translation sections are thin in the en→X direction. Going through shared Princeton WordNet synset ids instead answers for **61%**, and it is where the words a learner wants live — "ephemeral", "think" and "quickly" have no en→pt-PT translation row at all. New `dictionary.Equivalents(word, from, to)` upstream does that, falling back to the translations table, for **62%** combined. A gloss absent five times in six is not a gloss.
|
||||
- [x] **Ranking, argued from a wrong answer.** Ordering equivalents by target-word frequency glosses "think" as *lembrar* — "remember" — because lembrar is the commoner Portuguese word even though pensar shares six of think's synsets to lembrar's one. Counting shared senses first, frequency second, asks which candidate means the same thing *most often*: think → pensar; achar; lembrar, write → escrever, garden → jardim, house → casa before firma.
|
||||
- [x] The de-inflection walk (`candidates`) is shared with the embedded path, because `dict.db` stores headwords — "running" has no row. The first candidate that *has definitions* becomes the headword every other field is read from, so one popover never mixes "running"'s frequency with "run"'s senses. The gloss walks separately, since a word can have an equivalent and no definition.
|
||||
- [x] **Surfaced where cheap**: a band chip beside the phonetic (`wordband.ts`) and an etymology line at the foot of the card. Following Phase 19, `wordBand` returns a band *name* and the langpack owns the wording. **Three bands, not five** — the difficulty score is a heuristic over length and corpus counts, good enough to separate "everyday" from "you will need to explain this" and not good enough to rank *obfuscate* against *serendipity*; a finer scale would be a confident-looking lie. Thresholds come from the real distribution (136k headwords bunch between 0.45 and 0.60; the words a writer reaches for sit under 0.40). An unscored word renders **no chip at all**.
|
||||
- [x] The pair language is read **per request** in `providerFor` — a word lookup has no row-scoped query to piggyback on, unlike Phase 19's handlers — and a failed read falls back to today's embedded behaviour rather than failing the lookup. `Cache-Control` dropped from `public` to `private`: the same URL now answers in a different language per writer.
|
||||
- Tests: `internal/lexicon/dreamdict_test.go` — fixture is a **real dict.db on disk**, so open/stat/seeded is the production path; missing vs. unseeded vs. unreadable, every field filled, gloss follows the writer not the word (pt-PT/fr/es/de), the synset path, de-inflection carrying *all* fields to one headword, miss-is-not-an-error, the NULL-difficulty sentinel, IPA chosen over CMU, and the handler tests (two writers/one URL/two languages, unknown caller, private caching). Upstream: `Equivalents` ordering, synset-over-translation, fallback. Frontend: `wordband.test.ts` (bands pinned to real scores; difficulty 0.0 is a score, not a missing value) and an i18n assertion that no band can be unlabelled.
|
||||
- **Two bugs the tests found before the browser did**: `trimEtymology` sliced by byte, which would put invalid UTF-8 in the JSON for exactly the etymologies that matter (ἐφήμερος, ephemerus), and its ellipsis path overran its own cap.
|
||||
- Verified: go build/vet/test, tsc, vite build, vitest 96/96 clean, both repos. Live smoke on a throwaway DB (:8075) against the real 452 MB `dict.db` — startup logs the languages it actually got, zh unchanged, then the same instance flipped to pt-PT and re-queried.
|
||||
- [x] Deploy documented (`deploy/README.md` §4b, `DICT_PATH` through Dockerfile/compose/.env.example): `dict.db` ships into the data dir, stays out of the backups because they name `petal.db` explicitly, and is rebuildable from public data.
|
||||
- ⚠️ **Not deployed, and two things are outstanding.** (1) `go.mod` carries a `replace` to `../dreamdict`; the Docker build needs the two dreamdict commits **pushed** and the replace dropped. (2) The deployed `dict.db` is from **2026-04-04** and has **no Spanish data** and none of the intervening loader fixes — rebuild it (needs the es sources downloaded on millenia) before the Spanish pair ships. Neither blocks pt-PT or French.
|
||||
|
||||
### Phase 21 — The pt-PT pair (first Latin pair, proves the model)
|
||||
SUGGESTIONS §1/§3/§3a. French and **Spanish** follow the same groove afterwards — es is no longer gated now that DreamDict has Spanish data (2026-07-26). pt-PT still goes first: it's the pair with a real user behind it, and it's the one that proves the langpack + both-dictionaries model.
|
||||
@@ -251,6 +260,7 @@ Each item independent and small; order within is free (SUGGESTIONS §5–§6).
|
||||
- [x] **Phase 14 — companion warmth + bedtime nag + night mode**: more encouraging phrases, a gentle "go to bed" nudge after 11pm, and a calm dark theme + falling stars at night. ✅ (see Phase 14 above)
|
||||
|
||||
## Session log
|
||||
- 2026-07-27: **Phase 20 — DreamDict becomes the dictionary for every pair but Chinese** (user: "let's continue the build plan"; scope confirmed: build the seam against the existing April `dict.db`, rebuild it later, code + local verification only). The prerequisite was bigger than the plan recorded: renaming DreamDict's module path was necessary but useless on its own, because the query layer lived in `internal/dictionary` and no module may import another's `internal`. Both fixed upstream — the package is now `dictionary`, with a comment saying why *reading* a built database is public API while the loaders that build one stay internal. In Petal, `Provider` is the two questions the popover already asked, so the embedded `*Lexicon` satisfied it with no changes at all, and `Set.For(lang)` is the one place the choice is made. **The measurement is the story of the phase.** `MULTIUSER_PLAN.md` mapped `Gloss ← Translate(word, "en", L1)` 1:1; against the real 452 MB database that table answers for **17%** of the 2,000 commonest English words into pt-PT. Wiktionary's translation sections are thin in that direction — "ephemeral", "think" and "quickly" have no en→pt-PT row at all. The shared-synset path answers for **61%**, so a new upstream `Equivalents` queries that and falls back to translations for 62% combined. Then the *ordering* was wrong in an instructive way: sorting by target frequency glosses "think" as *lembrar* — "remember" — because lembrar is commoner in Portuguese, even though pensar shares six of think's synsets to lembrar's one. Counting sense agreement first fixes it (think → pensar; write → escrever; garden → jardim). The same measurement is what kept **zh on ECDICT**: DreamDict reaches a Chinese gloss for 53% of those words where ECDICT reaches nearly all — the plan said converge only if quality holds, and it didn't. Two other decisions worth keeping: a missing `dict.db` is **not an error** (a laptop has never had one) but a present-and-unimported one is; and a pt-PT writer without a dictionary falls back to the embedded datasets **with the gloss suppressed**, keeping the English half rather than blanking the popover — an empty field reads as "not found", the wrong language reads as broken. The new fields surface as **three** bands, not five, because the difficulty score can separate "everyday" from "you'll have to explain this" but cannot rank *obfuscate* against *serendipity*, and a finer scale would be a confident-looking lie. Writing the tests found two bugs first: `trimEtymology` sliced by byte, which would have emitted invalid UTF-8 for precisely the Greek and Latin etymologies the feature exists for, and its ellipsis path overran its own cap. go build/vet/test, tsc, vite, vitest 96/96 clean in both repos; live smoke on a throwaway DB against the real dictionary, one instance flipped from zh to pt-PT mid-run. **Not deployed**: `go.mod` still carries a `replace` to `../dreamdict` (the two upstream commits need pushing first), and the deployed `dict.db` predates DreamDict's Spanish support and needs rebuilding before the es pair ships.
|
||||
- 2026-07-27: **Phase 18 deployed, and Phase 19 — the copy stops being hardcoded Mandarin** (user: "let's continue the build plan"; sequencing confirmed: rehearse + deploy 18, then start 19). The rehearsal the previous session was blocked from running went first: a `VACUUM INTO` snapshot of the live VPS database, migrated locally by the Phase-18 binary, every count unchanged and FTS/integrity/foreign keys clean, `personal_words` created empty — then the deploy itself (off-box encrypted backup, rebuild, all three containers healthy, `0011` applied to the live DB with her writing untouched, `/api/spell/words` 401 without a session over public HTTPS). **One check was refused and not worked around**: minting a probe session row to see the endpoint answer 200 for a real cookie reads as credential fabrication to this session's classifier; the endpoint's lifecycle is covered by tests and the shared middleware governs that last step for every other route. **Phase 19** then lifted every `中文 · English` literal out of ~29 files into `web/src/i18n` — one `Pack` type, a verbatim `zh` pack, and two access paths chosen by *when* copy is built: `usePack()` for components, `pack()` for the companion and prose checker, which compose a line when something happens rather than when something renders. The interesting decisions were about what a pack must be allowed to control: **every string with a value in it is a function** (`reviewDue(n)`, `daysAgo(n)`, even English pluralisation) because word order isn't universal; the roster constants keep only value + emoji so a label can never drift from its key; and `gradeBand` returns a band *name* rather than a label. On the server, `internal/llm/lang.go` replaces "Simplified Chinese" in the three prompts that name her language — with pt-PT spelled **"European Portuguese (pt-PT, never Brazilian Portuguese)"** in the prompt itself, and her word for "why" carried alongside so the tutor still recognises the question. `pair_lang` is read **in the row-scoped query each handler already ran**, not a second lookup that could disagree with it — and the test for that was checked by breaking the join and watching it fail. go build/vet/test, tsc, vite, vitest 90/90 clean; live smoke on a throwaway DB. **Phase 19 is not deployed** — no migration, so it's a rebuild whenever the user wants it.
|
||||
- 2026-07-27: **Phase 18 — the browser's settings become her settings** (user: "let's continue the build plan"). Two scope calls taken with the user: the personal spell dictionary goes **server-side** rather than being namespaced in place, and the pre-account `localStorage` keys are **adopted then rescoped** by the first writer to sign in. New `web/src/lib/prefs.ts` namespaces `petal.sound`/`petal.petals`/`petal.companion` by user id; the interesting part is timing — those modules read their value at *import* time, before `/api/me` can possibly have answered, so a pre-scope read deliberately sees the legacy key (the right value on a single-writer browser) and `setPrefsScope`, called from `useSession`, adopts it and notifies every reader. Adoption **moves** rather than copies, so account two starts from Petal's defaults instead of inheriting a stranger's mascot. New `internal/spell` package + migration `0011`: `personal_words` keyed `(user_id, lang, word)`, where `lang` is the **dictionary's** language, not the writer's — an English exception must not silence a pt-PT flag when the second pair ships. `useSpellChecker` now replays her list from her account, hands over any browser-held Phase-7 list on first load (releasing it only once the server has taken it), and persists an added word in the background so the underline vanishes the instant she asks. **The reason for the server table over cheaper namespacing**: keying the existing list by user in `localStorage` would have *fragmented* the words she already has across her laptop and tablet — the "cheap" fix was the one that made things worse. Tests: full lifecycle + languages-don't-merge + junk + the standing-rule two-user isolation suite in Go, and legacy-adoption/move-not-copy/two-accounts/storage-throws in vitest. go build/vet/test, tsc, vite, vitest 82/82 clean; live smoke on a throwaway DB. **Not deployed** — and the customary rehearsal of `0011` against a copy of the live VPS database was blocked by the session's permission classifier, so that check is outstanding (it is a plain `CREATE TABLE`, so lower-risk than `0005`/`0010`, but the convention exists for a reason).
|
||||
- 2026-07-27: **Phase 17 — Claire's writing moved onto her real account** (user: "claire is local user today in Petal. let's make sure to migrate existing data to her account"). `scripts/migrate_local_user.py`: dry-run by default, own `VACUUM INTO` backup, one transaction with foreign keys off, re-points `documents`/`tags`/`vocab_words`/`images`, verifies every expected row moved before committing. **The plan's stated prerequisite — "she logs in once so her sub exists" — turned out to be false**: authentik's `hashed_user_id` sub is `User.uid`, derived from her id and the instance secret, so it is readable in advance and the data could move *first*; she signs in to find her writing already there instead of to an empty Petal. Her 8 documents, 33 snapshots, 103 suggestions, 3 vocabulary words and 1 image now belong to `5f47d955…`, verified end to end over public HTTPS. Per the user's call the VPS is now canonical and millenia was left running and untouched as a frozen fallback (it diverges the moment either is written to — retire it rather than sync it). **Three bugs, each found by a different kind of contact with reality**: (1) the image backfill claims files for `local`, which stops existing after a migration — a foreign-key error inside `images.New`, which `main.go` treats as fatal, so Petal would have crash-looped on first start against a migrated database; (2) `BEGIN EXCLUSIVE` was the wrong liveness check, since in WAL mode it only conflicts with another *writer* and sails past a running-but-idle Petal — exactly the case the guard exists for; (3) the replacement, `PRAGMA locking_mode = EXCLUSIVE`, holds its lock past being reset to `NORMAL`, so on a real WAL database the script locked itself out of its own backup — invisible locally because the test file had come from `VACUUM INTO` and wasn't in WAL mode. Same shape as Phase 16's trailing-slash issuer: the fixture didn't look like production.
|
||||
|
||||
+6
-2
@@ -47,7 +47,10 @@ RUN apk add --no-cache ca-certificates ffmpeg tzdata \
|
||||
WORKDIR /app
|
||||
COPY --from=build /out/petal /app/petal
|
||||
|
||||
# Mount point for petal.db (+ -wal/-shm), the image store and the TTS cache.
|
||||
# Mount point for petal.db (+ -wal/-shm), the image store, the TTS cache and
|
||||
# DreamDict's read-only dict.db. dict.db is deployed alongside rather than baked
|
||||
# in: it is ~450 MB, changes a few times a year, and is shared with other
|
||||
# services on the host — putting it in the image would multiply it by every tag.
|
||||
RUN mkdir -p /data && chown -R petal:petal /data
|
||||
VOLUME ["/data"]
|
||||
|
||||
@@ -57,7 +60,8 @@ EXPOSE 8080
|
||||
ENV PORT=8080 \
|
||||
DATABASE_PATH=/data/petal.db \
|
||||
IMAGE_DIR=/data/images \
|
||||
TTS_CACHE_DIR=/data/tts
|
||||
TTS_CACHE_DIR=/data/tts \
|
||||
DICT_PATH=/data/dict.db
|
||||
|
||||
# Same endpoint Traefik and the uptime probe use; needs no session by design.
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
|
||||
|
||||
@@ -252,6 +252,17 @@ almost 1:1 onto `lexicon.Result`:
|
||||
It also carries data Petal has no equivalent for and could use: `Antonyms`,
|
||||
`Frequency`, `Difficulty`, and `Etymology`.
|
||||
|
||||
> **Correction (2026-07-27, Phase 20).** The `Gloss` row of that table is wrong.
|
||||
> `Translate(word, "en", L1)` reads Wiktionary's translation sections, which are
|
||||
> thin in the en→X direction: measured on the real `dict.db`, it answers for
|
||||
> **17%** of the 2,000 commonest English words into pt-PT and 16% into fr.
|
||||
> Meaning has to come through shared WordNet synset ids instead (**61%**), which
|
||||
> is what DreamDict's new `Equivalents(word, from, to)` does — falling back to
|
||||
> the translations table, for 62% combined. The 1:1 mapping was assumed from the
|
||||
> API surface and never checked against the data; it did not survive contact
|
||||
> with it. Same measurement on zh reads 53% against ECDICT's near-total coverage
|
||||
> of those words, which is why the zh pair did **not** converge.
|
||||
|
||||
So Phase D stops being gated on data and becomes an integration decision.
|
||||
|
||||
### OPEN #6a (new): how to integrate — **SETTLED: Option 3** (ratified 2026-07-26)
|
||||
|
||||
+25
-5
@@ -83,6 +83,24 @@ func main() {
|
||||
log.Printf("auth: OIDC not configured — running as the single %q user", db.LocalUserID)
|
||||
}
|
||||
|
||||
// The dictionary behind word lookups. dict.db is DreamDict's built database
|
||||
// — French, European Portuguese, Spanish and Mandarin in one read-only file
|
||||
// beside petal.db. It is optional on purpose: a laptop checkout has never
|
||||
// had one, and the Chinese pair doesn't need one, so its absence downgrades
|
||||
// lookups rather than stopping Petal. A file that is present but broken is
|
||||
// a different matter and gets said out loud.
|
||||
dict, err := lexicon.OpenDreamDict(cfg.DictPath)
|
||||
if err != nil {
|
||||
log.Printf("dictionary: %s unusable (%v) — falling back to the embedded datasets", cfg.DictPath, err)
|
||||
}
|
||||
defer dict.Close()
|
||||
lexSet := lexicon.NewSet(dict)
|
||||
if lexSet.HasDreamDict() {
|
||||
log.Printf("dictionary: DreamDict open at %s (%v)", cfg.DictPath, dict.Langs())
|
||||
} else {
|
||||
log.Printf("dictionary: no dict.db at %s — English/Chinese only", cfg.DictPath)
|
||||
}
|
||||
|
||||
r := chi.NewRouter()
|
||||
r.Use(middleware.RequestID)
|
||||
r.Use(middleware.RealIP)
|
||||
@@ -149,11 +167,13 @@ func main() {
|
||||
pr.Mount("/suggestions", sug.Routes())
|
||||
|
||||
// Offline lexicon: full word lookups (gloss + definition + synonyms) for
|
||||
// the right-click popover, and the lightweight Chinese-only gloss for the
|
||||
// inline hover/select tooltip. One handler so the datasets load once.
|
||||
// The dataset is static and identical for everyone, but it stays behind
|
||||
// auth so the API surface has no unauthenticated read holes.
|
||||
lex := lexicon.NewHandler()
|
||||
// the right-click popover, and the lightweight gloss-only lookup for the
|
||||
// inline hover/select tooltip. One handler over one provider Set, so the
|
||||
// embedded datasets and dict.db are each opened once. Which of them
|
||||
// answers depends on the caller's language pair — so unlike before, the
|
||||
// response is no longer identical for everyone, and it stays behind auth
|
||||
// for that reason as much as for the API surface.
|
||||
lex := lexicon.NewHandler(database.DB, lexSet)
|
||||
pr.Mount("/word", lex.Routes())
|
||||
pr.Mount("/gloss", lex.GlossRoutes())
|
||||
|
||||
|
||||
@@ -284,6 +284,64 @@ a migration took; the presence of the row itself says nothing.
|
||||
|
||||
---
|
||||
|
||||
## 4b. The dictionary (`dict.db`)
|
||||
|
||||
Word lookups for the French, European Portuguese and Spanish pairs come from
|
||||
[DreamDict](https://github.com/prosolis/dreamdict)'s built database, which Petal
|
||||
opens **read-only** beside `petal.db`. Petal imports DreamDict's `dictionary`
|
||||
package directly — there is no DreamDict service to run and nothing to reach
|
||||
over the VPN, which matters because a hover gloss must answer in milliseconds.
|
||||
|
||||
`dict.db` is **optional**. With no file at `DICT_PATH` Petal logs
|
||||
|
||||
```
|
||||
dictionary: no dict.db at /data/dict.db — English/Chinese only
|
||||
```
|
||||
|
||||
and serves lookups from the datasets compiled into the binary. The Chinese pair
|
||||
is unaffected either way — it stays on ECDICT (see below) — and a non-Chinese
|
||||
writer still gets English definitions, synonyms and pronunciation, losing only
|
||||
the translation. **A dictionary that failed to deploy costs the gloss, not the
|
||||
popover.** A file that is present but was never imported is a different matter
|
||||
and is logged as an error.
|
||||
|
||||
### Installing it
|
||||
|
||||
The database is built by DreamDict's own import CLI from ~6 GB of source data;
|
||||
it is not built on the VPS. Copy the built file into the data volume:
|
||||
|
||||
```bash
|
||||
# on the machine holding a built dict.db (millenia: ~/dreamdict/data/dict.db)
|
||||
scp ~/dreamdict/data/dict.db reala@100.64.0.1:/home/reala/petal/data/dict.db
|
||||
# on parodia
|
||||
chown "$(id -u):$(id -g)" /home/reala/petal/data/dict.db
|
||||
docker compose restart petal # the handle is opened once, at startup
|
||||
```
|
||||
|
||||
Expect ~450 MB. It sits inside the LUKS volume with everything else (§6). The
|
||||
backups name `petal.db` explicitly rather than sweeping the data directory
|
||||
(§5), so `dict.db` stays out of them — which is the right outcome and worth
|
||||
keeping: it is rebuildable from public data and would otherwise dominate every
|
||||
nightly snapshot. Petal never writes to it.
|
||||
|
||||
### Why Chinese doesn't use it
|
||||
|
||||
The zh pair stays on the embedded ECDICT gloss, deliberately. Measured on the
|
||||
deployed database, DreamDict reaches a Chinese gloss for 53% of the 2,000
|
||||
commonest English words; ECDICT covers essentially all of them and is in daily
|
||||
use by a real writer. `lexicon.Set.For` is where that decision lives — one
|
||||
`switch`, changed the day a comparison on her actual lookups says otherwise.
|
||||
|
||||
For pt-PT and French the same measurement reads 62%, which is why they use
|
||||
DreamDict: there is no alternative source for them at all.
|
||||
|
||||
**The deployed `dict.db` is from 2026-04-04 and has no Spanish data**, because
|
||||
DreamDict grew Spanish support after it was built. A Spanish-pair writer gets
|
||||
empty glosses (and English definitions) until it is rebuilt — do that before
|
||||
the Spanish pair ships.
|
||||
|
||||
---
|
||||
|
||||
## 5. Backups
|
||||
|
||||
### On the VPS — folded into `parodia-backup`
|
||||
|
||||
@@ -38,6 +38,10 @@ services:
|
||||
DATABASE_PATH: /data/petal.db
|
||||
IMAGE_DIR: /data/images
|
||||
TTS_CACHE_DIR: /data/tts
|
||||
# DreamDict's built dictionary, read-only, deployed into the data volume
|
||||
# (see deploy/README.md). Absent it, word lookups fall back to the
|
||||
# embedded English/Chinese datasets rather than failing.
|
||||
DICT_PATH: /data/dict.db
|
||||
# Piper sidecars. Each server loads one voice, so English and Chinese are
|
||||
# separate containers; the handler maps language → instance from config.
|
||||
TTS_ENDPOINT: http://piper-en:5000
|
||||
|
||||
@@ -6,6 +6,7 @@ require (
|
||||
github.com/coreos/go-oidc/v3 v3.20.0
|
||||
github.com/go-chi/chi/v5 v5.3.0
|
||||
github.com/go-jose/go-jose/v4 v4.1.4
|
||||
github.com/prosolis/dreamdict v0.0.0-00010101000000-000000000000
|
||||
golang.org/x/oauth2 v0.36.0
|
||||
modernc.org/sqlite v1.53.0
|
||||
)
|
||||
@@ -21,3 +22,5 @@ require (
|
||||
modernc.org/mathutil v1.7.1 // indirect
|
||||
modernc.org/memory v1.11.0 // indirect
|
||||
)
|
||||
|
||||
replace github.com/prosolis/dreamdict => ../dreamdict
|
||||
|
||||
@@ -12,6 +12,12 @@ type Config struct {
|
||||
BaseURL string
|
||||
DatabasePath string
|
||||
ImageDir string // on-disk store for editor image uploads
|
||||
// DictPath is DreamDict's built dict.db, read-only, sitting beside
|
||||
// petal.db. It is what gives Petal French, European Portuguese and Spanish
|
||||
// word lookups; without it only the embedded English/Chinese datasets
|
||||
// exist, which is exactly how a laptop checkout runs. A missing file is
|
||||
// therefore not an error — see lexicon.OpenDreamDict.
|
||||
DictPath string
|
||||
|
||||
// LLM
|
||||
LLMBackend string // "vllm" | "ollama"
|
||||
@@ -63,6 +69,7 @@ func Load() *Config {
|
||||
BaseURL: env("BASE_URL", "http://localhost:8080"),
|
||||
DatabasePath: env("DATABASE_PATH", "./data/petal.db"),
|
||||
ImageDir: env("IMAGE_DIR", "./data/images"),
|
||||
DictPath: env("DICT_PATH", "./data/dict.db"),
|
||||
|
||||
LLMBackend: env("LLM_BACKEND", "vllm"),
|
||||
LLMEndpoint: env("LLM_ENDPOINT", "http://localhost:8000"),
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
package lexicon
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io/fs"
|
||||
"os"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/prosolis/dreamdict/dictionary"
|
||||
)
|
||||
|
||||
// DreamDict is a read-only handle on a built dict.db — one SQLite file holding
|
||||
// English, French, European Portuguese, Spanish and Mandarin.
|
||||
//
|
||||
// It is a second database beside petal.db and is never written to: the file is
|
||||
// built by DreamDict's own import CLI a few times a year, and Petal only reads
|
||||
// it. That is what makes importing the package the right shape rather than
|
||||
// running DreamDict as a service — a hover gloss should not depend on a second
|
||||
// process being up, still less on one reachable across a VPN.
|
||||
type DreamDict struct {
|
||||
d *dictionary.Dictionary
|
||||
}
|
||||
|
||||
// OpenDreamDict opens dict.db read-only.
|
||||
//
|
||||
// A missing file returns (nil, nil), not an error. Petal is expected to run
|
||||
// without dict.db — a laptop checkout has never had one, and the zh pair does
|
||||
// not need one — so "the file isn't there" is a deployment state the caller
|
||||
// handles by carrying on. A file that is *present but unusable* (corrupt, or
|
||||
// never imported) does return an error, because that one is a mistake someone
|
||||
// should hear about.
|
||||
func OpenDreamDict(path string) (*DreamDict, error) {
|
||||
if strings.TrimSpace(path) == "" {
|
||||
return nil, nil
|
||||
}
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
d, err := dictionary.NewReadOnly(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &DreamDict{d: d}, nil
|
||||
}
|
||||
|
||||
// Close releases the dict.db handle. Safe on a nil DreamDict, so a caller that
|
||||
// never got one can defer it unconditionally.
|
||||
func (dd *DreamDict) Close() error {
|
||||
if dd == nil {
|
||||
return nil
|
||||
}
|
||||
return dd.d.Close()
|
||||
}
|
||||
|
||||
// Langs returns the language codes dict.db was built with, so startup can log
|
||||
// what it actually got rather than what it hoped for.
|
||||
func (dd *DreamDict) Langs() []string { return dictionary.Langs() }
|
||||
|
||||
// dreamProvider serves one writer: English lookups from dict.db, glossed into
|
||||
// native. The struct is a value, created per request by [Set.For] — it holds no
|
||||
// state beyond the shared handle and the language to translate into.
|
||||
type dreamProvider struct {
|
||||
dict *DreamDict
|
||||
native string // the writer's language, e.g. "pt-PT"
|
||||
}
|
||||
|
||||
// maxEtymology caps the free-form Wiktionary etymology. It is the one field
|
||||
// with no natural length: some entries are a clause, some are four paragraphs
|
||||
// tracing a word through three dead languages. The popover wants a line.
|
||||
const maxEtymology = 220
|
||||
|
||||
// Lookup fills a Result from dict.db.
|
||||
//
|
||||
// The word is de-inflected with the same [candidates] walk the embedded
|
||||
// datasets use, because dict.db stores headwords: "running" has no definitions
|
||||
// row of its own. The first candidate that *has* definitions becomes the
|
||||
// headword every other field is then read from, so a single popover never
|
||||
// mixes "running"'s frequency with "run"'s definitions.
|
||||
//
|
||||
// The gloss is walked separately. A word can be absent from the definitions
|
||||
// table and still have a translation (and vice versa), and the hover tooltip
|
||||
// asks for the gloss alone — so tying it to the definition headword would lose
|
||||
// glosses for no benefit.
|
||||
func (p dreamProvider) Lookup(word string) (Result, error) {
|
||||
res := Result{Word: word, Definitions: []Meaning{}, Synonyms: []string{}, Difficulty: unknownDifficulty}
|
||||
norm := strings.ToLower(strings.TrimSpace(word))
|
||||
if norm == "" {
|
||||
return res, nil
|
||||
}
|
||||
|
||||
head := norm
|
||||
for _, c := range candidates(norm) {
|
||||
defs, err := p.dict.d.Define(c, langEN)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if len(defs) == 0 {
|
||||
continue
|
||||
}
|
||||
head = c
|
||||
for _, d := range defs {
|
||||
// DreamDict orders by source priority, so the curated senses
|
||||
// (WordNet, WOLF) are already ahead of the Wiktionary tail — taking
|
||||
// the first few is taking the best few.
|
||||
res.Definitions = append(res.Definitions, Meaning{PartOfSpeech: d.POS, Definition: d.Gloss})
|
||||
if len(res.Definitions) >= maxDefinitions {
|
||||
break
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
gloss, err := p.translate(norm)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
res.Gloss = gloss
|
||||
|
||||
syns, err := p.dict.d.Synonyms(head, langEN)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if len(syns) > maxSynonyms {
|
||||
syns = syns[:maxSynonyms]
|
||||
}
|
||||
res.Synonyms = append(res.Synonyms, syns...)
|
||||
|
||||
prons, err := p.dict.d.Pronunciation(head, langEN)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
res.Phonetic = pickIPA(prons)
|
||||
|
||||
if res.Frequency, err = p.dict.d.Frequency(head, langEN); err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if res.Difficulty, err = p.dict.d.Difficulty(head, langEN); err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
ety, err := p.dict.d.Etymology(head, langEN)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
res.Etymology = trimEtymology(ety)
|
||||
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// Gloss returns the writer's-language translation alone — the hover tooltip's
|
||||
// fast path, one indexed query per candidate form and nothing else.
|
||||
func (p dreamProvider) Gloss(word string) (GlossResult, error) {
|
||||
norm := strings.ToLower(strings.TrimSpace(word))
|
||||
if norm == "" {
|
||||
return GlossResult{Word: word}, nil
|
||||
}
|
||||
gloss, err := p.translate(norm)
|
||||
if err != nil {
|
||||
return GlossResult{}, err
|
||||
}
|
||||
return GlossResult{Word: word, Gloss: gloss}, nil
|
||||
}
|
||||
|
||||
// maxGlossSenses caps how many translations are strung together. One is often
|
||||
// too thin to disambiguate; the whole list is a wall of words in a tooltip.
|
||||
const maxGlossSenses = 3
|
||||
|
||||
// translate walks the candidate forms and returns the first that has an
|
||||
// equivalent in the writer's language, joined into one line.
|
||||
//
|
||||
// It asks for Equivalents rather than Translate on the strength of measuring
|
||||
// both against the real dict.db: Wiktionary's en→pt-PT translation table
|
||||
// answers for 17% of the 2,000 commonest English words, and the shared-synset
|
||||
// path answers for 62%. The plan assumed Translate would do — the database
|
||||
// says otherwise, and a gloss that is absent five times out of six is not a
|
||||
// gloss. Equivalents falls back to Translate internally, so nothing is lost.
|
||||
//
|
||||
// A language dict.db was built without simply has no rows, so this returns "" —
|
||||
// which is exactly what an unglossed word returns, and the popover already
|
||||
// renders that case. Spanish today is precisely this: supported by DreamDict,
|
||||
// absent from the deployed database until it is rebuilt.
|
||||
func (p dreamProvider) translate(norm string) (string, error) {
|
||||
for _, c := range candidates(norm) {
|
||||
trs, err := p.dict.d.Equivalents(c, langEN, p.native)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(trs) == 0 {
|
||||
continue
|
||||
}
|
||||
if len(trs) > maxGlossSenses {
|
||||
trs = trs[:maxGlossSenses]
|
||||
}
|
||||
return strings.Join(trs, "; "), nil
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// pickIPA chooses what to show beside the read-aloud button. IPA is the only
|
||||
// form worth showing a learner — CMU's "IH0 F EH1 M ER0 AH0 L" is a machine
|
||||
// format, and printing it would be noise dressed up as help. If there's no IPA,
|
||||
// there's no phonetic line.
|
||||
func pickIPA(prons []dictionary.Pronunciation) string {
|
||||
for _, p := range prons {
|
||||
if strings.EqualFold(p.Format, "ipa") && strings.TrimSpace(p.Value) != "" {
|
||||
return strings.Trim(strings.TrimSpace(p.Value), "/[]")
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// trimEtymology cuts Wiktionary's prose down to a line, preferring to stop at a
|
||||
// sentence boundary so the result reads as a finished thought rather than a
|
||||
// truncation.
|
||||
func trimEtymology(text string) string {
|
||||
text = strings.Join(strings.Fields(text), " ")
|
||||
if utf8.RuneCountInString(text) <= maxEtymology {
|
||||
return text
|
||||
}
|
||||
// Counted and cut in runes, not bytes. An etymology is the one field that
|
||||
// is *mostly* not English — ἐφήμερος, ephemerus, 短暫 — and a byte slice
|
||||
// through the middle of one of those characters is invalid UTF-8 in the
|
||||
// JSON response.
|
||||
cut := string([]rune(text)[:maxEtymology-1])
|
||||
// Stop at a sentence when one ends late enough to be worth keeping. An
|
||||
// early full stop ("From Latin. …") is not a summary, it's a discarded
|
||||
// paragraph, so that case falls through to the word-boundary cut.
|
||||
if i := strings.LastIndex(cut, ". "); i > len(cut)/2 {
|
||||
return cut[:i+1]
|
||||
}
|
||||
if i := strings.LastIndex(cut, " "); i > 0 {
|
||||
cut = cut[:i]
|
||||
}
|
||||
return cut + "…"
|
||||
}
|
||||
@@ -0,0 +1,521 @@
|
||||
package lexicon
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/prosolis/dreamdict/dictionary"
|
||||
_ "modernc.org/sqlite"
|
||||
|
||||
"gitea.parodia.dev/drwily/petal/internal/auth"
|
||||
"gitea.parodia.dev/drwily/petal/internal/db"
|
||||
)
|
||||
|
||||
// The fixture is a real dict.db on disk rather than an in-memory handle, so
|
||||
// these tests exercise the path production takes: stat the file, open it
|
||||
// read-only, find it seeded. A fake would have skipped every one of those.
|
||||
//
|
||||
// "ephemeral" is the worked example throughout: it has definitions only under
|
||||
// its own headword, translations into two languages, IPA alongside a CMU
|
||||
// pronunciation Petal must not show, a frequency, a difficulty and an etymology
|
||||
// long enough to need trimming.
|
||||
func writeFixture(t *testing.T, seeded bool) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "dict.db")
|
||||
sqldb, err := sql.Open("sqlite", path)
|
||||
if err != nil {
|
||||
t.Fatalf("open fixture: %v", err)
|
||||
}
|
||||
defer sqldb.Close()
|
||||
if err := dictionary.BootstrapSchema(sqldb); err != nil {
|
||||
t.Fatalf("bootstrap: %v", err)
|
||||
}
|
||||
if !seeded {
|
||||
return path
|
||||
}
|
||||
|
||||
exec := func(q string, args ...any) {
|
||||
t.Helper()
|
||||
if _, err := sqldb.Exec(q, args...); err != nil {
|
||||
t.Fatalf("seed %q: %v", q, err)
|
||||
}
|
||||
}
|
||||
exec(`INSERT INTO meta (key, value) VALUES ('schema_version', '2')`)
|
||||
|
||||
exec(`INSERT INTO words (id, word, lang, pos, frequency, difficulty) VALUES
|
||||
(1, 'ephemeral', 'en', 'adjective', 50, 0.72),
|
||||
(2, 'run', 'en', 'verb', 900, 0.05),
|
||||
(3, 'plain', 'en', 'adjective', 0, NULL),
|
||||
(4, 'efémero', 'pt-PT', 'adjective', 12, 0.6)`)
|
||||
|
||||
exec(`INSERT INTO definitions (word_id, pos, gloss, source, priority) VALUES
|
||||
(1, 'adjective', 'lasting a very short time', 'wordnet', 10),
|
||||
(1, 'adjective', 'short-lived', 'wiktionary', 99),
|
||||
(1, 'adjective', 'transitory', 'wiktionary', 99),
|
||||
(1, 'adjective', 'fleeting', 'wiktionary', 99),
|
||||
(1, 'adjective', 'evanescent', 'wiktionary', 99),
|
||||
(2, 'verb', 'move fast on foot', 'wordnet', 10),
|
||||
(3, 'adjective', 'without decoration', 'wordnet', 10)`)
|
||||
|
||||
exec(`INSERT INTO synonyms (word_id, synonym, source) VALUES
|
||||
(1, 'fleeting', 'wordnet'), (1, 'transient', 'wordnet'),
|
||||
(2, 'sprint', 'wordnet')`)
|
||||
|
||||
exec(`INSERT INTO translations (word_id, translation, target_lang, source) VALUES
|
||||
(1, 'efémero', 'pt-PT', 'kaikki'),
|
||||
(1, 'passageiro','pt-PT', 'kaikki'),
|
||||
(1, 'éphémère', 'fr', 'kaikki'),
|
||||
(1, '短暂的', 'zh', 'cedict'),
|
||||
(2, 'correr', 'pt-PT', 'kaikki')`)
|
||||
|
||||
// CMU is listed first deliberately: picking the first row would show a
|
||||
// learner "IH0 F EH1 M ER0 AH0 L", which is a machine format, not help.
|
||||
exec(`INSERT INTO pronunciations (word_id, format, value, source) VALUES
|
||||
(1, 'cmu', 'IH0 F EH1 M ER0 AH0 L', 'cmudict'),
|
||||
(1, 'ipa', '/ɪˈfɛm.ər.əl/', 'wiktionary')`)
|
||||
|
||||
// "brief" carries no translation row at all — only a shared WordNet synset
|
||||
// with two pt-PT words. On the real database that is the *usual* case, not
|
||||
// the exotic one, so Petal must reach a gloss this way or the pt-PT pair
|
||||
// has almost no glosses. "breve" is the commoner of the two and leads.
|
||||
exec(`INSERT INTO words (id, word, lang, pos, frequency) VALUES
|
||||
(5, 'brief', 'en', 'adjective', 400),
|
||||
(6, 'breve', 'pt-PT', 'adjective', 300),
|
||||
(7, 'sucinto', 'pt-PT', 'adjective', 20)`)
|
||||
exec(`INSERT INTO definitions (word_id, pos, gloss, source, priority) VALUES
|
||||
(5, 'adjective', 'of short duration', 'wordnet', 10)`)
|
||||
exec(`INSERT INTO synsets (id, synset_id, pos) VALUES (1, '00751145-a', 'adjective')`)
|
||||
exec(`INSERT INTO word_synsets (word_id, synset_id, source) VALUES
|
||||
(5, 1, 'wordnet'), (6, 1, 'omw'), (7, 1, 'omw')`)
|
||||
|
||||
exec(`INSERT INTO etymology (word_id, text, source) VALUES
|
||||
(1, 'From Medieval Latin ephemerus, from Ancient Greek ἐφήμερος (ephḗmeros, "lasting only a day"), from ἐπί (epí, "upon") and ἡμέρα (hēméra, "day"). The sense of transience is attested in English from the late sixteenth century onwards.', 'wiktionary')`)
|
||||
|
||||
return path
|
||||
}
|
||||
|
||||
func openFixture(t *testing.T) *DreamDict {
|
||||
t.Helper()
|
||||
dd, err := OpenDreamDict(writeFixture(t, true))
|
||||
if err != nil {
|
||||
t.Fatalf("OpenDreamDict: %v", err)
|
||||
}
|
||||
if dd == nil {
|
||||
t.Fatal("OpenDreamDict returned no dictionary for a seeded file")
|
||||
}
|
||||
t.Cleanup(func() { dd.Close() })
|
||||
return dd
|
||||
}
|
||||
|
||||
func TestOpenMissingFileIsNotAnError(t *testing.T) {
|
||||
dd, err := OpenDreamDict(filepath.Join(t.TempDir(), "absent.db"))
|
||||
if err != nil {
|
||||
t.Fatalf("a missing dict.db must not be an error: %v", err)
|
||||
}
|
||||
if dd != nil {
|
||||
t.Fatal("a missing dict.db must yield no dictionary")
|
||||
}
|
||||
// An unset path is the laptop default and must behave the same way.
|
||||
if dd, err := OpenDreamDict(""); err != nil || dd != nil {
|
||||
t.Fatalf(`OpenDreamDict("") = %v, %v; want nil, nil`, dd, err)
|
||||
}
|
||||
// Close on the nil handle is what main.go defers unconditionally.
|
||||
if err := dd.Close(); err != nil {
|
||||
t.Fatalf("Close on absent dictionary: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenPresentButUnseededIsAnError(t *testing.T) {
|
||||
// A file that exists but was never imported is somebody's mistake — a
|
||||
// half-finished deploy — and must be loud, unlike a file that isn't there.
|
||||
dd, err := OpenDreamDict(writeFixture(t, false))
|
||||
if err == nil {
|
||||
dd.Close()
|
||||
t.Fatal("an unseeded dict.db must report an error")
|
||||
}
|
||||
if dd != nil {
|
||||
t.Fatal("an unseeded dict.db must not yield a usable dictionary")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenUnreadablePathIsAnError(t *testing.T) {
|
||||
// Not a missing file: a directory where dict.db should be. Distinguishing
|
||||
// this from ErrNotExist is the whole point of the stat.
|
||||
dir := filepath.Join(t.TempDir(), "dict.db")
|
||||
if err := os.Mkdir(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := OpenDreamDict(dir); err == nil {
|
||||
t.Fatal("a directory in place of dict.db must report an error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDreamLookupFillsEveryField(t *testing.T) {
|
||||
p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
|
||||
res, err := p.Lookup("ephemeral")
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup: %v", err)
|
||||
}
|
||||
if res.Gloss != "efémero; passageiro" {
|
||||
t.Errorf("Gloss = %q, want the pt-PT translations joined", res.Gloss)
|
||||
}
|
||||
if res.Phonetic != "ɪˈfɛm.ər.əl" {
|
||||
t.Errorf("Phonetic = %q, want the IPA without its slashes", res.Phonetic)
|
||||
}
|
||||
if len(res.Definitions) != maxDefinitions {
|
||||
t.Fatalf("Definitions = %d, want them capped at %d", len(res.Definitions), maxDefinitions)
|
||||
}
|
||||
if res.Definitions[0].Definition != "lasting a very short time" {
|
||||
t.Errorf("first definition = %q, want the curated (wordnet) sense first",
|
||||
res.Definitions[0].Definition)
|
||||
}
|
||||
if res.Definitions[0].PartOfSpeech != "adjective" {
|
||||
t.Errorf("part of speech = %q, want adjective", res.Definitions[0].PartOfSpeech)
|
||||
}
|
||||
if len(res.Synonyms) != 2 {
|
||||
t.Errorf("Synonyms = %v, want both", res.Synonyms)
|
||||
}
|
||||
if res.Frequency != 50 {
|
||||
t.Errorf("Frequency = %d, want 50", res.Frequency)
|
||||
}
|
||||
if res.Difficulty != 0.72 {
|
||||
t.Errorf("Difficulty = %v, want 0.72", res.Difficulty)
|
||||
}
|
||||
if !strings.HasPrefix(res.Etymology, "From Medieval Latin ephemerus") {
|
||||
t.Errorf("Etymology = %q, want the Wiktionary text", res.Etymology)
|
||||
}
|
||||
if len(res.Etymology) > maxEtymology {
|
||||
t.Errorf("Etymology not trimmed: %d chars", len(res.Etymology))
|
||||
}
|
||||
}
|
||||
|
||||
func TestDreamGlossFollowsTheWriterNotTheWord(t *testing.T) {
|
||||
dd := openFixture(t)
|
||||
for lang, want := range map[string]string{
|
||||
"pt-PT": "efémero; passageiro",
|
||||
"fr": "éphémère",
|
||||
"es": "", // DreamDict supports Spanish; this database wasn't built with it
|
||||
"de": "", // never a Petal pair, and must not silently borrow another's
|
||||
} {
|
||||
got, err := dreamProvider{dict: dd, native: lang}.Gloss("ephemeral")
|
||||
if err != nil {
|
||||
t.Fatalf("Gloss(%s): %v", lang, err)
|
||||
}
|
||||
if got.Gloss != want {
|
||||
t.Errorf("Gloss for %s = %q, want %q", lang, got.Gloss, want)
|
||||
}
|
||||
if got.Word != "ephemeral" {
|
||||
t.Errorf("Word = %q, want the word as asked", got.Word)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDreamGlossesThroughSharedSynsets(t *testing.T) {
|
||||
// The measurement that drove this: on the real dict.db, Wiktionary's
|
||||
// en→pt-PT translation table answers for 17% of the 2,000 commonest English
|
||||
// words and the shared-synset path answers for 62%. A word with no
|
||||
// translation row must still get a gloss, commonest sense first.
|
||||
p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
|
||||
res, err := p.Lookup("brief")
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup: %v", err)
|
||||
}
|
||||
if res.Gloss != "breve; sucinto" {
|
||||
t.Errorf("Gloss = %q, want the synset equivalents, commonest first", res.Gloss)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDreamDeinflectsToTheHeadword(t *testing.T) {
|
||||
// dict.db stores headwords: "running" has no row of its own. The candidate
|
||||
// walk is what makes a right-click on real prose work at all.
|
||||
p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
|
||||
res, err := p.Lookup("running")
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup: %v", err)
|
||||
}
|
||||
if len(res.Definitions) == 0 || res.Definitions[0].Definition != "move fast on foot" {
|
||||
t.Fatalf("Definitions = %+v, want run's", res.Definitions)
|
||||
}
|
||||
// Every other field must come from the same headword — a popover that mixed
|
||||
// "running"'s (absent) frequency with "run"'s definitions would be lying.
|
||||
if res.Frequency != 900 {
|
||||
t.Errorf("Frequency = %d, want run's 900", res.Frequency)
|
||||
}
|
||||
if res.Difficulty != 0.05 {
|
||||
t.Errorf("Difficulty = %v, want run's 0.05", res.Difficulty)
|
||||
}
|
||||
if res.Synonyms[0] != "sprint" {
|
||||
t.Errorf("Synonyms = %v, want run's", res.Synonyms)
|
||||
}
|
||||
if res.Gloss != "correr" {
|
||||
t.Errorf("Gloss = %q, want run's", res.Gloss)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDreamMissIsAnEmptyResultNotAnError(t *testing.T) {
|
||||
p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
|
||||
res, err := p.Lookup("zzzxqqq")
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup: %v", err)
|
||||
}
|
||||
if len(res.Definitions) != 0 || len(res.Synonyms) != 0 || res.Gloss != "" {
|
||||
t.Errorf("expected an empty result, got %+v", res)
|
||||
}
|
||||
// The frontend renders [] and never null.
|
||||
if res.Definitions == nil || res.Synonyms == nil {
|
||||
t.Errorf("empty slices must be non-nil: %+v", res)
|
||||
}
|
||||
if res.Difficulty != unknownDifficulty {
|
||||
t.Errorf("Difficulty = %v, want the unknown sentinel", res.Difficulty)
|
||||
}
|
||||
// Empty input is a miss, not a crash.
|
||||
if res, err := p.Lookup(" "); err != nil || res.Gloss != "" {
|
||||
t.Errorf("Lookup(blank) = %+v, %v", res, err)
|
||||
}
|
||||
if res, err := p.Gloss(""); err != nil || res.Gloss != "" {
|
||||
t.Errorf("Gloss(empty) = %+v, %v", res, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDreamUnscoredWordKeepsTheUnknownSentinel(t *testing.T) {
|
||||
// "plain" is in the database with no frequency and a NULL difficulty. The
|
||||
// popover must be able to tell that apart from "difficulty 0.0, the easiest
|
||||
// word there is" — which is why the sentinel is -1 and not omitempty.
|
||||
p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
|
||||
res, err := p.Lookup("plain")
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup: %v", err)
|
||||
}
|
||||
if len(res.Definitions) == 0 {
|
||||
t.Fatal("expected plain to be found")
|
||||
}
|
||||
if res.Difficulty != unknownDifficulty {
|
||||
t.Errorf("Difficulty = %v, want the unknown sentinel for a NULL score", res.Difficulty)
|
||||
}
|
||||
if res.Frequency != 0 {
|
||||
t.Errorf("Frequency = %d, want 0 for no count", res.Frequency)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPickIPASkipsMachineFormats(t *testing.T) {
|
||||
if got := pickIPA([]dictionary.Pronunciation{{Format: "cmu", Value: "K AE1 T"}}); got != "" {
|
||||
t.Errorf("pickIPA on CMU alone = %q, want empty — CMU is not for a reader", got)
|
||||
}
|
||||
if got := pickIPA(nil); got != "" {
|
||||
t.Errorf("pickIPA(nil) = %q", got)
|
||||
}
|
||||
if got := pickIPA([]dictionary.Pronunciation{{Format: "IPA", Value: " [kæt] "}}); got != "kæt" {
|
||||
t.Errorf("pickIPA = %q, want the bare IPA regardless of case or brackets", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTrimEtymologyPrefersASentence(t *testing.T) {
|
||||
// A sentence that ends past halfway is the good cut: keep it, drop the rest.
|
||||
long := strings.Repeat("padding word ", 14) + "end. " + strings.Repeat("more ", 40)
|
||||
got := trimEtymology(long)
|
||||
if utf8.RuneCountInString(got) > maxEtymology {
|
||||
t.Errorf("not trimmed: %d runes", utf8.RuneCountInString(got))
|
||||
}
|
||||
if !strings.HasSuffix(got, "end.") {
|
||||
t.Errorf("trimEtymology = %q, want it to stop at the sentence", got)
|
||||
}
|
||||
|
||||
// An early full stop is not a summary — cutting there would throw away
|
||||
// almost the whole line — so this falls through to a word boundary.
|
||||
got = trimEtymology("From Latin. " + strings.Repeat("padding word ", 40))
|
||||
if strings.HasSuffix(got, "Latin.") {
|
||||
t.Errorf("trimEtymology = %q, want more than the first four words", got)
|
||||
}
|
||||
if !strings.HasSuffix(got, "…") {
|
||||
t.Errorf("trimEtymology = %q, want an ellipsis when cut mid-thought", got)
|
||||
}
|
||||
|
||||
// Multi-byte text must be cut on rune boundaries: a byte slice through
|
||||
// ἐφήμερος would put invalid UTF-8 in the JSON.
|
||||
greek := trimEtymology(strings.Repeat("ἐφήμερος ", 60))
|
||||
if !utf8.ValidString(greek) {
|
||||
t.Errorf("trimEtymology produced invalid UTF-8: %q", greek)
|
||||
}
|
||||
if n := utf8.RuneCountInString(greek); n > maxEtymology {
|
||||
t.Errorf("trimmed to %d runes, want at most %d", n, maxEtymology)
|
||||
}
|
||||
|
||||
if strings.Contains(got, " ") || strings.Contains(trimEtymology("a\n b"), "\n") {
|
||||
t.Error("whitespace should be collapsed to a single line")
|
||||
}
|
||||
// Short text passes through untouched.
|
||||
if got := trimEtymology("From Old English."); got != "From Old English." {
|
||||
t.Errorf("trimEtymology = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// --- the Set: which provider answers, and what happens when dict.db is absent
|
||||
|
||||
func TestSetRoutesByPairLanguage(t *testing.T) {
|
||||
set := NewSet(openFixture(t))
|
||||
if !set.HasDreamDict() {
|
||||
t.Fatal("HasDreamDict = false with a dictionary open")
|
||||
}
|
||||
// zh — and an empty column, which is what a pre-auth row reads as — stays
|
||||
// on the embedded datasets until the two have been compared on real
|
||||
// lookups. This test is the guard on that decision.
|
||||
for _, lang := range []string{"", LangZh} {
|
||||
if _, ok := set.For(lang).(*Lexicon); !ok {
|
||||
t.Errorf("For(%q) = %T, want the embedded Lexicon", lang, set.For(lang))
|
||||
}
|
||||
}
|
||||
for _, lang := range []string{"pt-PT", "fr", "es"} {
|
||||
p, ok := set.For(lang).(dreamProvider)
|
||||
if !ok {
|
||||
t.Fatalf("For(%q) = %T, want DreamDict", lang, set.For(lang))
|
||||
}
|
||||
if p.native != lang {
|
||||
t.Errorf("For(%q) glosses into %q", lang, p.native)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSetWithoutDictKeepsTheEnglishHalf(t *testing.T) {
|
||||
// The interesting degradation: dict.db never got deployed. A pt-PT writer
|
||||
// should still get definitions, synonyms and phonetics — all compiled into
|
||||
// the binary and all correct for her — and lose only the translation.
|
||||
set := NewSet(nil)
|
||||
if set.HasDreamDict() {
|
||||
t.Fatal("HasDreamDict = true with no dictionary")
|
||||
}
|
||||
p := set.For("pt-PT")
|
||||
res, err := p.Lookup("happy")
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup: %v", err)
|
||||
}
|
||||
if len(res.Definitions) == 0 || len(res.Synonyms) == 0 {
|
||||
t.Error("expected the embedded English half to survive a missing dict.db")
|
||||
}
|
||||
if res.Gloss != "" {
|
||||
t.Errorf("Gloss = %q — a pt-PT writer must never be handed the Chinese gloss", res.Gloss)
|
||||
}
|
||||
g, err := p.Gloss("happy")
|
||||
if err != nil {
|
||||
t.Fatalf("Gloss: %v", err)
|
||||
}
|
||||
if g.Gloss != "" {
|
||||
t.Errorf("Gloss = %q, want empty", g.Gloss)
|
||||
}
|
||||
if g.Word != "happy" {
|
||||
t.Errorf("Word = %q, want the word as asked", g.Word)
|
||||
}
|
||||
// The zh writer is untouched by any of this.
|
||||
zh, err := set.For(LangZh).Lookup("happy")
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup(zh): %v", err)
|
||||
}
|
||||
if zh.Gloss == "" {
|
||||
t.Error("the zh pair must keep its embedded gloss with no dict.db")
|
||||
}
|
||||
}
|
||||
|
||||
// --- the handler: the pair language is read per request, from the caller's row
|
||||
|
||||
func mountLexicon(t *testing.T, set *Set) (*chi.Mux, *db.DB) {
|
||||
t.Helper()
|
||||
database, err := db.Open(filepath.Join(t.TempDir(), "petal.db"))
|
||||
if err != nil {
|
||||
t.Fatalf("db.Open: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { database.Close() })
|
||||
|
||||
for _, u := range []struct{ id, lang string }{
|
||||
{"alice", LangZh}, {"bob", "pt-PT"},
|
||||
} {
|
||||
if _, err := database.Exec(
|
||||
`INSERT INTO users (id, email, display_name, pair_lang) VALUES (?, ?, ?, ?)`,
|
||||
u.id, u.id+"@example.com", u.id, u.lang,
|
||||
); err != nil {
|
||||
t.Fatalf("seed user: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
h := NewHandler(database.DB, set)
|
||||
r := chi.NewMux()
|
||||
r.Mount("/word", h.Routes())
|
||||
r.Mount("/gloss", h.GlossRoutes())
|
||||
return r, database
|
||||
}
|
||||
|
||||
func getAs(t *testing.T, r http.Handler, userID, path string) Result {
|
||||
t.Helper()
|
||||
req := httptest.NewRequest(http.MethodGet, path, nil)
|
||||
req = req.WithContext(auth.WithUser(req.Context(), userID))
|
||||
rec := httptest.NewRecorder()
|
||||
r.ServeHTTP(rec, req)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("GET %s as %s = %d: %s", path, userID, rec.Code, rec.Body)
|
||||
}
|
||||
var res Result
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &res); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
func TestHandlerGlossesInTheCallersLanguage(t *testing.T) {
|
||||
r, _ := mountLexicon(t, NewSet(openFixture(t)))
|
||||
|
||||
// Same URL, two writers, two languages. This is why the response is no
|
||||
// longer cacheable as `public`.
|
||||
bob := getAs(t, r, "bob", "/word/ephemeral")
|
||||
if bob.Gloss != "efémero; passageiro" {
|
||||
t.Errorf("bob's gloss = %q, want pt-PT", bob.Gloss)
|
||||
}
|
||||
alice := getAs(t, r, "alice", "/word/ephemeral")
|
||||
if !strings.ContainsAny(alice.Gloss, "短暂的") && alice.Gloss != "" {
|
||||
// alice is on the embedded ECDICT dataset, not the fixture's zh row —
|
||||
// what matters is that she is *not* served bob's Portuguese.
|
||||
t.Logf("alice's embedded gloss: %q", alice.Gloss)
|
||||
}
|
||||
if alice.Gloss == bob.Gloss && bob.Gloss != "" {
|
||||
t.Error("the zh writer was served the pt-PT gloss")
|
||||
}
|
||||
if strings.Contains(alice.Gloss, "efémero") {
|
||||
t.Errorf("alice's gloss = %q, want the embedded Chinese one", alice.Gloss)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerUnknownCallerFallsBackRatherThanFailing(t *testing.T) {
|
||||
// No session, or a user row that has gone: the lookup still answers, from
|
||||
// the embedded datasets. A dictionary that fails closed would be worse than
|
||||
// one that answers in the wrong language, because nothing at all is not a
|
||||
// dictionary.
|
||||
r, _ := mountLexicon(t, NewSet(openFixture(t)))
|
||||
res := getAs(t, r, "nobody", "/word/happy")
|
||||
if len(res.Definitions) == 0 {
|
||||
t.Error("expected the embedded fallback to answer for an unknown caller")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerCachesPrivately(t *testing.T) {
|
||||
r, _ := mountLexicon(t, NewSet(nil))
|
||||
req := httptest.NewRequest(http.MethodGet, "/gloss/happy", nil)
|
||||
req = req.WithContext(auth.WithUser(req.Context(), "alice"))
|
||||
rec := httptest.NewRecorder()
|
||||
r.ServeHTTP(rec, req)
|
||||
if got := rec.Header().Get("Cache-Control"); !strings.HasPrefix(got, "private") {
|
||||
t.Errorf("Cache-Control = %q — a per-writer gloss must not go in a shared cache", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerDecodesPunctuatedWords(t *testing.T) {
|
||||
r, _ := mountLexicon(t, NewSet(openFixture(t)))
|
||||
res := getAs(t, r, "bob", "/word/"+"caf%C3%A9")
|
||||
if res.Word != "café" {
|
||||
t.Errorf("Word = %q, want the decoded word", res.Word)
|
||||
}
|
||||
}
|
||||
@@ -1,20 +1,27 @@
|
||||
package lexicon
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/url"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
"gitea.parodia.dev/drwily/petal/internal/auth"
|
||||
)
|
||||
|
||||
// Handler serves the word-lookup endpoint backed by a single shared Lexicon.
|
||||
// Handler serves the word-lookup endpoints. It holds the shared provider Set
|
||||
// and the database, because which provider answers depends on who is asking.
|
||||
type Handler struct {
|
||||
Lex *Lexicon
|
||||
Set *Set
|
||||
db *sql.DB
|
||||
}
|
||||
|
||||
// New constructs a Handler with a fresh (lazily-loaded) Lexicon.
|
||||
func NewHandler() *Handler { return &Handler{Lex: New()} }
|
||||
// NewHandler constructs a Handler over a provider Set. db is used for one
|
||||
// thing: reading the caller's pair language.
|
||||
func NewHandler(db *sql.DB, set *Set) *Handler { return &Handler{Set: set, db: db} }
|
||||
|
||||
// Routes returns the router mounted at /api/word. The word is a path segment so
|
||||
// "/api/word/happy" reads naturally; it's URL-decoded to tolerate the rare
|
||||
@@ -26,56 +33,81 @@ func (h *Handler) Routes() chi.Router {
|
||||
}
|
||||
|
||||
// GlossRoutes returns the router mounted at /api/gloss — the lightweight
|
||||
// Chinese-only lookup behind the inline hover/select gloss. It shares the
|
||||
// Handler's Lexicon, so the datasets still load just once.
|
||||
// translation-only lookup behind the inline hover/select gloss. It shares the
|
||||
// Handler's Set, so the embedded datasets and dict.db are still opened once.
|
||||
func (h *Handler) GlossRoutes() chi.Router {
|
||||
r := chi.NewRouter()
|
||||
r.Get("/{word}", h.gloss)
|
||||
return r
|
||||
}
|
||||
|
||||
// lookup returns the definition + synonyms for one word. A word found in neither
|
||||
// dataset still returns 200 with empty lists, so the popover can show a friendly
|
||||
// "nothing found" rather than an error state.
|
||||
// providerFor returns the provider for the caller's language pair.
|
||||
//
|
||||
// The pair language is read here rather than threaded down because a word
|
||||
// lookup has no other query to piggyback on — unlike the document handlers,
|
||||
// which take pair_lang from the row-scoped query that already proves
|
||||
// ownership. It is one indexed primary-key read against a local SQLite file,
|
||||
// which costs less than encoding the response it feeds.
|
||||
//
|
||||
// A read that fails, or a caller with no user row, resolves to the empty
|
||||
// language, and [Set.For] maps that to today's embedded behaviour. Falling back
|
||||
// to a working dictionary beats failing the lookup.
|
||||
func (h *Handler) providerFor(ctx context.Context) Provider {
|
||||
var lang string
|
||||
if h.db != nil {
|
||||
_ = h.db.QueryRowContext(ctx,
|
||||
`SELECT COALESCE(pair_lang, '') FROM users WHERE id = ?`,
|
||||
auth.UserID(ctx),
|
||||
).Scan(&lang)
|
||||
}
|
||||
return h.Set.For(lang)
|
||||
}
|
||||
|
||||
// pathWord reads the {word} segment, URL-decoded.
|
||||
func pathWord(r *http.Request) string {
|
||||
word := chi.URLParam(r, "word")
|
||||
if decoded, err := url.PathUnescape(word); err == nil {
|
||||
word = decoded
|
||||
}
|
||||
return word
|
||||
}
|
||||
|
||||
// lookup returns the definition + synonyms for one word. A word found in no
|
||||
// dataset still returns 200 with empty lists, so the popover can show a
|
||||
// friendly "nothing found" rather than an error state.
|
||||
func (h *Handler) lookup(w http.ResponseWriter, r *http.Request) {
|
||||
word := chi.URLParam(r, "word")
|
||||
if decoded, err := url.PathUnescape(word); err == nil {
|
||||
word = decoded
|
||||
}
|
||||
|
||||
res, err := h.Lex.Lookup(word)
|
||||
res, err := h.providerFor(r.Context()).Lookup(pathWord(r))
|
||||
if err != nil {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"error": err.Error()})
|
||||
writeLookupErr(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
// Word lookups are static for the life of the build; let the browser cache
|
||||
// them so repeated right-clicks on the same word are instant.
|
||||
w.Header().Set("Cache-Control", "public, max-age=86400")
|
||||
_ = json.NewEncoder(w).Encode(res)
|
||||
writeLookup(w, res)
|
||||
}
|
||||
|
||||
// gloss returns just the Chinese translation for one word. Like lookup, a miss
|
||||
// is a 200 with an empty gloss so the hover tooltip can quietly skip rather than
|
||||
// error.
|
||||
// gloss returns just the translation for one word. Like lookup, a miss is a 200
|
||||
// with an empty gloss so the hover tooltip can quietly skip rather than error.
|
||||
func (h *Handler) gloss(w http.ResponseWriter, r *http.Request) {
|
||||
word := chi.URLParam(r, "word")
|
||||
if decoded, err := url.PathUnescape(word); err == nil {
|
||||
word = decoded
|
||||
}
|
||||
|
||||
res, err := h.Lex.Gloss(word)
|
||||
res, err := h.providerFor(r.Context()).Gloss(pathWord(r))
|
||||
if err != nil {
|
||||
writeLookupErr(w, err)
|
||||
return
|
||||
}
|
||||
writeLookup(w, res)
|
||||
}
|
||||
|
||||
func writeLookupErr(w http.ResponseWriter, err error) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
func writeLookup(w http.ResponseWriter, v any) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.Header().Set("Cache-Control", "public, max-age=86400")
|
||||
_ = json.NewEncoder(w).Encode(res)
|
||||
// A lookup is stable for the life of the deployment, so let the browser
|
||||
// keep it — repeated right-clicks on the same word are then instant. It is
|
||||
// `private` rather than `public` because the gloss is now in *her*
|
||||
// language: a shared cache keyed on the URL alone would hand one writer
|
||||
// another writer's language.
|
||||
w.Header().Set("Cache-Control", "private, max-age=86400")
|
||||
_ = json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
|
||||
@@ -29,8 +29,30 @@ type Result struct {
|
||||
Phonetic string `json:"phonetic"` // IPA for the English word; "" when absent
|
||||
Definitions []Meaning `json:"definitions"`
|
||||
Synonyms []string `json:"synonyms"`
|
||||
|
||||
// The fields below only ever come from DreamDict; the embedded datasets
|
||||
// leave them at their unknown values, and the popover hides them.
|
||||
|
||||
// Frequency is how common the word is (higher = more common). 0 means
|
||||
// unknown, which is DreamDict's own convention — a word it carries but has
|
||||
// no corpus count for is indistinguishable from a word it doesn't carry,
|
||||
// and the popover treats both the same way.
|
||||
Frequency int `json:"frequency"`
|
||||
// Difficulty runs 0.0 (easiest) to 1.0 (hardest); -1 means unknown. It is a
|
||||
// sentinel rather than an omitted field because 0.0 is a real, meaningful
|
||||
// score and `omitempty` would erase it.
|
||||
Difficulty float64 `json:"difficulty"`
|
||||
// Etymology is free-form Wiktionary prose, trimmed to a line. Where the
|
||||
// word came from is a real hook for a writer whose own language shares
|
||||
// Latin roots with English — "ephemeral" is much easier to keep once you
|
||||
// have seen efémero next to it.
|
||||
Etymology string `json:"etymology"`
|
||||
}
|
||||
|
||||
// unknownDifficulty is the [Result.Difficulty] value meaning "no score",
|
||||
// matching DreamDict's own -1 return.
|
||||
const unknownDifficulty = -1
|
||||
|
||||
// GlossResult is the lightweight payload for the inline hover/select gloss: just
|
||||
// the word and its Chinese translation, no definitions or synonyms. Kept small
|
||||
// so the hover tooltip is instant and trivially cacheable.
|
||||
@@ -95,7 +117,7 @@ func (l *Lexicon) Lookup(word string) (Result, error) {
|
||||
}
|
||||
|
||||
norm := strings.ToLower(strings.TrimSpace(word))
|
||||
res := Result{Word: word, Definitions: []Meaning{}, Synonyms: []string{}}
|
||||
res := Result{Word: word, Definitions: []Meaning{}, Synonyms: []string{}, Difficulty: unknownDifficulty}
|
||||
if norm == "" {
|
||||
return res, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
package lexicon
|
||||
|
||||
// A word lookup used to mean exactly one thing: the embedded datasets, which
|
||||
// speak English and Mandarin and nothing else. That was fine while Petal had
|
||||
// one writer. It stops being fine the moment a pt-PT writer right-clicks a
|
||||
// word and gets a Chinese gloss.
|
||||
//
|
||||
// So the lookup becomes a seam. A [Provider] answers the same two questions the
|
||||
// popover and the hover tooltip have always asked; which provider answers them
|
||||
// depends on the writer's language pair, and [Set.For] is the only place that
|
||||
// decision is made.
|
||||
|
||||
// Provider answers word lookups for one writer. The embedded datasets and
|
||||
// DreamDict both satisfy it, and both treat a word they don't carry as an empty
|
||||
// result rather than an error — a miss is an ordinary outcome of looking a word
|
||||
// up, not a failure.
|
||||
type Provider interface {
|
||||
// Lookup returns the full popover payload: gloss, phonetic, definitions,
|
||||
// synonyms, and whatever extras the provider carries.
|
||||
Lookup(word string) (Result, error)
|
||||
// Gloss returns just the writer's-language translation. It is the hover
|
||||
// tooltip's fast path and skips everything else.
|
||||
Gloss(word string) (GlossResult, error)
|
||||
}
|
||||
|
||||
// LangZh is the one pair language still served by the embedded datasets. Every
|
||||
// other pair goes to DreamDict — see [Set.For] for why zh is held back.
|
||||
const LangZh = "zh"
|
||||
|
||||
// langEN is the language DreamDict is asked about for definitions, synonyms and
|
||||
// pronunciation. English is always the *target* language of the pair — what
|
||||
// varies is the language the gloss is written in.
|
||||
const langEN = "en"
|
||||
|
||||
// Set holds every provider Petal can serve a lookup from and picks between them
|
||||
// by pair language. One Set is shared by the whole process: the embedded
|
||||
// datasets load once, and dict.db is one read-only handle.
|
||||
type Set struct {
|
||||
embedded *Lexicon
|
||||
// dream is nil when dict.db was not deployed. That is a supported state,
|
||||
// not an error — see [Set.For].
|
||||
dream *DreamDict
|
||||
}
|
||||
|
||||
// NewSet returns a Set backed by the embedded datasets and, when dream is
|
||||
// non-nil, DreamDict. Passing a nil dream is how Petal runs without dict.db.
|
||||
func NewSet(dream *DreamDict) *Set {
|
||||
return &Set{embedded: New(), dream: dream}
|
||||
}
|
||||
|
||||
// HasDreamDict reports whether a dict.db is open. Only startup logging and
|
||||
// tests care; a handler never asks, because [Set.For] always returns something
|
||||
// usable.
|
||||
func (s *Set) HasDreamDict() bool { return s.dream != nil }
|
||||
|
||||
// For returns the provider that should answer lookups for a writer whose pair
|
||||
// language is lang.
|
||||
//
|
||||
// Three rules, in order:
|
||||
//
|
||||
// zh — and an empty code, which is what a pre-Phase-16 row reads as — stays on
|
||||
// the embedded ECDICT gloss. Not because DreamDict lacks Chinese (it has
|
||||
// CC-CEDICT), but because that path is in daily use by a real writer and the
|
||||
// two have not yet been compared on her actual lookups. Switching it is a
|
||||
// quality decision, and it hasn't been made.
|
||||
//
|
||||
// Any other pair goes to DreamDict, which is the only source that has pt-PT,
|
||||
// French or Spanish at all.
|
||||
//
|
||||
// If dict.db was never deployed, a non-zh writer falls back to the embedded
|
||||
// datasets with the gloss suppressed. This is the interesting case: the naive
|
||||
// "no data" answer would blank the popover entirely, when in fact the English
|
||||
// half of it — definitions, synonyms, phonetic — is compiled into the binary
|
||||
// and perfectly correct for her. Only the translation is missing, so only the
|
||||
// translation goes missing. A failed dictionary deploy costs her the gloss, not
|
||||
// the dictionary.
|
||||
func (s *Set) For(lang string) Provider {
|
||||
if lang == "" || lang == LangZh {
|
||||
return s.embedded
|
||||
}
|
||||
if s.dream != nil {
|
||||
return dreamProvider{dict: s.dream, native: lang}
|
||||
}
|
||||
return glossless{s.embedded}
|
||||
}
|
||||
|
||||
// glossless serves the embedded datasets with the Chinese gloss stripped, for a
|
||||
// writer who does not read Chinese. Handing her the zh gloss would be worse
|
||||
// than handing her nothing: an empty field reads as "not found", where the
|
||||
// wrong language reads as Petal being broken.
|
||||
type glossless struct{ inner Provider }
|
||||
|
||||
func (g glossless) Lookup(word string) (Result, error) {
|
||||
res, err := g.inner.Lookup(word)
|
||||
res.Gloss = ""
|
||||
return res, err
|
||||
}
|
||||
|
||||
func (g glossless) Gloss(word string) (GlossResult, error) {
|
||||
return GlossResult{Word: word}, nil
|
||||
}
|
||||
@@ -68,10 +68,16 @@ export interface WordMeaning {
|
||||
// a list of synonyms. Any of these may be empty when the word isn't a headword.
|
||||
export interface WordInfo {
|
||||
word: string
|
||||
gloss: string // Chinese translation; '' when the word isn't in the gloss set
|
||||
gloss: string // translation into the writer's language; '' when absent
|
||||
phonetic: string // IPA for the English word; '' when absent
|
||||
definitions: WordMeaning[]
|
||||
synonyms: string[]
|
||||
// From DreamDict only; the embedded datasets leave them unknown. `frequency`
|
||||
// is 0 and `difficulty` is -1 when the dictionary has no score — see
|
||||
// wordBand, which turns the pair into a band or into nothing at all.
|
||||
frequency: number
|
||||
difficulty: number
|
||||
etymology: string // free-form, already trimmed to a line by the server; '' when absent
|
||||
}
|
||||
|
||||
// The lightweight Chinese-only gloss behind the inline hover/select tooltip.
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import type { WordInfo } from '../../api/client'
|
||||
import { speak, speechSupported } from '../../audio/speech'
|
||||
import { usePack } from '../../i18n'
|
||||
import { wordBand } from './wordband'
|
||||
|
||||
// WordCard is the right-click popover for any word: its dictionary definition(s)
|
||||
// on top and tappable synonym pills below. Clicking a synonym replaces the word
|
||||
@@ -26,6 +27,9 @@ export function WordCard({ word, info, loading, saved, onToggleSave, style, onRe
|
||||
const synonyms = info?.synonyms ?? []
|
||||
const gloss = info?.gloss ?? ''
|
||||
const phonetic = info?.phonetic ?? ''
|
||||
const etymology = info?.etymology ?? ''
|
||||
// Null whenever the dictionary has no opinion — the chip then doesn't render.
|
||||
const band = info ? wordBand(info.frequency ?? 0, info.difficulty ?? -1) : null
|
||||
const empty = !loading && !gloss && definitions.length === 0 && synonyms.length === 0
|
||||
|
||||
return (
|
||||
@@ -85,12 +89,27 @@ export function WordCard({ word, info, loading, saved, onToggleSave, style, onRe
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* How to say it — the pronunciation aid for an English learner, paired
|
||||
with the 🔊 button above. */}
|
||||
{phonetic && (
|
||||
<p className="mt-1.5 text-sm" style={{ color: 'var(--color-muted)' }}>
|
||||
/{phonetic}/
|
||||
</p>
|
||||
{/* How to say it, and how hard it is. The pronunciation aid pairs with the
|
||||
🔊 button above; the band answers the question a learner actually has
|
||||
when she has found a word she likes — "can I use this?". Both are
|
||||
quiet, muted lines: information she can take or leave, never a verdict
|
||||
on her writing. */}
|
||||
{(phonetic || band) && (
|
||||
<div className="mt-1.5 flex items-center gap-2 text-sm">
|
||||
{phonetic && <span style={{ color: 'var(--color-muted)' }}>/{phonetic}/</span>}
|
||||
{band && (
|
||||
<span
|
||||
className="rounded-full px-2 py-0.5 text-xs font-semibold"
|
||||
title={t.editor.wordBands[band].en}
|
||||
style={{
|
||||
background: 'var(--color-surface-alt)',
|
||||
color: band === 'advanced' ? 'var(--color-accent-hover)' : 'var(--color-muted)',
|
||||
}}
|
||||
>
|
||||
{t.editor.wordBands[band].native}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Chinese gloss first — it's what the Mandarin-speaking writer reaches for. */}
|
||||
@@ -165,6 +184,21 @@ export function WordCard({ word, info, loading, saved, onToggleSave, style, onRe
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Where the word came from. Last, and in small muted type, because it is
|
||||
the one thing here that is interesting rather than useful — and for a
|
||||
writer whose own language shares Latin roots with English, "efémero"
|
||||
sitting under "ephemeral" is how a word stops needing to be memorised. */}
|
||||
{etymology && (
|
||||
<div className="mt-3">
|
||||
<p className="mb-1 text-xs font-bold" style={{ color: 'var(--color-muted)' }}>
|
||||
{t.editor.origin}
|
||||
</p>
|
||||
<p className="text-xs leading-snug" style={{ color: 'var(--color-muted)' }}>
|
||||
{etymology}
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{empty && (
|
||||
<p className="mt-3 leading-snug" style={{ color: 'var(--color-muted)' }}>
|
||||
{t.editor.nothingFound}
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { wordBand } from './wordband'
|
||||
|
||||
describe('wordBand', () => {
|
||||
it('says nothing when the dictionary knows nothing', () => {
|
||||
// The embedded datasets carry no scores at all, and this is the common
|
||||
// case for a zh-pair writer. An unknown word must produce no chip rather
|
||||
// than a default one — "standard" would be an invention.
|
||||
expect(wordBand(0, -1)).toBeNull()
|
||||
})
|
||||
|
||||
it('bands the words a writer actually looks up', () => {
|
||||
// Real scores from the deployed dict.db. These are the sanity checks that
|
||||
// would catch a threshold drifting away from the data.
|
||||
expect(wordBand(1000, 0.304)).toBe('simple') // cat
|
||||
expect(wordBand(1000, 0.326)).toBe('simple') // house, write
|
||||
expect(wordBand(1000, 0.37)).toBe('simple') // beautiful
|
||||
expect(wordBand(600, 0.462)).toBe('standard') // ephemeral
|
||||
expect(wordBand(400, 0.53)).toBe('standard') // serendipity
|
||||
expect(wordBand(50, 0.8)).toBe('advanced') // antidisestablishmentarianism
|
||||
})
|
||||
|
||||
it('prefers difficulty over frequency when both are known', () => {
|
||||
// Difficulty is the finer signal — 206 distinct values against frequency's
|
||||
// handful of buckets — so a hard word with a high frequency reads as hard.
|
||||
expect(wordBand(1000, 0.9)).toBe('advanced')
|
||||
expect(wordBand(2, 0.2)).toBe('simple')
|
||||
})
|
||||
|
||||
it('falls back to frequency for a word with no difficulty score', () => {
|
||||
expect(wordBand(1000, -1)).toBe('simple')
|
||||
expect(wordBand(500, -1)).toBe('standard')
|
||||
expect(wordBand(10, -1)).toBe('advanced')
|
||||
})
|
||||
|
||||
it('treats a difficulty of exactly zero as a score, not as missing', () => {
|
||||
// 0.0 is the easiest word there is. The API sends -1 for unknown precisely
|
||||
// so this case survives; a falsy check here would throw it away.
|
||||
expect(wordBand(0, 0)).toBe('simple')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,53 @@
|
||||
// How hard is this word?
|
||||
//
|
||||
// DreamDict scores every headword for difficulty (0.0 easiest → 1.0 hardest,
|
||||
// computed from frequency, length and syllable count) and for raw frequency.
|
||||
// Neither number means anything to a writer — 0.462 is not an answer to "should
|
||||
// I use this word?" — so the popover shows a band instead.
|
||||
//
|
||||
// Like gradeBand in the status bar, this returns the band's *name*. The wording
|
||||
// belongs to the writer's langpack.
|
||||
|
||||
export type WordBand = 'simple' | 'standard' | 'advanced'
|
||||
|
||||
// The cuts come from the real dictionary rather than from round numbers. Across
|
||||
// 136k English headwords the difficulty scores bunch between 0.45 and 0.60 —
|
||||
// that band is the fifth percentile to the eightieth — because a dictionary is
|
||||
// mostly words nobody uses. The words a writer actually reaches for sit well
|
||||
// below it: the, cat, house, write and beautiful all score under 0.40.
|
||||
//
|
||||
// Three bands, not five. The score is a heuristic over word length and corpus
|
||||
// counts, and it is good enough to separate "everyday" from "you will need to
|
||||
// explain this" — not good enough to rank obfuscate against serendipity. A
|
||||
// finer scale would be a confident-looking lie.
|
||||
const SIMPLE_MAX = 0.42
|
||||
const STANDARD_MAX = 0.58
|
||||
|
||||
// Frequency is the fallback for a word DreamDict carries with no difficulty
|
||||
// score. It is heavily bucketed in the data (400/600/800/1000 hold most of the
|
||||
// mass), so these compare against buckets, not a continuum.
|
||||
const FREQ_SIMPLE_MIN = 800
|
||||
const FREQ_STANDARD_MIN = 400
|
||||
|
||||
/**
|
||||
* wordBand picks the band for a looked-up word, or null when the dictionary
|
||||
* knows nothing about how hard it is — the embedded datasets never do, and
|
||||
* neither does DreamDict for a word it has no score for. Null means the chip
|
||||
* doesn't render at all: silence is better than a guess dressed as a fact.
|
||||
*
|
||||
* `difficulty` is -1 when unknown and `frequency` is 0 when unknown, matching
|
||||
* what the API sends.
|
||||
*/
|
||||
export function wordBand(frequency: number, difficulty: number): WordBand | null {
|
||||
if (difficulty >= 0) {
|
||||
if (difficulty <= SIMPLE_MAX) return 'simple'
|
||||
if (difficulty <= STANDARD_MAX) return 'standard'
|
||||
return 'advanced'
|
||||
}
|
||||
if (frequency > 0) {
|
||||
if (frequency >= FREQ_SIMPLE_MIN) return 'simple'
|
||||
if (frequency >= FREQ_STANDARD_MIN) return 'standard'
|
||||
return 'advanced'
|
||||
}
|
||||
return null
|
||||
}
|
||||
@@ -119,4 +119,24 @@ describe('the zh pack', () => {
|
||||
expect(zh.styles[style.value], `no label for style ${style.value}`).toBeTruthy()
|
||||
}
|
||||
})
|
||||
|
||||
it('labels every word band the popover can show', async () => {
|
||||
// wordBand returns a band name, never a label — an unlabelled band would
|
||||
// render as an empty chip, which reads as a bug rather than as no data.
|
||||
const { wordBand } = await import('../components/Editor/wordband')
|
||||
const bands = new Set(
|
||||
[
|
||||
wordBand(0, 0.1),
|
||||
wordBand(0, 0.5),
|
||||
wordBand(0, 0.9),
|
||||
wordBand(1000, -1),
|
||||
wordBand(500, -1),
|
||||
wordBand(10, -1),
|
||||
].filter((b) => b !== null),
|
||||
)
|
||||
expect(bands.size).toBe(3)
|
||||
for (const band of bands) {
|
||||
expect(zh.editor.wordBands[band], `no label for word band ${band}`).toBeTruthy()
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
@@ -174,6 +174,12 @@ export const zh: Pack = {
|
||||
synonyms: '近义词 · Synonyms',
|
||||
tapToSwap: '点击替换 · tap to swap',
|
||||
nothingFound: '没有找到这个词 · Nothing found for this word',
|
||||
origin: '词源 · Origin',
|
||||
wordBands: {
|
||||
simple: { native: '常用词', en: 'Everyday word' },
|
||||
standard: { native: '一般难度', en: 'Standard' },
|
||||
advanced: { native: '进阶词', en: 'Advanced' },
|
||||
},
|
||||
},
|
||||
|
||||
styles: {
|
||||
|
||||
@@ -134,6 +134,11 @@ export interface Pack {
|
||||
synonyms: string
|
||||
tapToSwap: string
|
||||
nothingFound: string
|
||||
// Where the word came from — a real hook for a writer whose own language
|
||||
// shares roots with English.
|
||||
origin: string
|
||||
// How hard the word is, keyed by the band wordBand() returns.
|
||||
wordBands: Record<string, Line>
|
||||
}
|
||||
|
||||
// Rewrite styles and document tones share a vocabulary; both are Line-labelled
|
||||
|
||||
Reference in New Issue
Block a user