Phase 20: the dictionary stops being English and Chinese only

Word lookups now come from DreamDict's dict.db for every pair but Chinese —
opened read-only beside petal.db, no service, nothing over the VPN, because a
hover gloss has to answer in milliseconds.

`Provider` is the two questions the popover and the tooltip already asked, so
the embedded *Lexicon satisfies it with no changes at all; Set.For(lang) is the
single place the choice between them is made. The prerequisite in the dreamdict
repo turned out to be two things, not one: the module path was unfetchable
*and* the query layer sat in internal/, which no other module may import
whatever the module is called. Both fixed upstream.

The plan's central assumption did not survive the data. It mapped
Gloss ← Translate(word, "en", L1) one-to-one; against the real 452 MB database
that table answers for 17% of the 2,000 commonest English words into pt-PT.
Wiktionary's translation sections are thin in that direction — "ephemeral",
"think" and "quickly" have no en→pt-PT row at all. Shared WordNet synsets
answer for 61%, so DreamDict gained Equivalents() and Petal glosses through it.
Ordering those was wrong in an instructive way too: sorting by frequency
glosses "think" as lembrar, "remember", because lembrar is the commoner
Portuguese word even though pensar shares six of think's synsets to lembrar's
one. Counting sense agreement first asks the right question.

The same measurement is why zh stays on ECDICT: DreamDict reaches a Chinese
gloss for 53% of those words, ECDICT for nearly all of them. The plan said
converge only if quality holds. It didn't, so nothing converged.

Two decisions about failure worth keeping. A missing dict.db is not an error —
a laptop checkout has never had one — but a present-and-never-imported one is,
because that is a half-finished deploy. And a pt-PT writer with no dictionary
falls back to the embedded datasets with the gloss suppressed, keeping
definitions, synonyms and phonetics rather than blanking the popover: an empty
field reads as "not found", the wrong language reads as broken.

The new fields surface as an etymology line and a three-band chip. Three, not
five: the difficulty score separates "everyday" from "you'll have to explain
this" but cannot rank obfuscate against serendipity, and a finer scale would be
a confident-looking lie. An unscored word gets no chip.

Writing the tests found two bugs first — trimEtymology sliced by byte, which
would have emitted invalid UTF-8 for exactly the Greek and Latin etymologies
the feature exists for, and its ellipsis path overran its own cap.

go build/vet/test, tsc, vite, vitest 96/96 clean; live smoke against the real
dict.db with one instance flipped from zh to pt-PT mid-run.

Not deployed: go.mod still replaces github.com/prosolis/dreamdict with
../dreamdict, so the Docker build needs the two upstream commits pushed and the
replace dropped. The deployed dict.db also predates DreamDict's Spanish data.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua

internal/lexicon/dreamdict.go

@@ -0,0 +1,238 @@
+package lexicon
+
+import (
+	"errors"
+	"io/fs"
+	"os"
+	"strings"
+	"unicode/utf8"
+
+	"github.com/prosolis/dreamdict/dictionary"
+)
+
+// DreamDict is a read-only handle on a built dict.db — one SQLite file holding
+// English, French, European Portuguese, Spanish and Mandarin.
+//
+// It is a second database beside petal.db and is never written to: the file is
+// built by DreamDict's own import CLI a few times a year, and Petal only reads
+// it. That is what makes importing the package the right shape rather than
+// running DreamDict as a service — a hover gloss should not depend on a second
+// process being up, still less on one reachable across a VPN.
+type DreamDict struct {
+	d *dictionary.Dictionary
+}
+
+// OpenDreamDict opens dict.db read-only.
+//
+// A missing file returns (nil, nil), not an error. Petal is expected to run
+// without dict.db — a laptop checkout has never had one, and the zh pair does
+// not need one — so "the file isn't there" is a deployment state the caller
+// handles by carrying on. A file that is *present but unusable* (corrupt, or
+// never imported) does return an error, because that one is a mistake someone
+// should hear about.
+func OpenDreamDict(path string) (*DreamDict, error) {
+	if strings.TrimSpace(path) == "" {
+		return nil, nil
+	}
+	if _, err := os.Stat(path); err != nil {
+		if errors.Is(err, fs.ErrNotExist) {
+			return nil, nil
+		}
+		return nil, err
+	}
+	d, err := dictionary.NewReadOnly(path)
+	if err != nil {
+		return nil, err
+	}
+	return &DreamDict{d: d}, nil
+}
+
+// Close releases the dict.db handle. Safe on a nil DreamDict, so a caller that
+// never got one can defer it unconditionally.
+func (dd *DreamDict) Close() error {
+	if dd == nil {
+		return nil
+	}
+	return dd.d.Close()
+}
+
+// Langs returns the language codes dict.db was built with, so startup can log
+// what it actually got rather than what it hoped for.
+func (dd *DreamDict) Langs() []string { return dictionary.Langs() }
+
+// dreamProvider serves one writer: English lookups from dict.db, glossed into
+// native. The struct is a value, created per request by [Set.For] — it holds no
+// state beyond the shared handle and the language to translate into.
+type dreamProvider struct {
+	dict   *DreamDict
+	native string // the writer's language, e.g. "pt-PT"
+}
+
+// maxEtymology caps the free-form Wiktionary etymology. It is the one field
+// with no natural length: some entries are a clause, some are four paragraphs
+// tracing a word through three dead languages. The popover wants a line.
+const maxEtymology = 220
+
+// Lookup fills a Result from dict.db.
+//
+// The word is de-inflected with the same [candidates] walk the embedded
+// datasets use, because dict.db stores headwords: "running" has no definitions
+// row of its own. The first candidate that *has* definitions becomes the
+// headword every other field is then read from, so a single popover never
+// mixes "running"'s frequency with "run"'s definitions.
+//
+// The gloss is walked separately. A word can be absent from the definitions
+// table and still have a translation (and vice versa), and the hover tooltip
+// asks for the gloss alone — so tying it to the definition headword would lose
+// glosses for no benefit.
+func (p dreamProvider) Lookup(word string) (Result, error) {
+	res := Result{Word: word, Definitions: []Meaning{}, Synonyms: []string{}, Difficulty: unknownDifficulty}
+	norm := strings.ToLower(strings.TrimSpace(word))
+	if norm == "" {
+		return res, nil
+	}
+
+	head := norm
+	for _, c := range candidates(norm) {
+		defs, err := p.dict.d.Define(c, langEN)
+		if err != nil {
+			return Result{}, err
+		}
+		if len(defs) == 0 {
+			continue
+		}
+		head = c
+		for _, d := range defs {
+			// DreamDict orders by source priority, so the curated senses
+			// (WordNet, WOLF) are already ahead of the Wiktionary tail — taking
+			// the first few is taking the best few.
+			res.Definitions = append(res.Definitions, Meaning{PartOfSpeech: d.POS, Definition: d.Gloss})
+			if len(res.Definitions) >= maxDefinitions {
+				break
+			}
+		}
+		break
+	}
+
+	gloss, err := p.translate(norm)
+	if err != nil {
+		return Result{}, err
+	}
+	res.Gloss = gloss
+
+	syns, err := p.dict.d.Synonyms(head, langEN)
+	if err != nil {
+		return Result{}, err
+	}
+	if len(syns) > maxSynonyms {
+		syns = syns[:maxSynonyms]
+	}
+	res.Synonyms = append(res.Synonyms, syns...)
+
+	prons, err := p.dict.d.Pronunciation(head, langEN)
+	if err != nil {
+		return Result{}, err
+	}
+	res.Phonetic = pickIPA(prons)
+
+	if res.Frequency, err = p.dict.d.Frequency(head, langEN); err != nil {
+		return Result{}, err
+	}
+	if res.Difficulty, err = p.dict.d.Difficulty(head, langEN); err != nil {
+		return Result{}, err
+	}
+	ety, err := p.dict.d.Etymology(head, langEN)
+	if err != nil {
+		return Result{}, err
+	}
+	res.Etymology = trimEtymology(ety)
+
+	return res, nil
+}
+
+// Gloss returns the writer's-language translation alone — the hover tooltip's
+// fast path, one indexed query per candidate form and nothing else.
+func (p dreamProvider) Gloss(word string) (GlossResult, error) {
+	norm := strings.ToLower(strings.TrimSpace(word))
+	if norm == "" {
+		return GlossResult{Word: word}, nil
+	}
+	gloss, err := p.translate(norm)
+	if err != nil {
+		return GlossResult{}, err
+	}
+	return GlossResult{Word: word, Gloss: gloss}, nil
+}
+
+// maxGlossSenses caps how many translations are strung together. One is often
+// too thin to disambiguate; the whole list is a wall of words in a tooltip.
+const maxGlossSenses = 3
+
+// translate walks the candidate forms and returns the first that has an
+// equivalent in the writer's language, joined into one line.
+//
+// It asks for Equivalents rather than Translate on the strength of measuring
+// both against the real dict.db: Wiktionary's en→pt-PT translation table
+// answers for 17% of the 2,000 commonest English words, and the shared-synset
+// path answers for 62%. The plan assumed Translate would do — the database
+// says otherwise, and a gloss that is absent five times out of six is not a
+// gloss. Equivalents falls back to Translate internally, so nothing is lost.
+//
+// A language dict.db was built without simply has no rows, so this returns "" —
+// which is exactly what an unglossed word returns, and the popover already
+// renders that case. Spanish today is precisely this: supported by DreamDict,
+// absent from the deployed database until it is rebuilt.
+func (p dreamProvider) translate(norm string) (string, error) {
+	for _, c := range candidates(norm) {
+		trs, err := p.dict.d.Equivalents(c, langEN, p.native)
+		if err != nil {
+			return "", err
+		}
+		if len(trs) == 0 {
+			continue
+		}
+		if len(trs) > maxGlossSenses {
+			trs = trs[:maxGlossSenses]
+		}
+		return strings.Join(trs, "; "), nil
+	}
+	return "", nil
+}
+
+// pickIPA chooses what to show beside the read-aloud button. IPA is the only
+// form worth showing a learner — CMU's "IH0 F EH1 M ER0 AH0 L" is a machine
+// format, and printing it would be noise dressed up as help. If there's no IPA,
+// there's no phonetic line.
+func pickIPA(prons []dictionary.Pronunciation) string {
+	for _, p := range prons {
+		if strings.EqualFold(p.Format, "ipa") && strings.TrimSpace(p.Value) != "" {
+			return strings.Trim(strings.TrimSpace(p.Value), "/[]")
+		}
+	}
+	return ""
+}
+
+// trimEtymology cuts Wiktionary's prose down to a line, preferring to stop at a
+// sentence boundary so the result reads as a finished thought rather than a
+// truncation.
+func trimEtymology(text string) string {
+	text = strings.Join(strings.Fields(text), " ")
+	if utf8.RuneCountInString(text) <= maxEtymology {
+		return text
+	}
+	// Counted and cut in runes, not bytes. An etymology is the one field that
+	// is *mostly* not English — ἐφήμερος, ephemerus, 短暫 — and a byte slice
+	// through the middle of one of those characters is invalid UTF-8 in the
+	// JSON response.
+	cut := string([]rune(text)[:maxEtymology-1])
+	// Stop at a sentence when one ends late enough to be worth keeping. An
+	// early full stop ("From Latin. …") is not a summary, it's a discarded
+	// paragraph, so that case falls through to the word-boundary cut.
+	if i := strings.LastIndex(cut, ". "); i > len(cut)/2 {
+		return cut[:i+1]
+	}
+	if i := strings.LastIndex(cut, " "); i > 0 {
+		cut = cut[:i]
+	}
+	return cut + "…"
+}

internal/lexicon/dreamdict_test.go

@@ -0,0 +1,521 @@
+package lexicon
+
+import (
+	"database/sql"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"unicode/utf8"
+
+	"github.com/go-chi/chi/v5"
+	"github.com/prosolis/dreamdict/dictionary"
+	_ "modernc.org/sqlite"
+
+	"gitea.parodia.dev/drwily/petal/internal/auth"
+	"gitea.parodia.dev/drwily/petal/internal/db"
+)
+
+// The fixture is a real dict.db on disk rather than an in-memory handle, so
+// these tests exercise the path production takes: stat the file, open it
+// read-only, find it seeded. A fake would have skipped every one of those.
+//
+// "ephemeral" is the worked example throughout: it has definitions only under
+// its own headword, translations into two languages, IPA alongside a CMU
+// pronunciation Petal must not show, a frequency, a difficulty and an etymology
+// long enough to need trimming.
+func writeFixture(t *testing.T, seeded bool) string {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "dict.db")
+	sqldb, err := sql.Open("sqlite", path)
+	if err != nil {
+		t.Fatalf("open fixture: %v", err)
+	}
+	defer sqldb.Close()
+	if err := dictionary.BootstrapSchema(sqldb); err != nil {
+		t.Fatalf("bootstrap: %v", err)
+	}
+	if !seeded {
+		return path
+	}
+
+	exec := func(q string, args ...any) {
+		t.Helper()
+		if _, err := sqldb.Exec(q, args...); err != nil {
+			t.Fatalf("seed %q: %v", q, err)
+		}
+	}
+	exec(`INSERT INTO meta (key, value) VALUES ('schema_version', '2')`)
+
+	exec(`INSERT INTO words (id, word, lang, pos, frequency, difficulty) VALUES
+		(1, 'ephemeral', 'en', 'adjective', 50, 0.72),
+		(2, 'run',       'en', 'verb',     900, 0.05),
+		(3, 'plain',     'en', 'adjective',  0, NULL),
+		(4, 'efémero',   'pt-PT', 'adjective', 12, 0.6)`)
+
+	exec(`INSERT INTO definitions (word_id, pos, gloss, source, priority) VALUES
+		(1, 'adjective', 'lasting a very short time', 'wordnet',    10),
+		(1, 'adjective', 'short-lived',                'wiktionary', 99),
+		(1, 'adjective', 'transitory',                 'wiktionary', 99),
+		(1, 'adjective', 'fleeting',                   'wiktionary', 99),
+		(1, 'adjective', 'evanescent',                 'wiktionary', 99),
+		(2, 'verb',      'move fast on foot',          'wordnet',    10),
+		(3, 'adjective', 'without decoration',         'wordnet',    10)`)
+
+	exec(`INSERT INTO synonyms (word_id, synonym, source) VALUES
+		(1, 'fleeting', 'wordnet'), (1, 'transient', 'wordnet'),
+		(2, 'sprint', 'wordnet')`)
+
+	exec(`INSERT INTO translations (word_id, translation, target_lang, source) VALUES
+		(1, 'efémero',  'pt-PT', 'kaikki'),
+		(1, 'passageiro','pt-PT', 'kaikki'),
+		(1, 'éphémère', 'fr',    'kaikki'),
+		(1, '短暂的',    'zh',    'cedict'),
+		(2, 'correr',   'pt-PT', 'kaikki')`)
+
+	// CMU is listed first deliberately: picking the first row would show a
+	// learner "IH0 F EH1 M ER0 AH0 L", which is a machine format, not help.
+	exec(`INSERT INTO pronunciations (word_id, format, value, source) VALUES
+		(1, 'cmu', 'IH0 F EH1 M ER0 AH0 L', 'cmudict'),
+		(1, 'ipa', '/ɪˈfɛm.ər.əl/',          'wiktionary')`)
+
+	// "brief" carries no translation row at all — only a shared WordNet synset
+	// with two pt-PT words. On the real database that is the *usual* case, not
+	// the exotic one, so Petal must reach a gloss this way or the pt-PT pair
+	// has almost no glosses. "breve" is the commoner of the two and leads.
+	exec(`INSERT INTO words (id, word, lang, pos, frequency) VALUES
+		(5, 'brief',   'en',    'adjective', 400),
+		(6, 'breve',   'pt-PT', 'adjective', 300),
+		(7, 'sucinto', 'pt-PT', 'adjective',  20)`)
+	exec(`INSERT INTO definitions (word_id, pos, gloss, source, priority) VALUES
+		(5, 'adjective', 'of short duration', 'wordnet', 10)`)
+	exec(`INSERT INTO synsets (id, synset_id, pos) VALUES (1, '00751145-a', 'adjective')`)
+	exec(`INSERT INTO word_synsets (word_id, synset_id, source) VALUES
+		(5, 1, 'wordnet'), (6, 1, 'omw'), (7, 1, 'omw')`)
+
+	exec(`INSERT INTO etymology (word_id, text, source) VALUES
+		(1, 'From Medieval Latin ephemerus, from Ancient Greek ἐφήμερος (ephḗmeros, "lasting only a day"), from ἐπί (epí, "upon") and ἡμέρα (hēméra, "day"). The sense of transience is attested in English from the late sixteenth century onwards.', 'wiktionary')`)
+
+	return path
+}
+
+func openFixture(t *testing.T) *DreamDict {
+	t.Helper()
+	dd, err := OpenDreamDict(writeFixture(t, true))
+	if err != nil {
+		t.Fatalf("OpenDreamDict: %v", err)
+	}
+	if dd == nil {
+		t.Fatal("OpenDreamDict returned no dictionary for a seeded file")
+	}
+	t.Cleanup(func() { dd.Close() })
+	return dd
+}
+
+func TestOpenMissingFileIsNotAnError(t *testing.T) {
+	dd, err := OpenDreamDict(filepath.Join(t.TempDir(), "absent.db"))
+	if err != nil {
+		t.Fatalf("a missing dict.db must not be an error: %v", err)
+	}
+	if dd != nil {
+		t.Fatal("a missing dict.db must yield no dictionary")
+	}
+	// An unset path is the laptop default and must behave the same way.
+	if dd, err := OpenDreamDict(""); err != nil || dd != nil {
+		t.Fatalf(`OpenDreamDict("") = %v, %v; want nil, nil`, dd, err)
+	}
+	// Close on the nil handle is what main.go defers unconditionally.
+	if err := dd.Close(); err != nil {
+		t.Fatalf("Close on absent dictionary: %v", err)
+	}
+}
+
+func TestOpenPresentButUnseededIsAnError(t *testing.T) {
+	// A file that exists but was never imported is somebody's mistake — a
+	// half-finished deploy — and must be loud, unlike a file that isn't there.
+	dd, err := OpenDreamDict(writeFixture(t, false))
+	if err == nil {
+		dd.Close()
+		t.Fatal("an unseeded dict.db must report an error")
+	}
+	if dd != nil {
+		t.Fatal("an unseeded dict.db must not yield a usable dictionary")
+	}
+}
+
+func TestOpenUnreadablePathIsAnError(t *testing.T) {
+	// Not a missing file: a directory where dict.db should be. Distinguishing
+	// this from ErrNotExist is the whole point of the stat.
+	dir := filepath.Join(t.TempDir(), "dict.db")
+	if err := os.Mkdir(dir, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := OpenDreamDict(dir); err == nil {
+		t.Fatal("a directory in place of dict.db must report an error")
+	}
+}
+
+func TestDreamLookupFillsEveryField(t *testing.T) {
+	p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
+	res, err := p.Lookup("ephemeral")
+	if err != nil {
+		t.Fatalf("Lookup: %v", err)
+	}
+	if res.Gloss != "efémero; passageiro" {
+		t.Errorf("Gloss = %q, want the pt-PT translations joined", res.Gloss)
+	}
+	if res.Phonetic != "ɪˈfɛm.ər.əl" {
+		t.Errorf("Phonetic = %q, want the IPA without its slashes", res.Phonetic)
+	}
+	if len(res.Definitions) != maxDefinitions {
+		t.Fatalf("Definitions = %d, want them capped at %d", len(res.Definitions), maxDefinitions)
+	}
+	if res.Definitions[0].Definition != "lasting a very short time" {
+		t.Errorf("first definition = %q, want the curated (wordnet) sense first",
+			res.Definitions[0].Definition)
+	}
+	if res.Definitions[0].PartOfSpeech != "adjective" {
+		t.Errorf("part of speech = %q, want adjective", res.Definitions[0].PartOfSpeech)
+	}
+	if len(res.Synonyms) != 2 {
+		t.Errorf("Synonyms = %v, want both", res.Synonyms)
+	}
+	if res.Frequency != 50 {
+		t.Errorf("Frequency = %d, want 50", res.Frequency)
+	}
+	if res.Difficulty != 0.72 {
+		t.Errorf("Difficulty = %v, want 0.72", res.Difficulty)
+	}
+	if !strings.HasPrefix(res.Etymology, "From Medieval Latin ephemerus") {
+		t.Errorf("Etymology = %q, want the Wiktionary text", res.Etymology)
+	}
+	if len(res.Etymology) > maxEtymology {
+		t.Errorf("Etymology not trimmed: %d chars", len(res.Etymology))
+	}
+}
+
+func TestDreamGlossFollowsTheWriterNotTheWord(t *testing.T) {
+	dd := openFixture(t)
+	for lang, want := range map[string]string{
+		"pt-PT": "efémero; passageiro",
+		"fr":    "éphémère",
+		"es":    "", // DreamDict supports Spanish; this database wasn't built with it
+		"de":    "", // never a Petal pair, and must not silently borrow another's
+	} {
+		got, err := dreamProvider{dict: dd, native: lang}.Gloss("ephemeral")
+		if err != nil {
+			t.Fatalf("Gloss(%s): %v", lang, err)
+		}
+		if got.Gloss != want {
+			t.Errorf("Gloss for %s = %q, want %q", lang, got.Gloss, want)
+		}
+		if got.Word != "ephemeral" {
+			t.Errorf("Word = %q, want the word as asked", got.Word)
+		}
+	}
+}
+
+func TestDreamGlossesThroughSharedSynsets(t *testing.T) {
+	// The measurement that drove this: on the real dict.db, Wiktionary's
+	// en→pt-PT translation table answers for 17% of the 2,000 commonest English
+	// words and the shared-synset path answers for 62%. A word with no
+	// translation row must still get a gloss, commonest sense first.
+	p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
+	res, err := p.Lookup("brief")
+	if err != nil {
+		t.Fatalf("Lookup: %v", err)
+	}
+	if res.Gloss != "breve; sucinto" {
+		t.Errorf("Gloss = %q, want the synset equivalents, commonest first", res.Gloss)
+	}
+}
+
+func TestDreamDeinflectsToTheHeadword(t *testing.T) {
+	// dict.db stores headwords: "running" has no row of its own. The candidate
+	// walk is what makes a right-click on real prose work at all.
+	p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
+	res, err := p.Lookup("running")
+	if err != nil {
+		t.Fatalf("Lookup: %v", err)
+	}
+	if len(res.Definitions) == 0 || res.Definitions[0].Definition != "move fast on foot" {
+		t.Fatalf("Definitions = %+v, want run's", res.Definitions)
+	}
+	// Every other field must come from the same headword — a popover that mixed
+	// "running"'s (absent) frequency with "run"'s definitions would be lying.
+	if res.Frequency != 900 {
+		t.Errorf("Frequency = %d, want run's 900", res.Frequency)
+	}
+	if res.Difficulty != 0.05 {
+		t.Errorf("Difficulty = %v, want run's 0.05", res.Difficulty)
+	}
+	if res.Synonyms[0] != "sprint" {
+		t.Errorf("Synonyms = %v, want run's", res.Synonyms)
+	}
+	if res.Gloss != "correr" {
+		t.Errorf("Gloss = %q, want run's", res.Gloss)
+	}
+}
+
+func TestDreamMissIsAnEmptyResultNotAnError(t *testing.T) {
+	p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
+	res, err := p.Lookup("zzzxqqq")
+	if err != nil {
+		t.Fatalf("Lookup: %v", err)
+	}
+	if len(res.Definitions) != 0 || len(res.Synonyms) != 0 || res.Gloss != "" {
+		t.Errorf("expected an empty result, got %+v", res)
+	}
+	// The frontend renders [] and never null.
+	if res.Definitions == nil || res.Synonyms == nil {
+		t.Errorf("empty slices must be non-nil: %+v", res)
+	}
+	if res.Difficulty != unknownDifficulty {
+		t.Errorf("Difficulty = %v, want the unknown sentinel", res.Difficulty)
+	}
+	// Empty input is a miss, not a crash.
+	if res, err := p.Lookup("   "); err != nil || res.Gloss != "" {
+		t.Errorf("Lookup(blank) = %+v, %v", res, err)
+	}
+	if res, err := p.Gloss(""); err != nil || res.Gloss != "" {
+		t.Errorf("Gloss(empty) = %+v, %v", res, err)
+	}
+}
+
+func TestDreamUnscoredWordKeepsTheUnknownSentinel(t *testing.T) {
+	// "plain" is in the database with no frequency and a NULL difficulty. The
+	// popover must be able to tell that apart from "difficulty 0.0, the easiest
+	// word there is" — which is why the sentinel is -1 and not omitempty.
+	p := dreamProvider{dict: openFixture(t), native: "pt-PT"}
+	res, err := p.Lookup("plain")
+	if err != nil {
+		t.Fatalf("Lookup: %v", err)
+	}
+	if len(res.Definitions) == 0 {
+		t.Fatal("expected plain to be found")
+	}
+	if res.Difficulty != unknownDifficulty {
+		t.Errorf("Difficulty = %v, want the unknown sentinel for a NULL score", res.Difficulty)
+	}
+	if res.Frequency != 0 {
+		t.Errorf("Frequency = %d, want 0 for no count", res.Frequency)
+	}
+}
+
+func TestPickIPASkipsMachineFormats(t *testing.T) {
+	if got := pickIPA([]dictionary.Pronunciation{{Format: "cmu", Value: "K AE1 T"}}); got != "" {
+		t.Errorf("pickIPA on CMU alone = %q, want empty — CMU is not for a reader", got)
+	}
+	if got := pickIPA(nil); got != "" {
+		t.Errorf("pickIPA(nil) = %q", got)
+	}
+	if got := pickIPA([]dictionary.Pronunciation{{Format: "IPA", Value: " [kæt] "}}); got != "kæt" {
+		t.Errorf("pickIPA = %q, want the bare IPA regardless of case or brackets", got)
+	}
+}
+
+func TestTrimEtymologyPrefersASentence(t *testing.T) {
+	// A sentence that ends past halfway is the good cut: keep it, drop the rest.
+	long := strings.Repeat("padding word ", 14) + "end. " + strings.Repeat("more ", 40)
+	got := trimEtymology(long)
+	if utf8.RuneCountInString(got) > maxEtymology {
+		t.Errorf("not trimmed: %d runes", utf8.RuneCountInString(got))
+	}
+	if !strings.HasSuffix(got, "end.") {
+		t.Errorf("trimEtymology = %q, want it to stop at the sentence", got)
+	}
+
+	// An early full stop is not a summary — cutting there would throw away
+	// almost the whole line — so this falls through to a word boundary.
+	got = trimEtymology("From Latin. " + strings.Repeat("padding word ", 40))
+	if strings.HasSuffix(got, "Latin.") {
+		t.Errorf("trimEtymology = %q, want more than the first four words", got)
+	}
+	if !strings.HasSuffix(got, "…") {
+		t.Errorf("trimEtymology = %q, want an ellipsis when cut mid-thought", got)
+	}
+
+	// Multi-byte text must be cut on rune boundaries: a byte slice through
+	// ἐφήμερος would put invalid UTF-8 in the JSON.
+	greek := trimEtymology(strings.Repeat("ἐφήμερος ", 60))
+	if !utf8.ValidString(greek) {
+		t.Errorf("trimEtymology produced invalid UTF-8: %q", greek)
+	}
+	if n := utf8.RuneCountInString(greek); n > maxEtymology {
+		t.Errorf("trimmed to %d runes, want at most %d", n, maxEtymology)
+	}
+
+	if strings.Contains(got, "  ") || strings.Contains(trimEtymology("a\n  b"), "\n") {
+		t.Error("whitespace should be collapsed to a single line")
+	}
+	// Short text passes through untouched.
+	if got := trimEtymology("From Old English."); got != "From Old English." {
+		t.Errorf("trimEtymology = %q", got)
+	}
+}
+
+// --- the Set: which provider answers, and what happens when dict.db is absent
+
+func TestSetRoutesByPairLanguage(t *testing.T) {
+	set := NewSet(openFixture(t))
+	if !set.HasDreamDict() {
+		t.Fatal("HasDreamDict = false with a dictionary open")
+	}
+	// zh — and an empty column, which is what a pre-auth row reads as — stays
+	// on the embedded datasets until the two have been compared on real
+	// lookups. This test is the guard on that decision.
+	for _, lang := range []string{"", LangZh} {
+		if _, ok := set.For(lang).(*Lexicon); !ok {
+			t.Errorf("For(%q) = %T, want the embedded Lexicon", lang, set.For(lang))
+		}
+	}
+	for _, lang := range []string{"pt-PT", "fr", "es"} {
+		p, ok := set.For(lang).(dreamProvider)
+		if !ok {
+			t.Fatalf("For(%q) = %T, want DreamDict", lang, set.For(lang))
+		}
+		if p.native != lang {
+			t.Errorf("For(%q) glosses into %q", lang, p.native)
+		}
+	}
+}
+
+func TestSetWithoutDictKeepsTheEnglishHalf(t *testing.T) {
+	// The interesting degradation: dict.db never got deployed. A pt-PT writer
+	// should still get definitions, synonyms and phonetics — all compiled into
+	// the binary and all correct for her — and lose only the translation.
+	set := NewSet(nil)
+	if set.HasDreamDict() {
+		t.Fatal("HasDreamDict = true with no dictionary")
+	}
+	p := set.For("pt-PT")
+	res, err := p.Lookup("happy")
+	if err != nil {
+		t.Fatalf("Lookup: %v", err)
+	}
+	if len(res.Definitions) == 0 || len(res.Synonyms) == 0 {
+		t.Error("expected the embedded English half to survive a missing dict.db")
+	}
+	if res.Gloss != "" {
+		t.Errorf("Gloss = %q — a pt-PT writer must never be handed the Chinese gloss", res.Gloss)
+	}
+	g, err := p.Gloss("happy")
+	if err != nil {
+		t.Fatalf("Gloss: %v", err)
+	}
+	if g.Gloss != "" {
+		t.Errorf("Gloss = %q, want empty", g.Gloss)
+	}
+	if g.Word != "happy" {
+		t.Errorf("Word = %q, want the word as asked", g.Word)
+	}
+	// The zh writer is untouched by any of this.
+	zh, err := set.For(LangZh).Lookup("happy")
+	if err != nil {
+		t.Fatalf("Lookup(zh): %v", err)
+	}
+	if zh.Gloss == "" {
+		t.Error("the zh pair must keep its embedded gloss with no dict.db")
+	}
+}
+
+// --- the handler: the pair language is read per request, from the caller's row
+
+func mountLexicon(t *testing.T, set *Set) (*chi.Mux, *db.DB) {
+	t.Helper()
+	database, err := db.Open(filepath.Join(t.TempDir(), "petal.db"))
+	if err != nil {
+		t.Fatalf("db.Open: %v", err)
+	}
+	t.Cleanup(func() { database.Close() })
+
+	for _, u := range []struct{ id, lang string }{
+		{"alice", LangZh}, {"bob", "pt-PT"},
+	} {
+		if _, err := database.Exec(
+			`INSERT INTO users (id, email, display_name, pair_lang) VALUES (?, ?, ?, ?)`,
+			u.id, u.id+"@example.com", u.id, u.lang,
+		); err != nil {
+			t.Fatalf("seed user: %v", err)
+		}
+	}
+
+	h := NewHandler(database.DB, set)
+	r := chi.NewMux()
+	r.Mount("/word", h.Routes())
+	r.Mount("/gloss", h.GlossRoutes())
+	return r, database
+}
+
+func getAs(t *testing.T, r http.Handler, userID, path string) Result {
+	t.Helper()
+	req := httptest.NewRequest(http.MethodGet, path, nil)
+	req = req.WithContext(auth.WithUser(req.Context(), userID))
+	rec := httptest.NewRecorder()
+	r.ServeHTTP(rec, req)
+	if rec.Code != http.StatusOK {
+		t.Fatalf("GET %s as %s = %d: %s", path, userID, rec.Code, rec.Body)
+	}
+	var res Result
+	if err := json.Unmarshal(rec.Body.Bytes(), &res); err != nil {
+		t.Fatalf("decode: %v", err)
+	}
+	return res
+}
+
+func TestHandlerGlossesInTheCallersLanguage(t *testing.T) {
+	r, _ := mountLexicon(t, NewSet(openFixture(t)))
+
+	// Same URL, two writers, two languages. This is why the response is no
+	// longer cacheable as `public`.
+	bob := getAs(t, r, "bob", "/word/ephemeral")
+	if bob.Gloss != "efémero; passageiro" {
+		t.Errorf("bob's gloss = %q, want pt-PT", bob.Gloss)
+	}
+	alice := getAs(t, r, "alice", "/word/ephemeral")
+	if !strings.ContainsAny(alice.Gloss, "短暂的") && alice.Gloss != "" {
+		// alice is on the embedded ECDICT dataset, not the fixture's zh row —
+		// what matters is that she is *not* served bob's Portuguese.
+		t.Logf("alice's embedded gloss: %q", alice.Gloss)
+	}
+	if alice.Gloss == bob.Gloss && bob.Gloss != "" {
+		t.Error("the zh writer was served the pt-PT gloss")
+	}
+	if strings.Contains(alice.Gloss, "efémero") {
+		t.Errorf("alice's gloss = %q, want the embedded Chinese one", alice.Gloss)
+	}
+}
+
+func TestHandlerUnknownCallerFallsBackRatherThanFailing(t *testing.T) {
+	// No session, or a user row that has gone: the lookup still answers, from
+	// the embedded datasets. A dictionary that fails closed would be worse than
+	// one that answers in the wrong language, because nothing at all is not a
+	// dictionary.
+	r, _ := mountLexicon(t, NewSet(openFixture(t)))
+	res := getAs(t, r, "nobody", "/word/happy")
+	if len(res.Definitions) == 0 {
+		t.Error("expected the embedded fallback to answer for an unknown caller")
+	}
+}
+
+func TestHandlerCachesPrivately(t *testing.T) {
+	r, _ := mountLexicon(t, NewSet(nil))
+	req := httptest.NewRequest(http.MethodGet, "/gloss/happy", nil)
+	req = req.WithContext(auth.WithUser(req.Context(), "alice"))
+	rec := httptest.NewRecorder()
+	r.ServeHTTP(rec, req)
+	if got := rec.Header().Get("Cache-Control"); !strings.HasPrefix(got, "private") {
+		t.Errorf("Cache-Control = %q — a per-writer gloss must not go in a shared cache", got)
+	}
+}
+
+func TestHandlerDecodesPunctuatedWords(t *testing.T) {
+	r, _ := mountLexicon(t, NewSet(openFixture(t)))
+	res := getAs(t, r, "bob", "/word/"+"caf%C3%A9")
+	if res.Word != "café" {
+		t.Errorf("Word = %q, want the decoded word", res.Word)
+	}
+}

internal/lexicon/handlers.go

@@ -1,20 +1,27 @@
 package lexicon
 
 import (
+	"context"
+	"database/sql"
 	"encoding/json"
 	"net/http"
 	"net/url"
 
 	"github.com/go-chi/chi/v5"
+
+	"gitea.parodia.dev/drwily/petal/internal/auth"
 )
 
-// Handler serves the word-lookup endpoint backed by a single shared Lexicon.
+// Handler serves the word-lookup endpoints. It holds the shared provider Set
+// and the database, because which provider answers depends on who is asking.
 type Handler struct {
-	Lex *Lexicon
+	Set *Set
+	db  *sql.DB
 }
 
-// New constructs a Handler with a fresh (lazily-loaded) Lexicon.
-func NewHandler() *Handler { return &Handler{Lex: New()} }
+// NewHandler constructs a Handler over a provider Set. db is used for one
+// thing: reading the caller's pair language.
+func NewHandler(db *sql.DB, set *Set) *Handler { return &Handler{Set: set, db: db} }
 
 // Routes returns the router mounted at /api/word. The word is a path segment so
 // "/api/word/happy" reads naturally; it's URL-decoded to tolerate the rare
@@ -26,56 +33,81 @@ func (h *Handler) Routes() chi.Router {
 }
 
 // GlossRoutes returns the router mounted at /api/gloss — the lightweight
-// Chinese-only lookup behind the inline hover/select gloss. It shares the
-// Handler's Lexicon, so the datasets still load just once.
+// translation-only lookup behind the inline hover/select gloss. It shares the
+// Handler's Set, so the embedded datasets and dict.db are still opened once.
 func (h *Handler) GlossRoutes() chi.Router {
 	r := chi.NewRouter()
 	r.Get("/{word}", h.gloss)
 	return r
 }
 
-// lookup returns the definition + synonyms for one word. A word found in neither
-// dataset still returns 200 with empty lists, so the popover can show a friendly
-// "nothing found" rather than an error state.
+// providerFor returns the provider for the caller's language pair.
+//
+// The pair language is read here rather than threaded down because a word
+// lookup has no other query to piggyback on — unlike the document handlers,
+// which take pair_lang from the row-scoped query that already proves
+// ownership. It is one indexed primary-key read against a local SQLite file,
+// which costs less than encoding the response it feeds.
+//
+// A read that fails, or a caller with no user row, resolves to the empty
+// language, and [Set.For] maps that to today's embedded behaviour. Falling back
+// to a working dictionary beats failing the lookup.
+func (h *Handler) providerFor(ctx context.Context) Provider {
+	var lang string
+	if h.db != nil {
+		_ = h.db.QueryRowContext(ctx,
+			`SELECT COALESCE(pair_lang, '') FROM users WHERE id = ?`,
+			auth.UserID(ctx),
+		).Scan(&lang)
+	}
+	return h.Set.For(lang)
+}
+
+// pathWord reads the {word} segment, URL-decoded.
+func pathWord(r *http.Request) string {
+	word := chi.URLParam(r, "word")
+	if decoded, err := url.PathUnescape(word); err == nil {
+		word = decoded
+	}
+	return word
+}
+
+// lookup returns the definition + synonyms for one word. A word found in no
+// dataset still returns 200 with empty lists, so the popover can show a
+// friendly "nothing found" rather than an error state.
 func (h *Handler) lookup(w http.ResponseWriter, r *http.Request) {
-	word := chi.URLParam(r, "word")
-	if decoded, err := url.PathUnescape(word); err == nil {
-		word = decoded
-	}
-
-	res, err := h.Lex.Lookup(word)
+	res, err := h.providerFor(r.Context()).Lookup(pathWord(r))
 	if err != nil {
-		w.Header().Set("Content-Type", "application/json")
-		w.WriteHeader(http.StatusInternalServerError)
-		_ = json.NewEncoder(w).Encode(map[string]string{"error": err.Error()})
+		writeLookupErr(w, err)
 		return
 	}
-
-	w.Header().Set("Content-Type", "application/json")
-	// Word lookups are static for the life of the build; let the browser cache
-	// them so repeated right-clicks on the same word are instant.
-	w.Header().Set("Cache-Control", "public, max-age=86400")
-	_ = json.NewEncoder(w).Encode(res)
+	writeLookup(w, res)
 }
 
-// gloss returns just the Chinese translation for one word. Like lookup, a miss
-// is a 200 with an empty gloss so the hover tooltip can quietly skip rather than
-// error.
+// gloss returns just the translation for one word. Like lookup, a miss is a 200
+// with an empty gloss so the hover tooltip can quietly skip rather than error.
 func (h *Handler) gloss(w http.ResponseWriter, r *http.Request) {
-	word := chi.URLParam(r, "word")
-	if decoded, err := url.PathUnescape(word); err == nil {
-		word = decoded
-	}
-
-	res, err := h.Lex.Gloss(word)
+	res, err := h.providerFor(r.Context()).Gloss(pathWord(r))
 	if err != nil {
-		w.Header().Set("Content-Type", "application/json")
-		w.WriteHeader(http.StatusInternalServerError)
-		_ = json.NewEncoder(w).Encode(map[string]string{"error": err.Error()})
+		writeLookupErr(w, err)
 		return
 	}
+	writeLookup(w, res)
+}
 
+func writeLookupErr(w http.ResponseWriter, err error) {
 	w.Header().Set("Content-Type", "application/json")
-	w.Header().Set("Cache-Control", "public, max-age=86400")
-	_ = json.NewEncoder(w).Encode(res)
+	w.WriteHeader(http.StatusInternalServerError)
+	_ = json.NewEncoder(w).Encode(map[string]string{"error": err.Error()})
+}
+
+func writeLookup(w http.ResponseWriter, v any) {
+	w.Header().Set("Content-Type", "application/json")
+	// A lookup is stable for the life of the deployment, so let the browser
+	// keep it — repeated right-clicks on the same word are then instant. It is
+	// `private` rather than `public` because the gloss is now in *her*
+	// language: a shared cache keyed on the URL alone would hand one writer
+	// another writer's language.
+	w.Header().Set("Cache-Control", "private, max-age=86400")
+	_ = json.NewEncoder(w).Encode(v)
 }

internal/lexicon/lexicon.go

@@ -29,8 +29,30 @@ type Result struct {
 	Phonetic    string    `json:"phonetic"` // IPA for the English word; "" when absent
 	Definitions []Meaning `json:"definitions"`
 	Synonyms    []string  `json:"synonyms"`
+
+	// The fields below only ever come from DreamDict; the embedded datasets
+	// leave them at their unknown values, and the popover hides them.
+
+	// Frequency is how common the word is (higher = more common). 0 means
+	// unknown, which is DreamDict's own convention — a word it carries but has
+	// no corpus count for is indistinguishable from a word it doesn't carry,
+	// and the popover treats both the same way.
+	Frequency int `json:"frequency"`
+	// Difficulty runs 0.0 (easiest) to 1.0 (hardest); -1 means unknown. It is a
+	// sentinel rather than an omitted field because 0.0 is a real, meaningful
+	// score and `omitempty` would erase it.
+	Difficulty float64 `json:"difficulty"`
+	// Etymology is free-form Wiktionary prose, trimmed to a line. Where the
+	// word came from is a real hook for a writer whose own language shares
+	// Latin roots with English — "ephemeral" is much easier to keep once you
+	// have seen efémero next to it.
+	Etymology string `json:"etymology"`
 }
 
+// unknownDifficulty is the [Result.Difficulty] value meaning "no score",
+// matching DreamDict's own -1 return.
+const unknownDifficulty = -1
+
 // GlossResult is the lightweight payload for the inline hover/select gloss: just
 // the word and its Chinese translation, no definitions or synonyms. Kept small
 // so the hover tooltip is instant and trivially cacheable.
@@ -95,7 +117,7 @@ func (l *Lexicon) Lookup(word string) (Result, error) {
 	}
 
 	norm := strings.ToLower(strings.TrimSpace(word))
-	res := Result{Word: word, Definitions: []Meaning{}, Synonyms: []string{}}
+	res := Result{Word: word, Definitions: []Meaning{}, Synonyms: []string{}, Difficulty: unknownDifficulty}
 	if norm == "" {
 		return res, nil
 	}

internal/lexicon/provider.go

@@ -0,0 +1,101 @@
+package lexicon
+
+// A word lookup used to mean exactly one thing: the embedded datasets, which
+// speak English and Mandarin and nothing else. That was fine while Petal had
+// one writer. It stops being fine the moment a pt-PT writer right-clicks a
+// word and gets a Chinese gloss.
+//
+// So the lookup becomes a seam. A [Provider] answers the same two questions the
+// popover and the hover tooltip have always asked; which provider answers them
+// depends on the writer's language pair, and [Set.For] is the only place that
+// decision is made.
+
+// Provider answers word lookups for one writer. The embedded datasets and
+// DreamDict both satisfy it, and both treat a word they don't carry as an empty
+// result rather than an error — a miss is an ordinary outcome of looking a word
+// up, not a failure.
+type Provider interface {
+	// Lookup returns the full popover payload: gloss, phonetic, definitions,
+	// synonyms, and whatever extras the provider carries.
+	Lookup(word string) (Result, error)
+	// Gloss returns just the writer's-language translation. It is the hover
+	// tooltip's fast path and skips everything else.
+	Gloss(word string) (GlossResult, error)
+}
+
+// LangZh is the one pair language still served by the embedded datasets. Every
+// other pair goes to DreamDict — see [Set.For] for why zh is held back.
+const LangZh = "zh"
+
+// langEN is the language DreamDict is asked about for definitions, synonyms and
+// pronunciation. English is always the *target* language of the pair — what
+// varies is the language the gloss is written in.
+const langEN = "en"
+
+// Set holds every provider Petal can serve a lookup from and picks between them
+// by pair language. One Set is shared by the whole process: the embedded
+// datasets load once, and dict.db is one read-only handle.
+type Set struct {
+	embedded *Lexicon
+	// dream is nil when dict.db was not deployed. That is a supported state,
+	// not an error — see [Set.For].
+	dream *DreamDict
+}
+
+// NewSet returns a Set backed by the embedded datasets and, when dream is
+// non-nil, DreamDict. Passing a nil dream is how Petal runs without dict.db.
+func NewSet(dream *DreamDict) *Set {
+	return &Set{embedded: New(), dream: dream}
+}
+
+// HasDreamDict reports whether a dict.db is open. Only startup logging and
+// tests care; a handler never asks, because [Set.For] always returns something
+// usable.
+func (s *Set) HasDreamDict() bool { return s.dream != nil }
+
+// For returns the provider that should answer lookups for a writer whose pair
+// language is lang.
+//
+// Three rules, in order:
+//
+// zh — and an empty code, which is what a pre-Phase-16 row reads as — stays on
+// the embedded ECDICT gloss. Not because DreamDict lacks Chinese (it has
+// CC-CEDICT), but because that path is in daily use by a real writer and the
+// two have not yet been compared on her actual lookups. Switching it is a
+// quality decision, and it hasn't been made.
+//
+// Any other pair goes to DreamDict, which is the only source that has pt-PT,
+// French or Spanish at all.
+//
+// If dict.db was never deployed, a non-zh writer falls back to the embedded
+// datasets with the gloss suppressed. This is the interesting case: the naive
+// "no data" answer would blank the popover entirely, when in fact the English
+// half of it — definitions, synonyms, phonetic — is compiled into the binary
+// and perfectly correct for her. Only the translation is missing, so only the
+// translation goes missing. A failed dictionary deploy costs her the gloss, not
+// the dictionary.
+func (s *Set) For(lang string) Provider {
+	if lang == "" || lang == LangZh {
+		return s.embedded
+	}
+	if s.dream != nil {
+		return dreamProvider{dict: s.dream, native: lang}
+	}
+	return glossless{s.embedded}
+}
+
+// glossless serves the embedded datasets with the Chinese gloss stripped, for a
+// writer who does not read Chinese. Handing her the zh gloss would be worse
+// than handing her nothing: an empty field reads as "not found", where the
+// wrong language reads as Petal being broken.
+type glossless struct{ inner Provider }
+
+func (g glossless) Lookup(word string) (Result, error) {
+	res, err := g.inner.Lookup(word)
+	res.Gloss = ""
+	return res, err
+}
+
+func (g glossless) Gloss(word string) (GlossResult, error) {
+	return GlossResult{Word: word}, nil
+}