dreamdict/dreamdict-blueprint.md

DreamDict — Standalone Dictionary Service: Implementation Blueprint

Overview

dreamdict is a self-contained HTTP dictionary service providing word validation, random word generation, definitions, synonyms, and cross-language translation for English (en), French (fr), and European Portuguese (pt-PT).

It runs as a plain Go binary managed by systemd on the same host as GogoBee and Ollama. GogoBee calls it over localhost HTTP. No Docker, no Traefik, no external network exposure.

If GogoBee later containerizes, only the service URL in GogoBee's config changes. The service itself is unmodified.

---

Motivation

GogoBee is approaching 50,000 lines of code across 47+ modules. A dictionary package with loaders, a SQLite dependency, and an import CLI would compound that bloat. Dictionary logic has no business living inside a Matrix bot. dreamdict is a clean extraction: self-contained infrastructure with a stable HTTP API, independently deployable, independently updatable, with no knowledge of Matrix, rooms, or bot commands.

---

Repository

Separate repository: dreamdict (not a GogoBee internal package).

dreamdict/
├── cmd/
│   ├── server/
│   │   └── main.go          # HTTP server binary
│   └── dictimport/
│       └── main.go          # One-time data import CLI
├── internal/
│   ├── dictionary/
│   │   ├── dict.go          # Core package: public API
│   │   ├── db.go            # SQLite connection + schema bootstrap
│   │   └── query.go         # IsValidWord, RandomWord, Define, Synonyms, Translate
│   └── loader/
│       ├── loader.go        # Shared interface + orchestrator
│       ├── scowl.go         # English word list
│       ├── hunspell.go      # French + pt-PT word lists (shared, parameterized)
│       ├── wordnet.go       # English definitions + synonyms
│       ├── lexique.go       # French POS + frequency enrichment
│       ├── wolf.go          # French definitions + synonyms
│       ├── dicionario.go    # pt-PT definitions
│       └── wiktionary.go    # All langs: supplemental definitions + translations
├── data/
│   └── .gitkeep             # Source data files (not committed)
├── dict.db                  # Generated SQLite database (not committed)
├── scripts/
│   └── download-dict-data.sh
├── deploy/
│   └── dreamdict.service     # systemd unit file
├── go.mod
└── README.md

---

HTTP API

The server listens on 127.0.0.1:7777 by default (configurable via --port flag or DREAMDICT_PORT env var). All responses are JSON. All endpoints are read-only GET requests.

GET /valid

Check if a word exists in a language's word list.

Query params: word (required), lang (required: en, fr, pt-PT)

Response:

{"valid": true}

---

GET /random

Return a random word for a language, with optional filters.

Query params:

• lang (required)
• pos (optional: noun, verb, adjective, adverb)
• min (optional: minimum word length, integer)
• max (optional: maximum word length, integer)
• min_freq (optional: minimum frequency score, integer; only meaningful for fr)

Response:

{"word": "ephemeral"}

Error (no words match filters):

{"error": "no_match", "message": "no words match the given filters"}

---

GET /define

Return all definitions for a word, ordered by source priority.

Query params: word (required), lang (required)

Response:

{
  "word": "ephemeral",
  "lang": "en",
  "definitions": [
    {"pos": "adjective", "gloss": "lasting for a very short time", "source": "wordnet", "priority": 10},
    {"pos": "adjective", "gloss": "lasting only for a day", "source": "wiktionary", "priority": 20}
  ]
}

Word exists but has no definitions:

{"word": "...", "lang": "...", "definitions": []}

---

GET /synonyms

Return deduplicated synonyms for a word.

Query params: word (required), lang (required)

Response:

{"word": "happy", "lang": "en", "synonyms": ["glad", "joyful", "content", "pleased"]}

---

GET /translate

Return known equivalents of a word in another language.

Query params: word (required), from (required), to (required)

Response:

{"word": "cat", "from": "en", "to": "fr", "translations": ["chat", "minet"]}

---

GET /health

Health check and database stats.

Response:

{
  "status": "ok",
  "db_path": "/opt/dreamdict/dict.db",
  "word_counts": {"en": 214832, "fr": 82104, "pt-PT": 71088},
  "def_counts":  {"en": 380000, "fr": 200000, "pt-PT": 120000},
  "imported_at": "2025-03-28T14:22:00Z",
  "schema_version": "1"
}

---

Core Package (internal/dictionary)

dict.go

package dictionary

type Options struct {
    MinLength    int
    MaxLength    int
    POS          string  // "", "noun", "verb", "adjective", "adverb"
    MinFrequency int     // 0 = no filter
}

type Definition struct {
    POS      string
    Gloss    string
    Source   string
    Priority int
}

type Dictionary struct { /* unexported */ }

func New(dbPath string) (*Dictionary, error)
func (d *Dictionary) Close() error
func (d *Dictionary) IsValidWord(word, lang string) (bool, error)
func (d *Dictionary) RandomWord(lang string, opts Options) (string, error)
func (d *Dictionary) Define(word, lang string) ([]Definition, error)
func (d *Dictionary) Synonyms(word, lang string) ([]string, error)
func (d *Dictionary) Translate(word, fromLang, toLang string) ([]string, error)
func (d *Dictionary) WordCount() (map[string]int, error)
func (d *Dictionary) SupportedLangs() ([]string, error)

func ValidLang(lang string) bool  // "en", "fr", "pt-PT"

var (
    ErrNoMatch   = errors.New("dictionary: no word matches filters")
    ErrNotSeeded = errors.New("dictionary: database not seeded, run: go run ./cmd/dictimport")
)

---

Database Schema

Use modernc.org/sqlite (pure Go, no CGO).

CREATE TABLE IF NOT EXISTS words (
    id        INTEGER PRIMARY KEY AUTOINCREMENT,
    word      TEXT    NOT NULL,
    lang      TEXT    NOT NULL,
    pos       TEXT,
    frequency INTEGER DEFAULT 0,
    UNIQUE(word, lang)
);

CREATE INDEX IF NOT EXISTS idx_words_lang     ON words(lang);
CREATE INDEX IF NOT EXISTS idx_words_lang_pos ON words(lang, pos);
CREATE INDEX IF NOT EXISTS idx_words_word     ON words(word);

CREATE TABLE IF NOT EXISTS definitions (
    id       INTEGER PRIMARY KEY AUTOINCREMENT,
    word_id  INTEGER NOT NULL REFERENCES words(id) ON DELETE CASCADE,
    pos      TEXT,
    gloss    TEXT    NOT NULL,
    source   TEXT    NOT NULL,
    priority INTEGER NOT NULL DEFAULT 99,
    UNIQUE(word_id, gloss)
);

CREATE INDEX IF NOT EXISTS idx_definitions_word_id          ON definitions(word_id);
CREATE INDEX IF NOT EXISTS idx_definitions_word_id_priority ON definitions(word_id, priority);

CREATE TABLE IF NOT EXISTS synonyms (
    id       INTEGER PRIMARY KEY AUTOINCREMENT,
    word_id  INTEGER NOT NULL REFERENCES words(id) ON DELETE CASCADE,
    synonym  TEXT    NOT NULL,
    source   TEXT    NOT NULL,
    UNIQUE(word_id, synonym)
);

CREATE INDEX IF NOT EXISTS idx_synonyms_word_id ON synonyms(word_id);

CREATE TABLE IF NOT EXISTS translations (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    word_id     INTEGER NOT NULL REFERENCES words(id) ON DELETE CASCADE,
    translation TEXT    NOT NULL,
    target_lang TEXT    NOT NULL,
    source      TEXT    NOT NULL,
    UNIQUE(word_id, translation, target_lang)
);

CREATE INDEX IF NOT EXISTS idx_translations_word_id ON translations(word_id);

CREATE TABLE IF NOT EXISTS meta (
    key   TEXT PRIMARY KEY,
    value TEXT NOT NULL
);

Definition Priority

Source	Priority
wordnet	10
wolf	10
dicionario-aberto	10
wiktionary	20

---

Server Binary (cmd/server/main.go)

Flags / Env Vars

--port      PORT     Listen port (default 7777)       DREAMDICT_PORT
--db        PATH     Path to dict.db                  DREAMDICT_DB
--host      HOST     Listen host (default 127.0.0.1)  DREAMDICT_HOST

Default host is 127.0.0.1 — the service binds localhost only. It is never exposed to the external network. If GogoBee containerizes in future, change host to 0.0.0.0 and control access at the Docker network level.

Startup Behaviour

• Open the SQLite database on startup; log and exit if unavailable or not seeded
• Register HTTP handlers for all six endpoints
• Log dreamdict listening on 127.0.0.1:7777 on successful start
• Handle SIGTERM / SIGINT for graceful shutdown (close DB, drain in-flight requests)

HTTP Handler Pattern

All handlers follow the same structure:

1. Parse and validate query params — return 400 with {"error": "...", "message": "..."} on bad input
2. Call the appropriate dictionary.* method
3. Return 200 with JSON response, or 404 if the word is not found, or 500 on unexpected errors
4. Log errors at ERROR level; log each request at DEBUG level (controllable via --log-level)

Use only standard library net/http — no framework needed for six endpoints.

---

Import CLI (cmd/dictimport/main.go)

Flags

--db        Path to output dict.db     (default: ./dict.db)
--data      Path to data directory     (default: ./data/)
--langs     Comma-separated langs      (default: en,fr,pt-PT)
--clean     Drop and recreate all tables before import
--skip      Comma-separated loader names to skip

Execution Order

English:

1. scowl — populates words
2. wordnet — definitions + synonyms
3. wiktionary-en — supplemental definitions, synonyms, translations

French:

1. hunspell-fr — populates words
2. lexique — enriches POS + frequency
3. wolf — definitions + synonyms
4. wiktionary-fr — supplemental definitions, synonyms, translations

pt-PT:

1. hunspell-pt — populates words
2. dicionario — definitions
3. wiktionary-pt — supplemental definitions, synonyms, translations

Use INSERT OR IGNORE throughout. Wrap each loader in a single transaction; commit on completion. Print per-loader row counts and elapsed time. Print final summary.

---

Data Sources & Loader Specs

---

English — SCOWL (loader/scowl.go)

Source: https://wordlist.aspell.net/ Files: All final/english-words.NN where NN ≤ 70

One word per line. Skip blank lines and # comments. Lowercase. Skip words with spaces, digits, hyphens, or apostrophes. INSERT OR IGNORE INTO words (word, lang) VALUES (?, 'en').

Expected: ~170,000–220,000 words.

---

French + pt-PT — Hunspell (loader/hunspell.go)

Source: https://github.com/LibreOffice/dictionaries Files: fr_FR/fr_FR.dic, pt_PT/pt_PT.dic

Skip line 1 (word count). Split each line on /, take left side as base word. Lowercase. Skip words with digits, spaces, or non-Latin characters.

Same function handles both languages: func Load(path string, lang string, db *sql.DB) error

Expected: ~60,000–90,000 words per language.

---

English Definitions + Synonyms — WordNet (loader/wordnet.go)

Source: https://wordnet.princeton.edu/download/current-version Files: dict/data.noun, dict/data.verb, dict/data.adj, dict/data.adv

Skip lines starting with two spaces (headers). Parse ss_type for POS. Extract all words in synset. Take gloss text before the first ; (drops usage examples). Insert definitions at priority=10. Insert intra-synset word pairs as synonyms. Skip words containing _ for synonym insertion.

---

French Enrichment — Lexique (loader/lexique.go)

Source: http://www.lexique.org/?page_id=319 File: Lexique383.tsv

Columns: 0=ortho, 3=cgram, 8=freqfilms2. POS map: NOM→noun, VER→verb, ADJ→adjective, ADV→adverb. Frequency = freqfilms2 × 1000 truncated to int. On conflict with existing Hunspell row: UPDATE ... SET pos=?, frequency=? WHERE pos IS NULL.

---

French Definitions + Synonyms — WOLF (loader/wolf.go)

Source: https://github.com/nicolashernandez/WOLF File: wolf-1.0b4.xml.bz2 → decompress to wolf.xml

Stream-parse XML with encoding/xml. For each <SYNSET>: collect <LITERAL> values, map <POS>, insert definitions at priority=10, insert intra-synset synonym pairs.

---

pt-PT Definitions — Dicionário Aberto (loader/dicionario.go)

Source: https://github.com/jorgecardoso/dicionario-aberto File: dicionario-aberto.xml

⚠️ Inspect the file before writing the parser:

head -100 data/dicionario-aberto.xml

Print distinct <pos> values before writing the POS map. Element names in the live file may differ from older documentation. Stream-parse XML. Insert definitions at priority=10.

---

All Languages — Wiktionary (loader/wiktionary.go)

Source: kaikki.org pre-parsed Wiktionary JSONL dumps

Target file	Lang	URL
kaikki-en.jsonl	en	https://kaikki.org/dictionary/English/kaikki.org-dictionary-English.json
kaikki-fr.jsonl	fr	https://kaikki.org/dictionary/French/kaikki.org-dictionary-French.json
kaikki-pt.jsonl	pt-PT	https://kaikki.org/dictionary/Portuguese/kaikki.org-dictionary-Portuguese.json

JSON structure per line:

{
  "word": "example",
  "pos": "noun",
  "lang_code": "en",
  "senses": [
    {"glosses": ["a representative instance"], "tags": []},
    {"glosses": ["past tense of exemplify"], "tags": ["form-of"]}
  ],
  "synonyms": [{"word": "instance"}],
  "translations": [
    {"code": "fr", "word": "exemple"},
    {"code": "pt", "word": "exemplo"}
  ]
}

POS mapping: noun→noun, verb→verb, adj→adjective, adv→adverb, name→noun.

Import logic:

• Use bufio.Scanner with 10MB buffer (default 64KB truncates long lines silently)
• Wrap in single transaction; commit every 10,000 rows
• Verify lang_code matches expected language; skip if not
• Lowercase word; skip if contains spaces or digits
• Skip senses tagged form-of or alt-of
• Skip glosses that are empty, start with (, or exceed 500 characters
• Insert definitions at priority=20; UNIQUE(word_id, gloss) handles deduplication silently
• Insert synonyms; UNIQUE(word_id, synonym) handles deduplication silently
• Translations: for each entry in translations[], if code maps to a supported lang and code != source lang, insert into translations table
  • Code mapping: fr→fr, pt→pt-PT, en→en
  • Skip translations where word contains spaces or digits

pt-PT Brazil filtering: Reject senses where tags contains "Brazil" or "Brazilian Portuguese". Accept untagged senses and senses tagged "Portugal" or "European Portuguese".

Expected additional definitions after deduplication:

• en: ~200,000–400,000
• fr: ~80,000–150,000
• pt-PT: ~40,000–80,000

---

Deployment (systemd)

File Locations

/usr/local/bin/dreamdict          # compiled binary
/opt/dreamdict/dict.db            # SQLite database
/opt/dreamdict/data/              # source data files (can be deleted after import)
/etc/dreamdict/dreamdict.env       # environment config
/var/log/dreamdict/               # logs (or use journald)

deploy/dreamdict.service

[Unit]
Description=DreamDict dictionary service
After=network.target
# If GogoBee ever runs as a service too:
# Before=dreamdict.service

[Service]
Type=simple
User=dreamdict
Group=dreamdict
EnvironmentFile=/etc/dreamdict/dreamdict.env
ExecStart=/usr/local/bin/dreamdict server \
    --host 127.0.0.1 \
    --port 7777 \
    --db /opt/dreamdict/dict.db
Restart=on-failure
RestartSec=5s

# Hardening
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/opt/dreamdict
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

/etc/dreamdict/dreamdict.env

DREAMDICT_PORT=7777
DREAMDICT_DB=/opt/dreamdict/dict.db

Install Steps

# Create user
useradd --system --no-create-home --shell /usr/sbin/nologin dreamdict

# Create dirs
mkdir -p /opt/dreamdict /etc/dreamdict /var/log/dreamdict
chown dreamdict:dreamdict /opt/dreamdict /var/log/dreamdict

# Build and install binary
go build -o /usr/local/bin/dreamdict ./cmd/server
chmod 755 /usr/local/bin/dreamdict

# Run import (as yourself, not dreamdict user)
go run ./cmd/dictimport --db /opt/dreamdict/dict.db --data /opt/dreamdict/data/
chown dreamdict:dreamdict /opt/dreamdict/dict.db

# Install and start service
cp deploy/dreamdict.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now dreamdict

# Verify
systemctl status dreamdict
curl http://127.0.0.1:7777/health

Updating the Database

# Stop service, rebuild database, restart
systemctl stop dreamdict
go run ./cmd/dictimport --db /opt/dreamdict/dict.db --clean
chown dreamdict:dreamdict /opt/dreamdict/dict.db
systemctl start dreamdict

---

GogoBee Integration

GogoBee gets a single DreamDictClient struct. Zero dictionary logic inside the bot.

Client Package

Add to GogoBee: internal/dreamclient/client.go

package dreamclient

import (
    "encoding/json"
    "fmt"
    "net/http"
    "net/url"
    "time"
)

type Client struct {
    baseURL    string
    httpClient *http.Client
}

func New(baseURL string) *Client {
    return &Client{
        baseURL: baseURL,
        httpClient: &http.Client{Timeout: 3 * time.Second},
    }
}

func (c *Client) IsValidWord(word, lang string) (bool, error)
func (c *Client) RandomWord(lang string, pos string, min, max int) (string, error)
func (c *Client) Define(word, lang string) ([]Definition, error)
func (c *Client) Synonyms(word, lang string) ([]string, error)
func (c *Client) Translate(word, fromLang, toLang string) ([]string, error)
func (c *Client) Health() (*HealthResponse, error)

Configuration

In GogoBee's config:

dictionary:
  url: "http://127.0.0.1:7777"

When GogoBee eventually containerizes, this becomes the container network address. No other changes required.

Replacing Wordnik Calls

Run grep -r -i wordnik . in the GogoBee repo to find all call sites. Replace:

Wordnik call	GogoBee replacement
GET /word/{word}/valid	dictClient.IsValidWord(word, lang)
GET /words/randomWord	dictClient.RandomWord(lang, pos, min, max)
GET /word/{word}/definitions	dictClient.Define(word, lang)
GET /word/{word}/relatedWords	dictClient.Synonyms(word, lang)

Calls previously hardcoded to English: keep lang = "en".

Startup Behaviour

On GogoBee startup, call /health to verify dreamdict is reachable. Log a warning (not a fatal error) if unavailable — dictionary-dependent commands should fail gracefully with a user-facing message rather than taking the whole bot down.

---

Data File Checklist

File	Lang	Source
scowl/final/english-words.*	en	wordlist.aspell.net
wordnet/dict/data.{noun,verb,adj,adv}	en	wordnet.princeton.edu
kaikki-en.jsonl	en	kaikki.org
fr_FR.dic	fr	github.com/LibreOffice/dictionaries
Lexique383.tsv	fr	lexique.org
wolf.xml	fr	github.com/nicolashernandez/WOLF
kaikki-fr.jsonl	fr	kaikki.org
pt_PT.dic	pt-PT	github.com/LibreOffice/dictionaries
dicionario-aberto.xml	pt-PT	github.com/jorgecardoso/dicionario-aberto
kaikki-pt.jsonl	pt-PT	kaikki.org

scripts/download-dict-data.sh automates all downloads. Idempotent — skips existing files unless --force passed. Handles tar extraction (SCOWL), bz2 decompression (WOLF), and git clones where no stable direct URL exists.

---

go.mod Dependencies

modernc.org/sqlite    # Pure-Go SQLite, no CGO

Standard library covers everything else: net/http, encoding/xml, encoding/json, bufio, compress/bzip2, database/sql, flag, log/slog.

---

Testing

Service Tests (internal/dictionary/dict_test.go)

Use :memory: SQLite. Cover:

1. IsValidWord — known word true, unknown false, case-insensitive
2. RandomWord — matches length + POS filters; returns ErrNoMatch on empty set
3. Define — priority ordering (wordnet before wiktionary); empty slice for no definitions
4. Synonyms — deduplicated; empty slice when none
5. Translate — returns correct target-lang translations; empty slice when none
6. ErrNotSeeded — when meta table has no rows
7. Gloss deduplication — identical gloss twice → one row
8. Synonym deduplication — identical synonym twice → one row
9. Translation deduplication — identical translation twice → one row
10. form-of filtering — senses tagged form-of not inserted
11. pt-PT Brazil filtering — Brazil-tagged senses rejected; untagged accepted

HTTP Handler Tests (cmd/server/)

Use httptest.NewServer. Cover:

1. /valid — 200 true/false, 400 on missing params
2. /random — 200 with word, 404 on no match
3. /define — 200 with ordered definitions, 200 with empty array on no defs
4. /synonyms — 200 with list, 200 with empty array
5. /translate — 200 with list, 200 with empty array
6. /health — 200 with counts

---

Expected Database Size

Language	Words	Definitions	Approx size
en	~220,000	~380,000	~180MB
fr	~90,000	~200,000	~80MB
pt-PT	~75,000	~120,000	~50MB
Total			~310MB

Add to .gitignore: dict.db, data/

---

Phase 2: Affix Expansion

Implement shortly after initial launch, not at initial launch.

The base-forms-only approach in the initial import means IsValidWord("running", "en") returns false because only "run" is stored. For Hangman this doesn't matter — DreamDict generates the puzzle word and it's always a base form. But for any command where a player submits a word for validation (UNO, word games, future puzzle modes), this is a real correctness bug, not a theoretical one.

Recommended approach: affix expansion at import time using go-hunspell (github.com/trustmaster/go-hunspell). Parse the .aff rules for each language and generate all valid inflected forms during dictimport, storing them in the words table alongside base forms. This keeps IsValidWord as a simple indexed lookup with no runtime stemming logic.

Do not implement stemming as an alternative. Stemming at query time is more complex, language-specific, and redundant if you expand affixes at import time. Pick one direction — affix expansion at import is the cleaner path for this architecture.

Sequencing: get the service running and validate data quality across all three languages first. Affix expansion is a dictimport change only — the HTTP API, schema, and GogoBee client are untouched.

---

Deferred: Traefik Route for Second Consumer

When a second consumer for DreamDict exists, adding a Traefik route on parodia.dev's internal network is trivial — one label, one DNS entry, twenty minutes of work. Do it then, not now. The service architecture requires no changes at that point.

---

Not Worth Doing: HTTP Caching Headers

The consumers are on localhost, there is no intermediary cache, and SQLite responses are already faster than any cache lookup. Drop this entirely.

---

Phase 2

Implement after initial launch is stable and data quality has been validated across all three languages. Each item below is scoped and sequenced independently -- they do not depend on each other unless noted.

---

P2-1: Hunspell Affix Expansion

See "Phase 2: Affix Expansion" section above. Implement first among Phase 2 items as it fixes a real correctness bug for player-submitted word validation.

---

P2-2: WordNet Synset Cross-Language Backing

The problem with the current translation approach: The current translations table is populated opportunistically from Wiktionary tags, which are inconsistently applied. Coverage is good for common words and thin everywhere else. This is a structural ceiling, not a data quality problem that more sources can fix.

The intentional approach: Princeton WordNet assigns every concept a stable synset ID. WOLF maps French words to those same synset IDs. The Open Multilingual Wordnet (OMW) does the same for Portuguese. Two words in different languages that share a synset ID are semantic equivalents with high confidence -- two independent projects made that mapping independently. DreamDict currently discards synset IDs during import. Preserving them unlocks proper cross-language backing.

Schema additions:

CREATE TABLE IF NOT EXISTS synsets (
    id        INTEGER PRIMARY KEY AUTOINCREMENT,
    synset_id TEXT NOT NULL UNIQUE,  -- Princeton WN ID e.g. '00914031-a'
    pos       TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS word_synsets (
    word_id   INTEGER NOT NULL REFERENCES words(id) ON DELETE CASCADE,
    synset_id INTEGER NOT NULL REFERENCES synsets(id) ON DELETE CASCADE,
    source    TEXT NOT NULL,         -- 'wordnet', 'wolf', 'omw'
    PRIMARY KEY (word_id, synset_id)
);

CREATE INDEX IF NOT EXISTS idx_word_synsets_word_id   ON word_synsets(word_id);
CREATE INDEX IF NOT EXISTS idx_word_synsets_synset_id ON word_synsets(synset_id);

Loader changes:

• wordnet.go -- store the synset offset as synset_id in synsets, link words via word_synsets with source='wordnet'
• wolf.go -- WOLF synset IDs are Princeton WN IDs; store and link with source='wolf'
• New loader omw.go -- Open Multilingual Wordnet Portuguese data; maps pt words to Princeton synset IDs; link with source='omw'
  • Source: https://github.com/omwn/omw-data
  • File: Portuguese tab file from the OMW release

New API method:

// EnglishBacking returns English words and their WordNet definitions that share
// a synset with the given word. Semantic equivalence, not translation guesswork.
func (d *Dictionary) EnglishBacking(word, lang string) ([]EnglishEquivalent, error)

type EnglishEquivalent struct {
    Word       string
    Definition string  // the WordNet gloss for that synset
    Synset     string
}

New endpoint:

GET /backing?word=éphémère&lang=fr
→ {
    "word": "éphémère",
    "lang": "fr",
    "equivalents": [
      {"word": "ephemeral", "definition": "lasting a very short time", "synset": "00914031-a"}
    ]
  }

Fallback behaviour: Words with no synset mapping (proper nouns, invented words, very obscure entries) fall back to the existing translations table as a secondary source. The two mechanisms complement each other -- synsets for authoritative coverage, Wiktionary translations for the rest.

GogoBee impact: Every French and Portuguese word with a WOLF or OMW synset mapping automatically gets authoritative English backing with a real WordNet definition attached -- not just a word, but the concept it represents. Update DreamDictClient with a EnglishBacking() method.

---

P2-3: Antonyms

Data source: Already in Princeton WordNet. The pointer data in data.* files includes antonym relations marked with ! as the pointer symbol. The current wordnet.go loader skips all pointer types. This is a loader change only.

Schema addition:

CREATE TABLE IF NOT EXISTS antonyms (
    id       INTEGER PRIMARY KEY AUTOINCREMENT,
    word_id  INTEGER NOT NULL REFERENCES words(id) ON DELETE CASCADE,
    antonym  TEXT    NOT NULL,
    source   TEXT    NOT NULL,   -- 'wordnet', 'wiktionary'
    UNIQUE(word_id, antonym)
);

CREATE INDEX IF NOT EXISTS idx_antonyms_word_id ON antonyms(word_id);

Loader changes:

• wordnet.go -- parse pointer lines with symbol !; for each antonym pointer, insert into antonyms with source='wordnet'
• wiktionary.go -- kaikki.org entries carry an antonyms[] array alongside synonyms[]; parse and insert with source='wiktionary'

New API method and endpoint:

func (d *Dictionary) Antonyms(word, lang string) ([]string, error)

GET /antonyms?word=happy&lang=en
→ {"word": "happy", "lang": "en", "antonyms": ["unhappy", "sad", "miserable"]}

GogoBee uses: Opposite-word puzzle modes, richer !define output, future word game mechanics that need semantic opposites.

---

P2-4: Frequency Weighting for English and Portuguese

The problem: RandomWord() for English and Portuguese is currently unweighted -- ORDER BY RANDOM() across the full word table. A Hangman puzzle is as likely to return "aardvark" as "house." French already has frequency data from Lexique. English and Portuguese need the same.

Data sources:

• English: SUBTLEX-US (Brysbaert & New, 2009) -- subtitle frequency corpus, freely available. Provides word frequency per million from film/TV subtitles. Same format as Lexique conceptually.

  • URL: http://www.ugent.be/pp/experimentele-psychologie/en/research/documents/subtlexus
  • File: SUBTLEX-US.xlsx (export to TSV before import)

• Portuguese: SUBTLEX-PT exists but coverage skews pt-BR. Better option for pt-PT is the frequency list derived from the CETEMPúblico corpus (European Portuguese newspaper corpus), available via Linguateca.

  • URL: https://www.linguateca.pt/acesso/corpus.php?corpus=CETEMPUBLICO
  • Alternative: use Wiktionary frequency tags as a rough proxy if CETEMPúblico access is cumbersome -- lower quality but available immediately.

Loader changes:

• New subtlex.go -- TSV import for English frequency, same pattern as lexique.go; UPDATE words SET frequency=? WHERE word=? AND lang='en' AND frequency=0
• New cetempublico.go (or wiktfreq.go as fallback) -- same pattern for pt-PT

Schema: No change -- frequency column already exists on words for all languages.

API change: MinFrequency in Options already exists. No API change needed -- this just makes the filter meaningful for en and pt-PT for the first time.

DreamDict dictimport change: Add both loaders to the English and pt-PT execution order, running after word list population.

---

Phase 3

Implement after Phase 2 is complete and stable. These are higher-effort or lower-urgency than Phase 2 items but all have clear value.

---

P3-1: Word Difficulty Scoring

What it is: A composite difficulty score derived from data already in the database -- no new sources required. Computed at import time and stored as a column.

Formula (tune after testing):

difficulty = normalize(1 / frequency) * 0.5
           + normalize(length) * 0.3
           + normalize(syllable_count) * 0.2

Syllable count can be approximated from the word string (vowel cluster counting) or pulled from CMU Pronouncing Dictionary (see P3-2) once that data exists.

Schema addition:

ALTER TABLE words ADD COLUMN difficulty REAL DEFAULT NULL;
-- NULL = not yet scored; 0.0 = easiest; 1.0 = hardest

API change:

Add to Options:

MaxDifficulty float64  // 0.0–1.0; 0 = no filter
MinDifficulty float64

Add to RandomWord() response (exposed via new WordDetail return type if needed) and as a metadata field on /random response:

{"word": "ephemeral", "difficulty": 0.74}

GogoBee uses: Tiered Hangman difficulty without hardcoded word lists. "Easy", "Medium", "Hard" modes map to difficulty ranges. Arena word puzzles can scale to player level.

---

P3-2: Pronunciation Data

Data source: CMU Pronouncing Dictionary (CMUdict)

• URL: http://www.speech.cs.cmu.edu/cgi-bin/cmudict
• Format: plain text, one entry per line: EPHEMERAL IH0 F EH1 M ER0 AH0 L
• Coverage: English only (~134,000 entries)
• License: BSD-style, freely usable

For French and Portuguese, IPA data is available in the kaikki.org Wiktionary dumps under a sounds array per entry -- the current wiktionary loader ignores this field.

Schema addition:

CREATE TABLE IF NOT EXISTS pronunciations (
    id       INTEGER PRIMARY KEY AUTOINCREMENT,
    word_id  INTEGER NOT NULL REFERENCES words(id) ON DELETE CASCADE,
    format   TEXT NOT NULL,   -- 'cmu', 'ipa'
    value    TEXT NOT NULL,
    source   TEXT NOT NULL,   -- 'cmudict', 'wiktionary'
    UNIQUE(word_id, format, source)
);

CREATE INDEX IF NOT EXISTS idx_pronunciations_word_id ON pronunciations(word_id);

Loader changes:

• New cmudict.go -- plain text import; match words to existing words table entries; insert with format='cmu', source='cmudict'
• wiktionary.go -- extend to parse sounds[].ipa field; insert with format='ipa', source='wiktionary'

New API method and endpoint:

func (d *Dictionary) Pronunciation(word, lang string) ([]Pronunciation, error)

type Pronunciation struct {
    Format string  // 'cmu', 'ipa'
    Value  string
    Source string
}

GET /pronunciation?word=ephemeral&lang=en
→ {
    "word": "ephemeral",
    "lang": "en",
    "pronunciations": [
      {"format": "cmu", "value": "IH0 F EH1 M ER0 AH0 L", "source": "cmudict"},
      {"format": "ipa", "value": "ɪˈfɛm.ər.əl", "source": "wiktionary"}
    ]
  }

GogoBee uses: !define output enrichment, syllable count for difficulty scoring (P3-1), future rhyme game mechanics (words sharing a CMU tail sequence are rhymes).

---

P3-3: Etymology

Data source: kaikki.org Wiktionary dumps already downloaded -- etymology data is in the etymology_text field per entry. The current wiktionary.go loader ignores it.

Schema addition:

CREATE TABLE IF NOT EXISTS etymology (
    id         INTEGER PRIMARY KEY AUTOINCREMENT,
    word_id    INTEGER NOT NULL REFERENCES words(id) ON DELETE CASCADE,
    text       TEXT NOT NULL,   -- free-form etymology string from Wiktionary
    source     TEXT NOT NULL,   -- 'wiktionary'
    UNIQUE(word_id, source)
);

CREATE INDEX IF NOT EXISTS idx_etymology_word_id ON etymology(word_id);

Loader change:

• wiktionary.go -- extend to parse etymology_text field; insert into etymology if non-empty and not a redirect/stub string (skip entries that are just "See X" or empty after trimming)

New API method and endpoint:

func (d *Dictionary) Etymology(word, lang string) (string, error)

GET /etymology?word=ephemeral&lang=en
→ {
    "word": "ephemeral",
    "lang": "en",
    "etymology": "From Medieval Latin ephemerus, from Ancient Greek ἐφήμερος (ephḗmeros, \"lasting only a day\"), from ἐπί (epí, \"on\") + ἡμέρα (hēméra, \"day\")."
  }

GogoBee uses: !etymology command for the community. Particularly relevant given the pt-PT context -- shared Latin/Greek roots between Portuguese, French, and English make etymology a genuine learning tool, not just trivia. Word of the Day can optionally append a one-line etymology note.

Quality note: Wiktionary etymology text is free-form and inconsistently structured. Surface it as-is rather than attempting to parse it into structured fields. A raw string is useful; a badly parsed structured field is not.

---

Roadmap Summary

Phase	Item	Effort	Payoff
2	Affix expansion	Medium	High -- fixes real validation bug
2	Synset cross-language backing	Medium	High -- authoritative multilingual semantics
2	Antonyms	Low	Medium -- already in WordNet data
2	Frequency for en + pt-PT	Low-Medium	High -- better random word quality
3	Difficulty scoring	Low	High -- derived, no new data
3	Pronunciation	Medium	Medium -- en only initially
3	Etymology	Low	High -- data already downloaded