dreamdict/README.md

DreamDict

A self-hosted, CGO-free HTTP dictionary service supporting English, French, European Portuguese, and Mandarin Chinese. Built to replace metered third-party APIs like Wordnik with something you own outright.

No API keys. No rate limits. No external runtime dependencies. Sub-millisecond response times on commodity hardware.

---

Why DreamDict exists

Most open source dictionary tooling is either a dataset (WordNet, Wiktionary dumps, Hunspell) or a desktop application. There is no self-hosted HTTP service that combines all of them into a programmable API — so we built one.

The pt-PT distinction matters and is handled explicitly. Most open NLP resources default to Brazilian Portuguese (pt-BR) or treat both variants as interchangeable. DreamDict uses the LibreOffice pt-PT Hunspell dictionary, Dicionário Aberto, CETEMPúblico frequency data, and Wiktionary entries filtered to European Portuguese. If you are building something for a European Portuguese-speaking audience, this is the only self-hosted option that takes that seriously.

Benchmark results (end-to-end, including HTTP and JSON serialization):

Endpoint	Latency	Throughput
GET /valid	~20 µs	~50,000 req/s
GET /random (filtered)	~187 µs	~5,300 req/s
GET /define	~50 µs	~20,000 req/s
GET /synonyms	~37 µs	~27,000 req/s
GET /translate	~56 µs	~18,000 req/s

---

Requirements

• Go 1.25+
• No CGO required — uses modernc.org/sqlite, a pure Go SQLite port
• Single static binary, no runtime dependencies

---

Quick Start

# Download all dictionary source data (~2-3 GB)
./scripts/download-dict-data.sh

# Build the database (~330 MB, one-time)
go run ./cmd/dictimport --data ./data --db ./dict.db

# Start the server
go run ./cmd/server --db ./dict.db

The server listens on 127.0.0.1:7777 by default.

---

API

All endpoints are read-only GET requests returning JSON. When authentication is enabled, all endpoints except /health require a bearer token (see Authentication).

GET /valid

Check if a word exists in a language's word list. Includes inflected forms via affix expansion — running and ran resolve correctly, not just base forms.

curl 'localhost:7777/valid?word=running&lang=en'
{"valid":true}

GET /random

Return a random word. Optional filters: pos (noun, verb, adjective, adverb), min/max (word length), min_freq (frequency score), min_difficulty/max_difficulty (0.0–1.0), variant (us or gb, English only).

curl 'localhost:7777/random?lang=en&pos=noun&min=5&max=8&max_difficulty=0.5'
{"word":"lantern","difficulty":0.42}

GET /define

Return all definitions ordered by source priority. Curated sources (WordNet, WOLF, Dicionário Aberto) appear before Wiktionary supplemental definitions. For English words, the response includes a variant field ("us", "gb", or omitted for common English) indicating regional spelling.

curl 'localhost:7777/define?word=ardor&lang=en'
{
  "word": "ardor",
  "lang": "en",
  "variant": "us",
  "definitions": [
    {"pos":"noun","gloss":"a feeling of strong eagerness","source":"wordnet","priority":10},
    {"pos":"noun","gloss":"feelings of great warmth and intensity","source":"wordnet","priority":10}
  ]
}

GET /synonyms

Return deduplicated synonyms sourced from WordNet, WOLF, and Wiktionary.

curl 'localhost:7777/synonyms?word=happy&lang=en'
{"word":"happy","lang":"en","synonyms":["content","glad","joyful","pleased"]}

GET /antonyms

Return antonyms sourced from WordNet and Wiktionary.

curl 'localhost:7777/antonyms?word=happy&lang=en'
{"word":"happy","lang":"en","antonyms":["sad","unhappy"]}

GET /translate

Return known translations between supported languages.

curl 'localhost:7777/translate?word=cat&from=en&to=fr'
{"word":"cat","from":"en","to":"fr","translations":["chat","minet"]}

GET /backing

Return English semantic equivalents for a non-English word via shared Princeton WordNet synset IDs. This is semantic equivalence — not translation guesswork. Falls back to the translations table when no synset mapping exists.

curl 'localhost:7777/backing?word=éphémère&lang=fr'
{
  "word": "éphémère",
  "lang": "fr",
  "equivalents": [
    {"word":"ephemeral","definition":"lasting a very short time","synset":"00914031-a"}
  ]
}

GET /pronunciation

Return pronunciation data. English gets CMU phonemes and IPA. All other languages get IPA from Wiktionary where available.

curl 'localhost:7777/pronunciation?word=ephemeral&lang=en'
{
  "word": "ephemeral",
  "lang": "en",
  "pronunciations": [
    {"format":"cmu","value":"IH0 F EH1 M ER0 AH0 L","source":"cmudict"},
    {"format":"ipa","value":"ɪˈfɛm.ər.əl","source":"wiktionary"}
  ]
}

GET /etymology

Return the etymology of a word from Wiktionary. Returned as free-form text — Wiktionary etymology is not structured consistently enough to parse reliably.

curl 'localhost:7777/etymology?word=ephemeral&lang=en'
{
  "word": "ephemeral",
  "lang": "en",
  "etymology": "From Medieval Latin ephemerus, from Ancient Greek ἐφήμερος (ephḗmeros, \"lasting only a day\")..."
}

GET /frequency

Return the frequency score for a word. Higher values = more common. Sourced from SUBTLEX-US (en), Lexique (fr), and CETEMPúblico (pt-PT).

curl 'localhost:7777/frequency?word=house&lang=en'
{"word":"house","lang":"en","frequency":800}

GET /frequency/batch

Batch frequency lookup. Comma-separated words, max 100 per request.

curl 'localhost:7777/frequency/batch?words=house,cat,ephemeral&lang=en'
{"lang":"en","frequencies":{"cat":800,"ephemeral":50,"house":800}}

GET /difficulty

Return the difficulty score for a word (0.0 = easiest, 1.0 = hardest).

curl 'localhost:7777/difficulty?word=ephemeral&lang=en'
{"word":"ephemeral","lang":"en","difficulty":0.72}

GET /words

Return a list of words matching filters. Same filter options as /random plus variant. Returns up to 20,000 results.

curl 'localhost:7777/words?lang=en&min=5&max=5&variant=gb&min_freq=500'
{"lang":"en","count":42,"words":["colour","fibre","honour",...]}

GET /rhyme

Return English rhyming words based on CMU phoneme tail matching. Optional limit parameter (default 10).

curl 'localhost:7777/rhyme?word=cat&limit=5'
{"word":"cat","rhymes":["bat","flat","hat","mat","sat"]}

GET /health

Health check with database statistics. Always unauthenticated — safe for monitoring tools.

curl localhost:7777/health
{
  "status": "ok",
  "db_path": "./dict.db",
  "word_counts": {"en":136615,"fr":56096,"pt-PT":136300,"zh":120883},
  "def_counts":  {"en":361640,"fr":137738,"pt-PT":84577,"zh":199929},
  "imported_at": "2026-04-04T21:31:17Z",
  "schema_version": "2"
}

Error Responses

{"error": "bad_request",   "message": "word and lang are required"}
{"error": "no_match",      "message": "no words match the given filters"}
{"error": "unauthorized",  "message": "invalid or missing bearer token"}

---

Configuration

Flag	Env Var	Default	Description
--port	DREAMDICT_PORT	7777	Listen port
--host	DREAMDICT_HOST	127.0.0.1	Listen host (localhost only by default)
--db	DREAMDICT_DB	./dict.db	Path to SQLite database
--token	DREAMDICT_TOKEN	(none)	Bearer token for API authentication
--log-level		info	debug, info, warn, error

Flags take precedence over environment variables.

Authentication

When --token or DREAMDICT_TOKEN is set, all endpoints except /health require a bearer token. When no token is configured, authentication is disabled.

# Start with auth enabled
DREAMDICT_TOKEN=your-secret-token go run ./cmd/server --db ./dict.db

# Authenticated request
curl -H 'Authorization: Bearer your-secret-token' 'localhost:7777/define?word=saudade&lang=pt-PT'

Requests without a valid token receive a 401 response. The error message does not distinguish between a missing token and an incorrect one.

---

Import CLI

go run ./cmd/dictimport [flags]

Flag	Default	Description
--db	./dict.db	Output database path
--data	./data	Source data directory
--langs	en,fr,pt-PT,zh	Languages to import
--clean	false	Drop and recreate all tables before import
--skip		Comma-separated loader names to skip

Loaders

Loader	Language	What it provides
scowl	en	Word list with frequency tiers and regional variant tagging (us/gb)
affix-en	en	Inflected form expansion
wordnet	en	Definitions, synonyms, antonyms, synset IDs
subtlex	en	SUBTLEX-US subtitle frequency data
cmudict	en	CMU phoneme pronunciation data
wiktionary-en	en	Supplemental definitions, synonyms, antonyms, translations, IPA, etymology
hunspell-fr	fr	Word list
affix-fr	fr	Inflected form expansion
lexique	fr	POS and frequency enrichment
wolf	fr	Definitions, synonyms, synset IDs
wiktionary-fr	fr	Supplemental definitions, synonyms, antonyms, translations, IPA, etymology
hunspell-pt	pt-PT	European Portuguese word list (not pt-BR)
affix-pt-PT	pt-PT	Inflected form expansion
dicionario	pt-PT	Definitions from Dicionário Aberto
cetempublico	pt-PT	Frequency data from CETEMPúblico corpus
omw-pt	pt-PT	Open Multilingual Wordnet synset mappings
wiktionary-pt	pt-PT	Supplemental definitions, synonyms, antonyms, translations, IPA, etymology (filtered to European Portuguese entries)
cedict	zh	Words, definitions, translations via CC-CEDICT
wiktionary-zh	zh	Supplemental Chinese data
prune-orphans	all	Removes ghost words with no definitions, frequency, or references
difficulty	all	Composite difficulty scoring — runs last

---

Data Sources

Source	Language	What it provides
SCOWL	en	Word list + frequency tiers
WordNet 3.0	en	Definitions, synonyms, antonyms, synset IDs
SUBTLEX-US	en	Subtitle-based word frequency
CMU Pronouncing Dictionary	en	Phoneme pronunciation
Hunspell/LibreOffice	fr, pt-PT	Word lists + affix rules
Lexique 3.83	fr	POS + frequency enrichment
WOLF	fr	Definitions, synonyms, synset IDs
Dicionário Aberto	pt-PT	Definitions
CETEMPúblico	pt-PT	European Portuguese word frequency
Open Multilingual Wordnet	pt-PT	Synset mappings to Princeton WordNet
CC-CEDICT	zh	Words, definitions, translations
Wiktionary via kaikki.org	all	Definitions, synonyms, antonyms, translations, IPA, etymology

Run ./scripts/download-dict-data.sh to fetch all sources automatically. Pass --force to re-download existing files.

---

Difficulty Scoring

Each word receives a composite difficulty score (0.0 = easiest, 1.0 = hardest) computed at import time from frequency, word length, and syllable count:

difficulty = inverse_frequency * 0.5 + word_length * 0.3 + syllable_count * 0.2

Use min_difficulty / max_difficulty on /random to select words by difficulty tier without maintaining hardcoded word lists.

---

Database Schema

Schema version 2. Full schema in internal/dictionary/db.go.

Table	Purpose
words	Core word inventory: word, lang, pos, frequency, difficulty, variant
definitions	Definitions with source and priority
synonyms	Synonym relationships
antonyms	Antonym relationships
translations	Cross-language translations
synsets	Princeton WordNet synset IDs
word_synsets	Word-to-synset mappings for cross-language semantic backing
pronunciations	CMU phonemes and IPA
etymology	Word origin text from Wiktionary
meta	Import metadata: schema version, import timestamp, word/definition counts

---

Deployment

A systemd unit file is provided at deploy/dreamdict.service. The service runs as a dedicated unprivileged system user and binds to localhost only by default.

# Create dedicated system user
useradd --system --no-create-home --shell /usr/sbin/nologin dreamdict

# Build and install binary
go build -o /usr/local/bin/dreamdict ./cmd/server
chmod 755 /usr/local/bin/dreamdict

# Build the database (run as yourself, not the dreamdict user)
mkdir -p /opt/dreamdict /etc/dreamdict
go run ./cmd/dictimport --db /opt/dreamdict/dict.db --data ./data
chown -R dreamdict:dreamdict /opt/dreamdict

# Configure
cat > /etc/dreamdict/dreamdict.env <<EOF
DREAMDICT_PORT=7777
DREAMDICT_DB=/opt/dreamdict/dict.db
DREAMDICT_TOKEN=changeme
EOF
chmod 600 /etc/dreamdict/dreamdict.env

# Install and start
cp deploy/dreamdict.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now dreamdict

# Verify
systemctl status dreamdict
curl localhost:7777/health

Updating the Database

systemctl stop dreamdict
go run ./cmd/dictimport --db /opt/dreamdict/dict.db --clean
chown dreamdict:dreamdict /opt/dreamdict/dict.db
systemctl start dreamdict

---

Testing

go test ./...

---

Project Structure

cmd/server/           HTTP server binary
cmd/dictimport/       One-time data import CLI
internal/dictionary/  Core package: schema, types, queries
internal/loader/      One loader per data source
scripts/              Data download script
deploy/               systemd unit file
data/                 Source data files (gitignored)

---

Expected Database Size

Language	Words	Definitions
en	~137k	~362k
fr	~56k	~138k
pt-PT	~136k	~85k
zh	~121k	~200k
Total	~450k	~785k

English words include ~3,800 US-only and ~3,900 GB-only regional variants. Word counts include inflected forms from affix expansion, with ghost words pruned automatically.

---

License

MIT