drwily/dreamdict
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rewrite README for public release

Browse Source 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
prosolis
2026-04-03 09:02:26 -07:00
parent 56e073f753
commit a0cca2cb19
1 changed files with 212 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files

335
README.md
View File

@@ -1,13 +1,36 @@
# DreamDict
A self-contained HTTP dictionary service providing word validation, random word generation, definitions, synonyms, antonyms, translations, pronunciation, etymology, and cross-language semantic backing for English (`en`), French (`fr`), European Portuguese (`pt-PT`), and Mandarin Chinese (`zh`).
A self-hosted, CGO-free HTTP dictionary service supporting English, French, **European Portuguese**, and Mandarin Chinese. Built to replace metered third-party APIs like Wordnik with something you own outright.
Runs as a plain Go binary managed by systemd. Binds to localhost only — no external network exposure. Designed as a dictionary backend for [GogoBee](https://github.com/reala-misaki/gogobee), replacing Wordnik.
No API keys. No rate limits. No external runtime dependencies. Sub-millisecond response times on commodity hardware.
---
## Why DreamDict exists
Most open source dictionary tooling is either a dataset (WordNet, Wiktionary dumps, Hunspell) or a desktop application. There is no self-hosted HTTP service that combines all of them into a programmable API — so we built one.
The pt-PT distinction matters and is handled explicitly. Most open NLP resources default to Brazilian Portuguese (pt-BR) or treat both variants as interchangeable. DreamDict uses the LibreOffice pt-PT Hunspell dictionary, Dicionário Aberto, CETEMPúblico frequency data, and Wiktionary entries filtered to European Portuguese. If you are building something for a European Portuguese-speaking audience, this is the only self-hosted option that takes that seriously.
**Benchmark results (end-to-end, including HTTP and JSON serialization):**
| Endpoint | Latency | Throughput |
|---|---|---|
| GET /valid | ~20 µs | ~50,000 req/s |
| GET /random (filtered) | ~187 µs | ~5,300 req/s |
| GET /define | ~50 µs | ~20,000 req/s |
| GET /synonyms | ~37 µs | ~27,000 req/s |
| GET /translate | ~56 µs | ~18,000 req/s |
---
## Requirements
- Go 1.25+
- No CGO required (uses [modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite))
- No CGO required — uses [modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite), a pure Go SQLite port
- Single static binary, no runtime dependencies
---
## Quick Start
@@ -15,7 +38,7 @@ Runs as a plain Go binary managed by systemd. Binds to localhost only — no ext
# Download all dictionary source data (~2-3 GB)
./scripts/download-dict-data.sh
# Build the database (~310 MB)
# Build the database (~330 MB, one-time)
go run ./cmd/dictimport --data ./data --db ./dict.db
# Start the server
@@ -24,124 +47,149 @@ go run ./cmd/server --db ./dict.db
The server listens on `127.0.0.1:7777` by default.
---
## API
All endpoints are read-only `GET` requests returning JSON.
All endpoints are read-only `GET` requests returning JSON. When authentication is enabled, all endpoints except `/health` require a bearer token (see [Authentication](#authentication)).
### `GET /valid?word=...&lang=...`
### `GET /valid`
Check if a word exists in a language's word list. Includes inflected forms via affix expansion.
Check if a word exists in a language's word list. Includes inflected forms via affix expansion — `running` and `ran` resolve correctly, not just base forms.
```
$ curl 'localhost:7777/valid?word=running&lang=en'
```bash
curl 'localhost:7777/valid?word=running&lang=en'
{"valid":true}
```
### `GET /random?lang=...`
### `GET /random`
Return a random word. Optional filters: `pos` (`noun`, `verb`, `adjective`, `adverb`), `min`/`max` (word length), `min_freq` (frequency score), `min_difficulty`/`max_difficulty` (0.0–1.0).
```
$ curl 'localhost:7777/random?lang=en&pos=noun&min=5&max=8'
```bash
curl 'localhost:7777/random?lang=en&pos=noun&min=5&max=8&max_difficulty=0.5'
{"word":"lantern","difficulty":0.42}
```
### `GET /define?word=...&lang=...`
### `GET /define`
Return all definitions, ordered by source priority.
Return all definitions ordered by source priority. Curated sources (WordNet, WOLF, Dicionário Aberto) appear before Wiktionary supplemental definitions.
```
$ curl 'localhost:7777/define?word=ephemeral&lang=en'
{"word":"ephemeral","lang":"en","definitions":[
```bash
curl 'localhost:7777/define?word=ephemeral&lang=en'
{
"word": "ephemeral",
"lang": "en",
"definitions": [
{"pos":"adjective","gloss":"lasting for a very short time","source":"wordnet","priority":10},
{"pos":"adjective","gloss":"lasting only for a day","source":"wiktionary","priority":20}
]}
]
}
```
### `GET /synonyms?word=...&lang=...`
### `GET /synonyms`
Return deduplicated synonyms.
Return deduplicated synonyms sourced from WordNet, WOLF, and Wiktionary.
```
$ curl 'localhost:7777/synonyms?word=happy&lang=en'
```bash
curl 'localhost:7777/synonyms?word=happy&lang=en'
{"word":"happy","lang":"en","synonyms":["content","glad","joyful","pleased"]}
```
### `GET /antonyms?word=...&lang=...`
### `GET /antonyms`
Return antonyms sourced from WordNet and Wiktionary.
```
$ curl 'localhost:7777/antonyms?word=happy&lang=en'
```bash
curl 'localhost:7777/antonyms?word=happy&lang=en'
{"word":"happy","lang":"en","antonyms":["sad","unhappy"]}
```
### `GET /translate?word=...&from=...&to=...`
### `GET /translate`
Return known translations between supported languages.
```
$ curl 'localhost:7777/translate?word=cat&from=en&to=fr'
```bash
curl 'localhost:7777/translate?word=cat&from=en&to=fr'
{"word":"cat","from":"en","to":"fr","translations":["chat","minet"]}
```
### `GET /backing?word=...&lang=...`
### `GET /backing`
Return English semantic equivalents for a non-English word via shared WordNet synset IDs. Falls back to the translations table when no synset mapping exists.
Return English semantic equivalents for a non-English word via shared Princeton WordNet synset IDs. This is semantic equivalence — not translation guesswork. Falls back to the translations table when no synset mapping exists.
```
$ curl 'localhost:7777/backing?word=éphémère&lang=fr'
{"word":"éphémère","lang":"fr","equivalents":[
```bash
curl 'localhost:7777/backing?word=éphémère&lang=fr'
{
"word": "éphémère",
"lang": "fr",
"equivalents": [
{"word":"ephemeral","definition":"lasting a very short time","synset":"00914031-a"}
]}
]
}
```
### `GET /pronunciation?word=...&lang=...`
### `GET /pronunciation`
Return pronunciation data (CMU phonemes for English, IPA from Wiktionary for all languages).
Return pronunciation data. English gets CMU phonemes and IPA. All other languages get IPA from Wiktionary where available.
```
$ curl 'localhost:7777/pronunciation?word=ephemeral&lang=en'
{"word":"ephemeral","lang":"en","pronunciations":[
```bash
curl 'localhost:7777/pronunciation?word=ephemeral&lang=en'
{
"word": "ephemeral",
"lang": "en",
"pronunciations": [
{"format":"cmu","value":"IH0 F EH1 M ER0 AH0 L","source":"cmudict"},
{"format":"ipa","value":"ɪˈfɛm.ər.əl","source":"wiktionary"}
]}
]
}
```
### `GET /etymology?word=...&lang=...`
### `GET /etymology`
Return the etymology of a word from Wiktionary.
Return the etymology of a word from Wiktionary. Returned as free-form text — Wiktionary etymology is not structured consistently enough to parse reliably.
```
$ curl 'localhost:7777/etymology?word=ephemeral&lang=en'
{"word":"ephemeral","lang":"en","etymology":"From Medieval Latin ephemerus, from Ancient Greek ἐφήμερος (ephḗmeros, \"lasting only a day\")..."}
```bash
curl 'localhost:7777/etymology?word=ephemeral&lang=en'
{
"word": "ephemeral",
"lang": "en",
"etymology": "From Medieval Latin ephemerus, from Ancient Greek ἐφήμερος (ephḗmeros, \"lasting only a day\")..."
}
```
### `GET /frequency?word=...&lang=...`
### `GET /frequency`
Return the frequency score for a word. Higher values = more common.
Return the frequency score for a word. Higher values = more common. Sourced from SUBTLEX-US (en), Lexique (fr), and CETEMPúblico (pt-PT).
```
$ curl 'localhost:7777/frequency?word=house&lang=en'
```bash
curl 'localhost:7777/frequency?word=house&lang=en'
{"word":"house","lang":"en","frequency":800}
```
### `GET /frequency/batch?words=...&lang=...`
### `GET /frequency/batch`
Batch frequency lookup. Comma-separated words, max 100.
Batch frequency lookup. Comma-separated words, max 100 per request.
```
$ curl 'localhost:7777/frequency/batch?words=house,cat,ephemeral&lang=en'
```bash
curl 'localhost:7777/frequency/batch?words=house,cat,ephemeral&lang=en'
{"lang":"en","frequencies":{"cat":800,"ephemeral":50,"house":800}}
```
### `GET /health`
Health check with database stats.
Health check with database statistics. Always unauthenticated — safe for monitoring tools.
```
$ curl localhost:7777/health
{"status":"ok","db_path":"./dict.db","word_counts":{"en":214832,"fr":82104,"pt-PT":71088},...}
```bash
curl localhost:7777/health
{
"status": "ok",
"db_path": "./dict.db",
"word_counts": {"en":214832,"fr":82104,"pt-PT":71088,"zh":120000},
"def_counts": {"en":380000,"fr":200000,"pt-PT":120000,"zh":80000},
"imported_at": "2026-03-28T14:22:00Z",
"schema_version": "2"
}
```
### Error Responses
@@ -149,42 +197,38 @@ $ curl localhost:7777/health
```json
{"error": "bad_request", "message": "word and lang are required"}
{"error": "no_match", "message": "no words match the given filters"}
{"error": "unauthorized", "message": "invalid or missing bearer token"}
```
## Server Configuration
---
## Configuration
| Flag | Env Var | Default | Description |
|---|---|---|---|
| `--port` | `DREAMDICT_PORT` | `7777` | Listen port |
| `--host` | `DREAMDICT_HOST` | `127.0.0.1` | Listen host |
| `--host` | `DREAMDICT_HOST` | `127.0.0.1` | Listen host (localhost only by default) |
| `--db` | `DREAMDICT_DB` | `./dict.db` | Path to SQLite database |
| `--token` | `DREAMDICT_TOKEN` | _(none)_ | Bearer token for API auth |
| `--token` | `DREAMDICT_TOKEN` | _(none)_ | Bearer token for API authentication |
| `--log-level` | | `info` | `debug`, `info`, `warn`, `error` |
Flags take precedence over environment variables.
### Authentication
When `--token` or `DREAMDICT_TOKEN` is set, all endpoints except `/health` require a bearer token:
When `--token` or `DREAMDICT_TOKEN` is set, all endpoints except `/health` require a bearer token. When no token is configured, authentication is disabled.
```bash
# Start with auth enabled
DREAMDICT_TOKEN=mysecret go run ./cmd/server --db ./dict.db
DREAMDICT_TOKEN=your-secret-token go run ./cmd/server --db ./dict.db
# Authenticated request
curl -H 'Authorization: Bearer mysecret' 'localhost:7777/define?word=casa&lang=pt-PT'
# Health is always public (for monitoring)
curl localhost:7777/health
curl -H 'Authorization: Bearer your-secret-token' 'localhost:7777/define?word=saudade&lang=pt-PT'
```
Requests without a valid token receive a `401` response:
Requests without a valid token receive a `401` response. The error message does not distinguish between a missing token and an incorrect one.
```json
{"error": "unauthorized", "message": "invalid or missing bearer token"}
```
When no token is configured, auth is disabled and all endpoints are open.
---
## Import CLI
@@ -197,102 +241,121 @@ go run ./cmd/dictimport [flags]
| `--db` | `./dict.db` | Output database path |
| `--data` | `./data` | Source data directory |
| `--langs` | `en,fr,pt-PT,zh` | Languages to import |
| `--clean` | `false` | Drop and recreate all tables first |
| `--clean` | `false` | Drop and recreate all tables before import |
| `--skip` | | Comma-separated loader names to skip |
### Loader Names
### Loaders
| Loader | Language | What it does |
| Loader | Language | What it provides |
|---|---|---|
| `scowl` | en | English word list with frequency tiers |
| `affix-en` | en | English inflected form expansion |
| `wordnet` | en | Definitions, synonyms, antonyms, synset mappings |
| `subtlex` | en | SUBTLEX-US frequency enrichment |
| `cmudict` | en | CMU pronunciation data |
| `scowl` | en | Word list with frequency tiers |
| `affix-en` | en | Inflected form expansion |
| `wordnet` | en | Definitions, synonyms, antonyms, synset IDs |
| `subtlex` | en | SUBTLEX-US subtitle frequency data |
| `cmudict` | en | CMU phoneme pronunciation data |
| `wiktionary-en` | en | Supplemental definitions, synonyms, antonyms, translations, IPA, etymology |
| `hunspell-fr` | fr | French word list |
| `affix-fr` | fr | French inflected form expansion |
| `lexique` | fr | French POS + frequency enrichment |
| `wolf` | fr | French definitions, synonyms, synset mappings |
| `hunspell-fr` | fr | Word list |
| `affix-fr` | fr | Inflected form expansion |
| `lexique` | fr | POS and frequency enrichment |
| `wolf` | fr | Definitions, synonyms, synset IDs |
| `wiktionary-fr` | fr | Supplemental definitions, synonyms, antonyms, translations, IPA, etymology |
| `hunspell-pt` | pt-PT | Portuguese word list |
| `affix-pt-PT` | pt-PT | Portuguese inflected form expansion |
| `dicionario` | pt-PT | Portuguese definitions |
| `cetempublico` | pt-PT | Portuguese frequency enrichment |
| `hunspell-pt` | pt-PT | European Portuguese word list (not pt-BR) |
| `affix-pt-PT` | pt-PT | Inflected form expansion |
| `dicionario` | pt-PT | Definitions from Dicionário Aberto |
| `cetempublico` | pt-PT | Frequency data from CETEMPúblico corpus |
| `omw-pt` | pt-PT | Open Multilingual Wordnet synset mappings |
| `wiktionary-pt` | pt-PT | Supplemental definitions, synonyms, antonyms, translations, IPA, etymology |
| `cedict` | zh | Chinese words, definitions, translations |
| `wiktionary-pt` | pt-PT | Supplemental definitions, synonyms, antonyms, translations, IPA, etymology (filtered to European Portuguese entries) |
| `cedict` | zh | Words, definitions, translations via CC-CEDICT |
| `wiktionary-zh` | zh | Supplemental Chinese data |
| `difficulty` | all | Composite difficulty scoring (runs last) |
| `difficulty` | all | Composite difficulty scoring — runs last |
---
## Data Sources
| Source | Language | What it provides |
|---|---|---|
| [SCOWL](https://wordlist.aspell.net/) | en | Word list + frequency tiers |
| [WordNet](https://wordnet.princeton.edu/) | en | Definitions, synonyms, antonyms, synset IDs |
| [WordNet 3.1](https://wordnet.princeton.edu/) | en | Definitions, synonyms, antonyms, synset IDs |
| [SUBTLEX-US](http://www.ugent.be/pp/experimentele-psychologie/en/research/documents/subtlexus) | en | Subtitle-based word frequency |
| [CMU Pronouncing Dictionary](http://www.speech.cs.cmu.edu/cgi-bin/cmudict) | en | Phoneme-based pronunciation |
| [CMU Pronouncing Dictionary](http://www.speech.cs.cmu.edu/cgi-bin/cmudict) | en | Phoneme pronunciation |
| [Hunspell/LibreOffice](https://github.com/LibreOffice/dictionaries) | fr, pt-PT | Word lists + affix rules |
| [Lexique](http://www.lexique.org/) | fr | POS + frequency enrichment |
| [Lexique 3.83](http://www.lexique.org/) | fr | POS + frequency enrichment |
| [WOLF](https://github.com/nicolashernandez/WOLF) | fr | Definitions, synonyms, synset IDs |
| [Dicionario Aberto](https://github.com/jorgecardoso/dicionario-aberto) | pt-PT | Definitions |
| [CETEMPúblico](https://www.linguateca.pt/acesso/corpus.php?corpus=CETEMPUBLICO) | pt-PT | Word frequency |
| [Dicionário Aberto](https://github.com/jorgecardoso/dicionario-aberto) | pt-PT | Definitions |
| [CETEMPúblico](https://www.linguateca.pt/acesso/corpus.php?corpus=CETEMPUBLICO) | pt-PT | European Portuguese word frequency |
| [Open Multilingual Wordnet](https://github.com/omwn/omw-data) | pt-PT | Synset mappings to Princeton WordNet |
| [CC-CEDICT](https://cc-cedict.org/) | zh | Words, definitions, translations |
| [Wiktionary (kaikki.org)](https://kaikki.org/) | all | Definitions, synonyms, antonyms, translations, IPA pronunciation, etymology |
| [Wiktionary via kaikki.org](https://kaikki.org/) | all | Definitions, synonyms, antonyms, translations, IPA, etymology |
Run `./scripts/download-dict-data.sh` to fetch everything. Pass `--force` to re-download existing files.
Run `./scripts/download-dict-data.sh` to fetch all sources automatically. Pass `--force` to re-download existing files.
## Database Schema (v2)
---
### Tables
## Difficulty Scoring
| Table | Purpose |
|---|---|
| `words` | Core word inventory (word, lang, pos, frequency, difficulty) |
| `definitions` | Word definitions with source priority |
| `synonyms` | Synonym relationships |
| `antonyms` | Antonym relationships |
| `translations` | Cross-language translations |
| `synsets` | Princeton WordNet synset IDs |
| `word_synsets` | Maps words to synsets (for cross-language semantic backing) |
| `pronunciations` | CMU phonemes and IPA data |
| `etymology` | Word origin text from Wiktionary |
| `meta` | Import metadata (imported_at, schema_version) |
### Difficulty Scoring
Each word receives a composite difficulty score (0.0 = easiest, 1.0 = hardest) computed at import time:
Each word receives a composite difficulty score (0.0 = easiest, 1.0 = hardest) computed at import time from frequency, word length, and syllable count:
```
difficulty = inverse_frequency * 0.5 + word_length * 0.3 + syllable_count * 0.2
```
Use `min_difficulty` / `max_difficulty` on `/random` to select words by difficulty tier.
Use `min_difficulty` / `max_difficulty` on `/random` to select words by difficulty tier without maintaining hardcoded word lists.
---
## Database Schema
Schema version 2. Full schema in `internal/dictionary/db.go`.
| Table | Purpose |
|---|---|
| `words` | Core word inventory: word, lang, pos, frequency, difficulty |
| `definitions` | Definitions with source and priority |
| `synonyms` | Synonym relationships |
| `antonyms` | Antonym relationships |
| `translations` | Cross-language translations |
| `synsets` | Princeton WordNet synset IDs |
| `word_synsets` | Word-to-synset mappings for cross-language semantic backing |
| `pronunciations` | CMU phonemes and IPA |
| `etymology` | Word origin text from Wiktionary |
| `meta` | Import metadata: schema version, import timestamp, word/definition counts |
---
## Deployment
A systemd unit file is provided at `deploy/dreamdict.service`.
A systemd unit file is provided at `deploy/dreamdict.service`. The service runs as a dedicated unprivileged system user and binds to localhost only by default.
```bash
# Create system user
# Create dedicated system user
useradd --system --no-create-home --shell /usr/sbin/nologin dreamdict
# Build and install
# Build and install binary
go build -o /usr/local/bin/dreamdict ./cmd/server
chmod 755 /usr/local/bin/dreamdict
# Build the database (run as yourself, not the dreamdict user)
mkdir -p /opt/dreamdict /etc/dreamdict
go run ./cmd/dictimport --db /opt/dreamdict/dict.db --data ./data
chown -R dreamdict:dreamdict /opt/dreamdict
# Install service
cp deploy/dreamdict.service /etc/systemd/system/
echo 'DREAMDICT_PORT=7777' > /etc/dreamdict/dreamdict.env
echo 'DREAMDICT_DB=/opt/dreamdict/dict.db' >> /etc/dreamdict/dreamdict.env
echo 'DREAMDICT_TOKEN=changeme' >> /etc/dreamdict/dreamdict.env
# Configure
cat > /etc/dreamdict/dreamdict.env <<EOF
DREAMDICT_PORT=7777
DREAMDICT_DB=/opt/dreamdict/dict.db
DREAMDICT_TOKEN=changeme
EOF
chmod 600 /etc/dreamdict/dreamdict.env
# Install and start
cp deploy/dreamdict.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now dreamdict
# Verify
systemctl status dreamdict
curl localhost:7777/health
```
### Updating the Database
@@ -304,24 +367,30 @@ chown dreamdict:dreamdict /opt/dreamdict/dict.db
systemctl start dreamdict
```
---
## Testing
```bash
go test ./...
```
---
## Project Structure
```
cmd/server/ HTTP server (12 endpoints)
cmd/server/ HTTP server binary
cmd/dictimport/ One-time data import CLI
internal/dictionary/ Core package: DB schema, types, queries
internal/loader/ Data source parsers (one per source)
scripts/ Download script for source data
internal/dictionary/ Core package: schema, types, queries
internal/loader/ One loader per data source
scripts/ Data download script
deploy/ systemd unit file
data/ Source data files (gitignored)
```
---
## Expected Database Size
| Language | Words | Definitions | Size |
@@ -332,4 +401,10 @@ data/ Source data files (gitignored)
| zh | varies | ~80k | ~20 MB |
| **Total** | | | **~330 MB** |
Word counts are higher than base forms due to affix expansion of inflected forms.
Word counts exceed base form counts due to affix expansion of inflected forms.
---
## License
MIT