diff --git a/README.md b/README.md index 2d37428..91ebc0a 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,36 @@ # DreamDict -A self-contained HTTP dictionary service providing word validation, random word generation, definitions, synonyms, antonyms, translations, pronunciation, etymology, and cross-language semantic backing for English (`en`), French (`fr`), European Portuguese (`pt-PT`), and Mandarin Chinese (`zh`). +A self-hosted, CGO-free HTTP dictionary service supporting English, French, **European Portuguese**, and Mandarin Chinese. Built to replace metered third-party APIs like Wordnik with something you own outright. -Runs as a plain Go binary managed by systemd. Binds to localhost only — no external network exposure. Designed as a dictionary backend for [GogoBee](https://github.com/reala-misaki/gogobee), replacing Wordnik. +No API keys. No rate limits. No external runtime dependencies. Sub-millisecond response times on commodity hardware. + +--- + +## Why DreamDict exists + +Most open source dictionary tooling is either a dataset (WordNet, Wiktionary dumps, Hunspell) or a desktop application. There is no self-hosted HTTP service that combines all of them into a programmable API — so we built one. + +The pt-PT distinction matters and is handled explicitly. Most open NLP resources default to Brazilian Portuguese (pt-BR) or treat both variants as interchangeable. DreamDict uses the LibreOffice pt-PT Hunspell dictionary, Dicionário Aberto, CETEMPúblico frequency data, and Wiktionary entries filtered to European Portuguese. If you are building something for a European Portuguese-speaking audience, this is the only self-hosted option that takes that seriously. + +**Benchmark results (end-to-end, including HTTP and JSON serialization):** + +| Endpoint | Latency | Throughput | +|---|---|---| +| GET /valid | ~20 µs | ~50,000 req/s | +| GET /random (filtered) | ~187 µs | ~5,300 req/s | +| GET /define | ~50 µs | ~20,000 req/s | +| GET /synonyms | ~37 µs | ~27,000 req/s | +| GET /translate | ~56 µs | ~18,000 req/s | + +--- ## Requirements - Go 1.25+ -- No CGO required (uses [modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite)) +- No CGO required — uses [modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite), a pure Go SQLite port +- Single static binary, no runtime dependencies + +--- ## Quick Start @@ -15,7 +38,7 @@ Runs as a plain Go binary managed by systemd. Binds to localhost only — no ext # Download all dictionary source data (~2-3 GB) ./scripts/download-dict-data.sh -# Build the database (~310 MB) +# Build the database (~330 MB, one-time) go run ./cmd/dictimport --data ./data --db ./dict.db # Start the server @@ -24,167 +47,188 @@ go run ./cmd/server --db ./dict.db The server listens on `127.0.0.1:7777` by default. +--- + ## API -All endpoints are read-only `GET` requests returning JSON. +All endpoints are read-only `GET` requests returning JSON. When authentication is enabled, all endpoints except `/health` require a bearer token (see [Authentication](#authentication)). -### `GET /valid?word=...&lang=...` +### `GET /valid` -Check if a word exists in a language's word list. Includes inflected forms via affix expansion. +Check if a word exists in a language's word list. Includes inflected forms via affix expansion — `running` and `ran` resolve correctly, not just base forms. -``` -$ curl 'localhost:7777/valid?word=running&lang=en' +```bash +curl 'localhost:7777/valid?word=running&lang=en' {"valid":true} ``` -### `GET /random?lang=...` +### `GET /random` Return a random word. Optional filters: `pos` (`noun`, `verb`, `adjective`, `adverb`), `min`/`max` (word length), `min_freq` (frequency score), `min_difficulty`/`max_difficulty` (0.0–1.0). -``` -$ curl 'localhost:7777/random?lang=en&pos=noun&min=5&max=8' +```bash +curl 'localhost:7777/random?lang=en&pos=noun&min=5&max=8&max_difficulty=0.5' {"word":"lantern","difficulty":0.42} ``` -### `GET /define?word=...&lang=...` +### `GET /define` -Return all definitions, ordered by source priority. +Return all definitions ordered by source priority. Curated sources (WordNet, WOLF, Dicionário Aberto) appear before Wiktionary supplemental definitions. -``` -$ curl 'localhost:7777/define?word=ephemeral&lang=en' -{"word":"ephemeral","lang":"en","definitions":[ - {"pos":"adjective","gloss":"lasting for a very short time","source":"wordnet","priority":10}, - {"pos":"adjective","gloss":"lasting only for a day","source":"wiktionary","priority":20} -]} +```bash +curl 'localhost:7777/define?word=ephemeral&lang=en' +{ + "word": "ephemeral", + "lang": "en", + "definitions": [ + {"pos":"adjective","gloss":"lasting for a very short time","source":"wordnet","priority":10}, + {"pos":"adjective","gloss":"lasting only for a day","source":"wiktionary","priority":20} + ] +} ``` -### `GET /synonyms?word=...&lang=...` +### `GET /synonyms` -Return deduplicated synonyms. +Return deduplicated synonyms sourced from WordNet, WOLF, and Wiktionary. -``` -$ curl 'localhost:7777/synonyms?word=happy&lang=en' +```bash +curl 'localhost:7777/synonyms?word=happy&lang=en' {"word":"happy","lang":"en","synonyms":["content","glad","joyful","pleased"]} ``` -### `GET /antonyms?word=...&lang=...` +### `GET /antonyms` Return antonyms sourced from WordNet and Wiktionary. -``` -$ curl 'localhost:7777/antonyms?word=happy&lang=en' +```bash +curl 'localhost:7777/antonyms?word=happy&lang=en' {"word":"happy","lang":"en","antonyms":["sad","unhappy"]} ``` -### `GET /translate?word=...&from=...&to=...` +### `GET /translate` Return known translations between supported languages. -``` -$ curl 'localhost:7777/translate?word=cat&from=en&to=fr' +```bash +curl 'localhost:7777/translate?word=cat&from=en&to=fr' {"word":"cat","from":"en","to":"fr","translations":["chat","minet"]} ``` -### `GET /backing?word=...&lang=...` +### `GET /backing` -Return English semantic equivalents for a non-English word via shared WordNet synset IDs. Falls back to the translations table when no synset mapping exists. +Return English semantic equivalents for a non-English word via shared Princeton WordNet synset IDs. This is semantic equivalence — not translation guesswork. Falls back to the translations table when no synset mapping exists. -``` -$ curl 'localhost:7777/backing?word=éphémère&lang=fr' -{"word":"éphémère","lang":"fr","equivalents":[ - {"word":"ephemeral","definition":"lasting a very short time","synset":"00914031-a"} -]} +```bash +curl 'localhost:7777/backing?word=éphémère&lang=fr' +{ + "word": "éphémère", + "lang": "fr", + "equivalents": [ + {"word":"ephemeral","definition":"lasting a very short time","synset":"00914031-a"} + ] +} ``` -### `GET /pronunciation?word=...&lang=...` +### `GET /pronunciation` -Return pronunciation data (CMU phonemes for English, IPA from Wiktionary for all languages). +Return pronunciation data. English gets CMU phonemes and IPA. All other languages get IPA from Wiktionary where available. -``` -$ curl 'localhost:7777/pronunciation?word=ephemeral&lang=en' -{"word":"ephemeral","lang":"en","pronunciations":[ - {"format":"cmu","value":"IH0 F EH1 M ER0 AH0 L","source":"cmudict"}, - {"format":"ipa","value":"ɪˈfɛm.ər.əl","source":"wiktionary"} -]} +```bash +curl 'localhost:7777/pronunciation?word=ephemeral&lang=en' +{ + "word": "ephemeral", + "lang": "en", + "pronunciations": [ + {"format":"cmu","value":"IH0 F EH1 M ER0 AH0 L","source":"cmudict"}, + {"format":"ipa","value":"ɪˈfɛm.ər.əl","source":"wiktionary"} + ] +} ``` -### `GET /etymology?word=...&lang=...` +### `GET /etymology` -Return the etymology of a word from Wiktionary. +Return the etymology of a word from Wiktionary. Returned as free-form text — Wiktionary etymology is not structured consistently enough to parse reliably. -``` -$ curl 'localhost:7777/etymology?word=ephemeral&lang=en' -{"word":"ephemeral","lang":"en","etymology":"From Medieval Latin ephemerus, from Ancient Greek ἐφήμερος (ephḗmeros, \"lasting only a day\")..."} +```bash +curl 'localhost:7777/etymology?word=ephemeral&lang=en' +{ + "word": "ephemeral", + "lang": "en", + "etymology": "From Medieval Latin ephemerus, from Ancient Greek ἐφήμερος (ephḗmeros, \"lasting only a day\")..." +} ``` -### `GET /frequency?word=...&lang=...` +### `GET /frequency` -Return the frequency score for a word. Higher values = more common. +Return the frequency score for a word. Higher values = more common. Sourced from SUBTLEX-US (en), Lexique (fr), and CETEMPúblico (pt-PT). -``` -$ curl 'localhost:7777/frequency?word=house&lang=en' +```bash +curl 'localhost:7777/frequency?word=house&lang=en' {"word":"house","lang":"en","frequency":800} ``` -### `GET /frequency/batch?words=...&lang=...` +### `GET /frequency/batch` -Batch frequency lookup. Comma-separated words, max 100. +Batch frequency lookup. Comma-separated words, max 100 per request. -``` -$ curl 'localhost:7777/frequency/batch?words=house,cat,ephemeral&lang=en' +```bash +curl 'localhost:7777/frequency/batch?words=house,cat,ephemeral&lang=en' {"lang":"en","frequencies":{"cat":800,"ephemeral":50,"house":800}} ``` ### `GET /health` -Health check with database stats. +Health check with database statistics. Always unauthenticated — safe for monitoring tools. -``` -$ curl localhost:7777/health -{"status":"ok","db_path":"./dict.db","word_counts":{"en":214832,"fr":82104,"pt-PT":71088},...} +```bash +curl localhost:7777/health +{ + "status": "ok", + "db_path": "./dict.db", + "word_counts": {"en":214832,"fr":82104,"pt-PT":71088,"zh":120000}, + "def_counts": {"en":380000,"fr":200000,"pt-PT":120000,"zh":80000}, + "imported_at": "2026-03-28T14:22:00Z", + "schema_version": "2" +} ``` ### Error Responses ```json -{"error": "bad_request", "message": "word and lang are required"} -{"error": "no_match", "message": "no words match the given filters"} +{"error": "bad_request", "message": "word and lang are required"} +{"error": "no_match", "message": "no words match the given filters"} +{"error": "unauthorized", "message": "invalid or missing bearer token"} ``` -## Server Configuration +--- + +## Configuration | Flag | Env Var | Default | Description | |---|---|---|---| | `--port` | `DREAMDICT_PORT` | `7777` | Listen port | -| `--host` | `DREAMDICT_HOST` | `127.0.0.1` | Listen host | +| `--host` | `DREAMDICT_HOST` | `127.0.0.1` | Listen host (localhost only by default) | | `--db` | `DREAMDICT_DB` | `./dict.db` | Path to SQLite database | -| `--token` | `DREAMDICT_TOKEN` | _(none)_ | Bearer token for API auth | +| `--token` | `DREAMDICT_TOKEN` | _(none)_ | Bearer token for API authentication | | `--log-level` | | `info` | `debug`, `info`, `warn`, `error` | Flags take precedence over environment variables. ### Authentication -When `--token` or `DREAMDICT_TOKEN` is set, all endpoints except `/health` require a bearer token: +When `--token` or `DREAMDICT_TOKEN` is set, all endpoints except `/health` require a bearer token. When no token is configured, authentication is disabled. ```bash # Start with auth enabled -DREAMDICT_TOKEN=mysecret go run ./cmd/server --db ./dict.db +DREAMDICT_TOKEN=your-secret-token go run ./cmd/server --db ./dict.db # Authenticated request -curl -H 'Authorization: Bearer mysecret' 'localhost:7777/define?word=casa&lang=pt-PT' - -# Health is always public (for monitoring) -curl localhost:7777/health +curl -H 'Authorization: Bearer your-secret-token' 'localhost:7777/define?word=saudade&lang=pt-PT' ``` -Requests without a valid token receive a `401` response: +Requests without a valid token receive a `401` response. The error message does not distinguish between a missing token and an incorrect one. -```json -{"error": "unauthorized", "message": "invalid or missing bearer token"} -``` - -When no token is configured, auth is disabled and all endpoints are open. +--- ## Import CLI @@ -197,102 +241,121 @@ go run ./cmd/dictimport [flags] | `--db` | `./dict.db` | Output database path | | `--data` | `./data` | Source data directory | | `--langs` | `en,fr,pt-PT,zh` | Languages to import | -| `--clean` | `false` | Drop and recreate all tables first | +| `--clean` | `false` | Drop and recreate all tables before import | | `--skip` | | Comma-separated loader names to skip | -### Loader Names +### Loaders -| Loader | Language | What it does | +| Loader | Language | What it provides | |---|---|---| -| `scowl` | en | English word list with frequency tiers | -| `affix-en` | en | English inflected form expansion | -| `wordnet` | en | Definitions, synonyms, antonyms, synset mappings | -| `subtlex` | en | SUBTLEX-US frequency enrichment | -| `cmudict` | en | CMU pronunciation data | +| `scowl` | en | Word list with frequency tiers | +| `affix-en` | en | Inflected form expansion | +| `wordnet` | en | Definitions, synonyms, antonyms, synset IDs | +| `subtlex` | en | SUBTLEX-US subtitle frequency data | +| `cmudict` | en | CMU phoneme pronunciation data | | `wiktionary-en` | en | Supplemental definitions, synonyms, antonyms, translations, IPA, etymology | -| `hunspell-fr` | fr | French word list | -| `affix-fr` | fr | French inflected form expansion | -| `lexique` | fr | French POS + frequency enrichment | -| `wolf` | fr | French definitions, synonyms, synset mappings | +| `hunspell-fr` | fr | Word list | +| `affix-fr` | fr | Inflected form expansion | +| `lexique` | fr | POS and frequency enrichment | +| `wolf` | fr | Definitions, synonyms, synset IDs | | `wiktionary-fr` | fr | Supplemental definitions, synonyms, antonyms, translations, IPA, etymology | -| `hunspell-pt` | pt-PT | Portuguese word list | -| `affix-pt-PT` | pt-PT | Portuguese inflected form expansion | -| `dicionario` | pt-PT | Portuguese definitions | -| `cetempublico` | pt-PT | Portuguese frequency enrichment | +| `hunspell-pt` | pt-PT | European Portuguese word list (not pt-BR) | +| `affix-pt-PT` | pt-PT | Inflected form expansion | +| `dicionario` | pt-PT | Definitions from Dicionário Aberto | +| `cetempublico` | pt-PT | Frequency data from CETEMPúblico corpus | | `omw-pt` | pt-PT | Open Multilingual Wordnet synset mappings | -| `wiktionary-pt` | pt-PT | Supplemental definitions, synonyms, antonyms, translations, IPA, etymology | -| `cedict` | zh | Chinese words, definitions, translations | +| `wiktionary-pt` | pt-PT | Supplemental definitions, synonyms, antonyms, translations, IPA, etymology (filtered to European Portuguese entries) | +| `cedict` | zh | Words, definitions, translations via CC-CEDICT | | `wiktionary-zh` | zh | Supplemental Chinese data | -| `difficulty` | all | Composite difficulty scoring (runs last) | +| `difficulty` | all | Composite difficulty scoring — runs last | + +--- ## Data Sources | Source | Language | What it provides | |---|---|---| | [SCOWL](https://wordlist.aspell.net/) | en | Word list + frequency tiers | -| [WordNet](https://wordnet.princeton.edu/) | en | Definitions, synonyms, antonyms, synset IDs | +| [WordNet 3.1](https://wordnet.princeton.edu/) | en | Definitions, synonyms, antonyms, synset IDs | | [SUBTLEX-US](http://www.ugent.be/pp/experimentele-psychologie/en/research/documents/subtlexus) | en | Subtitle-based word frequency | -| [CMU Pronouncing Dictionary](http://www.speech.cs.cmu.edu/cgi-bin/cmudict) | en | Phoneme-based pronunciation | +| [CMU Pronouncing Dictionary](http://www.speech.cs.cmu.edu/cgi-bin/cmudict) | en | Phoneme pronunciation | | [Hunspell/LibreOffice](https://github.com/LibreOffice/dictionaries) | fr, pt-PT | Word lists + affix rules | -| [Lexique](http://www.lexique.org/) | fr | POS + frequency enrichment | +| [Lexique 3.83](http://www.lexique.org/) | fr | POS + frequency enrichment | | [WOLF](https://github.com/nicolashernandez/WOLF) | fr | Definitions, synonyms, synset IDs | -| [Dicionario Aberto](https://github.com/jorgecardoso/dicionario-aberto) | pt-PT | Definitions | -| [CETEMPúblico](https://www.linguateca.pt/acesso/corpus.php?corpus=CETEMPUBLICO) | pt-PT | Word frequency | +| [Dicionário Aberto](https://github.com/jorgecardoso/dicionario-aberto) | pt-PT | Definitions | +| [CETEMPúblico](https://www.linguateca.pt/acesso/corpus.php?corpus=CETEMPUBLICO) | pt-PT | European Portuguese word frequency | | [Open Multilingual Wordnet](https://github.com/omwn/omw-data) | pt-PT | Synset mappings to Princeton WordNet | | [CC-CEDICT](https://cc-cedict.org/) | zh | Words, definitions, translations | -| [Wiktionary (kaikki.org)](https://kaikki.org/) | all | Definitions, synonyms, antonyms, translations, IPA pronunciation, etymology | +| [Wiktionary via kaikki.org](https://kaikki.org/) | all | Definitions, synonyms, antonyms, translations, IPA, etymology | -Run `./scripts/download-dict-data.sh` to fetch everything. Pass `--force` to re-download existing files. +Run `./scripts/download-dict-data.sh` to fetch all sources automatically. Pass `--force` to re-download existing files. -## Database Schema (v2) +--- -### Tables +## Difficulty Scoring -| Table | Purpose | -|---|---| -| `words` | Core word inventory (word, lang, pos, frequency, difficulty) | -| `definitions` | Word definitions with source priority | -| `synonyms` | Synonym relationships | -| `antonyms` | Antonym relationships | -| `translations` | Cross-language translations | -| `synsets` | Princeton WordNet synset IDs | -| `word_synsets` | Maps words to synsets (for cross-language semantic backing) | -| `pronunciations` | CMU phonemes and IPA data | -| `etymology` | Word origin text from Wiktionary | -| `meta` | Import metadata (imported_at, schema_version) | - -### Difficulty Scoring - -Each word receives a composite difficulty score (0.0 = easiest, 1.0 = hardest) computed at import time: +Each word receives a composite difficulty score (0.0 = easiest, 1.0 = hardest) computed at import time from frequency, word length, and syllable count: ``` difficulty = inverse_frequency * 0.5 + word_length * 0.3 + syllable_count * 0.2 ``` -Use `min_difficulty` / `max_difficulty` on `/random` to select words by difficulty tier. +Use `min_difficulty` / `max_difficulty` on `/random` to select words by difficulty tier without maintaining hardcoded word lists. + +--- + +## Database Schema + +Schema version 2. Full schema in `internal/dictionary/db.go`. + +| Table | Purpose | +|---|---| +| `words` | Core word inventory: word, lang, pos, frequency, difficulty | +| `definitions` | Definitions with source and priority | +| `synonyms` | Synonym relationships | +| `antonyms` | Antonym relationships | +| `translations` | Cross-language translations | +| `synsets` | Princeton WordNet synset IDs | +| `word_synsets` | Word-to-synset mappings for cross-language semantic backing | +| `pronunciations` | CMU phonemes and IPA | +| `etymology` | Word origin text from Wiktionary | +| `meta` | Import metadata: schema version, import timestamp, word/definition counts | + +--- ## Deployment -A systemd unit file is provided at `deploy/dreamdict.service`. +A systemd unit file is provided at `deploy/dreamdict.service`. The service runs as a dedicated unprivileged system user and binds to localhost only by default. ```bash -# Create system user +# Create dedicated system user useradd --system --no-create-home --shell /usr/sbin/nologin dreamdict -# Build and install +# Build and install binary go build -o /usr/local/bin/dreamdict ./cmd/server +chmod 755 /usr/local/bin/dreamdict + +# Build the database (run as yourself, not the dreamdict user) mkdir -p /opt/dreamdict /etc/dreamdict go run ./cmd/dictimport --db /opt/dreamdict/dict.db --data ./data chown -R dreamdict:dreamdict /opt/dreamdict -# Install service -cp deploy/dreamdict.service /etc/systemd/system/ -echo 'DREAMDICT_PORT=7777' > /etc/dreamdict/dreamdict.env -echo 'DREAMDICT_DB=/opt/dreamdict/dict.db' >> /etc/dreamdict/dreamdict.env -echo 'DREAMDICT_TOKEN=changeme' >> /etc/dreamdict/dreamdict.env +# Configure +cat > /etc/dreamdict/dreamdict.env <