Add Phase 2+3 features: antonyms, backing, pronunciation, etymology, difficulty, affix expansion

New endpoints: /antonyms, /backing, /pronunciation, /etymology with difficulty scoring on /random. Cross-language synset backing links French/Portuguese words to English equivalents via WordNet 3.0 synset IDs (matching WOLF and OMW offsets). New loaders: Hunspell affix expansion (fr, pt-PT), English programmatic inflector, CMU Pronouncing Dictionary, SUBTLEX-US frequency, CETEMPúblico frequency, Open Multilingual Wordnet (Portuguese), and SQL-based difficulty scoring. Schema v2 adds tables: antonyms, synsets, word_synsets, pronunciations, etymology with migration support for existing databases. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-02 01:01:06 -07:00
parent d5e4ce0d36
commit 207086da08
22 changed files with 3102 additions and 120 deletions
--- a/README.md
+++ b/README.md
@@ -1,6 +1,6 @@
 # DreamDict

-A self-contained HTTP dictionary service providing word validation, random word generation, definitions, synonyms, and cross-language translation for English (`en`), French (`fr`), and European Portuguese (`pt-PT`).
+A self-contained HTTP dictionary service providing word validation, random word generation, definitions, synonyms, antonyms, translations, pronunciation, etymology, and cross-language semantic backing for English (`en`), French (`fr`), European Portuguese (`pt-PT`), and Mandarin Chinese (`zh`).

 Runs as a plain Go binary managed by systemd. Binds to localhost only — no external network exposure. Designed as a dictionary backend for [GogoBee](https://github.com/reala-misaki/gogobee), replacing Wordnik.

@@ -30,20 +30,20 @@ All endpoints are read-only `GET` requests returning JSON.

 ### `GET /valid?word=...&lang=...`

-Check if a word exists in a language's word list.
+Check if a word exists in a language's word list. Includes inflected forms via affix expansion.

 ```
-$ curl 'localhost:7777/valid?word=ephemeral&lang=en'
+$ curl 'localhost:7777/valid?word=running&lang=en'
 {"valid":true}
 ```

 ### `GET /random?lang=...`

-Return a random word. Optional filters: `pos` (`noun`, `verb`, `adjective`, `adverb`), `min`/`max` (word length), `min_freq` (frequency score, meaningful for `fr`).
+Return a random word. Optional filters: `pos` (`noun`, `verb`, `adjective`, `adverb`), `min`/`max` (word length), `min_freq` (frequency score), `min_difficulty`/`max_difficulty` (0.0–1.0).

 ```
 $ curl 'localhost:7777/random?lang=en&pos=noun&min=5&max=8'
-{"word":"lantern"}
+{"word":"lantern","difficulty":0.42}
 ```

 ### `GET /define?word=...&lang=...`
@@ -67,6 +67,15 @@ $ curl 'localhost:7777/synonyms?word=happy&lang=en'
 {"word":"happy","lang":"en","synonyms":["content","glad","joyful","pleased"]}
 ```

+### `GET /antonyms?word=...&lang=...`
+
+Return antonyms sourced from WordNet and Wiktionary.
+
+```
+$ curl 'localhost:7777/antonyms?word=happy&lang=en'
+{"word":"happy","lang":"en","antonyms":["sad","unhappy"]}
+```
+
 ### `GET /translate?word=...&from=...&to=...`

 Return known translations between supported languages.
@@ -76,6 +85,56 @@ $ curl 'localhost:7777/translate?word=cat&from=en&to=fr'
 {"word":"cat","from":"en","to":"fr","translations":["chat","minet"]}
 ```

+### `GET /backing?word=...&lang=...`
+
+Return English semantic equivalents for a non-English word via shared WordNet synset IDs. Falls back to the translations table when no synset mapping exists.
+
+```
+$ curl 'localhost:7777/backing?word=éphémère&lang=fr'
+{"word":"éphémère","lang":"fr","equivalents":[
+  {"word":"ephemeral","definition":"lasting a very short time","synset":"00914031-a"}
+]}
+```
+
+### `GET /pronunciation?word=...&lang=...`
+
+Return pronunciation data (CMU phonemes for English, IPA from Wiktionary for all languages).
+
+```
+$ curl 'localhost:7777/pronunciation?word=ephemeral&lang=en'
+{"word":"ephemeral","lang":"en","pronunciations":[
+  {"format":"cmu","value":"IH0 F EH1 M ER0 AH0 L","source":"cmudict"},
+  {"format":"ipa","value":"ɪˈfɛm.ər.əl","source":"wiktionary"}
+]}
+```
+
+### `GET /etymology?word=...&lang=...`
+
+Return the etymology of a word from Wiktionary.
+
+```
+$ curl 'localhost:7777/etymology?word=ephemeral&lang=en'
+{"word":"ephemeral","lang":"en","etymology":"From Medieval Latin ephemerus, from Ancient Greek ἐφήμερος (ephḗmeros, \"lasting only a day\")..."}
+```
+
+### `GET /frequency?word=...&lang=...`
+
+Return the frequency score for a word. Higher values = more common.
+
+```
+$ curl 'localhost:7777/frequency?word=house&lang=en'
+{"word":"house","lang":"en","frequency":800}
+```
+
+### `GET /frequency/batch?words=...&lang=...`
+
+Batch frequency lookup. Comma-separated words, max 100.
+
+```
+$ curl 'localhost:7777/frequency/batch?words=house,cat,ephemeral&lang=en'
+{"lang":"en","frequencies":{"cat":800,"ephemeral":50,"house":800}}
+```
+
 ### `GET /health`

 Health check with database stats.
@@ -113,26 +172,81 @@ go run ./cmd/dictimport [flags]
 |---|---|---|
 | `--db` | `./dict.db` | Output database path |
 | `--data` | `./data` | Source data directory |
-| `--langs` | `en,fr,pt-PT` | Languages to import |
+| `--langs` | `en,fr,pt-PT,zh` | Languages to import |
 | `--clean` | `false` | Drop and recreate all tables first |
 | `--skip` | | Comma-separated loader names to skip |

-Loader names: `scowl`, `wordnet`, `wiktionary-en`, `hunspell-fr`, `lexique`, `wolf`, `wiktionary-fr`, `hunspell-pt`, `dicionario`, `wiktionary-pt`.
+### Loader Names
+
+| Loader | Language | What it does |
+|---|---|---|
+| `scowl` | en | English word list with frequency tiers |
+| `affix-en` | en | English inflected form expansion |
+| `wordnet` | en | Definitions, synonyms, antonyms, synset mappings |
+| `subtlex` | en | SUBTLEX-US frequency enrichment |
+| `cmudict` | en | CMU pronunciation data |
+| `wiktionary-en` | en | Supplemental definitions, synonyms, antonyms, translations, IPA, etymology |
+| `hunspell-fr` | fr | French word list |
+| `affix-fr` | fr | French inflected form expansion |
+| `lexique` | fr | French POS + frequency enrichment |
+| `wolf` | fr | French definitions, synonyms, synset mappings |
+| `wiktionary-fr` | fr | Supplemental definitions, synonyms, antonyms, translations, IPA, etymology |
+| `hunspell-pt` | pt-PT | Portuguese word list |
+| `affix-pt-PT` | pt-PT | Portuguese inflected form expansion |
+| `dicionario` | pt-PT | Portuguese definitions |
+| `cetempublico` | pt-PT | Portuguese frequency enrichment |
+| `omw-pt` | pt-PT | Open Multilingual Wordnet synset mappings |
+| `wiktionary-pt` | pt-PT | Supplemental definitions, synonyms, antonyms, translations, IPA, etymology |
+| `cedict` | zh | Chinese words, definitions, translations |
+| `wiktionary-zh` | zh | Supplemental Chinese data |
+| `difficulty` | all | Composite difficulty scoring (runs last) |

 ## Data Sources

 | Source | Language | What it provides |
 |---|---|---|
-| [SCOWL](https://wordlist.aspell.net/) | en | Word list |
-| [WordNet](https://wordnet.princeton.edu/) | en | Definitions + synonyms |
-| [Hunspell/LibreOffice](https://github.com/LibreOffice/dictionaries) | fr, pt-PT | Word lists |
+| [SCOWL](https://wordlist.aspell.net/) | en | Word list + frequency tiers |
+| [WordNet](https://wordnet.princeton.edu/) | en | Definitions, synonyms, antonyms, synset IDs |
+| [SUBTLEX-US](http://www.ugent.be/pp/experimentele-psychologie/en/research/documents/subtlexus) | en | Subtitle-based word frequency |
+| [CMU Pronouncing Dictionary](http://www.speech.cs.cmu.edu/cgi-bin/cmudict) | en | Phoneme-based pronunciation |
+| [Hunspell/LibreOffice](https://github.com/LibreOffice/dictionaries) | fr, pt-PT | Word lists + affix rules |
 | [Lexique](http://www.lexique.org/) | fr | POS + frequency enrichment |
-| [WOLF](https://github.com/nicolashernandez/WOLF) | fr | Definitions + synonyms |
+| [WOLF](https://github.com/nicolashernandez/WOLF) | fr | Definitions, synonyms, synset IDs |
 | [Dicionario Aberto](https://github.com/jorgecardoso/dicionario-aberto) | pt-PT | Definitions |
-| [Wiktionary (kaikki.org)](https://kaikki.org/) | all | Supplemental definitions, synonyms, translations |
+| [CETEMPúblico](https://www.linguateca.pt/acesso/corpus.php?corpus=CETEMPUBLICO) | pt-PT | Word frequency |
+| [Open Multilingual Wordnet](https://github.com/omwn/omw-data) | pt-PT | Synset mappings to Princeton WordNet |
+| [CC-CEDICT](https://cc-cedict.org/) | zh | Words, definitions, translations |
+| [Wiktionary (kaikki.org)](https://kaikki.org/) | all | Definitions, synonyms, antonyms, translations, IPA pronunciation, etymology |

 Run `./scripts/download-dict-data.sh` to fetch everything. Pass `--force` to re-download existing files.

+## Database Schema (v2)
+
+### Tables
+
+| Table | Purpose |
+|---|---|
+| `words` | Core word inventory (word, lang, pos, frequency, difficulty) |
+| `definitions` | Word definitions with source priority |
+| `synonyms` | Synonym relationships |
+| `antonyms` | Antonym relationships |
+| `translations` | Cross-language translations |
+| `synsets` | Princeton WordNet synset IDs |
+| `word_synsets` | Maps words to synsets (for cross-language semantic backing) |
+| `pronunciations` | CMU phonemes and IPA data |
+| `etymology` | Word origin text from Wiktionary |
+| `meta` | Import metadata (imported_at, schema_version) |
+
+### Difficulty Scoring
+
+Each word receives a composite difficulty score (0.0 = easiest, 1.0 = hardest) computed at import time:
+
+```
+difficulty = inverse_frequency * 0.5 + word_length * 0.3 + syllable_count * 0.2
+```
+
+Use `min_difficulty` / `max_difficulty` on `/random` to select words by difficulty tier.
+
 ## Deployment

 A systemd unit file is provided at `deploy/dreamdict.service`.
@@ -173,20 +287,23 @@ go test ./...
 ## Project Structure

 ```
-cmd/server/          HTTP server
-cmd/dictimport/      One-time data import CLI
+cmd/server/           HTTP server (12 endpoints)
+cmd/dictimport/       One-time data import CLI
 internal/dictionary/  Core package: DB schema, types, queries
 internal/loader/      Data source parsers (one per source)
-scripts/             Download script for source data
-deploy/              systemd unit file
-data/                Source data files (gitignored)
+scripts/              Download script for source data
+deploy/               systemd unit file
+data/                 Source data files (gitignored)
 ```

 ## Expected Database Size

 | Language | Words | Definitions | Size |
 |---|---|---|---|
-| en | ~220k | ~380k | ~180 MB |
-| fr | ~90k | ~200k | ~80 MB |
-| pt-PT | ~75k | ~120k | ~50 MB |
-| **Total** | | | **~310 MB** |
+| en | ~220k+ | ~380k | ~180 MB |
+| fr | ~90k+ | ~200k | ~80 MB |
+| pt-PT | ~75k+ | ~120k | ~50 MB |
+| zh | varies | ~80k | ~20 MB |
+| **Total** | | | **~330 MB** |
+
+Word counts are higher than base forms due to affix expansion of inflected forms.