Initial commit: DreamDict HTTP dictionary service
Go-based dictionary backend providing word validation, definitions, synonyms, and cross-language translation for English, French, and Portuguese. Designed as a self-hosted replacement for Wordnik, backing GogoBee. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
prosolis
763
dreamdict-blueprint.md
Normal file
763
dreamdict-blueprint.md
Normal file
@@ -0,0 +1,763 @@
|
||||
# DreamDict — Standalone Dictionary Service: Implementation Blueprint
|
||||
|
||||
## Overview
|
||||
|
||||
`dreamdict` is a self-contained HTTP dictionary service providing word validation, random
|
||||
word generation, definitions, synonyms, and cross-language translation for English (`en`),
|
||||
French (`fr`), and European Portuguese (`pt-PT`).
|
||||
|
||||
It runs as a plain Go binary managed by systemd on the same host as GogoBee and Ollama.
|
||||
GogoBee calls it over localhost HTTP. No Docker, no Traefik, no external network exposure.
|
||||
|
||||
If GogoBee later containerizes, only the service URL in GogoBee's config changes. The
|
||||
service itself is unmodified.
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
GogoBee is approaching 50,000 lines of code across 47+ modules. A dictionary package with
|
||||
loaders, a SQLite dependency, and an import CLI would compound that bloat. Dictionary logic
|
||||
has no business living inside a Matrix bot. `dreamdict` is a clean extraction: self-contained
|
||||
infrastructure with a stable HTTP API, independently deployable, independently updatable,
|
||||
with no knowledge of Matrix, rooms, or bot commands.
|
||||
|
||||
---
|
||||
|
||||
## Repository
|
||||
|
||||
Separate repository: `dreamdict` (not a GogoBee internal package).
|
||||
|
||||
```
|
||||
dreamdict/
|
||||
├── cmd/
|
||||
│ ├── server/
|
||||
│ │ └── main.go # HTTP server binary
|
||||
│ └── dictimport/
|
||||
│ └── main.go # One-time data import CLI
|
||||
├── internal/
|
||||
│ ├── dictionary/
|
||||
│ │ ├── dict.go # Core package: public API
|
||||
│ │ ├── db.go # SQLite connection + schema bootstrap
|
||||
│ │ └── query.go # IsValidWord, RandomWord, Define, Synonyms, Translate
|
||||
│ └── loader/
|
||||
│ ├── loader.go # Shared interface + orchestrator
|
||||
│ ├── scowl.go # English word list
|
||||
│ ├── hunspell.go # French + pt-PT word lists (shared, parameterized)
|
||||
│ ├── wordnet.go # English definitions + synonyms
|
||||
│ ├── lexique.go # French POS + frequency enrichment
|
||||
│ ├── wolf.go # French definitions + synonyms
|
||||
│ ├── dicionario.go # pt-PT definitions
|
||||
│ └── wiktionary.go # All langs: supplemental definitions + translations
|
||||
├── data/
|
||||
│ └── .gitkeep # Source data files (not committed)
|
||||
├── dict.db # Generated SQLite database (not committed)
|
||||
├── scripts/
|
||||
│ └── download-dict-data.sh
|
||||
├── deploy/
|
||||
│ └── dreamdict.service # systemd unit file
|
||||
├── go.mod
|
||||
└── README.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## HTTP API
|
||||
|
||||
The server listens on `127.0.0.1:7777` by default (configurable via `--port` flag or
|
||||
`DREAMDICT_PORT` env var). All responses are JSON. All endpoints are read-only GET requests.
|
||||
|
||||
### `GET /valid`
|
||||
|
||||
Check if a word exists in a language's word list.
|
||||
|
||||
**Query params:** `word` (required), `lang` (required: `en`, `fr`, `pt-PT`)
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{"valid": true}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `GET /random`
|
||||
|
||||
Return a random word for a language, with optional filters.
|
||||
|
||||
**Query params:**
|
||||
- `lang` (required)
|
||||
- `pos` (optional: `noun`, `verb`, `adjective`, `adverb`)
|
||||
- `min` (optional: minimum word length, integer)
|
||||
- `max` (optional: maximum word length, integer)
|
||||
- `min_freq` (optional: minimum frequency score, integer; only meaningful for `fr`)
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{"word": "ephemeral"}
|
||||
```
|
||||
|
||||
**Error (no words match filters):**
|
||||
```json
|
||||
{"error": "no_match", "message": "no words match the given filters"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `GET /define`
|
||||
|
||||
Return all definitions for a word, ordered by source priority.
|
||||
|
||||
**Query params:** `word` (required), `lang` (required)
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"word": "ephemeral",
|
||||
"lang": "en",
|
||||
"definitions": [
|
||||
{"pos": "adjective", "gloss": "lasting for a very short time", "source": "wordnet", "priority": 10},
|
||||
{"pos": "adjective", "gloss": "lasting only for a day", "source": "wiktionary", "priority": 20}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Word exists but has no definitions:**
|
||||
```json
|
||||
{"word": "...", "lang": "...", "definitions": []}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `GET /synonyms`
|
||||
|
||||
Return deduplicated synonyms for a word.
|
||||
|
||||
**Query params:** `word` (required), `lang` (required)
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{"word": "happy", "lang": "en", "synonyms": ["glad", "joyful", "content", "pleased"]}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `GET /translate`
|
||||
|
||||
Return known equivalents of a word in another language.
|
||||
|
||||
**Query params:** `word` (required), `from` (required), `to` (required)
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{"word": "cat", "from": "en", "to": "fr", "translations": ["chat", "minet"]}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `GET /health`
|
||||
|
||||
Health check and database stats.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"db_path": "/opt/dreamdict/dict.db",
|
||||
"word_counts": {"en": 214832, "fr": 82104, "pt-PT": 71088},
|
||||
"def_counts": {"en": 380000, "fr": 200000, "pt-PT": 120000},
|
||||
"imported_at": "2025-03-28T14:22:00Z",
|
||||
"schema_version": "1"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Core Package (`internal/dictionary`)
|
||||
|
||||
### `dict.go`
|
||||
|
||||
```go
|
||||
package dictionary
|
||||
|
||||
type Options struct {
|
||||
MinLength int
|
||||
MaxLength int
|
||||
POS string // "", "noun", "verb", "adjective", "adverb"
|
||||
MinFrequency int // 0 = no filter
|
||||
}
|
||||
|
||||
type Definition struct {
|
||||
POS string
|
||||
Gloss string
|
||||
Source string
|
||||
Priority int
|
||||
}
|
||||
|
||||
type Dictionary struct { /* unexported */ }
|
||||
|
||||
func New(dbPath string) (*Dictionary, error)
|
||||
func (d *Dictionary) Close() error
|
||||
func (d *Dictionary) IsValidWord(word, lang string) (bool, error)
|
||||
func (d *Dictionary) RandomWord(lang string, opts Options) (string, error)
|
||||
func (d *Dictionary) Define(word, lang string) ([]Definition, error)
|
||||
func (d *Dictionary) Synonyms(word, lang string) ([]string, error)
|
||||
func (d *Dictionary) Translate(word, fromLang, toLang string) ([]string, error)
|
||||
func (d *Dictionary) WordCount() (map[string]int, error)
|
||||
func (d *Dictionary) SupportedLangs() ([]string, error)
|
||||
|
||||
func ValidLang(lang string) bool // "en", "fr", "pt-PT"
|
||||
|
||||
var (
|
||||
ErrNoMatch = errors.New("dictionary: no word matches filters")
|
||||
ErrNotSeeded = errors.New("dictionary: database not seeded, run: go run ./cmd/dictimport")
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database Schema
|
||||
|
||||
Use `modernc.org/sqlite` (pure Go, no CGO).
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS words (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
word TEXT NOT NULL,
|
||||
lang TEXT NOT NULL,
|
||||
pos TEXT,
|
||||
frequency INTEGER DEFAULT 0,
|
||||
UNIQUE(word, lang)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_words_lang ON words(lang);
|
||||
CREATE INDEX IF NOT EXISTS idx_words_lang_pos ON words(lang, pos);
|
||||
CREATE INDEX IF NOT EXISTS idx_words_word ON words(word);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS definitions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
word_id INTEGER NOT NULL REFERENCES words(id) ON DELETE CASCADE,
|
||||
pos TEXT,
|
||||
gloss TEXT NOT NULL,
|
||||
source TEXT NOT NULL,
|
||||
priority INTEGER NOT NULL DEFAULT 99,
|
||||
UNIQUE(word_id, gloss)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_definitions_word_id ON definitions(word_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_definitions_word_id_priority ON definitions(word_id, priority);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS synonyms (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
word_id INTEGER NOT NULL REFERENCES words(id) ON DELETE CASCADE,
|
||||
synonym TEXT NOT NULL,
|
||||
source TEXT NOT NULL,
|
||||
UNIQUE(word_id, synonym)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_synonyms_word_id ON synonyms(word_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS translations (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
word_id INTEGER NOT NULL REFERENCES words(id) ON DELETE CASCADE,
|
||||
translation TEXT NOT NULL,
|
||||
target_lang TEXT NOT NULL,
|
||||
source TEXT NOT NULL,
|
||||
UNIQUE(word_id, translation, target_lang)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_translations_word_id ON translations(word_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS meta (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
### Definition Priority
|
||||
|
||||
| Source | Priority |
|
||||
|---|---|
|
||||
| `wordnet` | 10 |
|
||||
| `wolf` | 10 |
|
||||
| `dicionario-aberto` | 10 |
|
||||
| `wiktionary` | 20 |
|
||||
|
||||
---
|
||||
|
||||
## Server Binary (`cmd/server/main.go`)
|
||||
|
||||
### Flags / Env Vars
|
||||
|
||||
```
|
||||
--port PORT Listen port (default 7777) DREAMDICT_PORT
|
||||
--db PATH Path to dict.db DREAMDICT_DB
|
||||
--host HOST Listen host (default 127.0.0.1) DREAMDICT_HOST
|
||||
```
|
||||
|
||||
Default host is `127.0.0.1` — the service binds localhost only. It is never exposed to the
|
||||
external network. If GogoBee containerizes in future, change host to `0.0.0.0` and control
|
||||
access at the Docker network level.
|
||||
|
||||
### Startup Behaviour
|
||||
|
||||
- Open the SQLite database on startup; log and exit if unavailable or not seeded
|
||||
- Register HTTP handlers for all six endpoints
|
||||
- Log `dreamdict listening on 127.0.0.1:7777` on successful start
|
||||
- Handle `SIGTERM` / `SIGINT` for graceful shutdown (close DB, drain in-flight requests)
|
||||
|
||||
### HTTP Handler Pattern
|
||||
|
||||
All handlers follow the same structure:
|
||||
1. Parse and validate query params — return `400` with `{"error": "...", "message": "..."}` on bad input
|
||||
2. Call the appropriate `dictionary.*` method
|
||||
3. Return `200` with JSON response, or `404` if the word is not found, or `500` on
|
||||
unexpected errors
|
||||
4. Log errors at ERROR level; log each request at DEBUG level (controllable via `--log-level`)
|
||||
|
||||
Use only standard library `net/http` — no framework needed for six endpoints.
|
||||
|
||||
---
|
||||
|
||||
## Import CLI (`cmd/dictimport/main.go`)
|
||||
|
||||
### Flags
|
||||
|
||||
```
|
||||
--db Path to output dict.db (default: ./dict.db)
|
||||
--data Path to data directory (default: ./data/)
|
||||
--langs Comma-separated langs (default: en,fr,pt-PT)
|
||||
--clean Drop and recreate all tables before import
|
||||
--skip Comma-separated loader names to skip
|
||||
```
|
||||
|
||||
### Execution Order
|
||||
|
||||
**English:**
|
||||
1. `scowl` — populates words
|
||||
2. `wordnet` — definitions + synonyms
|
||||
3. `wiktionary-en` — supplemental definitions, synonyms, translations
|
||||
|
||||
**French:**
|
||||
1. `hunspell-fr` — populates words
|
||||
2. `lexique` — enriches POS + frequency
|
||||
3. `wolf` — definitions + synonyms
|
||||
4. `wiktionary-fr` — supplemental definitions, synonyms, translations
|
||||
|
||||
**pt-PT:**
|
||||
1. `hunspell-pt` — populates words
|
||||
2. `dicionario` — definitions
|
||||
3. `wiktionary-pt` — supplemental definitions, synonyms, translations
|
||||
|
||||
Use `INSERT OR IGNORE` throughout. Wrap each loader in a single transaction; commit on
|
||||
completion. Print per-loader row counts and elapsed time. Print final summary.
|
||||
|
||||
---
|
||||
|
||||
## Data Sources & Loader Specs
|
||||
|
||||
---
|
||||
|
||||
### English — SCOWL (`loader/scowl.go`)
|
||||
|
||||
**Source:** https://wordlist.aspell.net/
|
||||
**Files:** All `final/english-words.NN` where NN ≤ 70
|
||||
|
||||
One word per line. Skip blank lines and `#` comments. Lowercase. Skip words with
|
||||
spaces, digits, hyphens, or apostrophes. `INSERT OR IGNORE INTO words (word, lang) VALUES (?, 'en')`.
|
||||
|
||||
**Expected:** ~170,000–220,000 words.
|
||||
|
||||
---
|
||||
|
||||
### French + pt-PT — Hunspell (`loader/hunspell.go`)
|
||||
|
||||
**Source:** https://github.com/LibreOffice/dictionaries
|
||||
**Files:** `fr_FR/fr_FR.dic`, `pt_PT/pt_PT.dic`
|
||||
|
||||
Skip line 1 (word count). Split each line on `/`, take left side as base word. Lowercase.
|
||||
Skip words with digits, spaces, or non-Latin characters.
|
||||
|
||||
Same function handles both languages: `func Load(path string, lang string, db *sql.DB) error`
|
||||
|
||||
**Expected:** ~60,000–90,000 words per language.
|
||||
|
||||
---
|
||||
|
||||
### English Definitions + Synonyms — WordNet (`loader/wordnet.go`)
|
||||
|
||||
**Source:** https://wordnet.princeton.edu/download/current-version
|
||||
**Files:** `dict/data.noun`, `dict/data.verb`, `dict/data.adj`, `dict/data.adv`
|
||||
|
||||
Skip lines starting with two spaces (headers). Parse `ss_type` for POS. Extract all words
|
||||
in synset. Take gloss text before the first `;` (drops usage examples). Insert definitions
|
||||
at `priority=10`. Insert intra-synset word pairs as synonyms. Skip words containing `_`
|
||||
for synonym insertion.
|
||||
|
||||
---
|
||||
|
||||
### French Enrichment — Lexique (`loader/lexique.go`)
|
||||
|
||||
**Source:** http://www.lexique.org/?page_id=319
|
||||
**File:** `Lexique383.tsv`
|
||||
|
||||
Columns: 0=`ortho`, 3=`cgram`, 8=`freqfilms2`. POS map: `NOM→noun`, `VER→verb`,
|
||||
`ADJ→adjective`, `ADV→adverb`. Frequency = `freqfilms2 × 1000` truncated to int.
|
||||
On conflict with existing Hunspell row: `UPDATE ... SET pos=?, frequency=? WHERE pos IS NULL`.
|
||||
|
||||
---
|
||||
|
||||
### French Definitions + Synonyms — WOLF (`loader/wolf.go`)
|
||||
|
||||
**Source:** https://github.com/nicolashernandez/WOLF
|
||||
**File:** `wolf-1.0b4.xml.bz2` → decompress to `wolf.xml`
|
||||
|
||||
Stream-parse XML with `encoding/xml`. For each `<SYNSET>`: collect `<LITERAL>` values,
|
||||
map `<POS>`, insert definitions at `priority=10`, insert intra-synset synonym pairs.
|
||||
|
||||
---
|
||||
|
||||
### pt-PT Definitions — Dicionário Aberto (`loader/dicionario.go`)
|
||||
|
||||
**Source:** https://github.com/jorgecardoso/dicionario-aberto
|
||||
**File:** `dicionario-aberto.xml`
|
||||
|
||||
**⚠️ Inspect the file before writing the parser:**
|
||||
```bash
|
||||
head -100 data/dicionario-aberto.xml
|
||||
```
|
||||
Print distinct `<pos>` values before writing the POS map. Element names in the live file
|
||||
may differ from older documentation. Stream-parse XML. Insert definitions at `priority=10`.
|
||||
|
||||
---
|
||||
|
||||
### All Languages — Wiktionary (`loader/wiktionary.go`)
|
||||
|
||||
**Source:** kaikki.org pre-parsed Wiktionary JSONL dumps
|
||||
|
||||
| Target file | Lang | URL |
|
||||
|---|---|---|
|
||||
| `kaikki-en.jsonl` | `en` | https://kaikki.org/dictionary/English/kaikki.org-dictionary-English.json |
|
||||
| `kaikki-fr.jsonl` | `fr` | https://kaikki.org/dictionary/French/kaikki.org-dictionary-French.json |
|
||||
| `kaikki-pt.jsonl` | `pt-PT` | https://kaikki.org/dictionary/Portuguese/kaikki.org-dictionary-Portuguese.json |
|
||||
|
||||
**JSON structure per line:**
|
||||
```json
|
||||
{
|
||||
"word": "example",
|
||||
"pos": "noun",
|
||||
"lang_code": "en",
|
||||
"senses": [
|
||||
{"glosses": ["a representative instance"], "tags": []},
|
||||
{"glosses": ["past tense of exemplify"], "tags": ["form-of"]}
|
||||
],
|
||||
"synonyms": [{"word": "instance"}],
|
||||
"translations": [
|
||||
{"code": "fr", "word": "exemple"},
|
||||
{"code": "pt", "word": "exemplo"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**POS mapping:** `noun→noun`, `verb→verb`, `adj→adjective`, `adv→adverb`, `name→noun`.
|
||||
|
||||
**Import logic:**
|
||||
- Use `bufio.Scanner` with 10MB buffer (default 64KB truncates long lines silently)
|
||||
- Wrap in single transaction; commit every 10,000 rows
|
||||
- Verify `lang_code` matches expected language; skip if not
|
||||
- Lowercase word; skip if contains spaces or digits
|
||||
- Skip senses tagged `form-of` or `alt-of`
|
||||
- Skip glosses that are empty, start with `(`, or exceed 500 characters
|
||||
- Insert definitions at `priority=20`; `UNIQUE(word_id, gloss)` handles deduplication silently
|
||||
- Insert synonyms; `UNIQUE(word_id, synonym)` handles deduplication silently
|
||||
- **Translations:** for each entry in `translations[]`, if `code` maps to a supported lang
|
||||
and `code != source lang`, insert into `translations` table
|
||||
- Code mapping: `fr→fr`, `pt→pt-PT`, `en→en`
|
||||
- Skip translations where `word` contains spaces or digits
|
||||
|
||||
**pt-PT Brazil filtering:** Reject senses where `tags` contains `"Brazil"` or
|
||||
`"Brazilian Portuguese"`. Accept untagged senses and senses tagged `"Portugal"` or
|
||||
`"European Portuguese"`.
|
||||
|
||||
**Expected additional definitions after deduplication:**
|
||||
- en: ~200,000–400,000
|
||||
- fr: ~80,000–150,000
|
||||
- pt-PT: ~40,000–80,000
|
||||
|
||||
---
|
||||
|
||||
## Deployment (systemd)
|
||||
|
||||
### File Locations
|
||||
|
||||
```
|
||||
/usr/local/bin/dreamdict # compiled binary
|
||||
/opt/dreamdict/dict.db # SQLite database
|
||||
/opt/dreamdict/data/ # source data files (can be deleted after import)
|
||||
/etc/dreamdict/dreamdict.env # environment config
|
||||
/var/log/dreamdict/ # logs (or use journald)
|
||||
```
|
||||
|
||||
### `deploy/dreamdict.service`
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=DreamDict dictionary service
|
||||
After=network.target
|
||||
# If GogoBee ever runs as a service too:
|
||||
# Before=dreamdict.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=dreamdict
|
||||
Group=dreamdict
|
||||
EnvironmentFile=/etc/dreamdict/dreamdict.env
|
||||
ExecStart=/usr/local/bin/dreamdict server \
|
||||
--host 127.0.0.1 \
|
||||
--port 7777 \
|
||||
--db /opt/dreamdict/dict.db
|
||||
Restart=on-failure
|
||||
RestartSec=5s
|
||||
|
||||
# Hardening
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ReadWritePaths=/opt/dreamdict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
### `/etc/dreamdict/dreamdict.env`
|
||||
|
||||
```
|
||||
DREAMDICT_PORT=7777
|
||||
DREAMDICT_DB=/opt/dreamdict/dict.db
|
||||
```
|
||||
|
||||
### Install Steps
|
||||
|
||||
```bash
|
||||
# Create user
|
||||
useradd --system --no-create-home --shell /usr/sbin/nologin dreamdict
|
||||
|
||||
# Create dirs
|
||||
mkdir -p /opt/dreamdict /etc/dreamdict /var/log/dreamdict
|
||||
chown dreamdict:dreamdict /opt/dreamdict /var/log/dreamdict
|
||||
|
||||
# Build and install binary
|
||||
go build -o /usr/local/bin/dreamdict ./cmd/server
|
||||
chmod 755 /usr/local/bin/dreamdict
|
||||
|
||||
# Run import (as yourself, not dreamdict user)
|
||||
go run ./cmd/dictimport --db /opt/dreamdict/dict.db --data /opt/dreamdict/data/
|
||||
chown dreamdict:dreamdict /opt/dreamdict/dict.db
|
||||
|
||||
# Install and start service
|
||||
cp deploy/dreamdict.service /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now dreamdict
|
||||
|
||||
# Verify
|
||||
systemctl status dreamdict
|
||||
curl http://127.0.0.1:7777/health
|
||||
```
|
||||
|
||||
### Updating the Database
|
||||
|
||||
```bash
|
||||
# Stop service, rebuild database, restart
|
||||
systemctl stop dreamdict
|
||||
go run ./cmd/dictimport --db /opt/dreamdict/dict.db --clean
|
||||
chown dreamdict:dreamdict /opt/dreamdict/dict.db
|
||||
systemctl start dreamdict
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## GogoBee Integration
|
||||
|
||||
GogoBee gets a single `DreamDictClient` struct. Zero dictionary logic inside the bot.
|
||||
|
||||
### Client Package
|
||||
|
||||
Add to GogoBee: `internal/dreamclient/client.go`
|
||||
|
||||
```go
|
||||
package dreamclient
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Client struct {
|
||||
baseURL string
|
||||
httpClient *http.Client
|
||||
}
|
||||
|
||||
func New(baseURL string) *Client {
|
||||
return &Client{
|
||||
baseURL: baseURL,
|
||||
httpClient: &http.Client{Timeout: 3 * time.Second},
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Client) IsValidWord(word, lang string) (bool, error)
|
||||
func (c *Client) RandomWord(lang string, pos string, min, max int) (string, error)
|
||||
func (c *Client) Define(word, lang string) ([]Definition, error)
|
||||
func (c *Client) Synonyms(word, lang string) ([]string, error)
|
||||
func (c *Client) Translate(word, fromLang, toLang string) ([]string, error)
|
||||
func (c *Client) Health() (*HealthResponse, error)
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
In GogoBee's config:
|
||||
```yaml
|
||||
dictionary:
|
||||
url: "http://127.0.0.1:7777"
|
||||
```
|
||||
|
||||
When GogoBee eventually containerizes, this becomes the container network address.
|
||||
No other changes required.
|
||||
|
||||
### Replacing Wordnik Calls
|
||||
|
||||
Run `grep -r -i wordnik .` in the GogoBee repo to find all call sites. Replace:
|
||||
|
||||
| Wordnik call | GogoBee replacement |
|
||||
|---|---|
|
||||
| `GET /word/{word}/valid` | `dictClient.IsValidWord(word, lang)` |
|
||||
| `GET /words/randomWord` | `dictClient.RandomWord(lang, pos, min, max)` |
|
||||
| `GET /word/{word}/definitions` | `dictClient.Define(word, lang)` |
|
||||
| `GET /word/{word}/relatedWords` | `dictClient.Synonyms(word, lang)` |
|
||||
|
||||
Calls previously hardcoded to English: keep `lang = "en"`.
|
||||
|
||||
### Startup Behaviour
|
||||
|
||||
On GogoBee startup, call `/health` to verify dreamdict is reachable. Log a warning (not
|
||||
a fatal error) if unavailable — dictionary-dependent commands should fail gracefully with
|
||||
a user-facing message rather than taking the whole bot down.
|
||||
|
||||
---
|
||||
|
||||
## Data File Checklist
|
||||
|
||||
| File | Lang | Source |
|
||||
|---|---|---|
|
||||
| `scowl/final/english-words.*` | en | wordlist.aspell.net |
|
||||
| `wordnet/dict/data.{noun,verb,adj,adv}` | en | wordnet.princeton.edu |
|
||||
| `kaikki-en.jsonl` | en | kaikki.org |
|
||||
| `fr_FR.dic` | fr | github.com/LibreOffice/dictionaries |
|
||||
| `Lexique383.tsv` | fr | lexique.org |
|
||||
| `wolf.xml` | fr | github.com/nicolashernandez/WOLF |
|
||||
| `kaikki-fr.jsonl` | fr | kaikki.org |
|
||||
| `pt_PT.dic` | pt-PT | github.com/LibreOffice/dictionaries |
|
||||
| `dicionario-aberto.xml` | pt-PT | github.com/jorgecardoso/dicionario-aberto |
|
||||
| `kaikki-pt.jsonl` | pt-PT | kaikki.org |
|
||||
|
||||
`scripts/download-dict-data.sh` automates all downloads. Idempotent — skips existing files
|
||||
unless `--force` passed. Handles tar extraction (SCOWL), bz2 decompression (WOLF), and
|
||||
git clones where no stable direct URL exists.
|
||||
|
||||
---
|
||||
|
||||
## `go.mod` Dependencies
|
||||
|
||||
```
|
||||
modernc.org/sqlite # Pure-Go SQLite, no CGO
|
||||
```
|
||||
|
||||
Standard library covers everything else: `net/http`, `encoding/xml`, `encoding/json`,
|
||||
`bufio`, `compress/bzip2`, `database/sql`, `flag`, `log/slog`.
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Service Tests (`internal/dictionary/dict_test.go`)
|
||||
|
||||
Use `:memory:` SQLite. Cover:
|
||||
|
||||
1. `IsValidWord` — known word true, unknown false, case-insensitive
|
||||
2. `RandomWord` — matches length + POS filters; returns `ErrNoMatch` on empty set
|
||||
3. `Define` — priority ordering (wordnet before wiktionary); empty slice for no definitions
|
||||
4. `Synonyms` — deduplicated; empty slice when none
|
||||
5. `Translate` — returns correct target-lang translations; empty slice when none
|
||||
6. `ErrNotSeeded` — when `meta` table has no rows
|
||||
7. Gloss deduplication — identical gloss twice → one row
|
||||
8. Synonym deduplication — identical synonym twice → one row
|
||||
9. Translation deduplication — identical translation twice → one row
|
||||
10. `form-of` filtering — senses tagged `form-of` not inserted
|
||||
11. pt-PT Brazil filtering — `Brazil`-tagged senses rejected; untagged accepted
|
||||
|
||||
### HTTP Handler Tests (`cmd/server/`)
|
||||
|
||||
Use `httptest.NewServer`. Cover:
|
||||
|
||||
1. `/valid` — 200 true/false, 400 on missing params
|
||||
2. `/random` — 200 with word, 404 on no match
|
||||
3. `/define` — 200 with ordered definitions, 200 with empty array on no defs
|
||||
4. `/synonyms` — 200 with list, 200 with empty array
|
||||
5. `/translate` — 200 with list, 200 with empty array
|
||||
6. `/health` — 200 with counts
|
||||
|
||||
---
|
||||
|
||||
## Expected Database Size
|
||||
|
||||
| Language | Words | Definitions | Approx size |
|
||||
|---|---|---|---|
|
||||
| en | ~220,000 | ~380,000 | ~180MB |
|
||||
| fr | ~90,000 | ~200,000 | ~80MB |
|
||||
| pt-PT | ~75,000 | ~120,000 | ~50MB |
|
||||
| **Total** | | | **~310MB** |
|
||||
|
||||
Add to `.gitignore`: `dict.db`, `data/`
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Affix Expansion
|
||||
|
||||
**Implement shortly after initial launch, not at initial launch.**
|
||||
|
||||
The base-forms-only approach in the initial import means `IsValidWord("running", "en")`
|
||||
returns false because only `"run"` is stored. For Hangman this doesn't matter — DreamDict
|
||||
generates the puzzle word and it's always a base form. But for any command where a player
|
||||
submits a word for validation (UNO, word games, future puzzle modes), this is a real
|
||||
correctness bug, not a theoretical one.
|
||||
|
||||
**Recommended approach:** affix expansion at import time using `go-hunspell`
|
||||
(`github.com/trustmaster/go-hunspell`). Parse the `.aff` rules for each language and
|
||||
generate all valid inflected forms during `dictimport`, storing them in the `words` table
|
||||
alongside base forms. This keeps `IsValidWord` as a simple indexed lookup with no runtime
|
||||
stemming logic.
|
||||
|
||||
**Do not implement stemming as an alternative.** Stemming at query time is more complex,
|
||||
language-specific, and redundant if you expand affixes at import time. Pick one direction —
|
||||
affix expansion at import is the cleaner path for this architecture.
|
||||
|
||||
**Sequencing:** get the service running and validate data quality across all three languages
|
||||
first. Affix expansion is a `dictimport` change only — the HTTP API, schema, and GogoBee
|
||||
client are untouched.
|
||||
|
||||
---
|
||||
|
||||
## Deferred: Traefik Route for Second Consumer
|
||||
|
||||
When a second consumer for DreamDict exists, adding a Traefik route on parodia.dev's
|
||||
internal network is trivial — one label, one DNS entry, twenty minutes of work. Do it
|
||||
then, not now. The service architecture requires no changes at that point.
|
||||
|
||||
---
|
||||
|
||||
## Not Worth Doing: HTTP Caching Headers
|
||||
|
||||
The consumers are on localhost, there is no intermediary cache, and SQLite responses are
|
||||
already faster than any cache lookup. Drop this entirely.
|
||||
Reference in New Issue
Block a user