Variant words from seedTestData were silently inflating TestWords counts.
Comments now track them explicitly, and a new combined filter test verifies
variant + frequency + length compose correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Seed color/colour with us/gb variants in test data
- Add TestWordVariant: verifies us, gb, common, pt-PT, and missing words
- Add TestVariantFilter: tests RandomWord and Words with variant param
- Update TestWords counts for new seeded variant words
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SCOWL loader now loads american-words and british-words alongside
english-words, tagging each with variant "us", "gb", or NULL (common)
- Words appearing in both American and British lists get NULL variant
- Add variant column to words table with schema migration
- API: /define returns variant field for English, /random and /words
accept ?variant=us|gb filter
- Add total word count to server status line
- Fix rows.Close leak in PopulateCMUTails, add rows.Err() check
- Add tests for PopulateCMUTails, cmuTail, TotalWordCount
- Expand test seed data and assertions to cover pt-PT definitions,
synonyms, and validity checks
- Update README with variant docs, /difficulty, /words, /rhyme
endpoints, and current database statistics
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace copy-pasted hasLetter/containsDigit/containsSpace/containsNonLatin
checks across 9 loaders with a single isJunkWord() call. SCOWL and CEDICT
keep their own filtering for valid reasons (SCOWL rejects hyphens/apostrophes,
CEDICT handles non-Latin Chinese characters).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add PruneOrphanWords post-import step that deletes words with no
definitions, synonyms, translations, pronunciations, etymology,
synsets, or frequency data
- Remove INSERT path from CETEMPublico loader — frequency data should
annotate existing words, not create new ones (was inflating pt-PT
from 136K real words to 718K)
- Add hasLetter filter to hunspell, affix, and dicionario loaders to
block punctuation entries like ".", ",", "(" from entering the DB
- Move hasLetter helper to scowl.go alongside other shared filters
Impact: en 655K→129K, pt-PT 1.5M→136K, fr 80K→56K (on reimport)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests: 23 new tests across cetempublico, affix, and wiktionary loaders
covering column order detection, ISO-8859-1 encoding, junk filtering,
frequency scaling, affix expansion, and sense-level synonym extraction.
Fixes: header detection now checks either field for numeric value (was
only checking last field, breaking count-first format). Added hasLetter
filter to reject pure punctuation entries like "." and ",".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Bearer token auth via --token flag or DREAMDICT_TOKEN env var
- Timing-safe comparison with crypto/subtle
- /health exempt for monitoring without credentials
- 401 response for missing/invalid tokens
- Auth disabled when no token configured (backwards compatible)
- Remove debug logging from cetempublico loader
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extract sense-level synonyms/antonyms from Wiktionary (not just top-level),
dramatically improving Portuguese synonym coverage
- Add bidirectional translation lookup so pt→en queries find en→pt entries
stored by English Wiktionary loader, with reverse index for performance
- Fix frequency loader: auto-detect column order (count\tword vs word\tcount),
auto-detect ISO-8859-1 encoding and convert to UTF-8, insert missing words
instead of only updating existing ones
- Switch WordNet download from 3.1 to 3.0 to match WOLF/OMW synset offsets
- Add OpenSubtitles-derived Portuguese frequency list (hermitdave/FrequencyWords)
- Increase affix expansion cap from 30→80 forms per word for Portuguese gerunds
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New /difficulty endpoint returns word difficulty scores (0.0-1.0).
New /rhyme endpoint uses CMU phoneme tail matching to find rhyming words,
sorted by frequency. Adds cmu_tail column migration with batch population
at startup, and index for efficient reverse lookups.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New endpoints: /antonyms, /backing, /pronunciation, /etymology with difficulty
scoring on /random. Cross-language synset backing links French/Portuguese words
to English equivalents via WordNet 3.0 synset IDs (matching WOLF and OMW offsets).
New loaders: Hunspell affix expansion (fr, pt-PT), English programmatic inflector,
CMU Pronouncing Dictionary, SUBTLEX-US frequency, CETEMPúblico frequency,
Open Multilingual Wordnet (Portuguese), and SQL-based difficulty scoring.
Schema v2 adds tables: antonyms, synsets, word_synsets, pronunciations, etymology
with migration support for existing databases.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Display query count, error count, req/s, uptime, and last endpoint
hit on a single updating line to stdout. Also wire up the
/frequency/batch route that was missing from the mux.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SCOWL loader now sets frequency based on tier (10=1000 most common,
70=50 rare). New GET /frequency?word=X&lang=Y endpoint returns frequency
score for a word. Enables downstream consumers to identify rare/
sophisticated vocabulary from dictionary data rather than LLM guessing.
Also fix .gitignore matching cmd/server/ directory instead of just the
server binary.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add CC-CEDICT loader and Wiktionary Chinese as data sources for
Mandarin. Also fix all Wiktionary download sections to retain
compressed archives and save checksums for integrity verification.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Go-based dictionary backend providing word validation, definitions, synonyms,
and cross-language translation for English, French, and Portuguese. Designed
as a self-hosted replacement for Wordnik, backing GogoBee.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
