Fix Portuguese coverage gaps: synonyms, translations, frequency, gerunds

- Extract sense-level synonyms/antonyms from Wiktionary (not just top-level),
  dramatically improving Portuguese synonym coverage
- Add bidirectional translation lookup so pt→en queries find en→pt entries
  stored by English Wiktionary loader, with reverse index for performance
- Fix frequency loader: auto-detect column order (count\tword vs word\tcount),
  auto-detect ISO-8859-1 encoding and convert to UTF-8, insert missing words
  instead of only updating existing ones
- Switch WordNet download from 3.1 to 3.0 to match WOLF/OMW synset offsets
- Add OpenSubtitles-derived Portuguese frequency list (hermitdave/FrequencyWords)
- Increase affix expansion cap from 30→80 forms per word for Portuguese gerunds

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
prosolis
2026-04-03 08:48:22 -07:00
parent 25ed087c3e
commit 2cc1832439
6 changed files with 157 additions and 48 deletions

View File

@@ -58,6 +58,7 @@ CREATE TABLE IF NOT EXISTS translations (
);
CREATE INDEX IF NOT EXISTS idx_translations_word_id ON translations(word_id);
CREATE INDEX IF NOT EXISTS idx_translations_reverse ON translations(translation, target_lang);
CREATE TABLE IF NOT EXISTS meta (
key TEXT PRIMARY KEY,