Fix Portuguese coverage gaps: synonyms, translations, frequency, gerunds
- Extract sense-level synonyms/antonyms from Wiktionary (not just top-level), dramatically improving Portuguese synonym coverage - Add bidirectional translation lookup so pt→en queries find en→pt entries stored by English Wiktionary loader, with reverse index for performance - Fix frequency loader: auto-detect column order (count\tword vs word\tcount), auto-detect ISO-8859-1 encoding and convert to UTF-8, insert missing words instead of only updating existing ones - Switch WordNet download from 3.1 to 3.0 to match WOLF/OMW synset offsets - Add OpenSubtitles-derived Portuguese frequency list (hermitdave/FrequencyWords) - Increase affix expansion cap from 30→80 forms per word for Portuguese gerunds Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -205,9 +205,9 @@ func parseDicFile(path string) ([]dicEntry, error) {
|
||||
}
|
||||
|
||||
// maxFormsPerWord caps the number of inflected forms generated per base word.
|
||||
// Portuguese and French .aff files can produce hundreds of cross-product forms
|
||||
// per word; most are valid but the combinatorial explosion bloats the DB.
|
||||
const maxFormsPerWord = 30
|
||||
// Portuguese verbs can have 60+ valid conjugations (gerunds, subjunctives, etc.),
|
||||
// so the cap must be generous enough to avoid cutting off common forms.
|
||||
const maxFormsPerWord = 80
|
||||
|
||||
func expandWord(word, flags string, rules map[string]*affixRuleSet) []string {
|
||||
seen := map[string]bool{word: true} // base form already in DB
|
||||
|
||||
Reference in New Issue
Block a user