Prune ghost words from affix expansion and stop frequency loader creating words

- Add PruneOrphanWords post-import step that deletes words with no
  definitions, synonyms, translations, pronunciations, etymology,
  synsets, or frequency data
- Remove INSERT path from CETEMPublico loader — frequency data should
  annotate existing words, not create new ones (was inflating pt-PT
  from 136K real words to 718K)
- Add hasLetter filter to hunspell, affix, and dicionario loaders to
  block punctuation entries like ".", ",", "(" from entering the DB
- Move hasLetter helper to scowl.go alongside other shared filters

Impact: en 655K→129K, pt-PT 1.5M→136K, fr 80K→56K (on reimport)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
prosolis
2026-04-03 19:55:37 -07:00
parent 2045cc2ada
commit aff583dc28
9 changed files with 125 additions and 36 deletions

View File

@@ -48,7 +48,7 @@ func (a AffixLoader) Load(db *sql.DB, dataDir string) error {
seen := make(map[string]struct{})
for _, entry := range dicEntries {
for _, form := range expandWord(entry.word, entry.flags, rules) {
if containsDigit(form) || containsSpace(form) {
if !hasLetter(form) || containsDigit(form) || containsSpace(form) {
continue
}
if _, dup := seen[form]; dup {