prosolis
77f284f65c
`pair_lang` had always been answering a second question nobody asked: it says which two languages, and every surface built on it assumed English was the one being learned. That is why hanzi is never tokenized, never spell-checked, never glossed — correct for a Mandarin native practising English, backwards for an English native practising Mandarin. `users.direction` (migration 0016) separates the two questions; a `zh-learner` pair code would have been cheaper and would have made two directions of one pair look like two unrelated languages to every query. Segmentation is what replaces `wordAt` where there are no spaces: a shortest-path walk over log-probabilities, 232 ms and 14 MB for 188,522 words. The browser gets the word list because segmentation runs on hover; the server keeps the whole dictionary. Their coverage gates come out opposite on purpose — the client list is frequency-gated because the segmentation is measurably identical without the tail, and the dictionary is gated by nothing, because its only power is to explain and the word a learner stops on is the rare one. The 错别字 pack is 24 confusable pairs behind two mechanical gates. One admits a pair only if the wrong form is not a dictionary word and the right form is, which is why it refuses 自已 for 自己 — a real error whose wrong form is a headword. The other asks the segmenter whether the two characters already belong to two different words, without which 自己经常, 睡觉的时候 and 不知到底 would all be corrupted silently into text still made of real characters. Not deployed (this carries a migration), not seen in a browser, and no account has ever been in the learner direction. The IME composition guards were in scope and are not done — see BUILD_PLAN Phase 26.
49 lines
2.1 KiB
Go
49 lines
2.1 KiB
Go
// Package lexicon serves offline word lookups — a Chinese gloss, a definition,
|
|
// and a list of synonyms for a single word — from public datasets compiled into
|
|
// the binary. Definitions come from the Wordset dictionary (modern, concise
|
|
// glosses with a part of speech and example, which read kindly for an ESL
|
|
// writer); synonyms come from the Moby Thesaurus; the Chinese gloss comes from
|
|
// ECDICT. All are gzipped JSON, decompressed lazily on first use so a writer who
|
|
// never looks up a word pays nothing.
|
|
package lexicon
|
|
|
|
import _ "embed"
|
|
|
|
// definitionsGz is the gzipped Wordset definitions map: word → [[pos, def,
|
|
// example], …], lowercase keys. Built by the data-prep step (see commit notes).
|
|
//
|
|
//go:embed data/definitions.json.gz
|
|
var definitionsGz []byte
|
|
|
|
// synonymsGz is the gzipped Moby thesaurus map: headword → [synonym, …],
|
|
// lowercase keys, capped per word to keep the popover (and the binary) small.
|
|
//
|
|
//go:embed data/synonyms.json.gz
|
|
var synonymsGz []byte
|
|
|
|
// glossGz is the gzipped English→Chinese gloss map: word → 中文 gloss, lowercase
|
|
// keys. Built from ECDICT (scripts/build_gloss.py), filtered to common words so
|
|
// an ESL writer who speaks Mandarin gets an instant translation on hover/lookup.
|
|
//
|
|
//go:embed data/gloss.json.gz
|
|
var glossGz []byte
|
|
|
|
// phoneticGz is the gzipped English-phonetic map: word → IPA, lowercase keys.
|
|
// Built from ECDICT's phonetic column (scripts/build_phonetic.py). Shown beside
|
|
// the read-aloud button so the writer — a Mandarin speaker learning English —
|
|
// can see how to say the word. Ships with a small common-word seed; a full build
|
|
// broadens coverage.
|
|
//
|
|
//go:embed data/phonetic.json.gz
|
|
var phoneticGz []byte
|
|
|
|
// hanziGz is the gzipped Chinese→English map: simplified headword → [[pinyin,
|
|
// senses], …]. Built from CC-CEDICT (scripts/build_cedict.py), unfiltered — the
|
|
// word a learner stops on is the one they do not know, so this is the one
|
|
// dataset here with no frequency gate. Loaded on its own sync.Once (see
|
|
// hanzi.go), not with the four above, because only a learner-direction account
|
|
// ever asks for it.
|
|
//
|
|
//go:embed data/hanzi.json.gz
|
|
var hanziGz []byte
|