The zh pair's other direction, and a rule pack that mostly says no
`pair_lang` had always been answering a second question nobody asked: it says which two languages, and every surface built on it assumed English was the one being learned. That is why hanzi is never tokenized, never spell-checked, never glossed — correct for a Mandarin native practising English, backwards for an English native practising Mandarin. `users.direction` (migration 0016) separates the two questions; a `zh-learner` pair code would have been cheaper and would have made two directions of one pair look like two unrelated languages to every query. Segmentation is what replaces `wordAt` where there are no spaces: a shortest-path walk over log-probabilities, 232 ms and 14 MB for 188,522 words. The browser gets the word list because segmentation runs on hover; the server keeps the whole dictionary. Their coverage gates come out opposite on purpose — the client list is frequency-gated because the segmentation is measurably identical without the tail, and the dictionary is gated by nothing, because its only power is to explain and the word a learner stops on is the rare one. The 错别字 pack is 24 confusable pairs behind two mechanical gates. One admits a pair only if the wrong form is not a dictionary word and the right form is, which is why it refuses 自已 for 自己 — a real error whose wrong form is a headword. The other asks the segmenter whether the two characters already belong to two different words, without which 自己经常, 睡觉的时候 and 不知到底 would all be corrupted silently into text still made of real characters. Not deployed (this carries a migration), not seen in a browser, and no account has ever been in the learner direction. The IME composition guards were in scope and are not done — see BUILD_PLAN Phase 26.
This commit is contained in:
@@ -561,6 +561,34 @@ DROP TABLE suggestions;
|
||||
ALTER TABLE suggestions_new RENAME TO suggestions;
|
||||
CREATE INDEX idx_suggestions_doc_id ON suggestions(doc_id);
|
||||
CREATE INDEX idx_suggestions_resolved ON suggestions(status, resolved_at);
|
||||
`,
|
||||
},
|
||||
{
|
||||
// Which half of the pair is being learned.
|
||||
//
|
||||
// `pair_lang` (0010) has always answered "which two languages", and every
|
||||
// surface built on it assumed the answer to a second question nobody had
|
||||
// asked: that English is the language being *learned*. That assumption is
|
||||
// load-bearing in a dozen places — CJK is deliberately never tokenized,
|
||||
// never spell-checked, never glossed; the prompts explain English in her
|
||||
// language; the vocabulary garden captures English words. All correct for
|
||||
// a Mandarin native practising English, and all backwards for an English
|
||||
// native practising Mandarin.
|
||||
//
|
||||
// A second pair code ('zh-learner') was the cheaper option and is the
|
||||
// wrong shape: it would make the two directions of one pair look like two
|
||||
// unrelated languages to every query, and it would have to be repeated for
|
||||
// fr, es and pt-PT before any of them could turn around. A column keeps
|
||||
// the two questions separate, which is what they are.
|
||||
//
|
||||
// 'learning_en' is the default and is what every existing row means — the
|
||||
// backfill is the DEFAULT itself, and it is right rather than merely
|
||||
// convenient: all three accounts today are Mandarin natives writing
|
||||
// English.
|
||||
name: "0016_user_direction",
|
||||
stmt: `
|
||||
ALTER TABLE users ADD COLUMN direction TEXT NOT NULL DEFAULT 'learning_en'
|
||||
CHECK(direction IN ('learning_en','learning_pair'));
|
||||
`,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -15,6 +15,19 @@ type User struct {
|
||||
// today, "pt-PT"/"fr"/"es" once the langpacks land. It selects the UI copy
|
||||
// and dictionary set, not the language they may type in.
|
||||
PairLang string `json:"pair_lang"`
|
||||
|
||||
// Direction says which half of the pair is being *learned*. Every pair until
|
||||
// now assumed one answer: the writer is native in X and practising English,
|
||||
// so hanzi is never tokenized and English is what gets underlined. Turn it
|
||||
// around — a native English speaker learning Chinese — and the same pair
|
||||
// wants the opposite of nearly every default.
|
||||
//
|
||||
// It is a separate column from PairLang rather than a second pair code
|
||||
// ("zh-learner") because it is a genuinely separate question: the pair says
|
||||
// *which two languages*, this says *which way round*. Keeping them apart is
|
||||
// what lets fr, es and pt-PT inherit the learner direction later without a
|
||||
// second langpack each.
|
||||
Direction string `json:"direction"`
|
||||
}
|
||||
|
||||
// Document is a single piece of writing. `Content` is the Tiptap JSON document
|
||||
|
||||
Reference in New Issue
Block a user