The zh pair's other direction, and a rule pack that mostly says no
`pair_lang` had always been answering a second question nobody asked: it says which two languages, and every surface built on it assumed English was the one being learned. That is why hanzi is never tokenized, never spell-checked, never glossed — correct for a Mandarin native practising English, backwards for an English native practising Mandarin. `users.direction` (migration 0016) separates the two questions; a `zh-learner` pair code would have been cheaper and would have made two directions of one pair look like two unrelated languages to every query. Segmentation is what replaces `wordAt` where there are no spaces: a shortest-path walk over log-probabilities, 232 ms and 14 MB for 188,522 words. The browser gets the word list because segmentation runs on hover; the server keeps the whole dictionary. Their coverage gates come out opposite on purpose — the client list is frequency-gated because the segmentation is measurably identical without the tail, and the dictionary is gated by nothing, because its only power is to explain and the word a learner stops on is the rare one. The 错别字 pack is 24 confusable pairs behind two mechanical gates. One admits a pair only if the wrong form is not a dictionary word and the right form is, which is why it refuses 自已 for 自己 — a real error whose wrong form is a headword. The other asks the segmenter whether the two characters already belong to two different words, without which 自己经常, 睡觉的时候 and 不知到底 would all be corrupted silently into text still made of real characters. Not deployed (this carries a migration), not seen in a browser, and no account has ever been in the learner direction. The IME composition guards were in scope and are not done — see BUILD_PLAN Phase 26.
This commit is contained in:
@@ -0,0 +1,59 @@
|
||||
Chinese word list (segmentation)
|
||||
================================
|
||||
|
||||
`words.txt.gz` is not a spelling dictionary — Chinese has no spelling to check
|
||||
in the Hunspell sense. It is the word list Petal's segmenter walks, so that a
|
||||
sentence written without spaces has words in it to hover, look up and capture.
|
||||
Each line is `word frequency`. See scripts/build_cedict.py for how it is built
|
||||
and why it is gated where it is.
|
||||
|
||||
It is derived from two upstream sources, both redistributable, both credited
|
||||
here because the file itself has no room for a header.
|
||||
|
||||
|
||||
CC-CEDICT — the headwords
|
||||
-------------------------
|
||||
Community maintained free Chinese-English dictionary, published by MDBG.
|
||||
https://www.mdbg.net/chinese/dictionary?page=cedict
|
||||
|
||||
Licensed under the Creative Commons Attribution-ShareAlike 4.0 International
|
||||
License — https://creativecommons.org/licenses/by-sa/4.0/
|
||||
|
||||
Referenced works:
|
||||
CEDICT — Copyright (C) 1997, 1998 Paul Andrew Denisowski
|
||||
|
||||
CC-CEDICT is also the source of `internal/lexicon/data/hanzi.json.gz`, the
|
||||
pinyin and English senses embedded in the Petal binary. The same attribution and
|
||||
the same ShareAlike terms apply to that file; it is named here because it has
|
||||
nowhere of its own to say so.
|
||||
|
||||
|
||||
jieba — the frequencies
|
||||
-----------------------
|
||||
"结巴" Chinese word segmentation, by Sun Junyi.
|
||||
https://github.com/fxsjy/jieba
|
||||
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2013 Sun Junyi
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
||||
Only the word/frequency columns are used; jieba's part-of-speech tags and its
|
||||
algorithm are not (Petal's segmenter is its own, in web/src/lib/segment.ts).
|
||||
Binary file not shown.
Reference in New Issue
Block a user