047f4ae67f206e42e6ff9fe5f2f30f9d9ce21d1b
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 prosolis
|
77f284f65c |
The zh pair's other direction, and a rule pack that mostly says no
`pair_lang` had always been answering a second question nobody asked: it says which two languages, and every surface built on it assumed English was the one being learned. That is why hanzi is never tokenized, never spell-checked, never glossed — correct for a Mandarin native practising English, backwards for an English native practising Mandarin. `users.direction` (migration 0016) separates the two questions; a `zh-learner` pair code would have been cheaper and would have made two directions of one pair look like two unrelated languages to every query. Segmentation is what replaces `wordAt` where there are no spaces: a shortest-path walk over log-probabilities, 232 ms and 14 MB for 188,522 words. The browser gets the word list because segmentation runs on hover; the server keeps the whole dictionary. Their coverage gates come out opposite on purpose — the client list is frequency-gated because the segmentation is measurably identical without the tail, and the dictionary is gated by nothing, because its only power is to explain and the word a learner stops on is the rare one. The 错别字 pack is 24 confusable pairs behind two mechanical gates. One admits a pair only if the wrong form is not a dictionary word and the right form is, which is why it refuses 自已 for 自己 — a real error whose wrong form is a headword. The other asks the segmenter whether the two characters already belong to two different words, without which 自己经常, 睡觉的时候 and 不知到底 would all be corrupted silently into text still made of real characters. Not deployed (this carries a migration), not seen in a browser, and no account has ever been in the learner direction. The IME composition guards were in scope and are not done — see BUILD_PLAN Phase 26. |
||
|
prosolis
|
9224c44fff |
The es pair, and a dictionary that was quietly Spain's
Phase 25. Spanish was never built — the groundwork was all [x] (DreamDict data, the prompt language, the L1 rule gating, TTS env-discovery), which is why the plan read as though it had shipped. shippedPairs was the honest answer: the server had been refusing es on purpose. The langpack is neutral Latin American, chosen with the user: tú, ustedes, no vosotros, and the pan-American half of every vocabulary split. A vitest greps for the peninsular twins the way fr is greped for québécismes — including coger, which is not merely regional but obscene through most of Latin America. The dictionary is the story. Debian's hunspell-es symlinks twenty country codes to one file, which reads as pan-Hispanic; RLA publishes twenty-four builds per release, one per country plus a generic es that is the union, and Debian ships peninsular es_ES. The 58,622-form gap is essentially voseo, so the first version of this commit underlined vení and tenés as misspellings and called it a considered gap. The MUST_ACCEPT list was written to catch exactly that and structurally could not: it asserted the pan-Hispanic vocabulary, and every RLA variant carries the full pan-Hispanic vocabulary — only the paradigms are localised. The REP table cited as the second witness is shared by all builds too. Two independent-looking proofs, neither able to distinguish anything, agreeing with each other. The profile now demands what discriminates, each verified against the build it targets: voseo rejects es_ES and Debian, vosotros rejects es_MX, and arepa/chévere/bacán reject es_AR, which has both paradigms and would otherwise pass. 717,640 forms, 1.74 MB gzipped, 762 ms / 97 MB in a real nspell. fr and pt-PT rebuild byte-identical from their own upstream debs, so the shared script still means what it meant. Shipping the union is fr's call arrived at from the other side: coût and cout are both correct French, tienes and tenés are both correct Spanish. The dictionary holds every variety because underlining is all it can do; the copy picks a register because speaking requires one. Reviewed by four models at the usual >=2-of-4 threshold, 5 of 27 findings applied — one catching the bedtime proverb as fr's Qui dort dîne calqued into Spanish, gloss and all, which is the rule the fr header states. One below-threshold finding (a missing ¡, seen by 1 of 4 because an absent opening mark has no closing ! to look wrong against) was applied and turned into an assertion instead: the suite now rejects any native line that closes ? or ! without opening one. piper-es on es_MX-ald-medium, not the es_ES-davefx-medium the plan named — six of Piper's nine Spanish voices are peninsular, so the obvious pick was the pt-PT trap through a different door. go build/vet/test, tsc, vite, vitest 251/251. Not deployed, not seen in a browser, not read by a native speaker, and no es account exists. |
||
|
prosolis
|
be1ab5cef7 |
Her apostrophe was cutting French words in half
Typography.ts rewrites every ' typed in the editor into a curly ’, but both word regexes only counted the straight one. So "aujourd’hui" reached the dictionary as "aujourd" + "hui", neither of them a French word, and one of the commonest words in the language came back wearing two red underlines. Same for quelqu’un, presqu’île, prud’homme. l’arbre only survived by accident, because "l" happens to be a bare entry. withElision, written for exactly this, could only ever fire on pasted text. Both marks are word characters now, and combine() straightens on lookup — the one place every lookup passes through — since the shipped word lists spell theirs straight. Suggestions come back wearing whichever mark she actually used, so accepting a pill never swaps her apostrophe. œ was untokenizable too: U+0152/U+0153 sit outside the Latin-1 ranges, so "cœur" split into "c" + "ur" and the orphan was long enough to underline. 586 œ forms ship in fr.dic.gz and not one of them was reachable. In the dictionary builder, the two cross-product paths added their forms without the NEEDAFFIX check the single-affix paths apply, so a doubly-affixed form that is still "not a word on its own" was accepted anyway — the exact class of error the FLAG-aware rewrite exists to close. PFX and SFX are also separate flag namespaces, and one shared `cross` dict let the second block overwrite the first. Odd-length long-flag strings now stop the build instead of dropping a character and expanding through the wrong paradigm. The pt-PT and Québécois greps were case-sensitive against sentence-cased copy, which let a leading "Actualmente…" through the guard added to catch it. Note: this changes what the expander produces, but fr.dic.gz and pt-PT.dic.gz are vendored and were built with the old behaviour. Both want regenerating on a box that can fetch the upstream .deb, and BUILD_PLAN Phase 24's "pt-PT rebuild is byte-identical" claim re-checked — if those bytes move, the NEEDAFFIX gap was live in the Portuguese list too. Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua |
||
|
prosolis
|
071ea7b835 |
Petal learns French, and the pack that shipped was misspelling itself
Phase 24, the fr half: langpack, Hunspell dictionary, Piper voice, and the lexicon coverage that turned out to have been measured already (63.1%, better than pt-PT's 62.1%). No migration; not deployed. The plan recorded that build_ptpt_dictionary.py "generalizes" to French. It did not. It handled single-character flags and plain PFX/SFX and stopped on everything else, and fr.aff uses four of the things it stopped on. FLAG long is the dangerous one: French flags are two characters, so the old reader's set(flagstr) yields a bag of unrelated letters and expands every entry through the wrong paradigm without ever erroring. Plus continuation flags (French really does affix an affixed form), NEEDAFFIX on 68,075 of 84,140 stems, and FULLSTRIP. Renamed build_hunspell_dictionary.py with a per-language profile, asserting that CIRCUMFIX and FORBIDDENWORD are still unused rather than assuming it — and it rebuilds pt-PT byte-identical to the shipped asset, which is the only thing that makes "generalized" a claim rather than a hope. Elision was decided by building both halves and measuring. Keeping l'arbre and its thirty-three siblings: 3,159,832 forms, 8.25 MB gzipped. Dropping them: 473,326 and 1.19 MB. They are not new words, but the tokenizer keeps internal apostrophes, so they genuinely would have been underlined — so they moved out of the dictionary into withElision, which splits at a known clitic and still requires the remainder to be a word (l'zzzz stays flagged). Real nspell: 369 ms and 74 MB, against pt-PT's 842 ms and 139 MB, on the larger language. Where the regional trap lives is the mirror image of Portuguese's: every fr_* Piper voice is fr_FR and Debian's fr_FR/fr_CA/fr_BE dictionaries are one shared word list, so nothing can be quietly wrong about the country and the whole decision sits in the copy. What French has instead is the 1990 reform, packaged three ways; comprehensive ships, because Petal never corrects her French and coût and cout are both correct. Then the interim review pass, at the user's suggestion and explicitly "for now": four models read each Latin pack independently, and only findings at least two of them reached on their own were applied — five per pack. It earned its keep on the pack that was already live. pt-PT was carrying pre-Acordo spellings (adjectivos, actualmente) in a file whose own header commits to post-Acordo, plus Brazilian decepção, because the Phase 21 greps checked for Brazilian vocabulary and never checked the pack against its own spelling policy. That grep now exists and was confirmed to fail on the old text before being kept. Where reviewers agreed a line was wrong but split on the fix, the wording is mine and the reasoning is in BUILD_PLAN rather than averaged away. Still owed, and both packs now say so precisely: a quorum of models agreeing is agreement, not authority. No native speaker has read either pack, and none of this has been seen in a browser. go build/vet/test clean, tsc, vite build, vitest 190/190. Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua |
||
|
prosolis
|
ccb43e5a4d |
Phase 21: Petal learns to be an English+Portuguese pair
The plan said "Hunspell pt-PT vendored like en-US". Measuring that first is what saved it: nspell expands affixes eagerly on construction, and European Portuguese's 1,340 rules over 44,257 stems want over a gigabyte of browser heap — ~340 MB for the first 12,000 entries, and no return at all after three minutes on the whole file. So the expansion runs once at build time instead: 1,039,058 forms, 2.66 MB gzipped, read by the same nspell in 842 ms. The obvious npm package would also have shipped the wrong language. Both dictionary-pt and dictionary-pt-br carry VERO, the Brazilian word list, so vendoring by name puts pt-BR spellings behind a pt-PT label — the drift SUGGESTIONS §3 warns about, arriving through the packaging where no reviewer can see it. The source is Projecto Natura's, and the build script now asserts the fault lines (receção in, recepção out) before writing anything. Spellcheck consults both dictionaries and flags only what both reject, which is the no-detector answer to a pair with no script boundary. The word card does the same in the other direction: "data" is a word in both languages, so Petal shows both readings rather than guessing which she meant. Writing the tests caught the one real bug — extendedAlphabet was a snapshot while correct/suggest read live, and her dictionary arrives after English, so every lookup would have resolved "cora" while the underlines were already right. Not done, and not claimed: the pack has not been read by a pt-PT speaker, and the Piper voice is deferred with the deploy. Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua |
||
|
prosolis
|
84ee6bfb9c |
Phase 17: run the migration, and fix the guard that locked out its own backup
Claire's writing — 8 documents, 33 snapshots, 103 suggestions, 3 vocabulary words and an image — now belongs to her account rather than to the pre-auth 'local' user, and the VPS is canonical. millenia was left running and untouched as a frozen fallback; it diverges the moment either side is written to, so it wants retiring rather than syncing. The plan's stated prerequisite, that she log in once so her subject exists, turned out to be false. Authentik's hashed_user_id sub is the user's uid, derived from her id and the instance secret, so it can be read in advance — which means the data moves first and she signs in to find her writing already there, instead of to an empty Petal that fills in later. The fix here is to the liveness guard, and it is the second attempt at it. PRAGMA locking_mode = EXCLUSIVE goes on holding its lock after being set back to NORMAL — SQLite only lets go on that connection's next database access — so against a real WAL database the script locked itself out of its own VACUUM INTO backup. It passed locally because the test database had come out of VACUUM INTO and so was never in WAL mode: the fixture didn't look like production, the same way the stub identity provider's slashless issuer didn't. The probe now runs on its own connection and closes it. Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua |
||
|
prosolis
|
151df4565b |
Phase 17: a script to move the local user's writing onto a real account
The destination is an OIDC subject id, which the app cannot know — it belongs to the identity provider — so this runs deliberately, with Petal stopped and a backup taken, rather than as a startup migration. documents, tags, vocab_words and images carry user_id directly; versions, suggestions and tag assignments hang off their parents and follow, which is why it has to be one transaction with foreign keys off. Sessions for the old identity are deleted rather than moved: a session is proof someone signed in, and nobody ever signed in as 'local'. Dry run by default, VACUUM INTO backup first, and it verifies every row it expected to move actually moved — and that the source is left owning nothing — before committing. The 'is the app stopped?' guard took two attempts. BEGIN EXCLUSIVE, the obvious check, sails past a running-but-idle Petal because in WAL mode it only conflicts with another writer, which is precisely the case worth catching. PRAGMA locking_mode = EXCLUSIVE conflicts with any connection at all, since it locks the shared-memory index every WAL reader maps. Sequencing this also turned up a crash waiting to happen: the image backfill claims unowned files for 'local', which no longer exists after a migration, and the resulting foreign-key error is fatal inside images.New. Petal would have crash-looped the first time it started on a migrated database. It now skips a missing owner, which costs nothing — the migration moves the image rows itself. Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua |
||
|
prosolis
|
db737fa612 |
Editor: Find & Replace, read-aloud, backup, typography, phonetic, org niceties
Writer power-ups (Phase 11), plus the selection-bubble vs copy/paste fix. - Find & Replace (Ctrl/Cmd+F): SearchHighlight decoration extension + FindReplace bar (match-case, replace-all back-to-front, scroll without popping the selection bubble). - Read-aloud (Web Speech, offline) on the word card and selection bubble. - Keyboard/touch access to the ESL helpers: Ctrl/Cmd+D look up word at caret, Ctrl/Cmd+J rewrite selection, touch long-press lookup. Refactored the right-click handler into a shared openWordLookup(pos). - Whole-corpus backup: GET /api/docs/export-all zips every doc (md/docx), de-dupes filenames, dated name; sidebar download links. TestExportAll. - Smart typography input rules (curly quotes/em-dash/ellipsis), ASCII-only so CJK is untouched. - Duplicate doc, sidebar sort (Recent/Title/Longest), toolbar outline popover. - English phonetic (chosen over pinyin for an English learner): ECDICT-built phonetic.json.gz (46,579 words) + Result.Phonetic + WordCard IPA line; scripts/build_phonetic.py (full build + --seed fallback). - Selection bubble no longer blocks copy/paste: deferred to pointer-up and made click-through except on its buttons. Claude-Session: https://claude.ai/code/session_016Yr6jELuRc7hyzYLccQKZd |
||
|
prosolis
|
2487e73551 |
Ambient WebGL petals + cute notification sounds
Add a fixed, GPU-driven petal layer that drifts soft blossoms over the page (sparse, translucent, paused when hidden, skipped under prefers-reduced-motion). Add a synthesized cute sound palette — bubble pop, water droplet, soft bell, sparkle, cheer — generated as embedded WAV assets (scripts/gen_sounds.py) and played through a quiet Web Audio bus with soft-clipping and a persisted mute toggle in the status bar. Sounds fire per suggestion type when fresh advice arrives (staggered, old pending advice stays silent) and on companion bubbles by tone. Claude-Session: https://claude.ai/code/session_016Yr6jELuRc7hyzYLccQKZd |
||
|
prosolis
|
60eba25fee |
Phase 9: ESL superpowers — Chinese gloss + tone-rewrite
Inline Chinese gloss (offline) and a "say it more naturally" / tone-rewrite,
the two ESL features for the Mandarin-speaking writer.
Gloss: embedded English→Chinese dictionary (gloss.json.gz, 57k common words
built from ECDICT via scripts/build_gloss.py). lexicon gains Gloss()/Result.Gloss
and a lightweight GET /api/gloss/{word}; the right-click WordCard leads with the
中文; GlossTip shows it on a 350ms hover (reuses wordAt, so CJK is never glossed).
Offline + instant, works with the LLM down.
Rewrite: selecting text pops a SelectionBubble (✨更自然 + the tone vocabulary);
picking a style calls POST /api/docs/:id/rewrite (llm.RunRewrite, stateless,
owner-scoped) and shows a RewritePreview (original→rewrite, accept/cancel/retry).
Accept applies it in-editor.
Tests added in lexicon and suggestions. go build/vet/test, tsc, vite all clean;
live smoke vs a fake vLLM verified gloss + rewrite + 400/404/502 paths.
Claude-Session: https://claude.ai/code/session_016Yr6jELuRc7hyzYLccQKZd
|