Her apostrophe was cutting French words in half

Typography.ts rewrites every ' typed in the editor into a curly ’, but both
word regexes only counted the straight one. So "aujourd’hui" reached the
dictionary as "aujourd" + "hui", neither of them a French word, and one of the
commonest words in the language came back wearing two red underlines. Same for
quelqu’un, presqu’île, prud’homme. l’arbre only survived by accident, because
"l" happens to be a bare entry. withElision, written for exactly this, could
only ever fire on pasted text.

Both marks are word characters now, and combine() straightens on lookup — the
one place every lookup passes through — since the shipped word lists spell
theirs straight. Suggestions come back wearing whichever mark she actually
used, so accepting a pill never swaps her apostrophe.

œ was untokenizable too: U+0152/U+0153 sit outside the Latin-1 ranges, so
"cœur" split into "c" + "ur" and the orphan was long enough to underline. 586 œ
forms ship in fr.dic.gz and not one of them was reachable.

In the dictionary builder, the two cross-product paths added their forms
without the NEEDAFFIX check the single-affix paths apply, so a doubly-affixed
form that is still "not a word on its own" was accepted anyway — the exact
class of error the FLAG-aware rewrite exists to close. PFX and SFX are also
separate flag namespaces, and one shared `cross` dict let the second block
overwrite the first. Odd-length long-flag strings now stop the build instead of
dropping a character and expanding through the wrong paradigm.

The pt-PT and Québécois greps were case-sensitive against sentence-cased copy,
which let a leading "Actualmente…" through the guard added to catch it.

Note: this changes what the expander produces, but fr.dic.gz and pt-PT.dic.gz
are vendored and were built with the old behaviour. Both want regenerating on a
box that can fetch the upstream .deb, and BUILD_PLAN Phase 24's "pt-PT rebuild
is byte-identical" claim re-checked — if those bytes move, the NEEDAFFIX gap
was live in the Portuguese list too.

Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
This commit is contained in:
prosolis
2026-07-27 17:07:30 -07:00
parent 071ea7b835
commit be1ab5cef7
6 changed files with 92 additions and 19 deletions
@@ -50,6 +50,25 @@ describe('wordAt and the pair alphabet', () => {
expect(wordAt(doc, posOf(6), true)?.word).toBe('today')
})
it('keeps a curly apostrophe inside the word, because that is the only kind here', () => {
// Typography.ts rewrites every ' typed in this editor into , so a token
// that only matched the straight mark never saw an apostrophe at all.
// "aujourdhui" cut into "aujourd" + "hui" — neither one a French word, both
// underlined, and withElision never reached.
const doc = para('cest aujourdhui')
expect(wordAt(doc, posOf(8), true)?.word).toBe('aujourdhui')
// English pays the same debt: "dont" must stay whole to be looked up.
expect(wordAt(doc, posOf(1))?.word).toBe('cest')
})
it('reads œ as a letter, not as a word boundary', () => {
// U+0153 sits outside the Latin-1 ranges, so "cœur" tokenized as "c" + "ur"
// and the orphan "ur" was long enough to be underlined. 586 œ forms ship in
// the French word list and none of them were reachable.
const doc = para('mon cœur')
expect(wordAt(doc, posOf(5), true)?.word).toBe('cœur')
})
it('stops the wide alphabet at the maths symbols hiding in Latin-1', () => {
// × (U+00D7) and ÷ (U+00F7) sit inside the accented-letter block. A range
// written À-ÿ would swallow them and glue "3×4" into one token.
+15 -4
View File
@@ -32,8 +32,19 @@ interface PluginState {
// writer with no Latin second language, adding accented letters can only find
// new words to underline (the "café" and "naïve" she borrows), and finds no
// mistakes she has actually made.
const WORD_RE = /[A-Za-z][A-Za-z']*/g
const WORD_RE_LATIN = /[A-Za-zÀ-ÖØ-öø-ÿ][A-Za-zÀ-ÖØ-öø-ÿ']*/g
//
// Both apostrophes count as word characters, because Typography.ts rewrites
// every ' typed in this editor into a curly — so by the time text reaches the
// tokenizer the straight mark is only ever seen in pasted content. Matching only
// the straight one cut "aujourdhui" into "aujourd" + "hui", neither of which is
// a French word, and left the elision handling in useSpellChecker unreachable.
// The dictionaries spell theirs straight; `combine` normalises on lookup.
//
// Œ/œ are named explicitly: they sit at U+0152/U+0153, outside the Latin-1
// ranges below, so without them "cœur" tokenized as "c" + "ur" and the bare
// "ur" was underlined. Æ/æ need no help — they are inside À-Ö and Ø-ö.
const WORD_RE = /[A-Za-z][A-Za-z']*/g
const WORD_RE_LATIN = /[A-Za-zÀ-ÖØ-öø-ÿŒœ][A-Za-zÀ-ÖØ-öø-ÿŒœ']*/g
// wordRe returns a fresh matcher for the alphabet in force. Fresh because these
// are /g regexes carrying lastIndex, and two scans sharing one would interleave.
@@ -55,8 +66,8 @@ function isCheckable(word: string): boolean {
// lookup sees the bare token; returns the core plus how many chars were trimmed
// off the front (to re-anchor the decoration).
function coreOf(word: string): { core: string; lead: number } {
const lead = word.match(/^'+/)?.[0].length ?? 0
const trail = word.match(/'+$/)?.[0].length ?? 0
const lead = word.match(/^[']+/)?.[0].length ?? 0
const trail = word.match(/[']+$/)?.[0].length ?? 0
return { core: word.slice(lead, word.length - trail), lead }
}