Her apostrophe was cutting French words in half
Typography.ts rewrites every ' typed in the editor into a curly ’, but both word regexes only counted the straight one. So "aujourd’hui" reached the dictionary as "aujourd" + "hui", neither of them a French word, and one of the commonest words in the language came back wearing two red underlines. Same for quelqu’un, presqu’île, prud’homme. l’arbre only survived by accident, because "l" happens to be a bare entry. withElision, written for exactly this, could only ever fire on pasted text. Both marks are word characters now, and combine() straightens on lookup — the one place every lookup passes through — since the shipped word lists spell theirs straight. Suggestions come back wearing whichever mark she actually used, so accepting a pill never swaps her apostrophe. œ was untokenizable too: U+0152/U+0153 sit outside the Latin-1 ranges, so "cœur" split into "c" + "ur" and the orphan was long enough to underline. 586 œ forms ship in fr.dic.gz and not one of them was reachable. In the dictionary builder, the two cross-product paths added their forms without the NEEDAFFIX check the single-affix paths apply, so a doubly-affixed form that is still "not a word on its own" was accepted anyway — the exact class of error the FLAG-aware rewrite exists to close. PFX and SFX are also separate flag namespaces, and one shared `cross` dict let the second block overwrite the first. Odd-length long-flag strings now stop the build instead of dropping a character and expanding through the wrong paradigm. The pt-PT and Québécois greps were case-sensitive against sentence-cased copy, which let a leading "Actualmente…" through the guard added to catch it. Note: this changes what the expander produces, but fr.dic.gz and pt-PT.dic.gz are vendored and were built with the old behaviour. Both want regenerating on a box that can fetch the upstream .deb, and BUILD_PLAN Phase 24's "pt-PT rebuild is byte-identical" claim re-checked — if those bytes move, the NEEDAFFIX gap was live in the Portuguese list too. Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
This commit is contained in:
@@ -50,6 +50,25 @@ describe('wordAt and the pair alphabet', () => {
|
||||
expect(wordAt(doc, posOf(6), true)?.word).toBe('today')
|
||||
})
|
||||
|
||||
it('keeps a curly apostrophe inside the word, because that is the only kind here', () => {
|
||||
// Typography.ts rewrites every ' typed in this editor into ’, so a token
|
||||
// that only matched the straight mark never saw an apostrophe at all.
|
||||
// "aujourd’hui" cut into "aujourd" + "hui" — neither one a French word, both
|
||||
// underlined, and withElision never reached.
|
||||
const doc = para('c’est aujourd’hui')
|
||||
expect(wordAt(doc, posOf(8), true)?.word).toBe('aujourd’hui')
|
||||
// English pays the same debt: "don’t" must stay whole to be looked up.
|
||||
expect(wordAt(doc, posOf(1))?.word).toBe('c’est')
|
||||
})
|
||||
|
||||
it('reads œ as a letter, not as a word boundary', () => {
|
||||
// U+0153 sits outside the Latin-1 ranges, so "cœur" tokenized as "c" + "ur"
|
||||
// and the orphan "ur" was long enough to be underlined. 586 œ forms ship in
|
||||
// the French word list and none of them were reachable.
|
||||
const doc = para('mon cœur')
|
||||
expect(wordAt(doc, posOf(5), true)?.word).toBe('cœur')
|
||||
})
|
||||
|
||||
it('stops the wide alphabet at the maths symbols hiding in Latin-1', () => {
|
||||
// × (U+00D7) and ÷ (U+00F7) sit inside the accented-letter block. A range
|
||||
// written À-ÿ would swallow them and glue "3×4" into one token.
|
||||
|
||||
@@ -32,8 +32,19 @@ interface PluginState {
|
||||
// writer with no Latin second language, adding accented letters can only find
|
||||
// new words to underline (the "café" and "naïve" she borrows), and finds no
|
||||
// mistakes she has actually made.
|
||||
const WORD_RE = /[A-Za-z][A-Za-z']*/g
|
||||
const WORD_RE_LATIN = /[A-Za-zÀ-ÖØ-öø-ÿ][A-Za-zÀ-ÖØ-öø-ÿ']*/g
|
||||
//
|
||||
// Both apostrophes count as word characters, because Typography.ts rewrites
|
||||
// every ' typed in this editor into a curly ’ — so by the time text reaches the
|
||||
// tokenizer the straight mark is only ever seen in pasted content. Matching only
|
||||
// the straight one cut "aujourd’hui" into "aujourd" + "hui", neither of which is
|
||||
// a French word, and left the elision handling in useSpellChecker unreachable.
|
||||
// The dictionaries spell theirs straight; `combine` normalises on lookup.
|
||||
//
|
||||
// Œ/œ are named explicitly: they sit at U+0152/U+0153, outside the Latin-1
|
||||
// ranges below, so without them "cœur" tokenized as "c" + "ur" and the bare
|
||||
// "ur" was underlined. Æ/æ need no help — they are inside À-Ö and Ø-ö.
|
||||
const WORD_RE = /[A-Za-z][A-Za-z'’]*/g
|
||||
const WORD_RE_LATIN = /[A-Za-zÀ-ÖØ-öø-ÿŒœ][A-Za-zÀ-ÖØ-öø-ÿŒœ'’]*/g
|
||||
|
||||
// wordRe returns a fresh matcher for the alphabet in force. Fresh because these
|
||||
// are /g regexes carrying lastIndex, and two scans sharing one would interleave.
|
||||
@@ -55,8 +66,8 @@ function isCheckable(word: string): boolean {
|
||||
// lookup sees the bare token; returns the core plus how many chars were trimmed
|
||||
// off the front (to re-anchor the decoration).
|
||||
function coreOf(word: string): { core: string; lead: number } {
|
||||
const lead = word.match(/^'+/)?.[0].length ?? 0
|
||||
const trail = word.match(/'+$/)?.[0].length ?? 0
|
||||
const lead = word.match(/^['’]+/)?.[0].length ?? 0
|
||||
const trail = word.match(/['’]+$/)?.[0].length ?? 0
|
||||
return { core: word.slice(lead, word.length - trail), lead }
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user