Phase 21: Petal learns to be an English+Portuguese pair

The plan said "Hunspell pt-PT vendored like en-US". Measuring that first is what saved it: nspell expands affixes eagerly on construction, and European Portuguese's 1,340 rules over 44,257 stems want over a gigabyte of browser heap — ~340 MB for the first 12,000 entries, and no return at all after three minutes on the whole file. So the expansion runs once at build time instead: 1,039,058 forms, 2.66 MB gzipped, read by the same nspell in 842 ms. The obvious npm package would also have shipped the wrong language. Both dictionary-pt and dictionary-pt-br carry VERO, the Brazilian word list, so vendoring by name puts pt-BR spellings behind a pt-PT label — the drift SUGGESTIONS §3 warns about, arriving through the packaging where no reviewer can see it. The source is Projecto Natura's, and the build script now asserts the fault lines (receção in, recepção out) before writing anything. Spellcheck consults both dictionaries and flags only what both reject, which is the no-detector answer to a pair with no script boundary. The word card does the same in the other direction: "data" is a word in both languages, so Petal shows both readings rather than guessing which she meant. Writing the tests caught the one real bug — extendedAlphabet was a snapshot while correct/suggest read live, and her dictionary arrives after English, so every lookup would have resolved "cora" while the underlines were already right. Not done, and not claimed: the pack has not been read by a pt-PT speaker, and the Piper voice is deferred with the deploy. Claude-Session: https://claude.ai/code/session_016y6gyuHkQXPiEuW8RGQyua
2026-07-27 12:43:02 -07:00
parent 4de83d0da5
commit ccb43e5a4d
22 changed files with 1458 additions and 107 deletions
@@ -0,0 +1,59 @@
+import { describe, it, expect } from 'vitest'
+import { Schema, type Node as PMNode } from '@tiptap/pm/model'
+import { wordAt } from './SpellCheck'
+
+// Same minimal schema the suggestion-anchoring tests use: enough of a document
+// to walk textblocks, none of the editor.
+const schema = new Schema({
+  nodes: {
+    doc: { content: 'block+' },
+    paragraph: { group: 'block', content: 'inline*', toDOM: () => ['p', 0] },
+    text: { group: 'inline' },
+  },
+})
+
+const para = (text: string): PMNode =>
+  schema.node('doc', null, [schema.node('paragraph', null, text ? [schema.text(text)] : [])])
+
+// posOf turns a plain-text offset into a ProseMirror position inside the single
+// paragraph (+1 for the paragraph's opening token).
+const posOf = (offset: number) => offset + 1
+
+describe('wordAt and the pair alphabet', () => {
+  it('keeps English-only tokenizing when no accented dictionary is loaded', () => {
+    const doc = para('the river runs')
+    expect(wordAt(doc, posOf(5))?.word).toBe('river')
+  })
+
+  it('cuts an accented word into fragments on the narrow alphabet', () => {
+    // Not a hypothetical: this is what every surface did before the pt-PT pair,
+    // and it is why the alphabet had to become a property of the checker rather
+    // than a constant. "coração" tokenized as A-Z yields "cora" — a definition
+    // of which would be worse than no popover.
+    const doc = para('o coração dela')
+    expect(wordAt(doc, posOf(3))?.word).toBe('cora')
+  })
+
+  it('resolves the whole word once the pair widens the alphabet', () => {
+    const doc = para('o coração dela')
+    expect(wordAt(doc, posOf(3), true)?.word).toBe('coração')
+    // …from either side of the accented letters, not just before them.
+    expect(wordAt(doc, posOf(9), true)?.word).toBe('coração')
+  })
+
+  it('never tokenizes CJK, whichever alphabet is in force', () => {
+    // The zh pair's guarantee, and it must survive a change made for another
+    // pair entirely: Chinese is the source language, not something to spellcheck.
+    const doc = para('我在写作 today')
+    expect(wordAt(doc, posOf(1))).toBeNull()
+    expect(wordAt(doc, posOf(1), true)).toBeNull()
+    expect(wordAt(doc, posOf(6), true)?.word).toBe('today')
+  })
+
+  it('stops the wide alphabet at the maths symbols hiding in Latin-1', () => {
+    // × (U+00D7) and ÷ (U+00F7) sit inside the accented-letter block. A range
+    // written À-ÿ would swallow them and glue "3×4" into one token.
+    const doc = para('3×4 é isso')
+    expect(wordAt(doc, posOf(4), true)?.word).toBe('é')
+  })
+})