Curated against English had quietly become curated against writing

Seen live: "Esta manhã acordei cedo e fui correr ao longo da marginal.
O ar estava fresco e havia poucas pessoas na rua." — unremarkable
Portuguese, two marker hits, zero English hits, and a verdict of
English. Corrected as English, read aloud in an American voice.

The list was missing the ordinary machinery of the language: the
contractions (ao, à, num), the tenses a diary is written in (estava,
havia, fomos), the words that join two clauses (até, depois, então,
onde). Each clears the bar the list already set — an English sentence
has no reason to contain them — so their absence bought nothing.

The floor stays at three. What changed is that three is now reachable
by prose rather than only by a paragraph that argues its own case. fr
and es get the same additions by analogy; neither has an account yet to
catch it live, which is exactly how this one survived. "sin" and "tan"
stay out of the es list: both are English words.

Two regression tests, pointed in opposite directions — ordinary
Portuguese must read as hers, and English about Portugal, English
quoting Portuguese, and a plain English diary must all still read as
English, held verdict included.

Claude-Session: https://claude.ai/code/session_01GJHNvirh7Hzhc9RL3HAvz7
This commit is contained in:
prosolis
2026-07-29 00:28:32 -07:00
parent e67f77eb05
commit acb35108c0
2 changed files with 87 additions and 0 deletions
+35
View File
@@ -155,6 +155,19 @@ func distinctMarkers(s string, markers map[string]bool) int {
//
// A single marker is not enough (see readsAsPairLang), so these lists are read
// as evidence to be corroborated rather than as a decision.
//
// **Curated against English is not the same as curated thinly**, and the first
// version of these lists confused the two. Seen live 2026-07-29: "Esta manhã
// acordei cedo e fui correr ao longo da marginal. O ar estava fresco e havia
// poucas pessoas na rua." — unremarkable Portuguese, two marker hits, *zero*
// English hits, and a verdict of English, because the document-level floor wants
// three. The list was missing the ordinary machinery of the language: the
// contractions (ao, à, num), the past tenses a diary is written in (estava,
// havia, fomos), and the words that join two clauses (até, depois, então,
// onde). Every one of them clears the bar above — an English sentence has no
// reason to contain them — so their absence bought nothing and cost the verdict.
// The floor stays at three; what changed is that three is now reachable by
// prose rather than only by a paragraph that happens to argue with itself.
var latinMarkers = map[string]map[string]bool{
"fr": words(
"je", "tu", "il", "elle", "ils", "elles", "nous", "vous", "est", "sont",
@@ -164,6 +177,10 @@ var latinMarkers = map[string]map[string]bool{
"beaucoup", "toujours", "jamais", "quand", "bien", "chose", "temps",
"moi", "toi", "lui", "peux", "veux", "sais", "faire", "dit", "aujourd",
"hui", "quelque", "chez", "tout", "tous", "rien", "déjà", "encore",
// The same gap the pt-PT list was caught with, closed by analogy rather
// than by observation — no fr account exists yet to catch it live.
"aux", "après", "où", "avait", "étaient", "depuis", "jusqu", "chaque",
"autre", "même", "hier", "demain", "matin", "soir", "nôtre", "leurs",
),
"pt-PT": words(
"eu", "você", "ele", "ela", "eles", "elas", "nós", "são", "uma", "os",
@@ -173,6 +190,18 @@ var latinMarkers = map[string]map[string]bool{
"nunca", "bem", "obrigado", "obrigada", "gosto", "tenho", "tem", "foi",
"ser", "ter", "mais", "já", "ainda", "aqui", "ali", "nada", "tudo",
"todos", "para", "pela", "pelo", "sobre", "assim",
// The contractions, which no English sentence has any use for.
"ao", "aos", "à", "às", "num", "numa", "dum", "duma", "pelos", "pelas",
"neste", "nesta", "disso", "deste", "desta",
// The tenses a journal is actually written in.
"estava", "estavam", "estão", "estamos", "havia", "houve", "era", "eram",
"fui", "fomos", "foram", "vai", "vamos", "tinha", "tinham",
// The joins between two clauses.
"até", "depois", "antes", "onde", "então", "enquanto", "embora",
"sem", "quem", "entre",
// And the everyday determiners and time words a diary can hardly avoid.
"nosso", "nossa", "outro", "outra", "mesmo", "mesma", "tão",
"muitos", "muitas", "poucos", "poucas", "hoje", "ontem", "amanhã",
),
"es": words(
"yo", "él", "ella", "ellos", "ellas", "nosotros", "una", "los", "las",
@@ -181,6 +210,12 @@ var latinMarkers = map[string]map[string]bool{
"hacer", "siempre", "nunca", "bien", "gracias", "tengo", "tiene", "fue",
"ser", "tener", "más", "aquí", "allí", "nada", "todos",
"para", "sobre", "así", "hola", "señor", "usted", "muchas",
// Likewise by analogy: no es account exists yet either. "sin" and "tan"
// stay out — both are English words, which is the one disqualification.
"al", "después", "antes", "donde", "entonces", "mientras", "aunque",
"estaba", "estaban", "están", "había", "hubo", "fuimos", "fueron",
"nuestro", "nuestra", "otro", "otra", "mismo", "misma", "quién", "quien",
"muchos", "pocas", "pocos", "hoy", "ayer", "mañana",
),
}