Checkpoint: raise output cap to 4096 + salvage truncated JSON

The grammar checkpoint capped num_predict at 1024, but qwen3.5:9b ignores
the prompt's "up to 5 issues" and emits ~17-20 suggestions (~2000 tokens)
on a 300+ word doc. The output hit the cap mid-array (done_reason=length),
the JSON never closed, and ParseCheckpoint found no parseable object -> a
502 in ~20s on every long doc (1024 tok @ ~50 tok/s, not a network
timeout). The repeated failures made the writing-assist helper look
permanently asleep.

Fix:
- Raise the cap to checkpointMaxTokens=4096. It is a ceiling, not a
  target -- the model stops at its JSON close, so shorter docs are
  unaffected; only genuinely long outputs use the headroom.
- Make ParseCheckpoint salvage the completed {...} suggestion objects
  from a truncated array (refactor extractJSONObject onto a shared
  firstBalancedObject scanner), so an over-long doc degrades to partial
  feedback instead of a hard 502.

Verified live on millenia: 'The Missing Key' (382 words) now returns 200
with 17 suggestions in ~38s, previously 502 every time.

Claude-Session: https://claude.ai/code/session_016Yr6jELuRc7hyzYLccQKZd
This commit is contained in:
prosolis
2026-06-26 14:16:07 -07:00
parent 0f2f753efa
commit 8bd2509bc2
2 changed files with 93 additions and 13 deletions

View File

@@ -42,6 +42,21 @@ func TestParseCheckpoint(t *testing.T) {
raw: `{"suggestions":[{"original":"use {x}","replacement":"use x","explanation":"drop the braces","type":"clarity"}]}`,
want: 1,
},
{
// The bug that 502'd every long doc: the model hit num_predict and the
// JSON never closed. Two suggestions completed; the third is cut off.
// Salvage keeps the two that completed instead of failing outright.
name: "truncated mid-array salvages completed objects",
raw: `{"suggestions":[{"original":"I has","replacement":"I have","explanation":"subject-verb agreement","type":"grammar"},{"original":"a apple","replacement":"an apple","explanation":"use an before a vowel","type":"grammar"},{"original":"she relised","replacement":"she real`,
want: 2,
},
{
// A truncated array where even the first object is incomplete has
// nothing to salvage → still an error (the caller releases + retries).
name: "truncated before any object closes",
raw: `{"suggestions":[{"original":"It was a thursdy morning when Clara`,
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {