Checkpoint: raise output cap to 4096 + salvage truncated JSON
The grammar checkpoint capped num_predict at 1024, but qwen3.5:9b ignores
the prompt's "up to 5 issues" and emits ~17-20 suggestions (~2000 tokens)
on a 300+ word doc. The output hit the cap mid-array (done_reason=length),
the JSON never closed, and ParseCheckpoint found no parseable object -> a
502 in ~20s on every long doc (1024 tok @ ~50 tok/s, not a network
timeout). The repeated failures made the writing-assist helper look
permanently asleep.
Fix:
- Raise the cap to checkpointMaxTokens=4096. It is a ceiling, not a
target -- the model stops at its JSON close, so shorter docs are
unaffected; only genuinely long outputs use the headroom.
- Make ParseCheckpoint salvage the completed {...} suggestion objects
from a truncated array (refactor extractJSONObject onto a shared
firstBalancedObject scanner), so an over-long doc degrades to partial
feedback instead of a hard 502.
Verified live on millenia: 'The Missing Key' (382 words) now returns 200
with 17 suggestions in ~38s, previously 502 every time.
Claude-Session: https://claude.ai/code/session_016Yr6jELuRc7hyzYLccQKZd
This commit is contained in:
@@ -42,6 +42,21 @@ func TestParseCheckpoint(t *testing.T) {
|
||||
raw: `{"suggestions":[{"original":"use {x}","replacement":"use x","explanation":"drop the braces","type":"clarity"}]}`,
|
||||
want: 1,
|
||||
},
|
||||
{
|
||||
// The bug that 502'd every long doc: the model hit num_predict and the
|
||||
// JSON never closed. Two suggestions completed; the third is cut off.
|
||||
// Salvage keeps the two that completed instead of failing outright.
|
||||
name: "truncated mid-array salvages completed objects",
|
||||
raw: `{"suggestions":[{"original":"I has","replacement":"I have","explanation":"subject-verb agreement","type":"grammar"},{"original":"a apple","replacement":"an apple","explanation":"use an before a vowel","type":"grammar"},{"original":"she relised","replacement":"she real`,
|
||||
want: 2,
|
||||
},
|
||||
{
|
||||
// A truncated array where even the first object is incomplete has
|
||||
// nothing to salvage → still an error (the caller releases + retries).
|
||||
name: "truncated before any object closes",
|
||||
raw: `{"suggestions":[{"original":"It was a thursdy morning when Clara`,
|
||||
wantErr: true,
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
|
||||
Reference in New Issue
Block a user