From 3f7e70502885c1a53eca10b60d12aa085a3b2303 Mon Sep 17 00:00:00 2001 From: prosolis <5590409+prosolis@users.noreply.github.com> Date: Thu, 25 Jun 2026 20:53:01 -0700 Subject: [PATCH] llm/ollama: disable thinking so reasoning models return content MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Qwen 3.5 — the spec's recommended model — is a reasoning model. With thinking on, Ollama streams its chain-of-thought into a separate `thinking` field and hits num_predict before emitting any answer into `content`, so Complete() got an empty string and the checkpoint failed with "no JSON object in model output". Sending `"think": false` on every /api/chat request fixes it; non-thinking models (qwen2.5) ignore the flag. Validated end-to-end on deployment hardware (Ollama, qwen3.5:9b): the grammar checkpoint now caught all five ESL errors in a 3-sentence sample with correct JSON and string-anchoring, ~8.5s warm. Claude-Session: https://claude.ai/code/session_016Yr6jELuRc7hyzYLccQKZd --- BUILD_PLAN.md | 1 + internal/llm/ollama.go | 7 +++++++ 2 files changed, 8 insertions(+) diff --git a/BUILD_PLAN.md b/BUILD_PLAN.md index 41d1a82..c76c9e0 100644 --- a/BUILD_PLAN.md +++ b/BUILD_PLAN.md @@ -7,6 +7,7 @@ Multi-session build. **Source of truth for what's done and what's next.** Update - **Copyleaks deferred** — needs a public webhook; skip Tier-2 plagiarism until there's a public endpoint. Tier-1 voice-consistency (local) is in scope. - **Traefik/deploy deferred** — local dev first. - **LLM**: Qwen 3.5 (256K context) on 64GB dual-GPU. Grammar checkpoint cap ~10K tokens (latency guard); voice pass sends whole document. + - **Reasoning models**: the Ollama client sends `"think": false` on every request. Qwen 3.5 is a reasoning model — left on, it streams chain-of-thought into a separate `thinking` field and exhausts `num_predict` before emitting any answer in `content` (empty response). Non-thinking models ignore the flag. (Validated on deployment hardware 2026-06-25.) - **Suggestion anchoring**: resolve by `original` string in ProseMirror coords at render time; stored `from_pos`/`to_pos` are plaintext offsets for server-side use only. (Spec Note #6.) - **Aesthetic is an acceptance criterion**: pretty, warm, Chinese-woman-friendly; CJK fonts first-class. diff --git a/internal/llm/ollama.go b/internal/llm/ollama.go index 581ef74..f467fed 100644 --- a/internal/llm/ollama.go +++ b/internal/llm/ollama.go @@ -16,10 +16,17 @@ type OllamaClient struct { // ollamaRequest mirrors Ollama's /api/chat body. Sampling parameters live under // "options" (see the cheatsheet in the spec). +// +// Think is sent false unconditionally: reasoning models (e.g. qwen3.5) otherwise +// stream their chain-of-thought into a separate "thinking" field and can exhaust +// num_predict before emitting any answer in "content" — leaving us an empty +// response. We want direct output (structured JSON for the checkpoint, concise +// prose for chat), not reasoning. Non-thinking models simply ignore the flag. type ollamaRequest struct { Model string `json:"model"` Messages []Message `json:"messages"` Stream bool `json:"stream"` + Think bool `json:"think"` Options ollamaOptions `json:"options"` }