drwily/petal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

llm/ollama: disable thinking so reasoning models return content

Browse Source 
Qwen 3.5 — the spec's recommended model — is a reasoning model. With
thinking on, Ollama streams its chain-of-thought into a separate
`thinking` field and hits num_predict before emitting any answer into
`content`, so Complete() got an empty string and the checkpoint failed
with "no JSON object in model output". Sending `"think": false` on every
/api/chat request fixes it; non-thinking models (qwen2.5) ignore the flag.

Validated end-to-end on deployment hardware (Ollama, qwen3.5:9b): the
grammar checkpoint now caught all five ESL errors in a 3-sentence sample
with correct JSON and string-anchoring, ~8.5s warm.

Claude-Session: https://claude.ai/code/session_016Yr6jELuRc7hyzYLccQKZd
This commit is contained in:
prosolis
2026-06-25 20:53:01 -07:00
parent a4069d5755
commit 3f7e705028
2 changed files with 8 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
BUILD_PLAN.md
View File

@@ -7,6 +7,7 @@ Multi-session build. **Source of truth for what's done and what's next.** Update
- **Copyleaks deferred** — needs a public webhook; skip Tier-2 plagiarism until there's a public endpoint. Tier-1 voice-consistency (local) is in scope. - **Copyleaks deferred** — needs a public webhook; skip Tier-2 plagiarism until there's a public endpoint. Tier-1 voice-consistency (local) is in scope.
- **Traefik/deploy deferred** — local dev first. - **Traefik/deploy deferred** — local dev first.
- **LLM**: Qwen 3.5 (256K context) on 64GB dual-GPU. Grammar checkpoint cap ~10K tokens (latency guard); voice pass sends whole document. - **LLM**: Qwen 3.5 (256K context) on 64GB dual-GPU. Grammar checkpoint cap ~10K tokens (latency guard); voice pass sends whole document.
- **Reasoning models**: the Ollama client sends `"think": false` on every request. Qwen 3.5 is a reasoning model — left on, it streams chain-of-thought into a separate `thinking` field and exhausts `num_predict` before emitting any answer in `content` (empty response). Non-thinking models ignore the flag. (Validated on deployment hardware 2026-06-25.)
- **Suggestion anchoring**: resolve by `original` string in ProseMirror coords at render time; stored `from_pos`/`to_pos` are plaintext offsets for server-side use only. (Spec Note #6.) - **Suggestion anchoring**: resolve by `original` string in ProseMirror coords at render time; stored `from_pos`/`to_pos` are plaintext offsets for server-side use only. (Spec Note #6.)
- **Aesthetic is an acceptance criterion**: pretty, warm, Chinese-woman-friendly; CJK fonts first-class. - **Aesthetic is an acceptance criterion**: pretty, warm, Chinese-woman-friendly; CJK fonts first-class.

7
internal/llm/ollama.go
View File

@@ -16,10 +16,17 @@ type OllamaClient struct {
// ollamaRequest mirrors Ollama's /api/chat body. Sampling parameters live under // ollamaRequest mirrors Ollama's /api/chat body. Sampling parameters live under
// "options" (see the cheatsheet in the spec). // "options" (see the cheatsheet in the spec).
//
// Think is sent false unconditionally: reasoning models (e.g. qwen3.5) otherwise
// stream their chain-of-thought into a separate "thinking" field and can exhaust
// num_predict before emitting any answer in "content" — leaving us an empty
// response. We want direct output (structured JSON for the checkpoint, concise
// prose for chat), not reasoning. Non-thinking models simply ignore the flag.
type ollamaRequest struct { type ollamaRequest struct {
Model string `json:"model"` Model string `json:"model"`
Messages []Message `json:"messages"` Messages []Message `json:"messages"`
Stream bool `json:"stream"` Stream bool `json:"stream"`
Think bool `json:"think"`
Options ollamaOptions `json:"options"` Options ollamaOptions `json:"options"`
} }