[hanna-paasivirta]

Short Description

Move to the Claude 5.5 models.

Closes OFN-4756

Implementation Details

Into release/3.5.0, which already has the bumps from #716.

Models

• IDs: claude-opus-5-5, claude-sonnet-5-5, claude-haiku-5-5 in models.py. Routes keep their tier.
• Effort: unchanged (medium on the planner, job_chat and workflow_chat).

Old behaviour kept

• Thinking off where it was off: the 5.5 models think when thinking is omitted; 4.x didn't. Router and status pass disabled (Haiku 5.5); docs retrieval, the test judge, vocab_mapper and doc_agent_chat pass between_tools (Sonnet 5.5 rejects disabled). Both settings live in models.py, and those calls keep their old max_tokens.

Fixes the new models need

• Text by block type: the router, docs retrieval, vocab_mapper and the test judge read content[0], which can be a thinking block.
• max_tokens raised on the calls that think: planner, job_chat, workflow_chat 32768. workflow_chat gets the explicit timeout the SDK requires above ~21k.
• Refusals: a decline is a 200 with stop_reason: "refusal". The planner, job_chat and workflow_chat raise 422 MODEL_REFUSAL with the category instead of an empty-output error; workflow_chat no longer retries it.
• Planner narration: notes between tool calls come back as thinking blocks, so the planner sets display: "updates" and streams them as narration.

Results

29 global_chat acceptance specs, no judges, one run each, one at a time: release/3.5.0 (old models) against this branch. Cost per call from Langfuse.

Folder	Specs	Cost	Time	Same agents	Same tool order	Reply length (median, range)
gold_workflows	6	$1.28 → $1.58 (+24%)	436s → 389s (−11%)	6/6	4/6	1.5× (1.3–1.8×)
one_shot_workflows	6	$0.71 → $0.79 (+11%)	216s → 246s (+14%)	6/6	5/6	1.4× (1.2–1.9×)
multi_step_changes	5	$0.27 → $0.33 (+20%)	104s → 133s (+27%)	3/5	3/5	1.6× (0.9–5.2×)
job_code	11	$0.39 → $0.32 (−16%)	259s → 241s (−7%)	10/11	9/11	1.4× (0.7–9.4×)
bugs	1	$0.06 → $0.05 (−12%)	24s → 20s (−17%)	1/1	1/1	1.3×
All	29	$2.71 → $3.08 (+14%)	1040s → 1029s (−1%)	26/29	22/29	1.4× (0.7–9.4×)

Behaviour changes

• Debugging questions: explains the cause and asks before changing code. The old planner applied fixes nobody asked for.
• Router: same decisions, except a single-step edit for a step other than the one on the page now goes to the planner (3/3). Right result, ~2× time and cost. Left as is: the prompt fix tried sent planner-only requests to subagents.
• Missing key information ("back up the data somewhere"): picks a destination and builds, where the old model asked. 1 of 2 such specs.
• Short build requests: adds duplicate checks or incremental fetching unasked. Detailed specs are followed exactly.
• Workflow shape: one more step in 3 of 12 builds (e.g. a separate transform step). On kobo-openmrs-vague this is now the webhook plus duplicate lookup the spec asks for.
• Replies ~1.4× longer: mostly stated assumptions and caveats. One of 6 code edits also pasted the code into the chat.
• Reads more before answering: opens another step's code first in 2 job_code specs.
• Narration before tool calls: 9 of 15 runs, against 13 of 13.
• Docs retrieval: asks for docs on some debugging questions the old model answered without.

Not tested

• Skills (Add skills as slash commands #711) on the new models.
• False refusals on health-data requests.
• Customer keys: the 5.5 models have their own rate-limit pools.

AI Usage

• Yes, I have used AI
• No, I have not used AI

You can read more details in our
Responsible AI Policy