Repository navigation
fix(j5): j5_spawn_agent can spawn a model with no reasoning option - #915
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
This review includes 9 billable files and costs up to $2.25.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 6 minutes for your next included review. View limit details
CodeRabbit Verify
|
There was a problem hiding this comment.
Combined review from the panel: Claude Opus 5.5 (lead) and GPT Sol 6.1. We read this at 59cf03f280 and agreed these findings. It still holds at 89057caaf9: the rebase left both commits identical (git range-diff), and the only change since the review is the field text below.
No blocking findings. The one suggested wording change has been applied. The live check is still pending, as the PR body says, so this review doesn't mean the PR is ready to merge.
Applied in 75356ad11f: reasoning field text (apps/server/src/j5/a2a/mcp/tools.ts)
The earlier description defined a reasoning option as "one with string values, such as effort or reasoningEffort". The code is narrower: the option id must be one of the known reasoning ids, and its values must all be strings. contextWindow and OpenCode's agent are also string-valued but aren't reasoning options.
We didn't find a model where this causes a failure. Every Claude model with contextWindow also has effort, and OpenCode falls back to default variant values when a model reports none. If an agent did misread it, the refusal would tell it what to do.
We asked for the text to be exact:
Required when the model has a reasoning option in orchestrator_capabilities: effort, reasoningEffort, reasoning, variant, or a thinking option with string values. Pass one of its values. Omit it for any other model; a true/false thinking option is not one.
What we checked
- Models with reasoning options: omitting
reasoningis refused and names the levels. An unlisted value is still refused. - Models without one: omitting it spawns with no
options. Passing it is refused, names the options the model has, and isn't silently dropped. - #770 allowlist: it can't be bypassed by omitting
reasoning, because a model with levels is refused inselectSpawnModelbeforeassertEligibleruns.capabilityModelOptiondoesn't filter by the saved list, so a saved list can't make such a model look like it has no levels. A model with no reasoning option has a null level and follows the model list only. - Persona routes:
reasoningEffortis required on a route, so an omittedreasoningnever matches one and no level is inferred. Haiku with a persona is refused with the route list. client_request_id: adding or removingreasoningon a bound key is a conflict, the same call replays, and a refused attempt binds nothing.- Null
reasoning:McpHttpServerdecodes throughSchema.toCodecJson, sonullarrives as omitted. delegate_taskand Crew custom seats: neither shares this rule, so no change is needed.- Docs:
agent-tools.md,spawning-guide.mdandagent-personas/index.mdmatch the behavior.docs/j5/user/personas.mdstays accurate.
Test evidence
Sol ran spawnWorkspace.test.ts, mcp/handlers.test.ts and mcp/tools.test.ts at 59cf03f280: 74 of 74 pass.
We didn't run the new tests against j5/main. From reading them, each should fail there, because the old code refuses Haiku, uses different refusal text and requires the field.
Not reviewed yet
- Evals: none ran, by Jackson's decision, and this PR has no eval cases. No eval result covers the tool text, as the PR body says; the live check is what tests it with real agents.
- Live check: the tester's result isn't in yet.
- CI: still running when this was posted.
Reviewed by Claude Opus 5.5 and GPT Sol 6.1, running in J5 Code.
reasoning was required on every spawn, so an agent that picked Claude Haiku 4.5, which has only a thinking boolean, was refused and had to switch models. reasoning is now required only when the model has a reasoning option, and a refusal names what the model offers. Passing reasoning for a model with no option is refused rather than dropped. A persona route always names a level, so an omitted reasoning never matches one. The custom-member lists, delegate_task and Crew custom seats need no change. Closes #704 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"One with string values" also fit contextWindow and agent, which are not reasoning options. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
75356ad to
89057ca
Compare
Problem
j5_spawn_agentrequiredreasoningon every spawn, but some models have no reasoning option. Claude Haiku 4.5 has only athinkingboolean, so an agent that picked it was refused and switched to another model. Closes #704.What changed
reasoningis now required only when the chosen model has a reasoning option: a select option such aseffortorreasoningEffort.reasoningThe other paths that read
reasoning:reasoningnever matches one and no level is inferred. A persona on Haiku is refused with its route list, like any other off-route pick.client_request_id:reasoningbinds like any other field. The same call replays, and adding or removingreasoningon a bound key is a conflict. A refused attempt binds nothing, so a corrected retry can keep its key.reasoning: a harness that sendsnullfor optional fields gets the same result as omitting it.delegate_taskand Crew custom seats: they don't share this rule, because their options are optional and only validated when given. No change.Tool description and schema are updated, and
agent-tools.md,spawning-guide.mdandagent-personas/index.mdare rewritten to match. These are the agent-direction and doc changes for Jackson to read before merge.Evals
Not run, by Jackson's decision (2026-11-03).
evals/README.mdcalls for an eval when model-facing text changes, butmcp-tool-useneeds a pay-per-use Anthropic API key, and Jackson won't set one up. This PR has no eval cases and norun.tsedits, so no eval result covers the tool description or schema text.The live test checks the text instead: real Claude Haiku and GPT Luna agents read the new description and schema on an isolated server and choose whether to pass
reasoning. See Live testing below.Live testing
Server behaviour and recovery passed; first-attempt tool-text use was partial. Tested final head
89057caaf9with thej5-live-testskill on an isolated server and test repository. Real UI turns ran on Claude Haiku 5.5 (Low) and Codex GPT-6-Luna (Medium). Each fresh parent was asked to start a messageableclaude-haiku-4-5helper, then agpt-6-lunahelper, using supported catalog settings. The prompts did not mentionreasoning, supply levels, or tell them how to correct errors.reasoningarguments"true", then omittedthinking (false, true)and retry-without-reasoning guidance; self-corrected and spawned"medium""medium"low, medium, high, xhigh, max; self-corrected and spawnedNeither parent switched models or received a corrective prompt from the tester. All four spawned helpers ran on their requested models, wrote their own proof file containing
started, and saiddone; verified from persisted tool inputs/results, child messages, and files. Failed attempts created no child. Thus the server and refusal guidance work, but the new text did not produce correct first-attempt arguments in every live sample. Evals were skipped by Jackson's decision; these live samples are the real-agent text check, not a statistical eval.Earlier server checks on
59cf03f280verified omitted reasoning success, missing Luna reasoning refusal with levels, and Haikureasoning: "high"refusal namingthinking (false, true). A text-only update was retested on75356ad11fthrough Luna (serializedreasoning: null, accepted as omission); final-head coverage above supersedes those samples. At-rest restarts retained test state. Interrupted runs, migrations, first-run defaults, and peering were not exercised because those paths are unchanged. Persona routes, Crew lists, and idempotency were left to the focused tests. Desktop, mobile, relay/tunnel, and other providers were not exercised. No UI changed, so captures were not needed. All tester servers were stopped through their tracked process groups.Checklist
delegate_taskand Crew seats checked, no change. Contracts and clients unchangeddocs/j5/product/rewritten.docs/j5/user/personas.mdstays accurateBuilt by Claude Opus 5.5 in Claude Code, running in J5 Code.
🤖 Generated with Claude Code