Skip to content

fix(j5): j5_spawn_agent can spawn a model with no reasoning option - #915

Merged
Jacksondr5 merged 2 commits into
j5/mainfrom
j5/704-spawn-without-reasoning
Nov 3, 2026
Merged

Jacksondr5 merged 2 commits into
j5/mainfrom
j5/704-spawn-without-reasoning

Conversation

@Jacksondr5

@Jacksondr5 Jacksondr5 commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Problem

j5_spawn_agent required reasoning on every spawn, but some models have no reasoning option. Claude Haiku 4.5 has only a thinking boolean, so an agent that picked it was refused and switched to another model. Closes #704.

What changed

reasoning is now required only when the chosen model has a reasoning option: a select option such as effort or reasoningEffort.

Model reasoning Result
No reasoning option omitted Spawns on the provider's defaults
No reasoning option given Refused, naming the options the model has, with "retry without reasoning". The value is not silently dropped
Has a reasoning option omitted Refused, naming the levels
Has a reasoning option given Unchanged, including the refusal for an unlisted value

The other paths that read reasoning:

  • Persona routes: a route always names a level, so an omitted reasoning never matches one and no level is inferred. A persona on Haiku is refused with its route list, like any other off-route pick.
  • Custom-member lists (feat(j5): choose which reasoning levels custom Crew members can use #770): no code change. A model with no reasoning option has no level to check, so it follows the model list only. A model with levels can't reach the check without one, so the level list isn't weakened. Tests cover both through the spawn path.
  • client_request_id: reasoning binds like any other field. The same call replays, and adding or removing reasoning on a bound key is a conflict. A refused attempt binds nothing, so a corrected retry can keep its key.
  • Null reasoning: a harness that sends null for optional fields gets the same result as omitting it.
  • delegate_task and Crew custom seats: they don't share this rule, because their options are optional and only validated when given. No change.

Tool description and schema are updated, and agent-tools.md, spawning-guide.md and agent-personas/index.md are rewritten to match. These are the agent-direction and doc changes for Jackson to read before merge.

Evals

Not run, by Jackson's decision (2026-11-03). evals/README.md calls for an eval when model-facing text changes, but mcp-tool-use needs a pay-per-use Anthropic API key, and Jackson won't set one up. This PR has no eval cases and no run.ts edits, so no eval result covers the tool description or schema text.

The live test checks the text instead: real Claude Haiku and GPT Luna agents read the new description and schema on an isolated server and choose whether to pass reasoning. See Live testing below.

Live testing

Server behaviour and recovery passed; first-attempt tool-text use was partial. Tested final head 89057caaf9 with the j5-live-test skill on an isolated server and test repository. Real UI turns ran on Claude Haiku 5.5 (Low) and Codex GPT-6-Luna (Medium). Each fresh parent was asked to start a messageable claude-haiku-4-5 helper, then a gpt-6-luna helper, using supported catalog settings. The prompts did not mention reasoning, supply levels, or tell them how to correct errors.

Parent Requested helper Actual reasoning arguments Result
Claude Haiku 5.5 Claude Haiku 4.5 "true", then omitted First refused with thinking (false, true) and retry-without-reasoning guidance; self-corrected and spawned
Claude Haiku 5.5 GPT-6-Luna "medium" Spawned first try
Codex GPT-6-Luna Claude Haiku 4.5 omitted Spawned first try
Codex GPT-6-Luna GPT-6-Luna omitted, then "medium" First refused with levels low, medium, high, xhigh, max; self-corrected and spawned

Neither parent switched models or received a corrective prompt from the tester. All four spawned helpers ran on their requested models, wrote their own proof file containing started, and said done; verified from persisted tool inputs/results, child messages, and files. Failed attempts created no child. Thus the server and refusal guidance work, but the new text did not produce correct first-attempt arguments in every live sample. Evals were skipped by Jackson's decision; these live samples are the real-agent text check, not a statistical eval.

Earlier server checks on 59cf03f280 verified omitted reasoning success, missing Luna reasoning refusal with levels, and Haiku reasoning: "high" refusal naming thinking (false, true). A text-only update was retested on 75356ad11f through Luna (serialized reasoning: null, accepted as omission); final-head coverage above supersedes those samples. At-rest restarts retained test state. Interrupted runs, migrations, first-run defaults, and peering were not exercised because those paths are unchanged. Persona routes, Crew lists, and idempotency were left to the focused tests. Desktop, mobile, relay/tunnel, and other providers were not exercised. No UI changed, so captures were not needed. All tester servers were stopped through their tracked process groups.

Checklist

  • One concern: the description has no "also"
  • Tests cover the changed behavior (backend changes ship with focused tests)
  • UI changes: not applicable, no UI changes
  • Live testing: isolated Claude and Codex passes recorded above, including first-attempt limitations
  • Upstream-owned files: not applicable, this PR only touches J5-owned paths
  • Upstream product: not applicable, upstream's product is unchanged
  • Surfaces: agents only, through the MCP tool. Providers: the rule reads the catalog, so it applies to every adapter. delegate_task and Crew seats checked, no change. Contracts and clients unchanged
  • Docs: definitions under docs/j5/product/ rewritten. docs/j5/user/personas.md stays accurate

Built by Claude Opus 5.5 in Claude Code, running in J5 Code.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
j5-code Ready Ready Preview Nov 3, 2026 3:46am UTC

Request Review

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

  • Run on-demand review

This review includes 9 billable files and costs up to $2.25.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 6 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available. Your 56 included PR review attempts over the past 7 days set your current allowance at 2 reviews per hour.

Learn how review limits work.

Review configuration:

Run configuration
Commits

Reviewing files that changed from the base of the PR and between 75ca81e and 89057ca.

Files selected for processing: 9
  • apps/server/src/j5/a2a/SpawnAgentService.ts
  • apps/server/src/j5/a2a/mcp/handlers.test.ts
  • apps/server/src/j5/a2a/mcp/tools.test.ts
  • apps/server/src/j5/a2a/mcp/tools.ts
  • apps/server/src/j5/a2a/spawnWorkspace.test.ts
  • apps/server/src/j5/agents/agentPersonaSpawn.ts
  • docs/j5/product/a2a/agent-tools.md
  • docs/j5/product/agent-personas/index.md
  • docs/j5/product/features/spawning-guide.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

@github-actions github-actions Bot added the vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. label Nov 3, 2026
@github-actions github-actions Bot added the size:M 30-99 effective changed lines (test files excluded in mixed PRs). label Nov 3, 2026

@Jacksondr5 Jacksondr5 left a comment •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Combined review from the panel: Claude Opus 5.5 (lead) and GPT Sol 6.1. We read this at 59cf03f280 and agreed these findings. It still holds at 89057caaf9: the rebase left both commits identical (git range-diff), and the only change since the review is the field text below.

No blocking findings. The one suggested wording change has been applied. The live check is still pending, as the PR body says, so this review doesn't mean the PR is ready to merge.

Applied in 75356ad11f: reasoning field text (apps/server/src/j5/a2a/mcp/tools.ts)

The earlier description defined a reasoning option as "one with string values, such as effort or reasoningEffort". The code is narrower: the option id must be one of the known reasoning ids, and its values must all be strings. contextWindow and OpenCode's agent are also string-valued but aren't reasoning options.

We didn't find a model where this causes a failure. Every Claude model with contextWindow also has effort, and OpenCode falls back to default variant values when a model reports none. If an agent did misread it, the refusal would tell it what to do.

We asked for the text to be exact:

Required when the model has a reasoning option in orchestrator_capabilities: effort, reasoningEffort, reasoning, variant, or a thinking option with string values. Pass one of its values. Omit it for any other model; a true/false thinking option is not one.

What we checked

  • Models with reasoning options: omitting reasoning is refused and names the levels. An unlisted value is still refused.
  • Models without one: omitting it spawns with no options. Passing it is refused, names the options the model has, and isn't silently dropped.
  • #770 allowlist: it can't be bypassed by omitting reasoning, because a model with levels is refused in selectSpawnModel before assertEligible runs. capabilityModelOption doesn't filter by the saved list, so a saved list can't make such a model look like it has no levels. A model with no reasoning option has a null level and follows the model list only.
  • Persona routes: reasoningEffort is required on a route, so an omitted reasoning never matches one and no level is inferred. Haiku with a persona is refused with the route list.
  • client_request_id: adding or removing reasoning on a bound key is a conflict, the same call replays, and a refused attempt binds nothing.
  • Null reasoning: McpHttpServer decodes through Schema.toCodecJson, so null arrives as omitted.
  • delegate_task and Crew custom seats: neither shares this rule, so no change is needed.
  • Docs: agent-tools.md, spawning-guide.md and agent-personas/index.md match the behavior. docs/j5/user/personas.md stays accurate.

Test evidence

Sol ran spawnWorkspace.test.ts, mcp/handlers.test.ts and mcp/tools.test.ts at 59cf03f280: 74 of 74 pass.

We didn't run the new tests against j5/main. From reading them, each should fail there, because the old code refuses Haiku, uses different refusal text and requires the field.

Not reviewed yet

  • Evals: none ran, by Jackson's decision, and this PR has no eval cases. No eval result covers the tool text, as the PR body says; the live check is what tests it with real agents.
  • Live check: the tester's result isn't in yet.
  • CI: still running when this was posted.

Reviewed by Claude Opus 5.5 and GPT Sol 6.1, running in J5 Code.

Jacksondr5 and others added 2 commits November 2, 2026 22:43
reasoning was required on every spawn, so an agent that picked Claude
Haiku 4.5, which has only a thinking boolean, was refused and had to
switch models. reasoning is now required only when the model has a
reasoning option, and a refusal names what the model offers.

Passing reasoning for a model with no option is refused rather than
dropped. A persona route always names a level, so an omitted reasoning
never matches one. The custom-member lists, delegate_task and Crew
custom seats need no change.

Closes #704

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"One with string values" also fit contextWindow and agent, which are not
reasoning options.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jacksondr5
Jacksondr5 force-pushed the j5/704-spawn-without-reasoning branch from 75356ad to 89057ca Compare November 3, 2026 03:45
@Jacksondr5
Jacksondr5 merged commit 93b7c36 into j5/main Nov 3, 2026
37 checks passed
@Jacksondr5
Jacksondr5 deleted the j5/704-spawn-without-reasoning branch November 3, 2026 04:01

This branch was successfully deployed

1 active deployment
Preview — 89057caa Deployed Nov 3, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M 30-99 effective changed lines (test files excluded in mixed PRs). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

j5_spawn_agent can't spawn a model that has no reasoning option

1 participant