Skip to content

feat(j5): choose which models custom Crew members can use - #757

Merged
bryantderosier merged 18 commits into
j5/748-archive-removes-clean-worktreesfrom
j5/744-custom-member-models
Nov 3, 2026
Merged

bryantderosier merged 18 commits into
j5/748-archive-removes-clean-worktreesfrom
j5/744-custom-member-models

Conversation

@bryantderosier

@bryantderosier bryantderosier commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

A Captain staffing a custom member, or an agent delegating to another provider, can pick any model in the catalog, and once additions and auto-approved launches skip the roster card I won't get a chance to correct it. This adds one list of the models custom workers may run on, and the platform enforces it everywhere a custom worker gets a model at creation. Closes #744.

This is stacked on #748, so the base is j5/748-archive-removes-clean-worktrees. Reasoning levels are a separate follow-up in #770, stacked on this PR.

  • Setting: Agent Setup → Crews → Models for custom members, a project-scoped server setting (crewMemberModels) holding checked (providerInstanceId, model) pairs. If nothing has been saved, every model is eligible, so nothing changes for existing users. Once I save a list, only checked models are eligible and a model that shows up in the catalog later starts unchecked. A saved list with nothing checked means nothing is eligible, and the row says so.
  • The row: the button shows "N of M" for checked models in the catalog and opens one collapsible card per set-up provider (one open at a time, count in the header, Check all and Uncheck all). Each model is a same-size chip, solid blue when allowed, and the providers that aren't set up fold into one line linking to Providers. The Claude card drops the "Claude" prefix from its chips, and Cursor keeps it for the Claude models it lists. The row keeps entries for models that left the catalog, and has the usual scope indicator, inheritance popover and reset. It follows crewAutoApprove in the mixed-scope state.
  • One check: CrewMemberModels.assertEligible is called from CrewLaunchService.resolveSelection (so preview, approval, auto-approve, j5_request_crew_member and additions are all covered, including a custom seat that inherits the Captain's model), from j5_spawn_agent when no persona is named, and from delegate_task when no persona is named, including a model inherited from the parent.
  • Personas: a persona seat that keeps either of its persona's two model routes isn't checked. One whose model I changed is checked like a custom seat. I compare the seat's provider driver and model against both routes, because editing any field on the roster card freezes the persona's model into the seat, and a seat frozen on the fallback route while the primary was signed out must still count as the persona's own. The driver match means another instance of the same driver on the same model also counts as unchanged.
  • Refusals: a typed error naming the model, the project and the setting ("Models for custom members in Agent Setup"), with the eligible models listed so the agent can retry. The roster card only offers eligible models for a custom member, and approving an ineligible one is refused with the same message. A retry of something already accepted isn't checked again.
  • Creation only: the list is enforced at launch and when a member is added. A later model change, for example an agent calling t3_thread_configure, is not limited. That's written in the docs.
  • Native subagents: I looked at Codex, Claude and Cursor. Codex and Cursor adapters only record the subagent's model after the provider launches it, so that's "not supported here". Claude's canUseTool sees the call and the requested model, but I couldn't confirm it fires in bypass mode and denying would need an adapter change, so it's written up as unverified and treated as not supported. crews.md says it per provider and the Agent Setup note says the list doesn't cover native subagents.

Docs: crews.md AC24–AC28, the j5_propose_crew, j5_request_crew_member, delegate_task and j5_spawn_agent sections of agent-tools.md (one sentence each), and the persona docs' note on model allowlists.

Evidence

At 76eabab0be, on top of #748, after the stack was rebased onto j5/main. vp run -r typecheck (the CI command) exits 0 here:

  • 1280 server tests passed (1 skipped) across the Crew, agents and MCP areas:
    cd apps/server && vp test run src/j5/a2a src/j5/agents src/mcp
  • New cases cover no saved list (everything allowed), a checked and an unchecked model, a model new to the catalog after saving, a persona exception on either route and an overridden persona model, an inherited model on a custom seat and on delegate_task, a project override including null, a saved empty list, and an accepted retry. The persona case that failed on the old code is a fallback route frozen at proposal while the primary is back at approval. The real-orchestrator delegate_task test goes through the new hook.
  • 1142 web tests passed (src/j5 and src/components/settings), including a card test that a custom member's picker is filtered and a persona's isn't:
    cd apps/web && vp test run src/j5/crew src/j5/a2a src/components/settings
  • 725 contracts and 1678 shared tests passed (the shared observability.memory test failed once under load and passes alone) (vp test run in each package). projectSettings.test.ts now checks that a shorter list replaces rather than merges.

Not verified: approving a roster card that holds an ineligible model by hand, auto-approve, j5_request_crew_member and delegate_task on a live server (unit tests cover them), and a project-scoped override. I didn't run repo-wide checks.

Merge Danger

Door: two-way. It adds a setting that's null until someone saves it, and no migration, so a revert undoes it. A saved list would just sit unused.

Blast Radius: every path that gives a custom worker a model: Crew launch and additions, j5_spawn_agent, and delegate_task, but only in a project where someone has saved a list. With nothing saved, nothing changes.

Things worth knowing:

  • Because I apply the list across the project, a persona seat on an unchecked model that calls delegate_task without a target is refused, since the child would inherit that model. The refusal lists the eligible models so it can pass one in target.
  • A Captain whose own model is off the list gets a roster card whose preview fails, and every runtime line disappears until a model is picked. I didn't add a check at propose time because that would change today's behavior.
  • No Settings-search entry for the new row, since settingsSearch.ts is upstream and I wanted exactly one upstream hook. It's a small follow-up if you want it.

After merge: nothing.

UI changes

The Agent Setup row opens as one card per provider, and the roster card's model picker for a custom member is filtered. Both are new, so there's no before.

Before After
Agent Setup → Crews → Models for custom members: the Codex card open, two of 21 models allowed new Agent Setup → Crews → Models for custom members: the Codex card open, two of 21 models allowed, after
The Claude card open: chips drop the Claude prefix, and the providers that aren't set up fold into one line new The Claude card open: chips drop the Claude prefix, and the providers that aren't set up fold into one line, after
A custom member's model picker offers only the allowed model (Haiku 5.5) new A custom member's model picker offers only the allowed model (Haiku 5.5), after

Live testing

Two live passes in isolated test homes from j5-live-test, both before the stack was last rebased onto j5/main. Nothing touched my real install.

The first ran on the whole stack at 1d4ff9bdf6, before I reworked the settings row. The second ran at 73869cd83e, for the row.

  • Row: I unchecked everything with Check all, then checked GPT-6 Luna and Claude Haiku 5.5. The row went to "2 of 21" and showed the scope indicator and reset. Opening a card collapsed the other one, and the Claude card's chips showed without the "Claude" prefix.
  • Picker: a Captain proposed a Crew with two persona seats, both on listed models. In Add member, a custom member's provider list was Codex and Claude, and its model list was only Haiku 5.5.
  • Refusal: I asked an agent in plain words to start a helper on Claude Opus 5.5 with no persona. It reported the spawn was refused, naming Opus 5.5, the project and the setting, and listing the two eligible models. I checked the stored state, not just its account: no thread exists on claude-opus-5-5.

Not exercised live: the cases listed under Not verified, and Cursor and the other providers (not set up on the test host). The Claude native-subagent control is still "unverified", as written above.

Upstream impact

All recorded in FORK.md case 107, plus register entry D42 in docs/j5/product/upstream.md:

  • packages/contracts/src/settings.ts: crewMemberModels added in the same four places as crewAutoApprove, plus the nullable-override list.
  • apps/server/src/mcp/OrchestratorMcpService.ts: an optional authorizeTarget hook in delegateTask, called right after the target resolves. About five lines. I approved this one in the issue discussion because J5's wrapper can't see the resolved model otherwise. With no hook, upstream behavior is unchanged.
  • apps/server/src/mcp/toolkits/worktree/registration.test.ts (one stub line) and packages/shared/src/projectSettings.test.ts (new cases).

Checklist

  • One concern: the description has no "also"
  • Tests cover the changed behavior (backend changes ship with focused tests)
  • UI changes: before/after screenshots above, and a video for motion or interaction — new states, so no before; no motion
  • Live testing: the section above says what ran live and what did not
  • Upstream-owned files: each one is recorded in FORK.md (case text and file-table row) in this PR
  • Upstream product: any change to what upstream's product does has a human decision linked above and a register entry in docs/j5/product/upstream.md — D42
  • Surfaces: entry points, clients, providers, contracts, reverse states, connection modes (see AGENTS.md) — the setting, picker and spawn refusal ran live on Claude; providers' native subagents are "not supported here"/unverified; the other paths are covered by tests
  • Docs: definitions under docs/j5/product/ and user docs rewritten where this changes them

Claude Sonnet 5.5 through the Claude Code harness in J5 Code.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
j5-code Ready Ready Preview Nov 2, 2026 10:50pm UTC

Request Review

@bryantderosier bryantderosier self-assigned this Oct 27, 2026
@bryantderosier
bryantderosier added this pull request to stack #753 October 27, 2026 18:15
@github-actions github-actions Bot added size:XL 500-999 effective changed lines (test files excluded in mixed PRs). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. labels Oct 27, 2026
@bryantderosier
bryantderosier force-pushed the j5/744-custom-member-models branch from 77ca1d6 to 5923fc3 Compare October 27, 2026 21:17
bryantderosier added a commit that referenced this pull request Oct 27, 2026
@bryantderosier
bryantderosier marked this pull request as ready for review October 27, 2026 22:36
@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

bryantderosier added a commit that referenced this pull request Oct 27, 2026
@github-actions github-actions Bot added size:XXL 1,000+ effective changed lines (test files excluded in mixed PRs). and removed size:XL 500-999 effective changed lines (test files excluded in mixed PRs). labels Oct 28, 2026
bryantderosier added a commit that referenced this pull request Oct 28, 2026
@bryantderosier bryantderosier changed the title feat(j5): choose which models custom Crew members can use feat(j5): choose which models and reasoning levels custom Crew members can use Oct 28, 2026
@bryantderosier
bryantderosier force-pushed the j5/744-custom-member-models branch from 4fa701e to bd50f86 Compare October 28, 2026 02:17
@Jacksondr5

Copy link
Copy Markdown
Owner

Panel review: #757

Live (Astra, stack tip): with the list restricted, a custom member on an excluded model was refused on all four paths: propose, j5_request_crew_member, delegate_task and j5_spawn_agent. Each refusal was actionable and stored, and a persona seat on the same model still launched (settings).

Process: FORK file table. Case 90 is written, but the file-table rows don't include it:

  • apps/server/src/mcp/OrchestratorMcpService.ts (FORK.md:778, still only "Saved-agent mentions")
  • mcp/toolkits/worktree/registration.test.ts (:779, "4, 45, 48")
  • packages/contracts/src/settings.ts (:883, "42, B, 88")

packages/shared/src/projectSettings.test.ts is upstream-owned and appears nowhere in FORK.md. D39's 2026-10-27 history line also sits above the 2026-10-20 and 2026-10-24 entries.

Docs nit: say the lists apply at creation only. t3_thread_configure, which agents can call, sets any thread's model after launch with only a thread-write check (mcp/toolkits/thread/handlers.ts:182-195). crews.md:20 already says "at launch and when added". Add a sentence there and in D39 that later model changes aren't limited, so "the platform enforces the list" isn't read as ongoing.

Scope. #744 asked for models only. The reasoning-level list adds:

  • crewMemberReasoning.ts and its tests
  • a second upstream settings key
  • the Reasoning chips
  • CrewMemberReasoningNotEligibleError
  • level filtering on the card

The panel would cut it or split it into its own PR. We've also raised it with Jackson.

Nit: a persona seat can be refused after its route changes. CrewLaunchService.ts:480-491 compares the frozen seat selection with the persona's current route. Suppose the persona's primary provider is signed out at proposal, so the card freezes the fallback, and the primary is back by approval. The seat is then treated as custom and refused, even though it runs the persona's own fallback.


Review panel for the #750–#762 stack, judged as the code that lands at da64b2795c: Claude Opus 5.5 (lead and three code reviewers), GPT-6.1-Sol (process, docs, CI) and GPT-6-Astra (live test on an isolated server, Haiku 5.5 and GPT-6 Luna turns). Every finding here was agreed by the whole panel. Product questions are with Jackson and are not listed. Posted by Claude Opus 5.5 via J5 Code.

@bryantderosier
bryantderosier removed this pull request from stack #753 October 28, 2026 11:45
@bryantderosier
bryantderosier force-pushed the j5/744-custom-member-models branch from bd50f86 to 423085c Compare October 28, 2026 11:45
@bryantderosier bryantderosier changed the title feat(j5): choose which models and reasoning levels custom Crew members can use feat(j5): choose which models custom Crew members can use Oct 28, 2026
bryantderosier and others added 9 commits October 29, 2026 08:18
Add a project-scoped crewMemberModels setting (checked provider/model pairs,
null until saved) and one J5 service that decides whether a resolved model may
run a custom worker. Crew launches and additions, a plain j5_spawn_agent and a
plain delegate_task call it; a persona that keeps its own model skips it. A
refusal names the model, the project and the setting and lists what is eligible.
delegate_task gets an optional authorizeTarget hook in upstream's delegateTask so
the check sees the model the child will really run on.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…up (#744)

A Models for custom members row in the Crews section of Agent Setup shows how
many catalog models are checked, opens in place, lists each set-up provider's
models with checkboxes, warns when a saved list leaves nothing eligible, and
notes that personas and native subagents are outside the list. The roster card
offers a custom member only the models its project allows.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…mits (#744)

Rewrite the Crews definition, the agent tool contracts for j5_propose_crew,
j5_request_crew_member, delegate_task and j5_spawn_agent, the persona follow-up
note and the user guide. Record per provider that native subagents sit outside
the list, and divergence D39 for the delegateTask hook. Each tool description
gains one sentence on the eligible models.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
#744)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
)

A flat checklist of every model was long and hard to scan. Each set-up
provider is now a card with its count, one open at a time, with models as
toggle chips, and providers that aren't set up fold into one line.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…#744)

Cursor keeps it for the Claude models it lists.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
#744)

A card that was drawn while the persona's primary provider was signed out freezes the fallback route into the seat. When the primary was back at approval the seat was compared with the primary alone and wrongly held to the models list. A persona seat now counts as unchanged when its provider and model match either of the persona's two routes; any other model is still checked.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lies at creation only (#744)

Add case 90 to the file-table rows of OrchestratorMcpService.ts, registration.test.ts and settings.ts, record projectSettings.test.ts, put D39's history line in date order, and say that a later model change, for example through t3_thread_configure, is not limited.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
bryantderosier and others added 5 commits November 2, 2026 09:18
…-models layer

The models decision moves to register D42 and FORK.md case 107, since
j5/main took D39 and case 90 for other decisions.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…-models layer

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
bryantderosier and others added 2 commits November 2, 2026 17:30
…ing exported

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@bryantderosier
bryantderosier merged commit 307a758 into j5/main Nov 3, 2026
35 checks passed
@bryantderosier
bryantderosier deleted the j5/744-custom-member-models branch November 3, 2026 01:30

This branch was successfully deployed

1 active deployment
Preview — 72d1c339 Deployed Nov 2, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL 500-999 effective changed lines (test files excluded in mixed PRs). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Crews: choose which models custom members can use

2 participants