Skip to content

feat(j5): reviewers prefer a model that didn't write the work - #932

Merged
Jacksondr5 merged 5 commits into
feat/persona-pick-limitsfrom
feat/persona-reviewer-independence
Nov 8, 2026
Merged

Jacksondr5 merged 5 commits into
feat/persona-pick-limitsfrom
feat/persona-reviewer-independence

Conversation

@bryantderosier

@bryantderosier bryantderosier commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

A second review is worth a lot more on a different model, because the same model tends to miss what it missed the first time. Nothing steered a reviewer away from the author's model, though. Now a persona can be flagged as an independent reviewer: when the server picks its model, it prefers one the authors didn't run on. It's a preference, never a block. Closes #782.

  • The flag. There's an Independent reviewer toggle in Agent Setup create and edit on web, desktop and mobile, and an optional independentReviewer in the definition. Personas without it are untouched: no reordering, no record, no note. A definition without the key keeps its digest, and an edit that leaves the field out keeps the stored flag, so an older client can't clear it. j5_list_personas reports independent_reviewer.
  • Authors.
    • Solo: the caller's thread plus its delegate_task children. Earlier reviewer children are skipped, and a provider-native child makes the authors partly unknown.
    • In a Crew: every other seat and member that isn't a reviewer. Reviewer seats resolve after the rest of the roster, so a same-roster builder's model is known.
    • You can name them explicitly: author_threads on j5_spawn_agent, authorThreads on delegate_task, and reviews on Crew seats and additions.
    • An author thread that doesn't exist, a reviews name that isn't on the Crew, or a reviews name for another reviewer seat on the same roster is refused as bad input. Reviewers resolve after the other seats, so one reviewer can't see another's model.
  • Selection. Entries on an author's model, matched by model id alone, move to the end of the list before Personas: agents pick personas, and who picked sets the limits #781's and Personas: an ordered model list #778's checks run. A different model wins whenever one passes. If only an author's model can run, the reviewer launches on it anyway. This applies whenever the server picks the model, including an @persona mention and a roster card approved unchanged. A model a person set on the card, or one an agent pinned on j5_spawn_agent, is kept and noted.
  • Record and brief. The assignment gets reviewIndependence: the outcome (different-model, same-model-as-author or unknown-authors), the author models and threads, partlyUnknown, and who chose the model. The reviewer's first turn gets one matching line, for example "Review independence: you run on the same model as the author (claude-opus-5-5), because no other model in your list could run." It's decided once, at creation.
  • j5_spawn_agent with a persona no longer needs provider, model or reasoning. Left out, the server picks from the persona's list, the same way delegate_task and Crews already do.
pickAgentPersona(reviewAuthors) ── flagged? ── resolveReviewAuthors (threads + delegate_task children)
                                             └─ resolveAgentPersonaRoute(avoidModels): non-author entries first
                                └─ assignment.reviewIndependence ── withReviewIndependenceNote → first turn
CrewLaunchService.resolveSeats ── non-reviewer seats first, then reviewers (roster models + member threads)

The model-facing text changed for j5_spawn_agent, delegate_task, j5_propose_crew, j5_request_crew_member, j5_list_personas, the @persona invocation guidance and the Crew guidance. The canonical descriptions in docs/j5/product/a2a/agent-tools.md are updated to match.

UI changes

Before After
Agent Setup → Create persona, with Independent reviewer switched on Agent Setup → Create persona, with Independent reviewer switched on, before Agent Setup → Create persona, with Independent reviewer switched on, after

Before is feat/persona-pick-limits and after is this branch, both on a fresh dev server in the web app. The edit dialog uses the same fields.

Evidence

  • New tests:
    • agentInvocation.test.ts, "independent reviewer subagents":
      • three entries with the first on the caller's model and the second on a delegate_task child's, so the third wins, and an earlier reviewer child isn't counted;
      • falls back to the author's model with the note;
      • a person's @persona pick is still reordered, and a native child reads as unknown;
      • named authorThreads replace the default, and a missing one is invalid_request;
      • an unflagged persona is unchanged.
    • handlers.test.ts (j5_spawn_agent):
      • an unmodeled reviewer avoids the caller's model;
      • a pin on the caller's model is kept and noted "because it was chosen explicitly";
      • an outside agent naming no threads gets unknown-authors;
      • a plain spawn without a model is refused.
    • CrewLaunchService.test.ts:
      • a reviewer listed first still avoids the builder seats' models;
      • reviews narrows the authors;
      • a person's card model is kept and noted;
      • an addition reviews member threads;
      • the seat's brief carries the line.
    • CrewProposalService.test.ts: an unknown reviews name is refused, and so is one that names another reviewer seat on the roster.
    • crewProposalDraft.test.ts (web): editing a reviewer seat on the roster card keeps its reviews. Without the fix it fails, because the edit dropped it.
    • From Tyler's review (4a64763):
      • agentPersonaRouting.test.ts: anthropic/claude-opus-5-5 counts as the author's claude-opus-5-5, while gpt-6-sol and gpt-6.1-sol stay distinct.
      • CrewLaunchService.test.ts: reviews naming only a renamed seat falls back to the default authors, and a card-set model's note says "set on the roster card".
      • agentInvocation.test.ts: a named author thread from another project is refused, and a disabled reviewer reports that before any author thread.
    • agentPersonaLibrary.test.ts: an old definition keeps its digest, an edit without the field keeps the flag, and clearing it removes the key.
  • The touched server suites: 253 passed.
    vp test run src/j5/agents/agentPersonaRouting.test.ts src/j5/agents/agentInvocation.test.ts src/j5/a2a/CrewLaunchService.test.ts src/j5/a2a/CrewProposalService.test.ts src/j5/a2a/mcp/handlers.test.ts src/j5/a2a/mcp/tools.test.ts src/j5/agents/agentPersonaLibrary.test.ts src/j5/agents/agentSubagent.integration.test.ts src/j5/agents/agentPersonaLaunch.integration.test.ts src/j5/agents/agentDelegation.test.ts (from apps/server)
  • vp run check:changed: format, lint, knip, typecheck (32 packages) and tests (19 files) all pass.
  • mcp-tool-use on Sonnet 5.5 for the changed @persona and Crew guidance: 32 of 33 Claude orchestration cases pass, 65 of 68 samples (96%), for $4.07.
    node --env-file-if-exists=../.env mcp-tool-use/run.ts --model claude-sonnet-5-5 --cases mcp-tool-use/cases/orchestration.jsonl --out runs/782-sonnet (from evals/)
    • The case that failed, opus-review-too-via-delegate (1 of 3), fails on the base branches too, so it isn't from this change. On j5/main (a216acc) it passed 1 of 3, and on feat/persona-pick-limits (a2fc1c2) 0 of 3. Those two re-runs cost $0.56 and $0.58, using the same command with --case opus-review-too-via-delegate,thread-mention-without-link-after-read --samples 3.
    • thread-mention-without-link-after-read passed 2 of 3 here and 3 of 3 on both bases. It still counts as passing, and the sentences I added say nothing about thread links.

Not verified:

  • No live run on a real server with real providers.
  • The mobile toggle wasn't exercised on a device; mobile typechecks. Web is captured above.
  • The evals ran only the Claude orchestration cases on Sonnet, not Codex models or Opus. The mcp-tool-use eval doesn't load J5's own tool schemas, so nothing checks that an agent leaves the model out for a reviewer on j5_spawn_agent.

Merge Danger

Door: two-way. independentReviewer, reviewIndependence and reviews are optional fields, so stored definitions, assignments and proposals still decode after a revert.

Blast Radius: launches of flagged personas only. If it's wrong, a reviewer lands on a less preferred model or its brief says the wrong thing. No launch is refused because of independence. The one new refusal is a named author thread or reviews name that doesn't exist.

After merge: once a release with this is out, set the flag on the team's reviewer personas (the #780 follow-up). #784 builds seat replacement on reviews and the outcome.

Checklist

  • One concern: the description has no "also"
  • Tests cover the changed behavior (backend changes ship with focused tests)
  • UI changes: before/after screenshots above, and a video for motion or interaction (web create dialog; a static switch needs no video; mobile not captured)
  • Live testing: the section above says what ran live and what did not (not live-tested; see Not verified)
  • Upstream-owned files: each one is recorded in FORK.md (case text and file-table row) in this PR (none edited; every path is J5-owned)
  • Upstream product: any change to what upstream's product does has a human decision linked above and a register entry in docs/j5/product/upstream.md (none)
  • Surfaces: entry points, clients, providers, contracts, reverse states, connection modes (see AGENTS.md)
  • Docs: definitions under docs/j5/product/ and user docs rewritten where this changes them

Surfaces walked:

  • Entry points: delegate_task, @persona, j5_spawn_agent (thread and outside callers), roster card, auto roster, Captain addition. A composer persona launch has no authors and starts on your own text, so it's left alone.
  • Clients: the toggle is on web (desktop wraps it) and mobile. The roster card carries reviews through edits. Mobile has no roster card.
  • Providers: selection is provider-agnostic, and model ids are compared across providers.
  • Contracts: all the new fields are optional.
  • Reverse states: the flag toggles off, which removes the key.
  • Connection modes: server-side only.

Made with Claude Opus 5.5 in Claude Code (J5 Code).

🤖 Generated with Claude Code

@vercel

vercel Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
j5-code Ready Ready Preview Nov 7, 2026 11:19pm UTC

Request Review

@bryantderosier bryantderosier added the enhancement New feature or request label Nov 6, 2026
@bryantderosier bryantderosier self-assigned this Nov 6, 2026
@github-actions github-actions Bot added the vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. label Nov 6, 2026
@bryantderosier
bryantderosier added this pull request to stack #933 November 6, 2026 14:49
@github-actions github-actions Bot added the size:XL 500-999 effective changed lines (test files excluded in mixed PRs). label Nov 6, 2026
@bryantderosier
bryantderosier marked this pull request as ready for review November 6, 2026 15:27
@coderabbitai

coderabbitai Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

  • Run on-demand review

This review includes 38 billable files and costs up to $9.50.

Or wait 46 seconds for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between cffd7b9 and c0d28ce.

Files selected for processing: 38

  • apps/mobile/src/j5/agents/AgentCreateScreen.tsx
  • apps/mobile/src/j5/agents/AgentDefinitionFields.tsx
  • apps/server/src/j5/a2a/AgentCrewProposalService.ts
  • apps/server/src/j5/a2a/CrewLaunchService.test.ts
  • apps/server/src/j5/a2a/CrewLaunchService.ts
  • apps/server/src/j5/a2a/CrewProposalService.test.ts
  • apps/server/src/j5/a2a/CrewProposalService.ts
  • apps/server/src/j5/a2a/SpawnAgentService.ts
  • apps/server/src/j5/a2a/mcp/handlers.test.ts
  • apps/server/src/j5/a2a/mcp/handlers.ts
  • apps/server/src/j5/a2a/mcp/tools.test.ts
  • apps/server/src/j5/a2a/mcp/tools.ts
  • apps/server/src/j5/agents/agentDelegation.ts
  • apps/server/src/j5/agents/agentInvocation.test.ts
  • apps/server/src/j5/agents/agentInvocation.ts
  • apps/server/src/j5/agents/agentPersonaLaunch.ts
  • apps/server/src/j5/agents/agentPersonaLibrary.test.ts
  • apps/server/src/j5/agents/agentPersonaLibrary.ts
  • apps/server/src/j5/agents/agentPersonaPick.ts
  • apps/server/src/j5/agents/agentPersonaReviewIndependence.ts
  • apps/server/src/j5/agents/agentPersonaRouting.test.ts
  • apps/server/src/j5/agents/agentPersonaRouting.ts
  • apps/server/src/j5/agents/agentPersonas.ts
  • apps/web/src/j5/agents/AgentCreateDialog.tsx
  • apps/web/src/j5/agents/AgentDefinitionFields.tsx
  • apps/web/src/j5/crew/crewProposalDraft.test.ts
  • apps/web/src/j5/crew/crewProposalDraft.ts
  • apps/web/src/j5/crew/crewSeatRuntime.ts
  • docs/j5/product/a2a/agent-tools.md
  • docs/j5/product/agent-personas/index.md
  • docs/j5/product/features/crews.md
  • docs/j5/user/personas.md
  • packages/client-runtime/src/j5/agentPersonas.test.ts
  • packages/client-runtime/src/j5/agentPersonas.ts
  • packages/contracts/src/j5.ts
  • packages/contracts/src/j5/agentPersona.ts
  • packages/provider-core/src/j5/agentInvocationInstructions.ts
  • packages/provider-core/src/j5/orchestrationInstructions.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@tyler-barton-horizon tyler-barton-horizon left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review of this layer of stack pingdotgg#933 (diff against its base branch). Findings were not individually verified against the source, so please treat them as likely issues and push back on any that are wrong.

🤖 Generated with Claude Code

threadIds.push(next.threadId);
for (const model of threadModels(records.thread, records.runs)) models.add(model);
for (const child of records.subagents) {
if (child.origin === "provider_native") partlyUnknown = true;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any provider-native child of an author thread sets partlyUnknown, and the default solo authors are every delegate_task child the caller ever ran.

A Claude orchestrator used the native Task tool once for exploration and then delegates to a reviewer. The reviewer lands on a different model, but the outcome is unknown-authors and the brief says it's not a verified independent review. A GPT child delegated hours earlier for unrelated research also keeps GPT in avoidModels for every later review from that thread.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's how we decided it, on purpose. A native subagent's model isn't recorded anywhere, so if one did any of the work we'd rather say 'not a verified independent review' than claim independence we can't check. Same for past delegate children: we can't tell which ones touched the work under review, so they all count by default. You can narrow it with authorThreads/author_threads when you know who wrote it.

const isReviewer = (name: string) => {
const agentId = seats.find((other) => other.seat === name)?.agentId;
return catalog.definitions.some(
({ id, independentReviewer }) => id === agentId && independentReviewer === true,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rule against naming another independent reviewer in reviews only checks seats on the same roster.

j5_request_crew_member adds a reviewer with reviews: ["code-review"], where code-review is an existing reviewer member. Validation passes, and because skipReviewer is false for explicit names, that reviewer's model goes into avoidModels, the opposite of what the roster rule enforces.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one's intentional. The same-roster refusal isn't a ban on reviewing a reviewer; it's because a reviewer seat's model isn't picked until after the seats it reviews, so we can't see it yet. An existing member's model is right there on its thread, so if the Captain explicitly names a reviewer member (say it pushed fixes), avoiding its model is what was asked for. The default still skips reviewers. I added a line to crews.md saying an existing reviewer member can be named (4a64763).

bryantderosier added a commit that referenced this pull request Nov 6, 2026
…s and foreign threads

From Tyler's review of #932:
- a reviewer whose reviews names only seats renamed or removed on the
  roster card falls back to the default authors instead of reviewing nobody;
- a model set on the roster card is described as "set on the roster card",
  not "chosen explicitly";
- an author's model is matched ignoring a provider prefix, so
  anthropic/claude-opus-5-5 counts as claude-opus-5-5;
- a pick reads the persona catalog once and shares it with the launch, and a
  Crew passes authors only for reviewer seats;
- each level of the author tree is read in parallel;
- a disabled or unknown persona fails before any author thread is read, and a
  named author thread from another project is refused;
- a failed library read is a launch error, not the first seat's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bryantderosier

Copy link
Copy Markdown
Collaborator Author

Restacked onto #931's final head (0742a8b); no changes beyond resolving #931's refactors.

@bryantderosier
bryantderosier force-pushed the feat/persona-reviewer-independence branch from 4a64763 to 2f2c9cf Compare November 6, 2026 23:17
bryantderosier added a commit that referenced this pull request Nov 6, 2026
…s and foreign threads

From Tyler's review of #932:
- a reviewer whose reviews names only seats renamed or removed on the
  roster card falls back to the default authors instead of reviewing nobody;
- a model set on the roster card is described as "set on the roster card",
  not "chosen explicitly";
- an author's model is matched ignoring a provider prefix, so
  anthropic/claude-opus-5-5 counts as claude-opus-5-5;
- a pick reads the persona catalog once and shares it with the launch, and a
  Crew passes authors only for reviewer seats;
- each level of the author tree is read in parallel;
- a disabled or unknown persona fails before any author thread is read, and a
  named author thread from another project is refused;
- a failed library read is a launch error, not the first seat's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bryantderosier

Copy link
Copy Markdown
Collaborator Author

Restacked onto #931's head (2064c40) for its code-block mention fix; no other changes.

@bryantderosier
bryantderosier force-pushed the feat/persona-reviewer-independence branch from 2f2c9cf to 82025de Compare November 7, 2026 00:17
bryantderosier added a commit that referenced this pull request Nov 7, 2026
…s and foreign threads

From Tyler's review of #932:
- a reviewer whose reviews names only seats renamed or removed on the
  roster card falls back to the default authors instead of reviewing nobody;
- a model set on the roster card is described as "set on the roster card",
  not "chosen explicitly";
- an author's model is matched ignoring a provider prefix, so
  anthropic/claude-opus-5-5 counts as claude-opus-5-5;
- a pick reads the persona catalog once and shares it with the launch, and a
  Crew passes authors only for reviewer seats;
- each level of the author tree is read in parallel;
- a disabled or unknown persona fails before any author thread is read, and a
  named author thread from another project is refused;
- a failed library read is a launch error, not the first seat's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bryantderosier and others added 2 commits November 7, 2026 17:56
A persona can be flagged as an independent reviewer in Agent Setup (web,
desktop and mobile). When the server chooses its model, entries on a model
the authors ran on move to the end of its list, so a different model wins
whenever one can run; if only an author's model can, the reviewer launches
on it anyway. It's a preference and never blocks or escalates a launch.

Authors default to the caller's thread and its delegate_task children, or,
in a Crew, every other seat and member that isn't a reviewer. Launches can
name them: author_threads on j5_spawn_agent, authorThreads on
delegate_task, and reviews on Crew seats and additions. A provider-native
subagent makes the authors partly unknown. A model a person set on the
roster card, or an agent pinned on j5_spawn_agent, is kept and noted.

The assignment records the outcome in reviewIndependence and the
reviewer's first turn carries one line saying the same. With a persona,
j5_spawn_agent's provider and model are now optional, and
j5_list_personas reports independent_reviewer. Unflagged personas are
unaffected.

Closes #782

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
saveSeat rebuilt an edited seat from the draft and carried only the Captain's reason, persona swaps and steps, so an edited reviewer silently fell back to reviewing every non-reviewer seat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bryantderosier and others added 3 commits November 7, 2026 17:56
Definitions don't narrate who decided what; the History section already records #782.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewer seats resolve after every other seat, so a reviews entry naming another reviewer on the same roster could never see that seat's model and was recorded as unknown authors. The Captain's proposal or request is now refused with the reason instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s and foreign threads

From Tyler's review of #932:
- a reviewer whose reviews names only seats renamed or removed on the
  roster card falls back to the default authors instead of reviewing nobody;
- a model set on the roster card is described as "set on the roster card",
  not "chosen explicitly";
- an author's model is matched ignoring a provider prefix, so
  anthropic/claude-opus-5-5 counts as claude-opus-5-5;
- a pick reads the persona catalog once and shares it with the launch, and a
  Crew passes authors only for reviewer seats;
- each level of the author tree is read in parallel;
- a disabled or unknown persona fails before any author thread is read, and a
  named author thread from another project is refused;
- a failed library read is a launch error, not the first seat's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@Jacksondr5 Jacksondr5 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved as part of the persona stack (#931 → #938), reviewed and live-tested as one unit. Details on #938.

@Jacksondr5
Jacksondr5 merged commit 80e0677 into j5/main Nov 8, 2026
56 of 68 checks passed
@Jacksondr5
Jacksondr5 deleted the feat/persona-reviewer-independence branch November 8, 2026 02:15

This branch was successfully deployed

1 active deployment
Preview — c0d28ce5 Deployed Nov 7, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:XL 500-999 effective changed lines (test files excluded in mixed PRs). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Personas: reviewers prefer a different model from the author

3 participants