Repository navigation
feat(j5): reviewers prefer a model that didn't write the work - #932
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
This review includes 38 billable files and costs up to $9.50. Or wait 46 seconds for your next included review. View limit details
|
tyler-barton-horizon
left a comment
There was a problem hiding this comment.
Automated review of this layer of stack pingdotgg#933 (diff against its base branch). Findings were not individually verified against the source, so please treat them as likely issues and push back on any that are wrong.
🤖 Generated with Claude Code
| threadIds.push(next.threadId); | ||
| for (const model of threadModels(records.thread, records.runs)) models.add(model); | ||
| for (const child of records.subagents) { | ||
| if (child.origin === "provider_native") partlyUnknown = true; |
There was a problem hiding this comment.
Any provider-native child of an author thread sets partlyUnknown, and the default solo authors are every delegate_task child the caller ever ran.
A Claude orchestrator used the native Task tool once for exploration and then delegates to a reviewer. The reviewer lands on a different model, but the outcome is unknown-authors and the brief says it's not a verified independent review. A GPT child delegated hours earlier for unrelated research also keeps GPT in avoidModels for every later review from that thread.
There was a problem hiding this comment.
That's how we decided it, on purpose. A native subagent's model isn't recorded anywhere, so if one did any of the work we'd rather say 'not a verified independent review' than claim independence we can't check. Same for past delegate children: we can't tell which ones touched the work under review, so they all count by default. You can narrow it with authorThreads/author_threads when you know who wrote it.
| const isReviewer = (name: string) => { | ||
| const agentId = seats.find((other) => other.seat === name)?.agentId; | ||
| return catalog.definitions.some( | ||
| ({ id, independentReviewer }) => id === agentId && independentReviewer === true, |
There was a problem hiding this comment.
The rule against naming another independent reviewer in reviews only checks seats on the same roster.
j5_request_crew_member adds a reviewer with reviews: ["code-review"], where code-review is an existing reviewer member. Validation passes, and because skipReviewer is false for explicit names, that reviewer's model goes into avoidModels, the opposite of what the roster rule enforces.
There was a problem hiding this comment.
This one's intentional. The same-roster refusal isn't a ban on reviewing a reviewer; it's because a reviewer seat's model isn't picked until after the seats it reviews, so we can't see it yet. An existing member's model is right there on its thread, so if the Captain explicitly names a reviewer member (say it pushed fixes), avoiding its model is what was asked for. The default still skips reviewers. I added a line to crews.md saying an existing reviewer member can be named (4a64763).
…s and foreign threads From Tyler's review of #932: - a reviewer whose reviews names only seats renamed or removed on the roster card falls back to the default authors instead of reviewing nobody; - a model set on the roster card is described as "set on the roster card", not "chosen explicitly"; - an author's model is matched ignoring a provider prefix, so anthropic/claude-opus-5-5 counts as claude-opus-5-5; - a pick reads the persona catalog once and shares it with the launch, and a Crew passes authors only for reviewer seats; - each level of the author tree is read in parallel; - a disabled or unknown persona fails before any author thread is read, and a named author thread from another project is refused; - a failed library read is a launch error, not the first seat's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4a64763 to
2f2c9cf
Compare
…s and foreign threads From Tyler's review of #932: - a reviewer whose reviews names only seats renamed or removed on the roster card falls back to the default authors instead of reviewing nobody; - a model set on the roster card is described as "set on the roster card", not "chosen explicitly"; - an author's model is matched ignoring a provider prefix, so anthropic/claude-opus-5-5 counts as claude-opus-5-5; - a pick reads the persona catalog once and shares it with the launch, and a Crew passes authors only for reviewer seats; - each level of the author tree is read in parallel; - a disabled or unknown persona fails before any author thread is read, and a named author thread from another project is refused; - a failed library read is a launch error, not the first seat's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2f2c9cf to
82025de
Compare
…s and foreign threads From Tyler's review of #932: - a reviewer whose reviews names only seats renamed or removed on the roster card falls back to the default authors instead of reviewing nobody; - a model set on the roster card is described as "set on the roster card", not "chosen explicitly"; - an author's model is matched ignoring a provider prefix, so anthropic/claude-opus-5-5 counts as claude-opus-5-5; - a pick reads the persona catalog once and shares it with the launch, and a Crew passes authors only for reviewer seats; - each level of the author tree is read in parallel; - a disabled or unknown persona fails before any author thread is read, and a named author thread from another project is refused; - a failed library read is a launch error, not the first seat's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A persona can be flagged as an independent reviewer in Agent Setup (web, desktop and mobile). When the server chooses its model, entries on a model the authors ran on move to the end of its list, so a different model wins whenever one can run; if only an author's model can, the reviewer launches on it anyway. It's a preference and never blocks or escalates a launch. Authors default to the caller's thread and its delegate_task children, or, in a Crew, every other seat and member that isn't a reviewer. Launches can name them: author_threads on j5_spawn_agent, authorThreads on delegate_task, and reviews on Crew seats and additions. A provider-native subagent makes the authors partly unknown. A model a person set on the roster card, or an agent pinned on j5_spawn_agent, is kept and noted. The assignment records the outcome in reviewIndependence and the reviewer's first turn carries one line saying the same. With a persona, j5_spawn_agent's provider and model are now optional, and j5_list_personas reports independent_reviewer. Unflagged personas are unaffected. Closes #782 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
saveSeat rebuilt an edited seat from the draft and carried only the Captain's reason, persona swaps and steps, so an edited reviewer silently fell back to reviewing every non-reviewer seat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Definitions don't narrate who decided what; the History section already records #782. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewer seats resolve after every other seat, so a reviews entry naming another reviewer on the same roster could never see that seat's model and was recorded as unknown authors. The Captain's proposal or request is now refused with the reason instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s and foreign threads From Tyler's review of #932: - a reviewer whose reviews names only seats renamed or removed on the roster card falls back to the default authors instead of reviewing nobody; - a model set on the roster card is described as "set on the roster card", not "chosen explicitly"; - an author's model is matched ignoring a provider prefix, so anthropic/claude-opus-5-5 counts as claude-opus-5-5; - a pick reads the persona catalog once and shares it with the launch, and a Crew passes authors only for reviewer seats; - each level of the author tree is read in parallel; - a disabled or unknown persona fails before any author thread is read, and a named author thread from another project is refused; - a failed library read is a launch error, not the first seat's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
82025de to
c0d28ce
Compare
Summary
A second review is worth a lot more on a different model, because the same model tends to miss what it missed the first time. Nothing steered a reviewer away from the author's model, though. Now a persona can be flagged as an independent reviewer: when the server picks its model, it prefers one the authors didn't run on. It's a preference, never a block. Closes #782.
independentReviewerin the definition. Personas without it are untouched: no reordering, no record, no note. A definition without the key keeps its digest, and an edit that leaves the field out keeps the stored flag, so an older client can't clear it.j5_list_personasreportsindependent_reviewer.delegate_taskchildren. Earlier reviewer children are skipped, and a provider-native child makes the authors partly unknown.author_threadsonj5_spawn_agent,authorThreadsondelegate_task, andreviewson Crew seats and additions.reviewsname that isn't on the Crew, or areviewsname for another reviewer seat on the same roster is refused as bad input. Reviewers resolve after the other seats, so one reviewer can't see another's model.@personamention and a roster card approved unchanged. A model a person set on the card, or one an agent pinned onj5_spawn_agent, is kept and noted.reviewIndependence: the outcome (different-model,same-model-as-authororunknown-authors), the author models and threads,partlyUnknown, and who chose the model. The reviewer's first turn gets one matching line, for example "Review independence: you run on the same model as the author (claude-opus-5-5), because no other model in your list could run." It's decided once, at creation.j5_spawn_agentwith a persona no longer needs provider, model or reasoning. Left out, the server picks from the persona's list, the same waydelegate_taskand Crews already do.The model-facing text changed for
j5_spawn_agent,delegate_task,j5_propose_crew,j5_request_crew_member,j5_list_personas, the@personainvocation guidance and the Crew guidance. The canonical descriptions indocs/j5/product/a2a/agent-tools.mdare updated to match.UI changes
Before is
feat/persona-pick-limitsand after is this branch, both on a fresh dev server in the web app. The edit dialog uses the same fields.Evidence
agentInvocation.test.ts, "independent reviewer subagents":delegate_taskchild's, so the third wins, and an earlier reviewer child isn't counted;@personapick is still reordered, and a native child reads as unknown;authorThreadsreplace the default, and a missing one isinvalid_request;handlers.test.ts(j5_spawn_agent):unknown-authors;CrewLaunchService.test.ts:reviewsnarrows the authors;CrewProposalService.test.ts: an unknownreviewsname is refused, and so is one that names another reviewer seat on the roster.crewProposalDraft.test.ts(web): editing a reviewer seat on the roster card keeps itsreviews. Without the fix it fails, because the edit dropped it.agentPersonaRouting.test.ts:anthropic/claude-opus-5-5counts as the author'sclaude-opus-5-5, whilegpt-6-solandgpt-6.1-solstay distinct.CrewLaunchService.test.ts:reviewsnaming only a renamed seat falls back to the default authors, and a card-set model's note says "set on the roster card".agentInvocation.test.ts: a named author thread from another project is refused, and a disabled reviewer reports that before any author thread.agentPersonaLibrary.test.ts: an old definition keeps its digest, an edit without the field keeps the flag, and clearing it removes the key.vp test run src/j5/agents/agentPersonaRouting.test.ts src/j5/agents/agentInvocation.test.ts src/j5/a2a/CrewLaunchService.test.ts src/j5/a2a/CrewProposalService.test.ts src/j5/a2a/mcp/handlers.test.ts src/j5/a2a/mcp/tools.test.ts src/j5/agents/agentPersonaLibrary.test.ts src/j5/agents/agentSubagent.integration.test.ts src/j5/agents/agentPersonaLaunch.integration.test.ts src/j5/agents/agentDelegation.test.ts(fromapps/server)vp run check:changed: format, lint, knip, typecheck (32 packages) and tests (19 files) all pass.mcp-tool-useon Sonnet 5.5 for the changed@personaand Crew guidance: 32 of 33 Claude orchestration cases pass, 65 of 68 samples (96%), for $4.07.node --env-file-if-exists=../.env mcp-tool-use/run.ts --model claude-sonnet-5-5 --cases mcp-tool-use/cases/orchestration.jsonl --out runs/782-sonnet(fromevals/)opus-review-too-via-delegate(1 of 3), fails on the base branches too, so it isn't from this change. Onj5/main(a216acc) it passed 1 of 3, and onfeat/persona-pick-limits(a2fc1c2) 0 of 3. Those two re-runs cost $0.56 and $0.58, using the same command with--case opus-review-too-via-delegate,thread-mention-without-link-after-read --samples 3.thread-mention-without-link-after-readpassed 2 of 3 here and 3 of 3 on both bases. It still counts as passing, and the sentences I added say nothing about thread links.Not verified:
mcp-tool-useeval doesn't load J5's own tool schemas, so nothing checks that an agent leaves the model out for a reviewer onj5_spawn_agent.Merge Danger
Door: two-way.
independentReviewer,reviewIndependenceandreviewsare optional fields, so stored definitions, assignments and proposals still decode after a revert.Blast Radius: launches of flagged personas only. If it's wrong, a reviewer lands on a less preferred model or its brief says the wrong thing. No launch is refused because of independence. The one new refusal is a named author thread or
reviewsname that doesn't exist.After merge: once a release with this is out, set the flag on the team's reviewer personas (the #780 follow-up). #784 builds seat replacement on
reviewsand the outcome.Checklist
FORK.md(case text and file-table row) in this PR (none edited; every path is J5-owned)docs/j5/product/upstream.md(none)AGENTS.md)docs/j5/product/and user docs rewritten where this changes themSurfaces walked:
delegate_task,@persona,j5_spawn_agent(thread and outside callers), roster card, auto roster, Captain addition. A composer persona launch has no authors and starts on your own text, so it's left alone.reviewsthrough edits. Mobile has no roster card.Made with Claude Opus 5.5 in Claude Code (J5 Code).
🤖 Generated with Claude Code