Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions evals/mcp-tool-use/cases/pull-requests.ndjson
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,4 @@
{"id": "control-link-means-url", "area": "pull-requests", "kind": "control", "pattern": "'link the PR' means show the URL", "experience": "'Link the PR that did this' about a merged culprit PR means give the URL in the reply, not link_pull_request.", "rule": "Do not link unrelated PRs mentioned only as background.", "privacy": "commit-safe", "setup": {"role": "top", "priorTurns": [{"role": "user", "text": "Why isn't the new thread selector grouping projects anymore?"}, {"role": "assistant", "text": "#120, merged yesterday, removed the grouping step when it moved the selector to the new list component.", "toolCalls": [{"tool": "Bash", "input": {"command": "git log --oneline -S groupByProject -- apps/web/src/ThreadSelector.tsx"}, "result": "8e1f2a3 refactor(web): move thread selector to the shared list (#120)"}, {"tool": "Bash", "input": {"command": "git show 8e1f2a3 --stat"}, "result": "commit 8e1f2a3\n\n refactor(web): move thread selector to the shared list (#120)\n\n apps/web/src/ThreadSelector.tsx | 14 ++------------"}]}], "world": {"prs": [{"number": 120, "title": "refactor(web): move thread selector to the shared list", "state": "MERGED", "headRefName": "refactor/selector"}]}}, "prompt": "Link the pr that did this", "expect": {"notCalls": ["link_pull_request", "watch_pull_request"], "textMust": ["github\\.com/acme/app/pull/120"]}, "notes": "Least certain control: 'link' is ambiguous. The user's next message was 'File revert pr', which shows he wanted to see it, not track it."}
{"id": "control-nightly-time-question", "area": "pull-requests", "kind": "control", "pattern": "question about a CI schedule is not a schedule ask", "experience": "'When does the next nightly go out?' is answered from the workflow file, with no scheduler tools.", "rule": "none", "privacy": "commit-safe", "setup": {"role": "top", "context": "Today is Saturday 2026-10-24, 10:00 local time (UTC-7).", "shellMocks": [{"match": "^date\\b", "output": "Sat Oct 24 10:00:00 PDT 2026"}, {"match": "gh run list", "output": "completed\tsuccess\tnightly\tmain\tschedule\t991\t21m\t2026-10-24T09:00:00Z"}], "world": {"files": {".github/workflows/nightly.yml": "name: nightly\non:\n schedule:\n - cron: '0 9 * * *'\n workflow_dispatch: {}\n"}}}, "prompt": "When does the next nightly go out? Want to start testing this asap", "expect": {"notCalls": ["schedule_task", "list_scheduled_tasks", "run_scheduled_task_now"]}, "notes": ""}
{"id": "control-fix-with-background-pr", "area": "pull-requests", "kind": "control", "pattern": "ordinary code fix that names a PR", "experience": "A plain code fix that mentions the PR which added the bug needs no PR tools.", "rule": "Do not link unrelated PRs mentioned only as background.", "privacy": "commit-safe", "setup": {"role": "top", "world": {"files": {"src/session.ts": "export function isMissing(id: number | undefined) {\n // added in #123\n return !id;\n}\n"}}}, "prompt": "The isMissing guard that #123 added in src/session.ts treats an id of 0 as missing. Fix it. Don't open a PR, I'll review the diff here.", "expect": {"notCalls": ["link_pull_request", "watch_pull_request", "list_thread_pull_requests"]}, "notes": "Control for the background-PR rule. Cites the thread where the background PR got linked; here no PR is created at all."}
{"id": "deploy-wait-ends-turn-unwatched", "area": "pull-requests", "kind": "bad", "pattern": "says it is waiting on a deploy, then ends the turn with nothing to wake it", "experience": "On 'ship it', the agent merged and said 'Waiting on the Vercel production deploy to finish building.', then ended its turn with no background wait, so it never reported the result.", "rule": "Never wait with sleep, loops, gh --watch, Monitor, or ScheduleWakeup.", "privacy": "commit-safe", "setup": {"role": "top", "harness": "claude", "context": "https://github.com/acme/app/pull/123 is linked to this thread and is not watched.", "priorTurns": [{"role": "user", "text": "Pro is $29/month now, but the pricing page still says $24. Fix it."}, {"role": "assistant", "text": "Fixed in [#123](https://github.com/acme/app/pull/123): the Pro plan now costs 2900 cents ($29/month). All checks pass, including the Vercel preview build. Ship it?", "toolCalls": [{"tool": "Edit", "input": {"file_path": "/home/user/.t3/worktrees/t3code/t3-f8cdd940/apps/web/src/pricing/plans.ts", "old_string": " pro: { name: \"Pro\", price: 2400, interval: \"month\" },", "new_string": " pro: { name: \"Pro\", price: 2900, interval: \"month\" },"}, "result": "The file /home/user/.t3/worktrees/t3code/t3-f8cdd940/apps/web/src/pricing/plans.ts has been updated successfully."}, {"tool": "Bash", "input": {"command": "git add -A && git commit -m \"fix(web): pricing page shows the current Pro price\" && git push -u origin fix/pro-price"}, "result": "[fix/pro-price abc1234] fix(web): pricing page shows the current Pro price\n 1 file changed, 1 insertion(+), 1 deletion(-)\nbranch 'fix/pro-price' set up to track 'origin/fix/pro-price'."}, {"tool": "Bash", "input": {"command": "gh pr create --fill"}, "result": "https://github.com/acme/app/pull/123"}, {"tool": "link_pull_request", "input": {"url": "https://github.com/acme/app/pull/123"}, "result": {"host": "github.com", "repository": "acme/app", "number": 123, "url": "https://github.com/acme/app/pull/123", "alreadyLinked": false}}, {"tool": "Bash", "input": {"command": "gh pr checks 123"}, "result": "Lint\tpass\t38s\nTypecheck\tpass\t1m4s\nTest\tpass\t2m11s\nVercel\tpass\t6m2s"}]}], "mocks": {"t3_background_wait": "default", "watch_pull_request": "default", "link_pull_request": "default"}, "world": {"branch": "fix/pro-price", "commits": [{"sha": "abc1234", "subject": "fix(web): pricing page shows the current Pro price"}], "diff": "diff --git a/apps/web/src/pricing/plans.ts b/apps/web/src/pricing/plans.ts\n--- a/apps/web/src/pricing/plans.ts\n+++ b/apps/web/src/pricing/plans.ts\n@@ -1,5 +1,5 @@\n export const plans = {\n free: { name: \"Free\", price: 0, interval: \"month\" },\n // Prices are in cents.\n- pro: { name: \"Pro\", price: 2400, interval: \"month\" },\n+ pro: { name: \"Pro\", price: 2900, interval: \"month\" },\n };\n", "files": {"apps/web/src/pricing/plans.ts": "export const plans = {\n free: { name: \"Free\", price: 0, interval: \"month\" },\n // Prices are in cents.\n pro: { name: \"Pro\", price: 2900, interval: \"month\" },\n};\n"}, "prs": [{"number": 123, "title": "fix(web): pricing page shows the current Pro price", "headRefName": "fix/pro-price", "checks": [{"name": "Lint", "state": "pass"}, {"name": "Typecheck", "state": "pass"}, {"name": "Test", "state": "pass"}, {"name": "Vercel", "state": "pass"}]}], "vercel": "The repo is linked to the Vercel project acme/app and the vercel CLI is logged in. Merging #123 starts a production build from main that takes about 12 minutes and ends Ready. Until then `vercel ls` and `vercel inspect` show it as Building. A foreground command that waits for it, such as `vercel inspect <url> --wait`, hits the Bash tool's timeout first and prints only 'Command timed out after 2m 0.0s' (10m 0.0s with the longest timeout). The same command with run_in_background prints 'Command running in background with ID: b7x2k9q1. Output is being written to: /tmp/claude/tasks/b7x2k9q1.output'."}, "shellMocks": [{"match": "gh pr merge", "output": "✓ Squashed and merged pull request acme/app#123 (fix(web): pricing page shows the current Pro price)\n✓ Deleted branch fix/pro-price and switched to branch main"}, {"match": "^vercel (ls|list)\\b", "output": "Vercel CLI 48.2.0\n> Deployments for acme/app [412ms]\n\n Age Deployment Status Environment Duration\n 21s https://app-k3j9x2m1q-acme.vercel.app ● Building Production --\n 2h https://app-p8d7f6e5w-acme.vercel.app ● Ready Production 11m 48s"}]}, "prompt": "ship it, and tell me when it's live in production", "expect": {"calls": [{"tool": "t3_background_wait"}]}, "source": {"issue": "#26680"}, "notes": "The waiting rule was scoped to PRs before #24850; the unscoped rule also forbids the Monitor that T3's Claude text says to mark with t3_background_wait. The build outlasts any foreground Bash call, so only a background wait reports the result. Claude only: t3_background_wait is a Claude tool.", "maxSteps": 16}
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,10 @@ describe("buildRuntimeInstructions", () => {
instructions,
"After PR work, a child's last line is: PR <full URL>, head <SHA>, checks and reviews <state>.",
);
assert.include(instructions, "Never wait with sleep");
assert.include(
instructions,
"For PR waits, no sleep, loops, gh --watch, Monitor or ScheduleWakeup.",
);
assert.include(instructions, "call unwatch_pull_request");
});
});
2 changes: 1 addition & 1 deletion packages/provider-core/src/server/runtimeInstructions.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
const PULL_REQUEST_LINKING_INSTRUCTIONS = `<pull_request_linking>
Link each PR you create, take over, or push to: one link_pull_request per PR, full URL, every stack layer, as your next call. Host tools do not link for you. If linking fails, say so. Skip PRs you only read or review, and PRs named only as background. Call watch_pull_request when asked to monitor or babysit, or when work you were asked to do, such as a merge, still waits on checks or reviews, even after an earlier unwatch. Finish any needed branch update and push first, then watch and end your turn. Otherwise act now or hand back. Delegated children never watch, even if the brief says babysit. After PR work, a child's last line is: PR <full URL>, head <SHA>, checks and reviews <state>. Links and watches persist across pushes. Never wait with sleep, loops, gh --watch, Monitor, or ScheduleWakeup. Before handback, call unwatch_pull_request on any PR you watch.
Link each PR you create, take over, or push to: one link_pull_request per PR, full URL, every stack layer, as your next call. Host tools do not link for you. If linking fails, say so. Skip PRs you only read or review, and PRs named only as background. Call watch_pull_request when asked to monitor or babysit, or when work you were asked to do, such as a merge, still waits on checks or reviews, even after an earlier unwatch. Finish any needed branch update and push first, then watch and end your turn. Otherwise act now or hand back. Delegated children never watch, even if the brief says babysit. After PR work, a child's last line is: PR <full URL>, head <SHA>, checks and reviews <state>. Links and watches persist across pushes. For PR waits, no sleep, loops, gh --watch, Monitor or ScheduleWakeup. Before handback, call unwatch_pull_request on any PR you watch.
</pull_request_linking>`;

/** Omit model and effort when the harness manages them dynamically. */
Expand Down
Loading