From 0a31ef6dae6c37ccb169bd5c0d0fe53851fbe798 Mon Sep 17 00:00:00 2001 From: Etan Joseph Heyman Date: Sun, 1 Nov 2026 03:25:35 +0200 Subject: [PATCH 1/4] docs(evals): full eval runs come after approval, on the change only The Cost section asked for "one full run per model before the PR merges" without saying when or on which tree. With the "compare against a baseline" rule, agents read it as a full suite on both main and the PR before asking for review, even when a maintainer asked only for targeted cases. Targeted cases on both sides are now the evidence for review. The full run per model comes after approval, on the change only. Co-Authored-By: Claude Opus 5.5 --- evals/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evals/README.md b/evals/README.md index 1aa5796b00378..2de5c4bfda917 100644 --- a/evals/README.md +++ b/evals/README.md @@ -44,7 +44,7 @@ Every eval follows the same shape, so a new one is easy to write and an old one ## Cost -Run only the cases your change touches until they pass. Then do one full run per model before the PR merges. Each eval's summary records what the run cost. +Run only the cases your change touches until they pass, then run them and their controls once on a same-model baseline. That is the evidence a PR needs for review or reconsideration. After a maintainer approves the PR, do one full run per model on your change before it merges. A full baseline run is needed only when every case changes, such as a template refresh. Each eval's summary records what the run cost. ## Add an eval From ecf366115f17a8f270184b3095461420d5829a9d Mon Sep 17 00:00:00 2001 From: EtanHey Date: Thu, 12 Nov 2026 09:09:59 +0200 Subject: [PATCH 2/4] docs(evals): cut the eval-run rule from 74 words to 49 Same rule, fewer tokens. Re-ran the planning repro on Claude Code and Codex subscriptions: with the 49-word text no sample planned an unneeded full run (Opus 0/8, Sol 0/5; main's text 8/8 and 4/5), and after approval none ran main in full (main's text 5/8 and 5/5). Co-Authored-By: Claude Fable 5.2 --- evals/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evals/README.md b/evals/README.md index 2de5c4bfda917..883bbc4481f3d 100644 --- a/evals/README.md +++ b/evals/README.md @@ -44,7 +44,7 @@ Every eval follows the same shape, so a new one is easy to write and an old one ## Cost -Run only the cases your change touches until they pass, then run them and their controls once on a same-model baseline. That is the evidence a PR needs for review or reconsideration. After a maintainer approves the PR, do one full run per model on your change before it merges. A full baseline run is needed only when every case changes, such as a template refresh. Each eval's summary records what the run cost. +Until review, run only the cases your change touches, and once with their controls on a same-model baseline. After a maintainer approves, do one full run per model on your change. A full baseline is only for changes to every case. Each eval's summary records what the run cost. ## Add an eval From 6070e71b7abc13f49d64fd4493969c70c4c4be34 Mon Sep 17 00:00:00 2001 From: EtanHey Date: Thu, 12 Nov 2026 09:16:04 +0200 Subject: [PATCH 3/4] docs(evals): make the eval-run rule unambiguous in 54 words Review found "Until review" unclear about the time between review and approval, and "until they pass" and "before merge" had been cut. This wording restores them at 54 words (main: 29, first PR text: 74). Re-ran the planning repro: no sample planned an unneeded full run (Opus 0/8, Sol 0/5), and none ran main in full after approval. Co-Authored-By: Claude Fable 5.2 --- evals/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evals/README.md b/evals/README.md index 883bbc4481f3d..5e477cde5a1be 100644 --- a/evals/README.md +++ b/evals/README.md @@ -44,7 +44,7 @@ Every eval follows the same shape, so a new one is easy to write and an old one ## Cost -Until review, run only the cases your change touches, and once with their controls on a same-model baseline. After a maintainer approves, do one full run per model on your change. A full baseline is only for changes to every case. Each eval's summary records what the run cost. +Before review, run only the touched cases until they pass, then run them plus controls once on main using the same model. After maintainer approval, run the full suite once per model on your change before merge. Run a full baseline only when every case changes. Each eval's summary records what the run cost. ## Add an eval From d8b890b64307157d8d440ac1f1d5d94ab2d7d0a4 Mon Sep 17 00:00:00 2001 From: EtanHey Date: Thu, 12 Nov 2026 09:29:34 +0200 Subject: [PATCH 4/4] docs(evals): the eval-run rule says "baseline", not "main" (52 words) A new case does not exist on main, so "run them once on main" could not be followed. The guide's term is a same-model baseline. Re-ran the planning repro on this text: no unneeded full run (Opus 0/8, Sol 0/5), none on main after approval. Co-Authored-By: Claude Fable 5.2 --- evals/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evals/README.md b/evals/README.md index 5e477cde5a1be..f753deda6fe77 100644 --- a/evals/README.md +++ b/evals/README.md @@ -44,7 +44,7 @@ Every eval follows the same shape, so a new one is easy to write and an old one ## Cost -Before review, run only the touched cases until they pass, then run them plus controls once on main using the same model. After maintainer approval, run the full suite once per model on your change before merge. Run a full baseline only when every case changes. Each eval's summary records what the run cost. +Before review, run only the touched cases until they pass, then run them plus controls once on a same-model baseline. After maintainer approval, run the full suite once per model on your change before merge. Run a full baseline only when every case changes. Each eval's summary records what the run cost. ## Add an eval