diff --git a/quest/m0/model-routing/README.md b/quest/m0/model-routing/README.md index 8d24777e..3040cd48 100644 --- a/quest/m0/model-routing/README.md +++ b/quest/m0/model-routing/README.md @@ -83,6 +83,12 @@ are mostly judgment. write at (XS medium, S high, M xhigh, L and XL xhigh); and coordinators size each `/quest-iterate` by its remaining work. The tables and the iterate decisions are in [Route by size](/quest/m0/model-routing/models.md). +- Revised again 2026-11-05 evening (user, /quest-plan): xhigh is XL's + alone. Haiku high writes XS; Opus writes S low, M medium, L high, XL + xhigh; Sol falls back one rung lower, now through L; Sol reviews follow + [Single-pass review](/quest/m0/token-waste/review-single-pass.md)'s + table. The tables and the benchmark evidence are in + [Route by size](/quest/m0/model-routing/models.md). - Which milestone? ✅ m0 (rejected: m1). It ranks first among m0's work after Kix, by the 2026-11-01 order: land work, then Kix, then the rest. diff --git a/quest/m0/model-routing/models.md b/quest/m0/model-routing/models.md index 80b268e8..1a1f5a15 100644 --- a/quest/m0/model-routing/models.md +++ b/quest/m0/model-routing/models.md @@ -27,9 +27,10 @@ Decided 2026-11-04 (user, /quest-plan); the evidence is in - One model per size, or a ranked list? - ✅ A ranked list, so a usage limit falls through to the next entry (user, replacing "skip the quest and report the outage") -- Which models? ✅ Opus writes the code at every size, with Sol as the - fallback through M; L and XL have no fallback and wait for Opus (user, - 2026-11-05, replacing the 2026-11-04 lists that gave XS to Grok). +- Which models? ✅ Haiku writes XS and Opus writes S and up, with Sol as + the fallback through L; XL has no fallback and waits for Opus (user, + 2026-11-05, revised below; replacing the 2026-11-04 lists that gave XS to + Grok). - Grok? ✅ Removed from routing and from every review (user, 2026-11-05: its weekly usage is close to zero and its results were weak), so `quest agent` gains no Grok harness; the "quest agent starts Grok" quest is @@ -38,20 +39,59 @@ Decided 2026-11-04 (user, /quest-plan); the evidence is in a `Maintainer:` item under its `## Waiting`. An XL stays XL. The lists, pinned in `src/cli/models.ts` beside the review lists -`quest review` reads (user, 2026-11-05): +`quest review` reads (user, 2026-11-05, revised the same evening below): | Size | Writes (spawn and iterate) | Reviews (`quest review`) | |---|---|---| -| XS | claude `claude-opus-5-5` low → codex `gpt-6.1-sol` medium | codex `gpt-6.1-sol` medium → claude `claude-opus-5-5` low | -| S | claude `claude-opus-5-5` medium → codex `gpt-6.1-sol` high | codex `gpt-6.1-sol` high → claude `claude-opus-5-5` medium | -| M | claude `claude-opus-5-5` high → codex `gpt-6.1-sol` xhigh | codex `gpt-6.1-sol` xhigh → claude `claude-opus-5-5` high | -| L, XL | claude `claude-opus-5-5` xhigh | codex `gpt-6.1-sol` xhigh → claude `claude-opus-5-5` xhigh | +| XS | claude `claude-haiku-5-5` high → codex `gpt-6.1-sol` low | codex `gpt-6.1-sol` low → claude `claude-opus-5-5` low | +| S | claude `claude-opus-5-5` low → codex `gpt-6.1-sol` medium | codex `gpt-6.1-sol` medium → claude `claude-opus-5-5` low | +| M | claude `claude-opus-5-5` medium → codex `gpt-6.1-sol` high | codex `gpt-6.1-sol` medium → claude `claude-opus-5-5` medium | +| L | claude `claude-opus-5-5` high → codex `gpt-6.1-sol` xhigh | codex `gpt-6.1-sol` high → claude `claude-opus-5-5` high | +| XL | claude `claude-opus-5-5` xhigh | codex `gpt-6.1-sol` xhigh → claude `claude-opus-5-5` xhigh | -The review list is the write list flipped: Sol reviews at the effort Sol -would write at, and Opus takes the review only when Sol wrote the code. This -raises every Sol review one step from -[moq-dev/quest#493](https://github.com/moq-dev/quest/pull/493)'s (XS low, -S medium, M high). +Opus takes a review only when Sol wrote the code, and never reviews XL in +practice, since Sol writes no XL. + +Revised 2026-11-05 evening (user, /quest-plan, while spawning the +[token waste](/quest/m0/token-waste/README.md) quests), replacing writes of +XS Opus low, S Opus medium, M Opus high, L and XL Opus xhigh, and Sol +fallbacks of medium, high, and xhigh for XS to M. Agents already running +keep the efforts they started with. Decision prompts, chosen options +marked ✅: + +- Goal: ✅ every size gets a cheaper writer, so xhigh is XL's alone. +- Opus writes: ✅ one rung lower, S low, M medium, L high, XL xhigh (the + user's proposal). +- XS writer: ✅ Haiku high, then Sol low (rejected: Haiku medium, Opus low, + Sol medium, DeepSeek through opencode). The user asked whether Haiku high + beats Opus low. No published benchmark measures Opus 5.5 at low effort. + [Bito's coding benchmark](https://bito.ai/blog/claude-haiku-5-5-vs-sonnet-5-5-vs-opus-5-5/) + (October 2026, judged by Opus) scored Haiku 5.5 high 87% on one-shot asks + and 82% on long builds against Opus 5.5 medium's 90% and 87%, at 1/4 to + 1/23 of the cost; Haiku medium scored 76-79% with about three times the + variance between sessions. Anthropic warns that low effort on long agent + prompts skips checks and follow-through. models.dev prices Haiku 5.5 at + $0.10/$0.50 per million input/output tokens, five times that for a + request over 100k tokens, against Opus 5.5's $4/$20. DeepSeek would need + an opencode harness and its sandbox in `quest agent`, which left on + 2026-11-04. The [model check](/quest/m1/model-check.md) moves XS back to + Opus if Haiku's review scores lag. +- Sol fallbacks: ✅ one rung lower, XS low, S medium, M high, and a new L + fallback at xhigh; XL keeps none (the user's pick; rejected: unchanged). +- Review efforts: ✅ [Single-pass review](/quest/m0/token-waste/review-single-pass.md)'s + Sol column (XS low, S medium, M medium, L high, XL xhigh) replaces the + rule that a review runs at the effort Sol would write at, since that + decision ([moq-dev/quest#509](https://github.com/moq-dev/quest/pull/509)) + is newer than this quest's + ([moq-dev/quest#507](https://github.com/moq-dev/quest/pull/507)) and + carries the cost data. That quest's PR changes `REVIEWERS`; this one adds + the write lists. Opus's review efforts follow its write efforts + (rejected: the write list mirrors the reviews). + +Agent's choice, for the implementer to revisit: an iterate whose writer is +missing from its size's list runs at that writer's effort for the nearest +size that lists it, so an XS iterate on an Opus PR runs Opus low and an L +iterate on a Haiku PR runs Haiku high. Iterates, decided 2026-11-05 (user, /quest-plan, after a `/quest-complete` run in moq-dev/moq sent many merge-and-record iterates): @@ -66,7 +106,7 @@ run in moq-dev/moq sent many merge-and-record iterates): `Implementer: Claude Code, claude-opus-5-5, low effort (iterate, XS)`, so the [model check](/quest/m1/model-check.md) can audit it. Not chosen: `quest` estimating it from those signals. -- What if both models have written a PR? ✅ They never do: one writer per +- What if two models have written a PR? ✅ They never do: one writer per PR (user, 2026-11-05, from a CodeRabbit finding on [moq-dev/quest#507](https://github.com/moq-dev/quest/pull/507)). Each review list holds only Sol and Opus, and `quest review` excludes every diff --git a/quest/m0/queue-watcher/README.md b/quest/m0/queue-watcher/README.md index 071c6c1e..11c28a90 100644 --- a/quest/m0/queue-watcher/README.md +++ b/quest/m0/queue-watcher/README.md @@ -138,6 +138,7 @@ Shared shape, for the children to follow: - [The other repositories follow](/quest/m0/queue-watcher/follow.md) - quest, rind and rust2 are cut over and landing; waits for a fork PR to land through one of their queues - [Harden the queue runners](/quest/m0/queue-watcher/runners.md) - each queue runner runs only its repository's queue workflow, and every outside fork PR needs approval - [Tests fit their timeout on the queue runner](/quest/m0/queue-watcher/queue-test-timeouts.md) - the root and Worker suites stop timing out or stalling in merge-group runs, so the queue stops ejecting PRs for them +- [The watcher writes flaky quests](/quest/m0/queue-watcher/flaky-quest.md) - a disable's follow-up quest comes from a template in `quest land watch`, so the headless land fixer writes no quest - [Daemons prune their logs](/quest/m0/queue-watcher/log-retention.md) - the watcher and gate delete their finished logs after a window, so state stays bounded (4.1 GiB on nixos on 2026-11-04) - [Retire the batching gate](/quest/m0/queue-watcher/retire.md) - once no repository lands through `quest land serve`, its batching code and host units go diff --git a/quest/m0/queue-watcher/flaky-quest.md b/quest/m0/queue-watcher/flaky-quest.md new file mode 100644 index 00000000..4d3c9f57 --- /dev/null +++ b/quest/m0/queue-watcher/flaky-quest.md @@ -0,0 +1,43 @@ +# [S] The watcher writes flaky quests + +## Goal + +When the queue watcher disables a test seen flaking twice, the follow-up +quest (`quest/m0/flaky/.md`, with its epic created when missing) +comes from a fixed template in `quest land watch`, not from the land fixer. +The fixer's `disable` commit holds only the skip, whose reason names the +quest path the watcher chose. No headless agent writes quest prose. + +## Plan + +Decided 2026-11-05 (user, /quest-plan). The user's rule: quests are written +only by a coordinator (`/quest-complete`, `/quest-spawn`) on Opus, never by a +sub-agent. The skills already follow it: only `/quest-merge` and +`/quest-complete` plan follow-ups, and `/quest-plan` refuses to run in an +agent that cannot prompt. The land fixer, a headless Codex agent, is the one +exception: `assets/land-fix.md` has it write the flaky quest and its +`quest/m0/flaky/README.md` link in the disable commit, and +`src/cli/land-watch-flaky.ts` requires exactly one added quest file. + +Decision prompts, chosen options marked ✅: + +- Who writes it: ✅ the watcher, from a template (rejected: the fixer + reports the test and a coordinator runs `/quest-plan` on it later, which + leaves the skipped test untracked until then; keep the exception). +- Milestone: ✅ m0, in the queue watcher epic, whose end-to-end run + disables a flaky test (rejected: m1). + +Recommendations for the implementer: + +- The watcher already knows the test (the `failure` the fixer names, from + its known-flaky input) and the `merge_group` jobs that showed it, so it + picks `` and passes the quest path in the fixer's input for the skip + reason, then adds the quest file to the disable commit itself, or as a + second commit on the disable branch. +- The template's Goal says to fix the test or delete it and links each job. + Its size is a fixed guess, such as `[S]`, for the next coordinator to + correct. +- [Epics list no children](/quest/m0/token-waste/child-lists.md) is in + flight and drops the epic README links; follow whichever format the base + has when this lands. +- `validateFix` then rejects a disable commit that touches `quest/` at all.