From 3f97e608fbe0174bbb0fbcdf507bc26a34f33c20 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 12:37:14 -0800 Subject: [PATCH] quest: plan the token waste epic An audit of this host's Claude and Codex transcripts and 30 days of moq-dev PRs found where sessions waste tokens: a doubled first review round, 3-4 turns of rule bootstrap that compaction drops, polling, coordinators past 400k context, and merge churn on epic READMEs and base merges. Co-Authored-By: Claude Opus 5.5 --- quest/m0/README.md | 1 + quest/m0/review-session-key.md | 6 ++ quest/m0/token-waste/README.md | 61 +++++++++++++++++++ quest/m0/token-waste/base-merges.md | 37 +++++++++++ quest/m0/token-waste/child-lists.md | 44 ++++++++++++++ quest/m0/token-waste/coordinator-waits.md | 47 ++++++++++++++ quest/m0/token-waste/review-single-pass.md | 71 ++++++++++++++++++++++ quest/m0/token-waste/rule-loading.md | 50 +++++++++++++++ 8 files changed, 317 insertions(+) create mode 100644 quest/m0/token-waste/README.md create mode 100644 quest/m0/token-waste/base-merges.md create mode 100644 quest/m0/token-waste/child-lists.md create mode 100644 quest/m0/token-waste/coordinator-waits.md create mode 100644 quest/m0/token-waste/review-single-pass.md create mode 100644 quest/m0/token-waste/rule-loading.md diff --git a/quest/m0/README.md b/quest/m0/README.md index 568de976..9ba989f3 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -96,6 +96,7 @@ ownership-manifest design: - [Review sessions keyed by upstream branch](/quest/m0/review-session-key.md) - a round from a local branch tracking the PR branch continues that branch's review session - [Kix decision UI](/quest/m0/kix/README.md) - planning and implementation through prompts and summaries, run on the maintainer's own runners - [Route quests to models by size](/quest/m0/model-routing/README.md) - `/quest-spawn` starts each quest on the first model of its size's ranked list that can run, reviews rank the same way, and PRs record size, implementer, and reviewer +- [Cut agent token waste](/quest/m0/token-waste/README.md) - sessions stop paying for loading rules, a second review pass, polling, oversized coordinators, and merge churn - [Model check](/quest/m0/model-check.md) - recurring: the first audit of the routed PRs retunes the built-in size lists - [Codex cloud off](/quest/m0/codex-cloud-off.md) - automatic Codex cloud reviews stop once no open PR relies on them - [Codex project-config skip](/quest/m0/codex-project-config.md) - once Codex keeps a supported way to skip a project's `.codex/` layer, reviews keep the user's config diff --git a/quest/m0/review-session-key.md b/quest/m0/review-session-key.md index 703f76f0..d60874db 100644 --- a/quest/m0/review-session-key.md +++ b/quest/m0/review-session-key.md @@ -45,3 +45,9 @@ Implementation guidance: base today, so the base's branch has to be known first. - Nothing is in production, so sessions already stored under temporary names need no migration. +- Added 2026-11-05 (/quest-plan, the + [token waste](/quest/m0/token-waste/README.md) audit): 80 `quest review` + runs in Claude sessions failed with "HEAD is detached". A detached + `quest review` without `--branch` names the PR branch to pass when it + can find one, such as a branch whose tip is the head, so the agent's next + call succeeds. diff --git a/quest/m0/token-waste/README.md b/quest/m0/token-waste/README.md new file mode 100644 index 00000000..dc72fa2a --- /dev/null +++ b/quest/m0/token-waste/README.md @@ -0,0 +1,61 @@ +# Cut agent token waste + +## Goal + +Quest sessions spend their tokens on the work itself, not on loading rules, +a second review pass, polling, oversized coordinator context, or merge churn. +The epic finishes by re-measuring the baseline below over a week of sessions +and recording the comparison in its closing PR. + +Non-goals: the online reviewers (CodeRabbit and Pullfrog are free and stay), +and turning prose rules into hooks or lint. + +## Plan + +Decided 2026-11-05 (user, /quest-plan), from an audit of this host's Claude +Code and Codex transcripts and the moq-dev PRs created 2026-10-06 to 11-05. + +Baseline: + +- Claude Code, 2026-10-28 to 11-05 (all the data the profile holds): about + $11k API-equivalent. Re-sent cache is 71% of weighted cost; the average + request carries 144k tokens of context; the 88 sessions that peaked above + 400k (2.6%) are 35% of cost. Auto-compaction ran 14 times, at a median of + 967k. By skill: iterate 24%, start 24%, complete 22%, spawn 10%. +- Codex, 30 days: 31.0B input tokens, 97% cached. `quest review` is 10% of + it (28% since 11-02); interactive sessions are 76%. The weekly limit ran + out on 11-02. +- PRs, 30 days, 7 repositories: 5,553 PRs, 93% merged. Base merges are 33% + of all commits (70% in moq.pro) and rose from 2.6 to 3.4 per merged PR + after 11-01. The merge queue ejected 723 PRs for conflicts. 21% of PRs + change only `quest/*.md`. + +Decision prompts, chosen options marked ✅: + +- Milestone: ✅ m0 (rejected: m1). +- Online reviewers: ✅ keep CodeRabbit and Pullfrog, which are free + (rejected: drop them). +- Rules to hooks or lint: ✅ not planned (rejected: plan it now). +- Applied outside the tree the same day: Claude's auto-compact window is + 400k for Opus (`modelSettings.claude-opus-5-5.autoCompactWindow`, the + value the Claude Code team recommends), and `BASH_MAX_TIMEOUT_MS` is + 3300000, just under the 1h prompt-cache TTL, in the maintainer's user + settings. The 1h TTL stays for main sessions and sub-agents: replayed over + the requests since 10-30, a 5m TTL would have cost 35% more cache spend + for main sessions and 14% more for sub-agents. +- The milestone re-cut the maintainer raised (m1 additive improvements, m2 + semver-breaking features) is moq-dev/moq's own plan: ✅ re-plan moq + separately (rejected: make it quest's convention; not now). + +## Required + +- [Single-pass review](/quest/m0/token-waste/review-single-pass.md) - a review round explores the change once, with the diff and REVIEW.md given up front +- [Rules load once](/quest/m0/token-waste/rule-loading.md) - skills and rules arrive in the first turn and survive compaction +- [Coordinators and waits](/quest/m0/token-waste/coordinator-waits.md) - coordinators lean on compaction, children report briefly, and waits block +- [Epics list no children](/quest/m0/token-waste/child-lists.md) - an epic's children are its directory, so finishing a quest edits no README +- [Base merges only when needed](/quest/m0/token-waste/base-merges.md) - a branch merges its base on a conflict, not before every queue attempt + +## Related + +- [Usage reports](/quest/m1/usage.md) - `quest usage` will report these numbers from session logs +- [Model check](/quest/m0/model-check.md) - audits the size tables the review quest changes diff --git a/quest/m0/token-waste/base-merges.md b/quest/m0/token-waste/base-merges.md new file mode 100644 index 00000000..fd8e846e --- /dev/null +++ b/quest/m0/token-waste/base-merges.md @@ -0,0 +1,37 @@ +# [S] Base merges only when needed + +## Goal + +A PR branch merges its base only when it conflicts with it or GitHub refuses +to enqueue it. `quest merge` enqueues a head that is behind the base on a +base that lands through GitHub's merge queue, since the queue tests every +PR against the live base. + +## Plan + +Decided 2026-11-05 (user, /quest-plan, the +[token waste](/quest/m0/token-waste/README.md) audit). + +Measured, moq-dev PRs created 2026-10-06 to 11-05: 16,013 of 48,907 commits +(33%) were base merges, 48% on merged PRs and 70% in moq.pro, rising from 2.6 +to 3.4 per merged PR after 11-01. Many moq CI runs were cancelled (1,527 of +5,735 PR runs), most likely superseded by the next push or base merge. + +`/quest-merge` says `quest merge` refuses a head that does not contain the +current base, which sends agents to `/quest-iterate` to merge it before +every queue attempt. That preflight came from +[moq-dev/quest#253](https://github.com/moq-dev/quest/pull/253) on 10-28, +before repositories moved to GitHub's merge queue. + +Decision prompts, chosen options marked ✅: + +- Merge churn: ✅ the watcher merges quest files, and agents merge the base + only on a real conflict or at queue time (rejected: stop editing READMEs, + which [Epics list no children](/quest/m0/token-waste/child-lists.md) now + does a cleaner way; membership from directories, adopted there too). The + watcher half already exists: it merges a dropped PR's conflict with the + quest merge driver before using a fixer slot + ([moq-dev/quest#389](https://github.com/moq-dev/quest/pull/389)). + +Keep the preflight on bases that land through `quest land`, if any remain, +and check what #253 guarded against before removing it on queue bases. diff --git a/quest/m0/token-waste/child-lists.md b/quest/m0/token-waste/child-lists.md new file mode 100644 index 00000000..3c35b226 --- /dev/null +++ b/quest/m0/token-waste/child-lists.md @@ -0,0 +1,44 @@ +# [L] Epics list no children + +## Goal + +An epic's children are the quests in its directory, and its `Required` +section lists only the work that blocks it. Finishing a quest deletes its +file and edits no epic or milestone README, so PRs that finish sibling +quests stop conflicting. `quest check --fix` strips the old child lists from +a repository's tree. + +## Plan + +Decided 2026-11-05 (user, /quest-plan, the +[token waste](/quest/m0/token-waste/README.md) audit). + +Measured, moq-dev PRs created 2026-10-06 to 11-05: + +- `quest/m1/README.md` was touched by 33% of merged moq-dev/moq PRs, and + milestone READMEs as a group by 45%; in moq-dev/rust2, `quest/m3/README.md` + by 290 of 585. +- GitHub's merge queue cannot run quest's merge driver and ejected 723 PRs + for conflicts. The [queue watcher](/quest/m0/queue-watcher/README.md) + re-merges such a PR with the driver since + [moq-dev/quest#389](https://github.com/moq-dev/quest/pull/389), but each + ejection still costs a requeue and a CI run, and the rate after 11-01 + stayed at 0.13 per merged PR. + +Decision prompts, chosen options marked ✅: + +- Ordering: ✅ drop the ordering. The maintainer proposed it. + `quest unblocked` already ranks by milestone, then downstream impact, and + only then by list order (`src/core/unblocked.ts`), so order is only a + tiebreak. Path order replaces it. +- Scope: ✅ every epic (rejected: milestones only; unordered lists). The + directory already is the tree and `Required` already holds blockers, so + epics stay the graph they are minus the copy of the directory. A child + that must precede a sibling becomes an explicit blocker in `Required`. + Parallel siblings spawned from one epic conflict on its README today, as + milestones do. + +The guide's Model, Format, and Creation sections, the skills that rank or +place work (`/quest-plan`, `/quest-audit`, `/quest-spawn`, +`/quest-recurring`, `/quest-import`), and `quest tree`'s output change with +it. The merge driver keeps resolving the blocker lists that remain. diff --git a/quest/m0/token-waste/coordinator-waits.md b/quest/m0/token-waste/coordinator-waits.md new file mode 100644 index 00000000..24cb99da --- /dev/null +++ b/quest/m0/token-waste/coordinator-waits.md @@ -0,0 +1,47 @@ +# [M] Coordinators and waits + +## Goal + +Long coordinators (`/quest-complete`, `/quest-spawn`, `/quest-afk`) keep +their state in GitHub and the quest tree and rely on compaction instead of +growing without bound. Sub-agent reports stay short. Every wait blocks on a +terminal state in one call instead of polling, and agents batch independent +reads and searches into one turn. + +## Plan + +Decided 2026-11-05 (user, /quest-plan, the +[token waste](/quest/m0/token-waste/README.md) audit). + +Measured, Claude Code 2026-10-28 to 11-05: + +- The most expensive session was a `/quest-complete` coordinator: 1,895 + turns at a mean context of 444k, peaking at 966k, with 374 delegated-task + reports of up to 37k characters each. `/quest-complete` and + `/quest-spawn` are 32% of all cost. +- Polling cost 264M (6.8%): 1,165 `until grep -q "^EXIT=" ...; do sleep` + loops, 2,701 sleep calls, 850 background waits that returned nothing, and + 1,204 `quest wait` calls. In Codex interactive sessions, polling requests + were 19.6% of input. +- Serial exploration: search and read calls trigger 35% of turn cost, at + 1.11 tool calls per request, so each read re-sends the whole context. +- T3 Code sessions never saw the dotfiles' `BASH_MAX_TIMEOUT_MS`, so + foreground waits hit the 10-minute cap. The maintainer's user settings now + set it to 3300000, just under the 1h cache TTL. + +Decision prompts, chosen options marked ✅: + +- Coordinator context: ✅ rely on compaction (rejected: a handoff and + restart after N picks). Claude Code reminds the session which background + agents still run after compacting, so finished children shrink to the + summary without a status file. Automatic compaction cannot be steered, so + state that must persist belongs in GitHub and the quest tree. + +Recommendations for the implementer: + +- Cap a child's report at the PR, its state, and the decisions it needs, + with details in the PR body. +- `quest setup` sets the Bash timeout cap for Claude Code under the cache + TTL, so no session depends on a shell profile. +- Name the banned shapes (sleep or `until` loops, short-timeout waits) once, + where waits are described. diff --git a/quest/m0/token-waste/review-single-pass.md b/quest/m0/token-waste/review-single-pass.md new file mode 100644 index 00000000..813aa7b0 --- /dev/null +++ b/quest/m0/token-waste/review-single-pass.md @@ -0,0 +1,71 @@ +# [M] Single-pass review + +## Goal + +A `quest review` round explores the change once. The reviewer starts with +the diff, the base's `REVIEW.md`, and the responses to earlier findings, and +never reloads `quest guide`, `quest context`, or the repository's +instruction files, or runs builds and tests. `quest review` with no flags +reads the size and implementing models from the PR body's `## Models` +record. Review efforts follow the new table below. + +## Plan + +Decided 2026-11-05 (user, /quest-plan, the +[token waste](/quest/m0/token-waste/README.md) audit). + +Measured, Codex reviews over 30 days: + +- Round 1 is two full reviews. `src/cli/review.ts` runs the harness's + built-in review (`codex exec review`, or Claude's `/code-review`), then + resumes it with `firstPrompt`'s "Report your review as JSON", and that turn + re-explores from scratch: 608M input (121M uncached) against the first + pass's 706M. The two passes agree on whether there are findings in 96.5% of + reviews; the second added findings 41 times (2.5%). Later rounds resume + cheaply (404M input, 28M uncached over 934 rounds). +- The reviewer ran `quest guide` 2,125 times and `quest context` 598 times, + plus about 7.8k `git show` reads of instruction files it already had: about + 163M tokens carried. +- Each review request carries a ~10k-token skills catalog (92 entries, many + repeated 4×), about 270M over 26,765 requests (estimate), and the + maintainer's global `AGENTS.md` leaks in despite `--ignore-user-config`. +- 26% of tool outputs hit the ~40k-character cap and get re-read in slices. +- Tests ran in 483 of 4,219 review turns and builds in 71. Few tokens, but + `quest gate` stalls in the read-only sandbox and the reviewer polls it. +- Claude sessions ran `quest review --help` 1,130 times, mostly to find + `--exclude-model` and `--size`. + +Decision prompts, chosen options marked ✅: + +- Review efforts: ✅ revisit in this quest (rejected: record the cost data + in the model check; leave the table). The Sol column becomes XS low, + S medium, M medium, L high, XL xhigh, replacing M high and L xhigh from + [moq-dev/quest#493](https://github.com/moq-dev/quest/pull/493). xhigh cost + 2-6× more per review, and only 4 large xhigh reviews exist, so nothing yet + shows it finds more. The Opus column is unchanged. This PR updates + [Model check](/quest/m0/model-check.md)'s Plan with the change, so its + audit compares findings by effort once reviews are single-pass. +- Where the size and implementing models come from: ✅ read the PR body's + `## Models` record when no flag is given; flags still override (rejected: + keep the flags only). The implementing agent writes that record and passes + the flags today, so trust is unchanged. #493's rule stands: the size never + comes from the head's files. + +Recommendations for the implementer: + +- `codex exec review` takes custom instructions as its prompt argument, and + Claude's `/code-review` accepts text after its target. Put the scope, + `REVIEW.md`, the responses, and the report shape into round 1 itself, so + the follow-up turn only shapes the answer. Prefer one turn when the harness + can return the schema directly. +- The prompt says that the repository's instruction files and `REVIEW.md` + are already given, and forbids builds, tests, and `quest` commands. +- Look for a supported way to drop the skills catalog and the user's global + `AGENTS.md` from the reviewer, such as a Codex `--disable` feature; verify + it in the transcript, not by assumption. +- Skip a round when the head has not changed since the last review. + +## Related + +- [Review sessions keyed by upstream branch](/quest/m0/review-session-key.md) - the detached-HEAD failures +- [Route quests to models by size](/quest/m0/model-routing/README.md) - shares the size tables diff --git a/quest/m0/token-waste/rule-loading.md b/quest/m0/token-waste/rule-loading.md new file mode 100644 index 00000000..f33dc5fa --- /dev/null +++ b/quest/m0/token-waste/rule-loading.md @@ -0,0 +1,50 @@ +# [L] Rules load once + +## Goal + +A quest skill's first turn already holds its body and the rules it needs, +and both survive compaction. A Claude Code skill invocation runs no +bootstrap commands, and a Codex one runs a single `quest skill `. +Implementers no longer load rules only reviewers or prompt authors need. + +## Plan + +Decided 2026-11-05 (user, /quest-plan, the +[token waste](/quest/m0/token-waste/README.md) audit). + +Measured, Claude Code 2026-10-28 to 11-05: + +- Each skill run takes 3-4 turns before real work: the installed stub + (`Skill.stub` in `src/cli/skills.ts`) says to run `quest skill `, + whose output says to run `quest guide` and `quest skill grill`, and the + guide says to run `quest context`. That is about 27KB (context 17KB, guide + 10KB), roughly 260 instruction sentences, against the 150-200 published + guidance says models follow reliably. +- `quest guide` ran 2,283 times, `quest skill` 3,057 times (695 sessions ran + it twice or more), and `quest context` 373 times: about 198M weighted + tokens. +- These arrive as Bash output, which compaction summarizes away. Claude Code + re-injects invoked skill bodies (up to 5k tokens each) and the output of + `SessionStart` hooks matching `compact`. +- `Reviews` is 36% of `assets/CONTEXT.md`, and every implementer loads it. + +Decision prompts, chosen options marked ✅: + +- Delivery: ✅ injection plus a user hook. Claude stubs inline + `` !`quest skill ` ``, which prints the skill with the guide and + context sections it needs. A user-level `SessionStart` hook (startup, + resume, and compact) that `quest setup` installs prints the always-on + rules, since repositories write no hooks + ([Agents get the dev shell from user settings](/quest/m0/user-dev-shell/README.md)). + Codex keeps a single `quest skill ` call. Rejected: install full + skill bodies (each repository's copy drifts from the CLI); one command + only (rules still vanish on compaction). +- Split `assets/CONTEXT.md` by role: the review procedure moves to + `/quest-review` and `quest review`'s own output, Prompting stays with the + path-scoped rule, and question mechanics go to the skills that ask. Say + each rule once, as Prompting requires. + +Recommendations: keep the important lines near the top of each skill, since +re-injection keeps the start of a skill past 5k tokens. A failing `!` +command aborts the skill, so `quest skill` must print the setup pointer +itself when it cannot run.