Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ ownership-manifest design:
- [Review sessions keyed by upstream branch](/quest/m0/review-session-key.md) - a round from a local branch tracking the PR branch continues that branch's review session
- [Kix decision UI](/quest/m0/kix/README.md) - planning and implementation through prompts and summaries, run on the maintainer's own runners
- [Route quests to models by size](/quest/m0/model-routing/README.md) - `/quest-spawn` starts each quest on the first model of its size's ranked list that can run, reviews rank the same way, and PRs record size, implementer, and reviewer
- [Cut agent token waste](/quest/m0/token-waste/README.md) - sessions stop paying for loading rules, a second review pass, polling, oversized coordinators, and merge churn
- [Model check](/quest/m0/model-check.md) - recurring: the first audit of the routed PRs retunes the built-in size lists
- [Codex cloud off](/quest/m0/codex-cloud-off.md) - automatic Codex cloud reviews stop once no open PR relies on them
- [Codex project-config skip](/quest/m0/codex-project-config.md) - once Codex keeps a supported way to skip a project's `.codex/` layer, reviews keep the user's config
Expand Down
6 changes: 6 additions & 0 deletions quest/m0/review-session-key.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,9 @@ Implementation guidance:
base today, so the base's branch has to be known first.
- Nothing is in production, so sessions already stored under temporary names
need no migration.
- Added 2026-11-05 (/quest-plan, the
[token waste](/quest/m0/token-waste/README.md) audit): 80 `quest review`
runs in Claude sessions failed with "HEAD is detached". A detached
`quest review` without `--branch` names the PR branch to pass when it
can find one, such as a branch whose tip is the head, so the agent's next
call succeeds.
61 changes: 61 additions & 0 deletions quest/m0/token-waste/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Cut agent token waste

## Goal

Quest sessions spend their tokens on the work itself, not on loading rules,
a second review pass, polling, oversized coordinator context, or merge churn.
The epic finishes by re-measuring the baseline below over a week of sessions
and recording the comparison in its closing PR.

Non-goals: the online reviewers (CodeRabbit and Pullfrog are free and stay),
and turning prose rules into hooks or lint.

## Plan

Decided 2026-11-05 (user, /quest-plan), from an audit of this host's Claude
Code and Codex transcripts and the moq-dev PRs created 2026-10-06 to 11-05.

Baseline:

- Claude Code, 2026-10-28 to 11-05 (all the data the profile holds): about
$11k API-equivalent. Re-sent cache is 71% of weighted cost; the average
request carries 144k tokens of context; the 88 sessions that peaked above
400k (2.6%) are 35% of cost. Auto-compaction ran 14 times, at a median of
967k. By skill: iterate 24%, start 24%, complete 22%, spawn 10%.
- Codex, 30 days: 31.0B input tokens, 97% cached. `quest review` is 10% of
it (28% since 11-02); interactive sessions are 76%. The weekly limit ran
out on 11-02.
- PRs, 30 days, 7 repositories: 5,553 PRs, 93% merged. Base merges are 33%
of all commits (70% in moq.pro) and rose from 2.6 to 3.4 per merged PR
after 11-01. The merge queue ejected 723 PRs for conflicts. 21% of PRs
change only `quest/*.md`.

Decision prompts, chosen options marked ✅:

- Milestone: ✅ m0 (rejected: m1).
- Online reviewers: ✅ keep CodeRabbit and Pullfrog, which are free
(rejected: drop them).
- Rules to hooks or lint: ✅ not planned (rejected: plan it now).
- Applied outside the tree the same day: Claude's auto-compact window is
400k for Opus (`modelSettings.claude-opus-5-5.autoCompactWindow`, the
value the Claude Code team recommends), and `BASH_MAX_TIMEOUT_MS` is
3300000, just under the 1h prompt-cache TTL, in the maintainer's user
settings. The 1h TTL stays for main sessions and sub-agents: replayed over
the requests since 10-30, a 5m TTL would have cost 35% more cache spend
for main sessions and 14% more for sub-agents.
- The milestone re-cut the maintainer raised (m1 additive improvements, m2
semver-breaking features) is moq-dev/moq's own plan: ✅ re-plan moq
separately (rejected: make it quest's convention; not now).

## Required

- [Single-pass review](/quest/m0/token-waste/review-single-pass.md) - a review round explores the change once, with the diff and REVIEW.md given up front
- [Rules load once](/quest/m0/token-waste/rule-loading.md) - skills and rules arrive in the first turn and survive compaction
- [Coordinators and waits](/quest/m0/token-waste/coordinator-waits.md) - coordinators lean on compaction, children report briefly, and waits block
- [Epics list no children](/quest/m0/token-waste/child-lists.md) - an epic's children are its directory, so finishing a quest edits no README
- [Base merges only when needed](/quest/m0/token-waste/base-merges.md) - a branch merges its base on a conflict, not before every queue attempt

## Related

- [Usage reports](/quest/m1/usage.md) - `quest usage` will report these numbers from session logs
- [Model check](/quest/m0/model-check.md) - audits the size tables the review quest changes
37 changes: 37 additions & 0 deletions quest/m0/token-waste/base-merges.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# [S] Base merges only when needed

## Goal

A PR branch merges its base only when it conflicts with it or GitHub refuses
to enqueue it. `quest merge` enqueues a head that is behind the base on a
base that lands through GitHub's merge queue, since the queue tests every
PR against the live base.

## Plan

Decided 2026-11-05 (user, /quest-plan, the
[token waste](/quest/m0/token-waste/README.md) audit).

Measured, moq-dev PRs created 2026-10-06 to 11-05: 16,013 of 48,907 commits
(33%) were base merges, 48% on merged PRs and 70% in moq.pro, rising from 2.6
to 3.4 per merged PR after 11-01. Many moq CI runs were cancelled (1,527 of
5,735 PR runs), most likely superseded by the next push or base merge.

`/quest-merge` says `quest merge` refuses a head that does not contain the
current base, which sends agents to `/quest-iterate` to merge it before
every queue attempt. That preflight came from
[moq-dev/quest#253](https://github.com/moq-dev/quest/pull/253) on 10-28,
before repositories moved to GitHub's merge queue.

Decision prompts, chosen options marked ✅:

- Merge churn: ✅ the watcher merges quest files, and agents merge the base
only on a real conflict or at queue time (rejected: stop editing READMEs,
which [Epics list no children](/quest/m0/token-waste/child-lists.md) now
does a cleaner way; membership from directories, adopted there too). The
watcher half already exists: it merges a dropped PR's conflict with the
quest merge driver before using a fixer slot
([moq-dev/quest#389](https://github.com/moq-dev/quest/pull/389)).

Keep the preflight on bases that land through `quest land`, if any remain,
and check what #253 guarded against before removing it on queue bases.
44 changes: 44 additions & 0 deletions quest/m0/token-waste/child-lists.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# [L] Epics list no children

## Goal

An epic's children are the quests in its directory, and its `Required`
section lists only the work that blocks it. Finishing a quest deletes its
file and edits no epic or milestone README, so PRs that finish sibling
quests stop conflicting. `quest check --fix` strips the old child lists from
a repository's tree.

## Plan

Decided 2026-11-05 (user, /quest-plan, the
[token waste](/quest/m0/token-waste/README.md) audit).

Measured, moq-dev PRs created 2026-10-06 to 11-05:

- `quest/m1/README.md` was touched by 33% of merged moq-dev/moq PRs, and
milestone READMEs as a group by 45%; in moq-dev/rust2, `quest/m3/README.md`
by 290 of 585.
- GitHub's merge queue cannot run quest's merge driver and ejected 723 PRs
for conflicts. The [queue watcher](/quest/m0/queue-watcher/README.md)
re-merges such a PR with the driver since
[moq-dev/quest#389](https://github.com/moq-dev/quest/pull/389), but each
ejection still costs a requeue and a CI run, and the rate after 11-01
stayed at 0.13 per merged PR.

Decision prompts, chosen options marked ✅:

- Ordering: ✅ drop the ordering. The maintainer proposed it.
`quest unblocked` already ranks by milestone, then downstream impact, and
only then by list order (`src/core/unblocked.ts`), so order is only a
tiebreak. Path order replaces it.
- Scope: ✅ every epic (rejected: milestones only; unordered lists). The
directory already is the tree and `Required` already holds blockers, so
epics stay the graph they are minus the copy of the directory. A child
that must precede a sibling becomes an explicit blocker in `Required`.
Parallel siblings spawned from one epic conflict on its README today, as
milestones do.

The guide's Model, Format, and Creation sections, the skills that rank or
place work (`/quest-plan`, `/quest-audit`, `/quest-spawn`,
`/quest-recurring`, `/quest-import`), and `quest tree`'s output change with
it. The merge driver keeps resolving the blocker lists that remain.
47 changes: 47 additions & 0 deletions quest/m0/token-waste/coordinator-waits.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# [M] Coordinators and waits

## Goal

Long coordinators (`/quest-complete`, `/quest-spawn`, `/quest-afk`) keep
their state in GitHub and the quest tree and rely on compaction instead of
growing without bound. Sub-agent reports stay short. Every wait blocks on a
terminal state in one call instead of polling, and agents batch independent
reads and searches into one turn.

## Plan

Decided 2026-11-05 (user, /quest-plan, the
[token waste](/quest/m0/token-waste/README.md) audit).

Measured, Claude Code 2026-10-28 to 11-05:

- The most expensive session was a `/quest-complete` coordinator: 1,895
turns at a mean context of 444k, peaking at 966k, with 374 delegated-task
reports of up to 37k characters each. `/quest-complete` and
`/quest-spawn` are 32% of all cost.
- Polling cost 264M (6.8%): 1,165 `until grep -q "^EXIT=" ...; do sleep`
loops, 2,701 sleep calls, 850 background waits that returned nothing, and
1,204 `quest wait` calls. In Codex interactive sessions, polling requests
were 19.6% of input.
- Serial exploration: search and read calls trigger 35% of turn cost, at
1.11 tool calls per request, so each read re-sends the whole context.
- T3 Code sessions never saw the dotfiles' `BASH_MAX_TIMEOUT_MS`, so
foreground waits hit the 10-minute cap. The maintainer's user settings now
set it to 3300000, just under the 1h cache TTL.

Decision prompts, chosen options marked ✅:

- Coordinator context: ✅ rely on compaction (rejected: a handoff and
restart after N picks). Claude Code reminds the session which background
agents still run after compacting, so finished children shrink to the
summary without a status file. Automatic compaction cannot be steered, so
state that must persist belongs in GitHub and the quest tree.

Recommendations for the implementer:

- Cap a child's report at the PR, its state, and the decisions it needs,
with details in the PR body.
- `quest setup` sets the Bash timeout cap for Claude Code under the cache
TTL, so no session depends on a shell profile.
- Name the banned shapes (sleep or `until` loops, short-timeout waits) once,
where waits are described.
71 changes: 71 additions & 0 deletions quest/m0/token-waste/review-single-pass.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# [M] Single-pass review

## Goal

A `quest review` round explores the change once. The reviewer starts with
the diff, the base's `REVIEW.md`, and the responses to earlier findings, and
never reloads `quest guide`, `quest context`, or the repository's
instruction files, or runs builds and tests. `quest review` with no flags
reads the size and implementing models from the PR body's `## Models`
record. Review efforts follow the new table below.

## Plan

Decided 2026-11-05 (user, /quest-plan, the
[token waste](/quest/m0/token-waste/README.md) audit).

Measured, Codex reviews over 30 days:

- Round 1 is two full reviews. `src/cli/review.ts` runs the harness's
built-in review (`codex exec review`, or Claude's `/code-review`), then
resumes it with `firstPrompt`'s "Report your review as JSON", and that turn
re-explores from scratch: 608M input (121M uncached) against the first
pass's 706M. The two passes agree on whether there are findings in 96.5% of
reviews; the second added findings 41 times (2.5%). Later rounds resume
cheaply (404M input, 28M uncached over 934 rounds).
- The reviewer ran `quest guide` 2,125 times and `quest context` 598 times,
plus about 7.8k `git show` reads of instruction files it already had: about
163M tokens carried.
- Each review request carries a ~10k-token skills catalog (92 entries, many
repeated 4×), about 270M over 26,765 requests (estimate), and the
maintainer's global `AGENTS.md` leaks in despite `--ignore-user-config`.
- 26% of tool outputs hit the ~40k-character cap and get re-read in slices.
- Tests ran in 483 of 4,219 review turns and builds in 71. Few tokens, but
`quest gate` stalls in the read-only sandbox and the reviewer polls it.
- Claude sessions ran `quest review --help` 1,130 times, mostly to find
`--exclude-model` and `--size`.

Decision prompts, chosen options marked ✅:

- Review efforts: ✅ revisit in this quest (rejected: record the cost data
in the model check; leave the table). The Sol column becomes XS low,
S medium, M medium, L high, XL xhigh, replacing M high and L xhigh from
[moq-dev/quest#493](https://github.com/moq-dev/quest/pull/493). xhigh cost
2-6× more per review, and only 4 large xhigh reviews exist, so nothing yet
shows it finds more. The Opus column is unchanged. This PR updates
[Model check](/quest/m0/model-check.md)'s Plan with the change, so its
audit compares findings by effort once reviews are single-pass.
- Where the size and implementing models come from: ✅ read the PR body's
`## Models` record when no flag is given; flags still override (rejected:
keep the flags only). The implementing agent writes that record and passes
the flags today, so trust is unchanged. #493's rule stands: the size never
comes from the head's files.

Recommendations for the implementer:

- `codex exec review` takes custom instructions as its prompt argument, and
Claude's `/code-review` accepts text after its target. Put the scope,
`REVIEW.md`, the responses, and the report shape into round 1 itself, so
the follow-up turn only shapes the answer. Prefer one turn when the harness
can return the schema directly.
- The prompt says that the repository's instruction files and `REVIEW.md`
are already given, and forbids builds, tests, and `quest` commands.
- Look for a supported way to drop the skills catalog and the user's global
`AGENTS.md` from the reviewer, such as a Codex `--disable` feature; verify
it in the transcript, not by assumption.
- Skip a round when the head has not changed since the last review.

## Related

- [Review sessions keyed by upstream branch](/quest/m0/review-session-key.md) - the detached-HEAD failures
- [Route quests to models by size](/quest/m0/model-routing/README.md) - shares the size tables
50 changes: 50 additions & 0 deletions quest/m0/token-waste/rule-loading.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# [L] Rules load once

## Goal

A quest skill's first turn already holds its body and the rules it needs,
and both survive compaction. A Claude Code skill invocation runs no
bootstrap commands, and a Codex one runs a single `quest skill <name>`.
Implementers no longer load rules only reviewers or prompt authors need.

## Plan

Decided 2026-11-05 (user, /quest-plan, the
[token waste](/quest/m0/token-waste/README.md) audit).

Measured, Claude Code 2026-10-28 to 11-05:

- Each skill run takes 3-4 turns before real work: the installed stub
(`Skill.stub` in `src/cli/skills.ts`) says to run `quest skill <name>`,
whose output says to run `quest guide` and `quest skill grill`, and the
guide says to run `quest context`. That is about 27KB (context 17KB, guide
10KB), roughly 260 instruction sentences, against the 150-200 published
guidance says models follow reliably.
- `quest guide` ran 2,283 times, `quest skill` 3,057 times (695 sessions ran
it twice or more), and `quest context` 373 times: about 198M weighted
tokens.
- These arrive as Bash output, which compaction summarizes away. Claude Code
re-injects invoked skill bodies (up to 5k tokens each) and the output of
`SessionStart` hooks matching `compact`.
- `Reviews` is 36% of `assets/CONTEXT.md`, and every implementer loads it.

Decision prompts, chosen options marked ✅:

- Delivery: ✅ injection plus a user hook. Claude stubs inline
`` !`quest skill <name>` ``, which prints the skill with the guide and
context sections it needs. A user-level `SessionStart` hook (startup,
resume, and compact) that `quest setup` installs prints the always-on
rules, since repositories write no hooks
([Agents get the dev shell from user settings](/quest/m0/user-dev-shell/README.md)).
Codex keeps a single `quest skill <name>` call. Rejected: install full
skill bodies (each repository's copy drifts from the CLI); one command
only (rules still vanish on compaction).
- Split `assets/CONTEXT.md` by role: the review procedure moves to
`/quest-review` and `quest review`'s own output, Prompting stays with the
path-scoped rule, and question mechanics go to the skills that ask. Say
each rule once, as Prompting requires.

Recommendations: keep the important lines near the top of each skill, since
re-injection keeps the start of a skill past 5k tokens. A failing `!`
command aborts the skill, so `quest skill` must print the setup pointer
itself when it cannot run.