Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions quest/m0/caused-by-culprits.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# [S] Caused by names only the PR that shipped the defect

## Goal

A fix's `## Caused by` names the PR whose diff introduced the defect, not
one that only touched the same lines, deferred the work to a planned
follow-up, or kept behavior older than it, so the escapes `quest usage`
counts are defects.

## Plan

The 2026-11-11 [mistake audit](/quest/m0/mistake-audit.md) found 4 of 14
moq-dev/moq culprit links named a PR that shipped no defect:
[moq-dev/moq#8294](https://github.com/moq-dev/moq/pull/8294) and
[moq-dev/moq#8440](https://github.com/moq-dev/moq/pull/8440) left the work
to a planned follow-up,
[moq-dev/moq#8362](https://github.com/moq-dev/moq/pull/8362) deferred it on
the maintainer's decision, and
[moq-dev/moq#8402](https://github.com/moq-dev/moq/pull/8402) kept an older
order. `quest context` says to find the culprit "by `git blame` on the
lines the fix changes", which finds the last PR to touch them.

Agent's choices, for the implementer to revisit:

- Reword that rule in `assets/CONTEXT.md` to confirm in the blamed PR's
diff that it introduced the defect, else follow blame further back or
say no single PR caused it. This is a prose rule because only judgment
can tell a defect from a deferral. `assets/CONTEXT.md` is at 9,933 of
its 10,000 characters, so the rewording must not grow it.
- Follow `quest guide prompting`.
62 changes: 62 additions & 0 deletions quest/m0/late-signals.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# [M] A finding or decision posted on a head holds its landing

## Goal

A PR does not land past a review finding, or a recorded decision, posted
on its head after the review its pick relied on, including one posted
while it waits in a merge queue. `quest merge` refuses such a head, naming
the finding or comment, and a queued PR that draws one leaves the queue
until it is fixed or refuted.

Non-goal: failed checks on the head, which `quest merge` already refuses
([moq-dev/quest#601](https://github.com/moq-dev/quest/pull/601)), and a
failed run on an earlier head, below.

## Plan

The 2026-11-11 [mistake audit](/quest/m0/mistake-audit.md) found 3 of the
week's 7 escapes landed past a signal already on the PR. The cases this
must refuse:

- [moq-dev/moq#8375](https://github.com/moq-dev/moq/pull/8375): an online
review posted 65 seconds after enqueue described the bug
[moq-dev/moq#8504](https://github.com/moq-dev/moq/pull/8504) later
fixed, and the PR merged 15 minutes after it.
- [moq-dev/moq#8431](https://github.com/moq-dev/moq/pull/8431): a comment
recording the maintainer's decision said the head lacked a timeout, and
the same head was enqueued 10 minutes later, fixed by
[moq-dev/moq#8466](https://github.com/moq-dev/moq/pull/8466).

Agent's choices, for the implementer to revisit:

- `quest merge` reads the reviews posted after the head's commit, and
the comments the decision under `## Waiting` picks, and refuses while
one holds a finding or decision the head has not answered.
`quest status` prints them, so `/quest-merge` sees them before it
picks.
- The land watcher re-reads queued PRs and dequeues one that draws a
finding, so CONTRIBUTING's "online findings already posted are addressed
at merge, never waited for" still holds: nothing waits for a reviewer,
but a finding that arrives before the merge takes the PR out of the
queue until it is fixed or refuted.
- [moq-dev/moq#8372](https://github.com/moq-dev/moq/pull/8372) merged with
a red `platform.yml` dispatch run on an earlier head, which no check on
its final head showed. Leave it to moq-dev/moq, since one case is not a
class.

## Waiting

- Maintainer: which comments hold a head? Options: (a) any comment by a
write-access account after the head's commit, until a push or a reply
answers it; (b) only comments that open with a fixed marker, such as
"Maintainer decision", which the skills that record a decision on a PR
start writing; (c) only review findings, leaving decision comments to
`/quest-merge`'s prose. Recommend (b): it would have caught
moq-dev/moq#8431, whose comment opened that way, without holding every
chat comment, and a check cannot tell a decision from discussion
otherwise. Check the answer in this Plan.

## Related

- [Mistake audit](/quest/m0/mistake-audit.md) - counts heads that landed
past a signal
118 changes: 84 additions & 34 deletions quest/m0/mistake-audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,11 @@ and `/reflect`):
`quest messages`
- Rejected: forge records only; transcripts through a script each run
- What gates each run?
- ✅ A week since the last run, as a `## Waiting` item, the way
[the release cut](/quest/m1/release-cut.md) waits
- ✅ A week since the last run, the way
[the release cut](/quest/m1/release-cut.md) waits. The decision named a
`## Waiting` item because quests could not yet be held until a date;
since `## Scheduled` exists, the run holds the date there (agent,
2026-11-11).
- Rejected: each milestone, as
[the model check](/quest/m1/model-check.md) runs, since m0 has been
open since 2026-09-26; each release cut
Expand All @@ -38,10 +41,21 @@ and `/reflect`):
Agent's choices, for the implementer to revisit. Each run:

- Read, since the last run's date: `quest usage --since <date> --json` in
moq-dev/moq, moq-dev/moq.pro, and moq-dev/quest, for each PR's review
rounds, findings fixed and refuted, hand-backs, and the later PRs naming
it under `## Caused by`; the diffs of those culprits and their fixes;
and the typed messages from `quest messages --since <date> --json`.
moq-dev/moq, moq-dev/moq.pro, and moq-dev/quest (each clone's root as
`--root`), for each PR's review rounds, findings fixed and refuted,
hand-backs, and the later PRs naming it under `## Caused by`; the diffs
of those culprits and their fixes; and the typed messages from
`quest messages --since <date> --json`.
- Read each culprit pair's `## Caused by` line before counting it. On
2026-11-11 only 7 of 20 pairs were escapes: `fixedBy` takes every link
in the section, including "No single PR" lines and unmerged PRs, until
[the token-max attribution](/quest/m0/routing-evidence/usage-token-max.md)
lands; the rest named a PR that left the work to a planned follow-up
or kept older behavior.
- `quest messages` returns every human turn, so set aside machine-written
ones yourself: bare skill invocations, pasted relays and logs, accepted
suggested prompts, `/compact`, `/clear`, and `/goal` commands, and
Codex's question replies (about 175 of 1,452 on 2026-11-11).
- Group them into classes by the agent behavior behind them. A class is
the same behavior in two or more PRs or messages; one failure is not a
class.
Expand All @@ -53,9 +67,6 @@ Agent's choices, for the implementer to revisit. Each run:
owns that gate, so it goes in the run PR's `## Follow-ups` for
`/quest-plan` there, as the guide's Creation section says (user,
2026-11-11, [routing evidence](/quest/m0/routing-evidence/README.md)).
The 2026-11-11 audit sampled 30 escapes: catchable 6, maybe 17, and not
from the diff 7. The 5 races fell across those columns, since race and
catchability were classified separately.
- Before proposing a fix, find the quest, check, or rule that already owns
the class, and report whether its count fell since that landed. A rule
whose class still recurs is the case for moving it up a level.
Expand All @@ -69,39 +80,78 @@ Agent's choices, for the implementer to revisit. Each run:
edits no `AGENTS.md`, `CONTRIBUTING.md`, or skill.
- Tally the decision prompts per skill: how often the answer was the
agent's own recommendation. Propose a change only where that is nearly
always.
always. `quest messages` drops Claude Code's and T3 Code's structured
answers and returns Codex's as text without the recommended option, so
this waits on [question answers](/quest/m0/question-answers.md); until
then, count only typed answers that just pick the recommendation.
- moq-dev/quest is public, and typed messages can name customers and
private repositories. The PR and its quests give counts and classes in
the agent's words, never a message's text; quotes stay in `.tmp/`.
the agent's words, never a message's text, and name no moq-dev/moq.pro
PR or path; quotes stay in `.tmp/`.
- Leave model and effort tallies to
[the model check](/quest/m1/model-check.md).
- The run's PR rewrites this quest at this path, or at the same path in
the lowest open milestone once m0 completes: the new baseline, what to
check first, and a `## Waiting` item for one week after the run's date,
checked with `date`. The first run has no such item.
check first, and a `## Scheduled` date one week after the run's date,
checked with `date`.

Baseline, from the hand scan of 2026-11-09 (typed messages from
2026-10-26 to 2026-11-10, classed by three sub-agents, counts unverified):
Check first next run:

- 579 mid-session messages, about 227 machine-written, about 352 typed, 68
of them corrections or redirects.
- Stalls until nudged: 17 typed, plus the coordinator relays that the
stop hook of [moq-dev/quest#600](https://github.com/moq-dev/quest/pull/600)
targets; check first whether they fell after it landed.
- A proposal the maintainer steered simpler: 10.
- Misread intent or target: 7.
- Output handed back without running or looking at it: 7, which
the verification rule in `quest context` owns.
- Reports a problem instead of fixing it, or fixes an arbitrary top few: 6.
- Asks a routine choice: 4, beside 14 answers that only picked the
recommendation.
- Repeated by hand: `/quest-complete` or `/quest-spawn` about 36 times,
"any new PRs?" 15, and away mode entered 14 times. A session on
2026-11-10 was already weighing one skill for the first; check what it
planned before proposing anything.
- Forge, since 2026-11-06: 65 of 416 merged moq-dev/moq PRs and 14 of 80
merged moq-dev/quest PRs carry "Caused by" in the body, a phrase search.
Use `quest usage`'s parsed links instead.
- Stalls after the stop hook of
[moq-dev/quest#600](https://github.com/moq-dev/quest/pull/600) (landed
2026-11-10T18:34Z): 3.8 per 100 typed messages after it against 6.5
before, on only 27 hours and 3 stalls.
- Hand-run `/quest-complete` after `/quest-run`
([moq-dev/quest#608](https://github.com/moq-dev/quest/pull/608), landed
2026-11-10T20:04Z): 6.6 a day in its first day against 5.2 before, with
3 `/quest-run` runs; `/quest-spawn` fell from 4.1 to 0.9 a day.
- Output handed back unchecked, after the verification rule of
[moq-dev/quest#625](https://github.com/moq-dev/quest/pull/625) landed
2026-11-11T18:43Z: no data yet.
- Heads that landed past a finding or decision, after
[late signals](/quest/m0/late-signals.md) lands.
- Whether moq-dev/moq and moq-dev/moq.pro planned the runtime follow-ups
of moq-dev/quest's 2026-11-11 run PR, and whether their escapes recurred.

Baseline from the 2026-11-11 run (PRs updated 2026-11-09 to 11-11; typed
messages 2026-10-26 to 11-11, classed by one sub-agent, approximate):

- PRs: moq-dev/moq 194 (158 merged), moq-dev/moq.pro 150 (99), and
moq-dev/quest 69 (59). Hand-backs: 4, 20, and 0. 57 of the 316 merged
PRs record implementers that `quest usage` drops, which
[implementer lines](/quest/m0/routing-evidence/implementer-lines.md)
fixes.
- Escapes: 7 real of 20 culprit pairs. Catchable 2, maybe 2, runtime 3;
races 2 (one maybe, one runtime). All 7 culprits passed a local Sol
review. The routing audit's sample of 30 earlier escapes split 6, 17,
and 7, with 5 races.
- Landed past an open signal: 3 in moq-dev/moq, 2 catchable and 1
maybe. A review finding posted after enqueue, a recorded decision the
head lacked, and a red platform run. [Late signals](/quest/m0/late-signals.md) takes the
first two; `quest merge`'s refusal of failed checks
([moq-dev/quest#601](https://github.com/moq-dev/quest/pull/601)) covers
none of them.
- A scoped gate skipped a test the change reached, runtime: 2, one each in
moq-dev/moq and moq-dev/moq.pro.
- A shell pipeline under `pipefail` whose reader exits early, a race: 1
this week in moq-dev/moq.pro, its third fix there.
- Named a PR that deferred the work, or kept older behavior, as a
culprit: 4 in moq-dev/moq, which
[Caused by culprits](/quest/m0/caused-by-culprits.md) takes.
- Typed mid-session messages, per 100, before 2026-11-09 (841) then after
(131): stalls until nudged 6.5 then 4.6; proposals steered simpler 4.8
then 0; misread intent 1.9 then 3.1; reported instead of fixed 0.7 then
2.3; routine choices asked 3.4 then 1.5; answers that only picked the
recommendation 7.7 then 3.8 (`/quest-plan` 11, `/quest-complete` 10,
`/quest-spawn` 6, `/quest-iterate` 1, unknown skill 42). The after
window is under 3 days.
- `quest messages` counts more than the 2026-11-09 hand scan: 1,088
mid-session messages to its 579 over the same days, likely from the
Codex and T3 Code threads and worktree directories it covers.

## Scheduled

- 2026-11-18 - one week after the 2026-11-11 run

## Related

Expand Down
28 changes: 28 additions & 0 deletions quest/m0/question-answers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# [S] quest messages prints the answers to agents' questions

## Goal

`quest messages` prints each answer a person gave to an agent's blocking
question, such as Claude Code's `AskUserQuestion`, Codex's
`request_user_input`, or T3 Code's structured questions, with the option
the agent marked recommended and the option picked. The
[mistake audit](/quest/m0/mistake-audit.md) then tallies how often each
skill's prompts were answered with the recommendation, the measure its
Plan asks for before any change to how agents ask.

## Plan

Today `quest messages` drops most of these answers: T3 Code records them
as server-created (`readT3Messages` in `src/cli/messages.ts`), and Claude
Code returns them as tool results, not human turns. Codex's replies come
through as text, without the option it recommended. The 2026-11-11 audit
found only 6 structured replies among 1,452 messages, and so could count
only typed answers that picked the recommendation.

Agent's choices, for the implementer to revisit:

- Print an answer as its own kind of record beside typed messages, with
the question's header and the skill the session ran, when its log names
one. Keep messages' rules: stdout only, nothing stored or posted.
- Read Codex's question from its log beside the reply `quest messages`
already returns, so its recommended option is known too.
39 changes: 39 additions & 0 deletions quest/m0/routing-evidence/implementer-lines.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# [S] quest usage reads the Implementer lines agents write

## Goal

`quest usage` and `quest review` read every `Implementer:` item in a PR's
`## Models` section the way agents write it: a model's display name maps
to its ID, and a missing effort reads as unknown. A line neither can read
fails loud: `quest usage` reports it on its PR instead of dropping it, and
`quest review` errors as it does for a line naming no model.

## Plan

The 2026-11-11 [mistake audit](/quest/m0/mistake-audit.md) found 57 of 316
PRs merged in moq-dev/moq, moq-dev/moq.pro, and moq-dev/quest since
2026-11-09 with a `## Models` record whose implementers `quest usage`
dropped, nearly all Claude Code sessions. `implementer()` in
`src/github/models.ts` matches the model as `[^,\s]+`, so it rejects
`Claude Code, Opus 5.5, <effort> effort` (36 lines, at every effort or
none),
`Claude Code, Claude Opus 5.5, ...` (9), `Claude, Haiku 5.5, ...`, and
`claude-haiku-5-5, effort not reported`. Every Claude row in usage's
accuracy by implementer, and so
[the writer A/B](/quest/m0/routing-evidence/writer-ab.md), is short.

Agent's choices, for the implementer to revisit:

- Map display names through the models `src/cli/models.ts` already lists,
rather than ask agents to write IDs: a rule would be a prose fix for a
mistake parsing removes.
- `quest review` avoids the latest writer by the model its line names;
check whether `Opus 5.5` failing to match `claude-opus-5-5` let a head
be reviewed by its own writer's model, and fix it with the same mapping.
- `implementers()` and `withModels()` in `src/github/models.ts` have no
caller outside tests; delete them unless this work uses them.

## Related

- [Writer A/B](/quest/m0/routing-evidence/writer-ab.md) - extends the
same line with each writer's arm
5 changes: 5 additions & 0 deletions quest/m0/routing-evidence/usage-token-max.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,11 @@ The audit found today's failures:
quest reads only the parent's log, so 72% of host spend was unattributed.
- `fixedBy` counted context links, such as moq-dev/quest#558 linking #494,
and fixes still open or closed unmerged, such as moq-dev/moq#8476.
The 2026-11-11 [mistake audit](/quest/m0/mistake-audit.md) found 9 of
20 culprit pairs that week were such artifacts: links on a "No single
PR" or "None" line, a "(fixed by #N)" aside, a note after the bullets,
a closed duplicate fix, and an unmerged culprit. Test against
moq-dev/moq#8471, #8426, #8483, #8427, #8371, and #8493.

## Waiting

Expand Down