diff --git a/quest/m0/caused-by-culprits.md b/quest/m0/caused-by-culprits.md new file mode 100644 index 00000000..5627f652 --- /dev/null +++ b/quest/m0/caused-by-culprits.md @@ -0,0 +1,30 @@ +# [S] Caused by names only the PR that shipped the defect + +## Goal + +A fix's `## Caused by` names the PR whose diff introduced the defect, not +one that only touched the same lines, deferred the work to a planned +follow-up, or kept behavior older than it, so the escapes `quest usage` +counts are defects. + +## Plan + +The 2026-11-11 [mistake audit](/quest/m0/mistake-audit.md) found 4 of 14 +moq-dev/moq culprit links named a PR that shipped no defect: +[moq-dev/moq#8294](https://github.com/moq-dev/moq/pull/8294) and +[moq-dev/moq#8440](https://github.com/moq-dev/moq/pull/8440) left the work +to a planned follow-up, +[moq-dev/moq#8362](https://github.com/moq-dev/moq/pull/8362) deferred it on +the maintainer's decision, and +[moq-dev/moq#8402](https://github.com/moq-dev/moq/pull/8402) kept an older +order. `quest context` says to find the culprit "by `git blame` on the +lines the fix changes", which finds the last PR to touch them. + +Agent's choices, for the implementer to revisit: + +- Reword that rule in `assets/CONTEXT.md` to confirm in the blamed PR's + diff that it introduced the defect, else follow blame further back or + say no single PR caused it. This is a prose rule because only judgment + can tell a defect from a deferral. `assets/CONTEXT.md` is at 9,933 of + its 10,000 characters, so the rewording must not grow it. +- Follow `quest guide prompting`. diff --git a/quest/m0/late-signals.md b/quest/m0/late-signals.md new file mode 100644 index 00000000..118a8a20 --- /dev/null +++ b/quest/m0/late-signals.md @@ -0,0 +1,62 @@ +# [M] A finding or decision posted on a head holds its landing + +## Goal + +A PR does not land past a review finding, or a recorded decision, posted +on its head after the review its pick relied on, including one posted +while it waits in a merge queue. `quest merge` refuses such a head, naming +the finding or comment, and a queued PR that draws one leaves the queue +until it is fixed or refuted. + +Non-goal: failed checks on the head, which `quest merge` already refuses +([moq-dev/quest#601](https://github.com/moq-dev/quest/pull/601)), and a +failed run on an earlier head, below. + +## Plan + +The 2026-11-11 [mistake audit](/quest/m0/mistake-audit.md) found 3 of the +week's 7 escapes landed past a signal already on the PR. The cases this +must refuse: + +- [moq-dev/moq#8375](https://github.com/moq-dev/moq/pull/8375): an online + review posted 65 seconds after enqueue described the bug + [moq-dev/moq#8504](https://github.com/moq-dev/moq/pull/8504) later + fixed, and the PR merged 15 minutes after it. +- [moq-dev/moq#8431](https://github.com/moq-dev/moq/pull/8431): a comment + recording the maintainer's decision said the head lacked a timeout, and + the same head was enqueued 10 minutes later, fixed by + [moq-dev/moq#8466](https://github.com/moq-dev/moq/pull/8466). + +Agent's choices, for the implementer to revisit: + +- `quest merge` reads the reviews posted after the head's commit, and + the comments the decision under `## Waiting` picks, and refuses while + one holds a finding or decision the head has not answered. + `quest status` prints them, so `/quest-merge` sees them before it + picks. +- The land watcher re-reads queued PRs and dequeues one that draws a + finding, so CONTRIBUTING's "online findings already posted are addressed + at merge, never waited for" still holds: nothing waits for a reviewer, + but a finding that arrives before the merge takes the PR out of the + queue until it is fixed or refuted. +- [moq-dev/moq#8372](https://github.com/moq-dev/moq/pull/8372) merged with + a red `platform.yml` dispatch run on an earlier head, which no check on + its final head showed. Leave it to moq-dev/moq, since one case is not a + class. + +## Waiting + +- Maintainer: which comments hold a head? Options: (a) any comment by a + write-access account after the head's commit, until a push or a reply + answers it; (b) only comments that open with a fixed marker, such as + "Maintainer decision", which the skills that record a decision on a PR + start writing; (c) only review findings, leaving decision comments to + `/quest-merge`'s prose. Recommend (b): it would have caught + moq-dev/moq#8431, whose comment opened that way, without holding every + chat comment, and a check cannot tell a decision from discussion + otherwise. Check the answer in this Plan. + +## Related + +- [Mistake audit](/quest/m0/mistake-audit.md) - counts heads that landed + past a signal diff --git a/quest/m0/mistake-audit.md b/quest/m0/mistake-audit.md index 62428528..a82ab787 100644 --- a/quest/m0/mistake-audit.md +++ b/quest/m0/mistake-audit.md @@ -28,8 +28,11 @@ and `/reflect`): `quest messages` - Rejected: forge records only; transcripts through a script each run - What gates each run? - - ✅ A week since the last run, as a `## Waiting` item, the way - [the release cut](/quest/m1/release-cut.md) waits + - ✅ A week since the last run, the way + [the release cut](/quest/m1/release-cut.md) waits. The decision named a + `## Waiting` item because quests could not yet be held until a date; + since `## Scheduled` exists, the run holds the date there (agent, + 2026-11-11). - Rejected: each milestone, as [the model check](/quest/m1/model-check.md) runs, since m0 has been open since 2026-09-26; each release cut @@ -38,10 +41,21 @@ and `/reflect`): Agent's choices, for the implementer to revisit. Each run: - Read, since the last run's date: `quest usage --since --json` in - moq-dev/moq, moq-dev/moq.pro, and moq-dev/quest, for each PR's review - rounds, findings fixed and refuted, hand-backs, and the later PRs naming - it under `## Caused by`; the diffs of those culprits and their fixes; - and the typed messages from `quest messages --since --json`. + moq-dev/moq, moq-dev/moq.pro, and moq-dev/quest (each clone's root as + `--root`), for each PR's review rounds, findings fixed and refuted, + hand-backs, and the later PRs naming it under `## Caused by`; the diffs + of those culprits and their fixes; and the typed messages from + `quest messages --since --json`. +- Read each culprit pair's `## Caused by` line before counting it. On + 2026-11-11 only 7 of 20 pairs were escapes: `fixedBy` takes every link + in the section, including "No single PR" lines and unmerged PRs, until + [the token-max attribution](/quest/m0/routing-evidence/usage-token-max.md) + lands; the rest named a PR that left the work to a planned follow-up + or kept older behavior. +- `quest messages` returns every human turn, so set aside machine-written + ones yourself: bare skill invocations, pasted relays and logs, accepted + suggested prompts, `/compact`, `/clear`, and `/goal` commands, and + Codex's question replies (about 175 of 1,452 on 2026-11-11). - Group them into classes by the agent behavior behind them. A class is the same behavior in two or more PRs or messages; one failure is not a class. @@ -53,9 +67,6 @@ Agent's choices, for the implementer to revisit. Each run: owns that gate, so it goes in the run PR's `## Follow-ups` for `/quest-plan` there, as the guide's Creation section says (user, 2026-11-11, [routing evidence](/quest/m0/routing-evidence/README.md)). - The 2026-11-11 audit sampled 30 escapes: catchable 6, maybe 17, and not - from the diff 7. The 5 races fell across those columns, since race and - catchability were classified separately. - Before proposing a fix, find the quest, check, or rule that already owns the class, and report whether its count fell since that landed. A rule whose class still recurs is the case for moving it up a level. @@ -69,39 +80,78 @@ Agent's choices, for the implementer to revisit. Each run: edits no `AGENTS.md`, `CONTRIBUTING.md`, or skill. - Tally the decision prompts per skill: how often the answer was the agent's own recommendation. Propose a change only where that is nearly - always. + always. `quest messages` drops Claude Code's and T3 Code's structured + answers and returns Codex's as text without the recommended option, so + this waits on [question answers](/quest/m0/question-answers.md); until + then, count only typed answers that just pick the recommendation. - moq-dev/quest is public, and typed messages can name customers and private repositories. The PR and its quests give counts and classes in - the agent's words, never a message's text; quotes stay in `.tmp/`. + the agent's words, never a message's text, and name no moq-dev/moq.pro + PR or path; quotes stay in `.tmp/`. - Leave model and effort tallies to [the model check](/quest/m1/model-check.md). - The run's PR rewrites this quest at this path, or at the same path in the lowest open milestone once m0 completes: the new baseline, what to - check first, and a `## Waiting` item for one week after the run's date, - checked with `date`. The first run has no such item. + check first, and a `## Scheduled` date one week after the run's date, + checked with `date`. -Baseline, from the hand scan of 2026-11-09 (typed messages from -2026-10-26 to 2026-11-10, classed by three sub-agents, counts unverified): +Check first next run: -- 579 mid-session messages, about 227 machine-written, about 352 typed, 68 - of them corrections or redirects. -- Stalls until nudged: 17 typed, plus the coordinator relays that the - stop hook of [moq-dev/quest#600](https://github.com/moq-dev/quest/pull/600) - targets; check first whether they fell after it landed. -- A proposal the maintainer steered simpler: 10. -- Misread intent or target: 7. -- Output handed back without running or looking at it: 7, which - the verification rule in `quest context` owns. -- Reports a problem instead of fixing it, or fixes an arbitrary top few: 6. -- Asks a routine choice: 4, beside 14 answers that only picked the - recommendation. -- Repeated by hand: `/quest-complete` or `/quest-spawn` about 36 times, - "any new PRs?" 15, and away mode entered 14 times. A session on - 2026-11-10 was already weighing one skill for the first; check what it - planned before proposing anything. -- Forge, since 2026-11-06: 65 of 416 merged moq-dev/moq PRs and 14 of 80 - merged moq-dev/quest PRs carry "Caused by" in the body, a phrase search. - Use `quest usage`'s parsed links instead. +- Stalls after the stop hook of + [moq-dev/quest#600](https://github.com/moq-dev/quest/pull/600) (landed + 2026-11-10T18:34Z): 3.8 per 100 typed messages after it against 6.5 + before, on only 27 hours and 3 stalls. +- Hand-run `/quest-complete` after `/quest-run` + ([moq-dev/quest#608](https://github.com/moq-dev/quest/pull/608), landed + 2026-11-10T20:04Z): 6.6 a day in its first day against 5.2 before, with + 3 `/quest-run` runs; `/quest-spawn` fell from 4.1 to 0.9 a day. +- Output handed back unchecked, after the verification rule of + [moq-dev/quest#625](https://github.com/moq-dev/quest/pull/625) landed + 2026-11-11T18:43Z: no data yet. +- Heads that landed past a finding or decision, after + [late signals](/quest/m0/late-signals.md) lands. +- Whether moq-dev/moq and moq-dev/moq.pro planned the runtime follow-ups + of moq-dev/quest's 2026-11-11 run PR, and whether their escapes recurred. + +Baseline from the 2026-11-11 run (PRs updated 2026-11-09 to 11-11; typed +messages 2026-10-26 to 11-11, classed by one sub-agent, approximate): + +- PRs: moq-dev/moq 194 (158 merged), moq-dev/moq.pro 150 (99), and + moq-dev/quest 69 (59). Hand-backs: 4, 20, and 0. 57 of the 316 merged + PRs record implementers that `quest usage` drops, which + [implementer lines](/quest/m0/routing-evidence/implementer-lines.md) + fixes. +- Escapes: 7 real of 20 culprit pairs. Catchable 2, maybe 2, runtime 3; + races 2 (one maybe, one runtime). All 7 culprits passed a local Sol + review. The routing audit's sample of 30 earlier escapes split 6, 17, + and 7, with 5 races. +- Landed past an open signal: 3 in moq-dev/moq, 2 catchable and 1 + maybe. A review finding posted after enqueue, a recorded decision the + head lacked, and a red platform run. [Late signals](/quest/m0/late-signals.md) takes the + first two; `quest merge`'s refusal of failed checks + ([moq-dev/quest#601](https://github.com/moq-dev/quest/pull/601)) covers + none of them. +- A scoped gate skipped a test the change reached, runtime: 2, one each in + moq-dev/moq and moq-dev/moq.pro. +- A shell pipeline under `pipefail` whose reader exits early, a race: 1 + this week in moq-dev/moq.pro, its third fix there. +- Named a PR that deferred the work, or kept older behavior, as a + culprit: 4 in moq-dev/moq, which + [Caused by culprits](/quest/m0/caused-by-culprits.md) takes. +- Typed mid-session messages, per 100, before 2026-11-09 (841) then after + (131): stalls until nudged 6.5 then 4.6; proposals steered simpler 4.8 + then 0; misread intent 1.9 then 3.1; reported instead of fixed 0.7 then + 2.3; routine choices asked 3.4 then 1.5; answers that only picked the + recommendation 7.7 then 3.8 (`/quest-plan` 11, `/quest-complete` 10, + `/quest-spawn` 6, `/quest-iterate` 1, unknown skill 42). The after + window is under 3 days. +- `quest messages` counts more than the 2026-11-09 hand scan: 1,088 + mid-session messages to its 579 over the same days, likely from the + Codex and T3 Code threads and worktree directories it covers. + +## Scheduled + +- 2026-11-18 - one week after the 2026-11-11 run ## Related diff --git a/quest/m0/question-answers.md b/quest/m0/question-answers.md new file mode 100644 index 00000000..907b95f1 --- /dev/null +++ b/quest/m0/question-answers.md @@ -0,0 +1,28 @@ +# [S] quest messages prints the answers to agents' questions + +## Goal + +`quest messages` prints each answer a person gave to an agent's blocking +question, such as Claude Code's `AskUserQuestion`, Codex's +`request_user_input`, or T3 Code's structured questions, with the option +the agent marked recommended and the option picked. The +[mistake audit](/quest/m0/mistake-audit.md) then tallies how often each +skill's prompts were answered with the recommendation, the measure its +Plan asks for before any change to how agents ask. + +## Plan + +Today `quest messages` drops most of these answers: T3 Code records them +as server-created (`readT3Messages` in `src/cli/messages.ts`), and Claude +Code returns them as tool results, not human turns. Codex's replies come +through as text, without the option it recommended. The 2026-11-11 audit +found only 6 structured replies among 1,452 messages, and so could count +only typed answers that picked the recommendation. + +Agent's choices, for the implementer to revisit: + +- Print an answer as its own kind of record beside typed messages, with + the question's header and the skill the session ran, when its log names + one. Keep messages' rules: stdout only, nothing stored or posted. +- Read Codex's question from its log beside the reply `quest messages` + already returns, so its recommended option is known too. diff --git a/quest/m0/routing-evidence/implementer-lines.md b/quest/m0/routing-evidence/implementer-lines.md new file mode 100644 index 00000000..0fba71fc --- /dev/null +++ b/quest/m0/routing-evidence/implementer-lines.md @@ -0,0 +1,39 @@ +# [S] quest usage reads the Implementer lines agents write + +## Goal + +`quest usage` and `quest review` read every `Implementer:` item in a PR's +`## Models` section the way agents write it: a model's display name maps +to its ID, and a missing effort reads as unknown. A line neither can read +fails loud: `quest usage` reports it on its PR instead of dropping it, and +`quest review` errors as it does for a line naming no model. + +## Plan + +The 2026-11-11 [mistake audit](/quest/m0/mistake-audit.md) found 57 of 316 +PRs merged in moq-dev/moq, moq-dev/moq.pro, and moq-dev/quest since +2026-11-09 with a `## Models` record whose implementers `quest usage` +dropped, nearly all Claude Code sessions. `implementer()` in +`src/github/models.ts` matches the model as `[^,\s]+`, so it rejects +`Claude Code, Opus 5.5, effort` (36 lines, at every effort or +none), +`Claude Code, Claude Opus 5.5, ...` (9), `Claude, Haiku 5.5, ...`, and +`claude-haiku-5-5, effort not reported`. Every Claude row in usage's +accuracy by implementer, and so +[the writer A/B](/quest/m0/routing-evidence/writer-ab.md), is short. + +Agent's choices, for the implementer to revisit: + +- Map display names through the models `src/cli/models.ts` already lists, + rather than ask agents to write IDs: a rule would be a prose fix for a + mistake parsing removes. +- `quest review` avoids the latest writer by the model its line names; + check whether `Opus 5.5` failing to match `claude-opus-5-5` let a head + be reviewed by its own writer's model, and fix it with the same mapping. +- `implementers()` and `withModels()` in `src/github/models.ts` have no + caller outside tests; delete them unless this work uses them. + +## Related + +- [Writer A/B](/quest/m0/routing-evidence/writer-ab.md) - extends the + same line with each writer's arm diff --git a/quest/m0/routing-evidence/usage-token-max.md b/quest/m0/routing-evidence/usage-token-max.md index 6b9dab46..7ad9e59e 100644 --- a/quest/m0/routing-evidence/usage-token-max.md +++ b/quest/m0/routing-evidence/usage-token-max.md @@ -39,6 +39,11 @@ The audit found today's failures: quest reads only the parent's log, so 72% of host spend was unattributed. - `fixedBy` counted context links, such as moq-dev/quest#558 linking #494, and fixes still open or closed unmerged, such as moq-dev/moq#8476. + The 2026-11-11 [mistake audit](/quest/m0/mistake-audit.md) found 9 of + 20 culprit pairs that week were such artifacts: links on a "No single + PR" or "None" line, a "(fixed by #N)" aside, a note after the bullets, + a closed duplicate fix, and an unmerged culprit. Test against + moq-dev/moq#8471, #8426, #8483, #8427, #8371, and #8493. ## Waiting