Skip to content
74 changes: 60 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -738,10 +738,12 @@ the unreported required `quest land` check must be verified on the live host.

On a base that lands through GitHub's merge queue, `quest land watch` runs on a
maintainer's host, one process per repository, and tells each PR the queue
drops unmerged why. It polls; the host needs no public endpoint.
drops unmerged why. With `--fixer`, it also runs the land fixer on each PR the
queue dropped for a failure or a conflict, and puts it back at the back of the
queue. It polls; the host needs no public endpoint.

```sh
QUEST_LAND_KEY=/run/credentials/quest-land/key quest land watch --app-id 1039466
QUEST_LAND_KEY=/run/credentials/quest-land/key quest land watch --app-id 1039466 --fixer /path/to/fixer-wrapper
```

Once a minute it reads the base's open PRs and their merge queue removals since
Expand All @@ -752,18 +754,62 @@ quotes each failing job, its failed step, and its log's tail; at build
concurrency 1 a group holds only that PR, so the failure is its own. A merge,
and the App's own removals, get no comment.

Each pick starts with an enqueue by a writer; the App re-enqueues only within
one. A failed build or a conflict is recorded for the land fixer in
`removal-<n>.json`, with the full log beside it. A push since the removal, a
person's removal, or a reason the watcher does not know ends the pick, and the
comment says the PR waits for a maintainer to enqueue it again. The state
directory remembers each removal handled, so a restart neither misses nor
repeats one; a crash between commenting and recording finds its own comment's
marker and posts nothing. The first pass reads the ten minutes before it.

It shares `quest land serve`'s App key, state directory, and locks, so the two
never run for the same repository at once. Its token asks for `contents: read`,
`pull_requests: write`, and `actions: read`, which the App needs for the logs.
Each pick starts with an enqueue, by a writer or by the App, as when a
workflow enqueues Dependabot's PRs with the App's token. An App enqueue of a
head the watcher itself put back in the queue continues the watcher's pick
instead, until a handback ends it: the watcher's re-enqueue, GitHub's
auto-merge of it, or such a workflow picking the fixer's push. A failed build
or a conflict is recorded for the land fixer in `removal-<n>.json`, with the
full log beside it. A push since the removal, a person's removal, or a reason
the watcher does not know ends the pick, and the comment says the PR waits for
a maintainer to enqueue it again. The state directory remembers each removal
handled, so a restart neither misses nor repeats one; a crash between
commenting and recording finds its own comment's marker and posts nothing. The
first pass reads the ten minutes before it.

With `--fixer`, the watcher takes each recorded removal, oldest first, while
its PR still waits as the queue dropped it:

- A conflict is first merged with the current base by the quest merge driver,
in the daemon's private repository, as the host gate merges a base. A clean
merge needs no fixer: the watcher lease-pushes it to the PR branch, records
it in the PR body's Review, and re-enqueues it.
- Otherwise the fixer runs in the background, as for the host gate: the same
credential-free wrapper, given the failure log and the PR's number, with the
dropped head on `pr-<n>`, starting from the base the `merge_group` build
was built on, or from the current base for a conflict, with the merge's
conflicts as its log. At most `--fixer-jobs` (2) run at once; the rest wait
for a slot. `--fixer-effort` (medium) sets the fixer's
`model_reasoning_effort`, passed to the wrapper in `QUEST_LAND_FIXER_EFFORT`,
which `quest land fix --effort` reads by default.

Each finished round is recorded in its pick's history, as the host gate
records its rounds: `fixed` for a fixer commit, `merged` for a clean merge,
`flaky` for a failed build the fixer requeued unchanged, and `unchanged` for a
conflict it requeued unchanged. Its failure signature is the test or step the
fixer names, falling back to the failing job and step, or, for a conflict, the
base it conflicted with. The pick keeps going while each round makes progress:
a round with no new commit for a failure that isn't flaky, the failure a fix
addressed recurring right after it, or three flaky rounds in a row hand it
back. A fix is validated, lease-pushed, and recorded in Review as for the host
gate, and a push GitHub refuses is handed back with its resolution for
`quest land apply`. The watcher then puts the fixed, merged, or flaky head back
in the queue with `expectedHeadOid` bound to it: it enqueues a head whose
required checks have passed, and arms auto-merge on one whose checks still
run, so GitHub enqueues it once they pass. Either joins the back of the queue.
It comments on the PR with what it did, after any note the fixer wrote. A
handback, from a stop rule, the fixer, a refusal, a crash, or a timeout, is
posted with its reason and a recommendation, and the PR leaves the watcher's
care until a maintainer enqueues it again. A round whose pick ended while it
ran, by a writer's push or enqueue, is not applied, and its comment says so.

It shares `quest land serve`'s App key, state directory, private repository,
and locks, so the two never run for the same repository at once. Without a
fixer its token asks for `contents: read`, `pull_requests: write`, and
`actions: read`, which the App needs for the logs. With one it also asks for
`merge_queues: write`, which `enqueuePullRequest` needs, and `contents: write`,
for arming auto-merge and pushing fixes; a push that carries a workflow change
asks for `workflows: write`.

## Gate expensive commands

Expand Down
4 changes: 2 additions & 2 deletions quest/m0/land-fixer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,8 +140,8 @@ Decision prompts, selected options marked ✅:
pick GitHub refuses fails `quest merge` instead), and flaky tests (now
[Flaky failures](/quest/m0/queue-watcher/flaky.md), without the rerun).
Deleted: the batch quest check, pre-checks, and throughput tuning, which
only served batches (the fixer cap and reasoning effort moved to
[Fix and re-enqueue](/quest/m0/queue-watcher/fix.md)); and queue status in
only served batches (the fixer cap and reasoning effort moved to the
watcher, as `quest land watch --fixer-jobs` and `--fixer-effort`); and queue status in
the required check, since GitHub's merge queue shows each PR's position
itself.

Expand Down
8 changes: 3 additions & 5 deletions quest/m0/land-fixer/pr-signal.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,6 @@ refuses such a head, `quest merge` fails when GitHub refuses it, and either
way the PR needs `/quest-iterate`.

Retargeted 2026-11-01 to the [queue watcher](/quest/m0/queue-watcher/README.md),
which replaces the batched host gate.

## Related

- [Fix and re-enqueue](/quest/m0/queue-watcher/fix.md) - where the fixer uses the signal
which replaces the batched host gate. The watcher starts the fixer on each PR
the queue drops in `take` (`src/cli/land-watch-fix.ts`), with the removal's
`merge_group` log; the PR's own checks go beside it.
1 change: 0 additions & 1 deletion quest/m0/queue-watcher/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,6 @@ Shared shape, for the children to follow:

## Required

- [Fix and re-enqueue](/quest/m0/queue-watcher/fix.md) - the fixer runs on each dropped PR in the background, and the watcher re-enqueues it at the back
- [Bad picks](/quest/m0/queue-watcher/bad-picks.md) - a pick whose auto-merge is armed on a ready PR GitHub never queued is enqueued at the head it was armed on
- [Flaky failures](/quest/m0/queue-watcher/flaky.md) - a flaky failure is re-enqueued unchanged, and a test seen flaking twice is disabled in a separate PR with a follow-up quest
- [Watcher on the host](/quest/m0/queue-watcher/host.md) - kixelated/nixos runs the watcher for moq-dev/moq beside its fixer container
Expand Down
4 changes: 4 additions & 0 deletions quest/m0/queue-watcher/bad-picks.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,7 @@ decision 8).
- An armed pick whose own required checks fail, or that turns conflicting,
is left alone: it needs `/quest-iterate` and a new pick, and
`quest status` shows its checks.
- The watcher's own re-enqueues are among these: it arms auto-merge on a head
the fixer pushed, or merged with the quest driver, while that head's
required checks run (`requeue` in `src/cli/land-watch-fix.ts`), and GitHub
may leave it out of the queue as it does any armed pick.
52 changes: 0 additions & 52 deletions quest/m0/queue-watcher/fix.md

This file was deleted.

4 changes: 0 additions & 4 deletions quest/m0/queue-watcher/flaky.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,3 @@ naming the follow-up quest: `#[ignore = "flaky: /quest/m0/flaky/<name>.md"]`,
fix the test or delete it, and links the `merge_group` runs that showed the
flake. A disabled test that is a quest's acceptance test still counts as its
quest's work.

## Required

- [Fix and re-enqueue](/quest/m0/queue-watcher/fix.md) - the fixer's flaky verdict and the re-enqueue
26 changes: 23 additions & 3 deletions quest/m0/queue-watcher/host.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,29 @@ Decided 2026-11-01 (maintainer, /quest-plan; see the
- The runner's own setup and the `merge_group` workflow are moq's switch,
not this quest.

## Required

- [Fix and re-enqueue](/quest/m0/queue-watcher/fix.md) - the watcher command this deploys
The watcher command, from
[moq-dev/quest#389](https://github.com/moq-dev/quest/pull/389), run as the
same user and with the same state directory as moq's `quest land serve`
instance, and with `quest` and `flock` on PATH as for it, since the watcher
merges conflicts with the quest merge driver in the same private repository:

```sh
quest land watch --app-id 1039466 --fixer <moq's fixer launcher client>
```

The fixer launcher client is the one `serve --fixer` uses, reaching
`/run/quest-land-fixer.sock` with the failure log and the PR's number. The
watcher needs no check sandbox: the `merge_group` builds run on GitHub's side.
`--fixer-jobs` (default 2) caps the watcher's own fixer jobs below the
launcher's shared slots, and `--fixer-effort` (default `medium`) sets
`QUEST_LAND_FIXER_EFFORT` in the wrapper's environment. The launcher passes
only the log and the PR numbers into the container, where `quest land fix`
defaults to `medium`, so a non-default effort needs the launcher to forward
that variable as `quest land fix --effort`. With a fixer, the watcher's
installation token asks for `merge_queues: write` (re-enqueueing, granted to
moq-bot on 2026-11-02), `contents: write` (arming auto-merge and pushing
fixes), `pull_requests: write`, and `actions: read`, and a push that carries a
workflow change asks for `workflows: write`.

## Related

Expand Down
66 changes: 40 additions & 26 deletions src/cli/hold.ts
Original file line number Diff line number Diff line change
Expand Up @@ -301,41 +301,55 @@ export async function land(
const PENDING = ["BLOCKED", "UNKNOWN"];

/** GraphQL error types that fault the token or the request, not the PR GitHub would enqueue. */
const UNREFUSED = ["FORBIDDEN", "NOT_FOUND", "INSUFFICIENT_SCOPES", "INTERNAL"];
export const UNREFUSED = ["FORBIDDEN", "NOT_FOUND", "INSUFFICIENT_SCOPES", "INTERNAL"];

/**
* Pick `snapshot`'s head to land through the base's merge queue, as {@link land} checked it:
* enqueue that head, or arm auto-merge bound to it while its required checks may be pending.
* The queue builds each PR on the current base, so a head behind it is enqueued as is.
* GitHub's queue events and auto-merge request record the pick, naming who made it.
* A pick GitHub refuses, such as one that conflicts or fails its required checks, fails:
* the PR needs a fix before it can land.
* Put `head` in the base's merge queue: enqueue it, or, while GitHub may still be running its
* required checks (`mergeState` pending), arm auto-merge bound to it, which enqueues it once they
* pass. Either joins the back of the queue. A refusal throws GitHub's error.
*/
async function pick(labels: Labels, snapshot: Snapshot, method: Method): Promise<void> {
export async function queueHead(
github: GitHub,
id: string,
head: string,
mergeState: string,
method: Method = "merge",
): Promise<void> {
const enqueuePull = () =>
labels.github.graphql(
github.graphql(
`mutation($id: ID!, $sha: GitObjectID!) {
enqueuePullRequest(input: { pullRequestId: $id, expectedHeadOid: $sha }) { clientMutationId }
}`,
{ id: snapshot.id, sha: snapshot.head },
{ id, sha: head },
);
if (!PENDING.includes(mergeState)) {
await enqueuePull();
return;
}
try {
if (!PENDING.includes(snapshot.mergeState)) {
await enqueuePull();
return;
}
try {
await labels.github.graphql(
`mutation($id: ID!, $sha: GitObjectID!, $method: PullRequestMergeMethod!) {
enablePullRequestAutoMerge(input: { pullRequestId: $id, expectedHeadOid: $sha, mergeMethod: $method }) { clientMutationId }
}`,
{ id: snapshot.id, sha: snapshot.head, method: method.toUpperCase() },
);
} catch (error) {
// An UNKNOWN state GitHub finished computing as ready: nothing is left to wait for.
if (!cleanAutoMergeRefusal(error instanceof Error ? error.message : null)) throw error;
await enqueuePull();
}
await github.graphql(
`mutation($id: ID!, $sha: GitObjectID!, $method: PullRequestMergeMethod!) {
enablePullRequestAutoMerge(input: { pullRequestId: $id, expectedHeadOid: $sha, mergeMethod: $method }) { clientMutationId }
}`,
{ id, sha: head, method: method.toUpperCase() },
);
} catch (error) {
// An UNKNOWN state GitHub finished computing as ready: nothing is left to wait for.
if (!cleanAutoMergeRefusal(error instanceof Error ? error.message : null)) throw error;
await enqueuePull();
}
}

/**
* Pick `snapshot`'s head to land through the base's merge queue, as {@link land} checked it, with
* {@link queueHead}. The queue builds each PR on the current base, so a head behind it is enqueued
* as is. GitHub's queue events and auto-merge request record the pick, naming who made it.
* A pick GitHub refuses, such as one that conflicts or fails its required checks, fails:
* the PR needs a fix before it can land.
*/
async function pick(labels: Labels, snapshot: Snapshot, method: Method): Promise<void> {
try {
await queueHead(labels.github, snapshot.id, snapshot.head, snapshot.mergeState, method);
} catch (error) {
if (!(error instanceof GitHubGraphQLError) || error.types.some((type) => UNREFUSED.includes(type ?? "")))
throw error;
Expand Down
12 changes: 11 additions & 1 deletion src/cli/land-fix.ts
Original file line number Diff line number Diff line change
Expand Up @@ -152,14 +152,24 @@ export function parseBaseFix(text: string): BaseFix {
return value as BaseFix;
}

/** Codex's reasoning efforts, for `model_reasoning_effort`. */
export const EFFORTS = ["minimal", "low", "medium", "high", "xhigh"];
/**
* The variable a daemon sets for its fixer wrapper to pass on as `quest land fix --effort`, since the
* wrapper's arguments are only the failure log and the PR numbers.
*/
export const EFFORT_VARIABLE = "QUEST_LAND_FIXER_EFFORT";

export async function fix(
root: string,
log: string,
numbers: number[],
agent = "codex",
args?: string[],
model?: string,
effort = "medium",
): Promise<void> {
if (!EFFORTS.includes(effort)) throw new Error(`fix effort must be one of ${EFFORTS.join(", ")}`);
// No PR numbers fixes the base alone, which fails its check without any PR.
if (
numbers.some((number) => !Number.isSafeInteger(number) || number <= 0) ||
Expand Down Expand Up @@ -216,7 +226,7 @@ export async function fix(
'model_providers.quest-fixer.wire_api="responses"',
// Without this, Codex runs the proxy's model with no reasoning at all ("reasoning effort: none").
"-c",
'model_reasoning_effort="medium"',
`model_reasoning_effort=${JSON.stringify(effort)}`,
...(model ? ["--model", model] : []),
"-",
];
Expand Down
Loading
Loading