Skip to content

feat(land): size batches from the measured failure rate, and fill past a base conflict - #376

Closed
kixelated wants to merge 4 commits into
mainfrom
quest/m0/land-fixer/tuning
Closed

kixelated wants to merge 4 commits into
mainfrom
quest/m0/land-fixer/tuning

Conversation

@kixelated

@kixelated kixelated commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Part of Throughput tuning in the land fixer epic.

Summary

  • Batch size from the measured failure rate (src/cli/land-policy.ts, new). After each checked batch, the daemon estimates the per-PR failure rate p by maximum likelihood over the last 30 checked batches: a batch of n PRs passes with probability (1 − p)^n. It then sizes the next batch to maximize the expected PRs landed per check, n(1 − p)^n, which peaks at n = −1/ln(1 − p). The existing cap of 1 + floor(log2 queue) still applies. batch-policy.json stays the state and now also holds history: [{size, passed}]. Halving on each failure is gone.
  • A base conflict no longer costs a round (build in src/cli/land.ts, serveBatch in src/cli/land-serve.ts). Before, a PR that would start the batch but conflicted with the base stopped the batch: it was parked, and the round ended without a check. Now build sets each such PR aside and tries the next one. serveBatch parks each set-aside PR for the fixer (or ejects it, without a fixer), the same way rejections already are, then checks the batch it built in the same round. Each check log is now named after the batch's own base PR.
  • README, quest land serve --help, the epic entry, and the quest are updated. The quest is rewritten to what remains (see below).

Numbers (moq-dev/moq gate, nixos:/var/lib/quest-land-moq/moq-dev-moq)

Sources: the 387 batch-*.log files and their tails (pass, error: recipe fail, or Conflicts with the base); the Failure log:/PRs: lines of 222 fixer job logs; and journalctl -u quest-land-moq from 2026-10-30. A passed batch's size is the count of landed events after its check. A failed batch's size is its fixer job's PR list, or, before parking, the burst of ejected events right after its check.

Window Checked batches (known size) PRs checked Batch failure rate Per-PR failure rate p (MLE) Mean check
2026-10-30 18:59 → 11-01 13:11 PST (before parking, 43 h) 164 536 48% 22.1% 11.4 min
2026-10-31 13:11 → 11-01 15:21 PST (last 27 h) 145 361 44% 24.3% 7.7 min
2026-11-01 13:11 → 15:21 PST (after #355's parking, 2.1 h) 11 26 45% 24.4% 9.9 min
  • Check time doesn't depend on size in the last 27 h. Median minutes by batch size: 6.9 (1 PR), 6.8 (2), 5.9 (3), 7.5 (4), 8.2 (5), 9.0 (6). So per-check throughput is the right thing to maximize.
  • Halving measured against the model. Simulating halving at p = 0.22 gives 1.16 PRs landed per check. The gate actually landed 196 PRs in 171 checks (1.15). The model matches the gate.
  • Base-conflict rounds: 156 in the last 27 h, and 183 over 45 h at a median 23 s each (1.3 h of gate time, about 0.7 h a day). These rounds ran no check, and this PR removes them.

Expected throughput

From a simulation of 40,000 rounds with the 1 + floor(log2 88) = 7 cap:

Policy p = 22.1% p = 24.3%
One PR per check (today's {"size":1}) 0.78 / check 0.76 / check
Halve on failure, +1 on a pass (before this PR) 1.16 / check 1.06 / check
This PR (maximum likelihood over 30 batches) 1.43 / check 1.28 / check
Best fixed size (n = 4, which an exact p would give) 1.47 / check 1.31 / check

At the 10–17 minute rounds seen today (~5 checks an hour), that is about 6.4–7 PRs an hour, against 3.8 at one PR per check and 5.3–5.8 with halving. That's about 1.7× today's rate and 1.2× halving's. The 88-PR queue would drain in about 13–14 hours rather than about 23. Simulated windows of 10/20/30/50/100 batches gave 1.38/1.43/1.45/1.46/1.47 PRs per check at p = 0.22. A window of 30 is about 3.5–6 hours of checks: within 2% of the best fixed size, and still quick to follow a changed rate.

Choices made without asking

  • Maximum likelihood over a fixed 30-batch window, no prior. With an empty history (as on deploy), the size stays whatever batch-policy.json holds. If the first batch fails, p reads as 1 and the next batch is one PR, until a pass arrives; the estimate settles within about ten batches. Simulations include this cold start.
  • Every failed check counts, including a red base or a flake. That inflates p only for as long as those batches stay in the window. quest check rejections are screened out before the check and don't count.
  • Only conflicts found before the batch has a base PR are parked. A later candidate that conflicts with the base is still skipped silently, as before, and is parked once it would start a batch. Parking several base conflicts in one round launches the same fixer jobs as one per ~23-second round did, only sooner.
  • The quest is rewritten, not deleted. It keeps two items:
    • The fixer's priority, which is host configuration in kixelated/nixos#16. Neither the fixer nor the check units set CPUWeight/IOWeight today. The maintainer's session pushed there minutes before this work, so I left it alone.
    • rerere, which looks like a no-op (below).
  • Dropped from this PR: the reasoning effort (moq-dev/quest#370 set medium) and the fixer cap. The host's quest-land-fixer.socket already accepts at most three fixers across every gate (kixelated/nixos@aa155d1).

Open question for the maintainer (does not block this PR)

rerere (recommend dropping). 43 of the 89 PRs that conflicted with the base conflicted more than once, but only 20 of the 96 repeats named the same files as the PR's previous conflict. More to the point, the gate never resolves a conflict: squash resets on any conflict, and the fixer resolves in a fresh container clone. The gate's rr-cache would stay empty. Making it replay would mean training rerere from the fixer's merges and letting the gate apply them in its own squashes. The gate would then land resolutions the PR's head never had. The rewritten quest records this and recommends deleting the item.

Deploying

No migration. The live {"order":"oldest","size":2} reads as size 2 with no history yet. The first checked batch after the deploy starts the measurement.

Follow-ups

  • The fixer job's 60-minute timeout runs from launch, so it includes time waiting behind the host's three-fixer socket cap. A burst of parked PRs can time out before it starts. This PR doesn't change how many jobs launch, only how close together.
  • Codex's round-1 finding on moq-dev/quest#348: src/cli/agent.ts:665 may launch a claude found through a relative PATH entry on POSIX. It came from main moving mid-review, not from this PR.

Checks

just check and just test pass on the merged head 389140a (823 passed, 2 skipped; worker suite 426 passed). Two agent.test.ts cases fail only when the caller's shell exports CLAUDE_CONFIG_DIR, both on this branch and on main's; with it unset, everything passes.

Review

mechanical fix by codex / Codex: Merged base and re·solved src/cli/main.ts help-text conflict preserving both intents; passed bun install --frozen-lockfile, bun run build, bun run check, bun run test (836 passed, 2 skipped), and bun dist/quest.js check. (head 09bc0bf)

  • Reviewer: Codex gpt-6.1-sol, medium effort, via quest review --base origin/main, session 01a1997d-e138-7d20-8816-a565b613e2c7.
  • Round 1, head e39b1d3: one finding, refuted. It concerned src/cli/agent.ts:665, which this branch doesn't touch; main moved to 97ed0ff (feat(agent): run billed agents on native Windows #348) during the round. The branch then merged main (389140a).
  • Round 2, head 389140a: no findings.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 3 commits November 1, 2026 15:21
…t a base conflict

The next batch's size now maximizes the expected PRs landed per check,
n(1 - p)^n, at the per-PR failure rate p estimated by maximum likelihood
over the last 30 checked batches, kept in batch-policy.json. Halving on
each failure held moq at one to three PRs at its measured 22-24% rate,
where four lands the most.

A PR that would start the batch but conflicts with the base is parked for
the fixer (ejected without one), and the batch starts from the next PR in
the same round instead of spending a round on the conflict.

The quest keeps what remains: the fixer's priority on the host, and
whether rerere can replay anything at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Tune batching and the fixer for throughput feat(land): size batches from the measured failure rate, and fill past a base conflict Nov 1, 2026
@kixelated
kixelated marked this pull request as ready for review November 1, 2026 23:47
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0a905ceb-21a8-48f4-b680-d47a12be55a9
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated
kixelated enabled auto-merge (squash) November 1, 2026 23:48
@moq-bot
moq-bot Bot disabled auto-merge November 2, 2026 00:22
@moq-bot
moq-bot Bot enabled auto-merge (squash) November 2, 2026 00:22
@kixelated
kixelated disabled auto-merge November 2, 2026 00:33
@kixelated

Copy link
Copy Markdown
Collaborator Author

Closing. On 2026-11-01 the maintainer decided to move moq-dev/moq (first) back to GitHub's merge queue, serialized, with the Codex land fixer fixing every failure asynchronously and re-enqueuing it. Work on the host gate is frozen apart from what the switch needs, so this batching improvement won't land. The measurements here (failure rate, check time, base-conflict rounds) inform that plan.

(Written by Claude Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant