From 870b9e91e3e0a42fd18a46dd7bbf244e518d0b43 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 22:10:55 -0800 Subject: [PATCH 01/10] chore(quest): claim quest/m1/bench-ci.md Quest-Claim: 619e8968e2f89a78 From 29cfa66af78392a4d854710a714ce6bc70947095 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 22:17:14 -0800 Subject: [PATCH 02/10] bench: say each paired revision needs its own target directory Co-Authored-By: Claude Opus 5.5 --- bench/README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/bench/README.md b/bench/README.md index 6566fa19ad9..5ca98a69227 100644 --- a/bench/README.md +++ b/bench/README.md @@ -904,6 +904,11 @@ Each run's evidence directory keeps both builds' estimates (`criterion` and `criterion-base`), the confirmation rounds under `confirm/`, and the run's `paired.json`. +Each revision builds in its own Cargo target directory (`base` and `current` +under `CARGO_TARGET_DIR`, default `target/bench`). An A/B outside `run.sh` +uses its base mode or a target directory per source tree: two trees sharing +one can leave a bench linked against the other tree's library. + ### CodSpeed The CodSpeed workflow adds instruction counts for every `main` commit that From 61d5887a070becae2d1852021002d787a3ee2bf0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 00:12:31 -0800 Subject: [PATCH 03/10] bench: calibrate the paired comparison from A/A and slowed-bench runs Two A/A Nightlies and one deliberately slowed-bench Nightly hold the 10% threshold, 20 confirmations and 3 rounds for the shared, supplemental and io_uring groups. Record them, and leave the quest to the moq-net and relay groups and the bias that makes rounds agree on unchanged code. Co-Authored-By: Claude Opus 5.5 --- bench/README.md | 12 +++- bench/compare.py | 2 +- quest/m1/README.md | 2 +- quest/m1/bench-ci.md | 145 ++++++++++++++++++------------------------- 4 files changed, 72 insertions(+), 89 deletions(-) diff --git a/bench/README.md b/bench/README.md index 5ca98a69227..8f0a5e88b93 100644 --- a/bench/README.md +++ b/bench/README.md @@ -883,8 +883,16 @@ measurement, 30 samples). A case whose rounds are all slower with a median beyond 10% is a confirmed regression and fails its job after the upload, so the Alert workflow posts the failed Nightly. A target the base cannot build or run fails the job too, rather than passing uncompared; a bench target the base does -not have yet is only reported as new. The threshold is a starting point until -A/A runs on the hosted runner measure its noise floor. +not have yet is only reported as new. + +Three runs on the hosted runner set these numbers. In two A/A runs of the +shared, supplemental and io_uring groups, 2 to 4% of cases crossed 10% either +way in the screen, at most 17 of them slower, and the rounds rejected every +one; no round set that agreed on a slowdown had a median above 6%. A third run, +whose base cut one case's work by a quarter, failed with that case alone at ++35%. Its rounds also agreed, at medians of +9.3 to +9.7%, on `stats/tick` cases whose source +was identical on both sides, so 10% is a floor rather than a margin. The +moq-net and relay groups are not calibrated yet. The table page shows tonight's comparison above the trend: confirmed regressions, targets not compared, cases faster in the screen, slowdowns the diff --git a/bench/compare.py b/bench/compare.py index 9f5b66f6786..00de468d467 100644 --- a/bench/compare.py +++ b/bench/compare.py @@ -157,7 +157,7 @@ def criterion_baseline(directory, discovery, benchmark): # A paired case beyond this change is flagged, and fails the nightly only when its rounds agree. -# Loose until A/A runs on the hosted runner measure its noise floor. +# Rounds on unchanged code have agreed at up to +9.7% on the hosted runner, so lower is unsafe. THRESHOLD_PCT = 10 # Each alternating round costs seconds per case, so a noisy night confirms only its worst slowdowns. CONFIRMATIONS = 20 diff --git a/quest/m1/README.md b/quest/m1/README.md index 23dc7549706..fadfa1c9064 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -171,7 +171,7 @@ their end report, so an embedder's drain alert pages on real cuts. - [crates.io trusts the workflow](/quest/m1/crates-trusted-publisher.md) - maintainer action: every published crate gets a trusted publisher for `release-rs.yml` - [Rust releases publish through OIDC](/quest/m1/release-rs-oidc.md) - release-plz publishes with trusted publishing and `CARGO_REGISTRY_TOKEN` is revoked - [Main is no slower than release](/quest/m1/release-perf/README.md) - every benchmark where main is slower than release is fixed or accepted with a reason, checked by a final paired run before the cut -- [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - A/A and deliberately slowed Nightly runs set the paired comparison's threshold, so its regression alerts can be trusted +- [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - the moq-net and relay groups get A/A runs and the bias that makes rounds agree on unchanged code is removed, so every paired regression alert can be trusted - [moq-net's benches run to completion](/quest/m1/bench-test-mode.md) - the `origin`, `session` and `memory` benches stop panicking in test mode and Nightly's measured run, and the maintainer picks where CI catches the next break - [Catalog contributions become includes on the root's timeline](/quest/m1/catalog-include.md) - rename the unreleased `contributions` map before the cut and state that included broadcasts share the root's timeline - [Unbounded groups](/quest/m1/unbounded-groups/README.md) - groups and frames lose their size caps; a group keeps a 32 MB tail, and readers behind it replay through paced flights or refresh diff --git a/quest/m1/bench-ci.md b/quest/m1/bench-ci.md index d2d41824766..f8d519aefb0 100644 --- a/quest/m1/bench-ci.md +++ b/quest/m1/bench-ci.md @@ -4,69 +4,73 @@ The paired Rust Criterion comparison each Nightly runs (see [Paired comparison](/bench/README.md#paired-comparison)) fails only on real -regressions: its threshold, confirmation budget, and round count come from A/A -runs on the hosted ARM runners, and a deliberately slowed bench fails the job. +regressions in every paired group: A/A runs on the hosted ARM runners hold its +threshold, confirmation budget, and round count for moq-net (`rust-bench`) +and `rust-relay` too, and the rounds stop agreeing on slowdowns in code that +did not change. ## Waiting -- Maintainer: dispatch the slowed-bench Nightly run below. Agents never - dispatch Nightly; the 2026-11-04 authorization covered the two A/A runs only. -- moq-dev/moq: once A/A run - [41921320690](https://github.com/moq-dev/moq/actions/runs/41921320690) - finishes, dispatch the second A/A run under the same authorization - (`gh workflow run nightly.yml --ref main -f bench-base=`; as of - 2026-11-04 it is not dispatched yet), then read both runs' `paired.json` (the - `bench-rust*` artifacts). Find the second with `gh run list --repo - moq-dev/moq --workflow nightly.yml --event workflow_dispatch`. +- Maintainer: authorize one agent dispatch of an A/A Nightly run on `main` + (`bench-base` set to its own commit) once + [moq-net's benches run to completion](/quest/m1/bench-test-mode.md) lands. + Agents dispatch Nightly only under a per-run authorization, since a dispatch + cancels any in-progress Nightly through its concurrency group. Neither + `rust-bench` nor `rust-relay` produced a `paired.json` in any run so far: + moq-net's `memory`, `origin` and `session` benches panic, and `rust-relay`'s + session bench includes moq-net's `benches/session/mod.rs` and panics at the + same `Evicted`. Check: that run's `bench-rust` and `bench-rust-relay` + artifacts each hold a `paired.json`. ## Plan -Decided 2026-11-04 (maintainer, /quest-unblock): ✅ authorize one-time agent -dispatch of the two A/A Nightly runs on `main`, with `bench-base` set to each -run's own commit. Not chosen: the maintainer dispatching them, or skipping. -Nightly's concurrency group cancels an in-progress run, so the second is -dispatched only after the first finishes. - -- A/A: dispatch Nightly on `main` with `bench-base` set to the run's own - commit, at least twice. Read each paired group's `paired.json` (the - `bench-rust*` artifacts, or `bench/paired.json` on the site): the spread of - screen deltas sets the screen threshold, and any confirmed regression in an - A/A run means the rounds or threshold are too loose. -- Slowed bench: dispatch with `bench-base` on a branch commit that speeds up - one cheap bench (so tonight's `main` reads as slower), and check the job fails - with that case alone. -- Tune `THRESHOLD_PCT`, `CONFIRMATIONS` (`bench/compare.py`) and the round count - (`BENCH_RUNTIME_ROUNDS`, default three) from what the runs show, and - update the README's numbers. Bimodal series (about 50% up and back across - nights) are the expected false positives; three alternating rounds pass one - only when all three land on the slow mode. -- Check the paired legs' durations against their hour; the base build relies - on the mbx cache for the previous night's artifacts, and a cold one roughly - doubles each leg's build time. -- Each revision in a paired comparison needs its own `CARGO_TARGET_DIR`. - `bench/run.sh` already splits them (`target/bench/current` and - `target/bench/base`); the stale artifacts while attributing - [moq-dev/moq#7141](https://github.com/moq-dev/moq/pull/7141), such as a bench - linked against the other tree's library, came from an ad-hoc A/B of two - source trees sharing one. Say so in `bench/README.md`'s paired section, so an - A/B outside `run.sh` uses its base mode or a target per tree (decided - 2026-10-28). - -- Decide whether `bench/run.sh` pins glibc's malloc thresholds for every - Criterion target, as [moq-dev/moq#6727](https://github.com/moq-dev/moq/pull/6727) - did for `ts_schedule`: some small paired deltas may be the same heap mode flip. +Calibrated 2026-11-05 for the shared, supplemental (`rust-extra`) and io_uring +groups; `bench/README.md` records the numbers. Threshold 10%, 20 +confirmations, and 3 rounds stay as they were: + +- A/A runs [41921320690](https://github.com/moq-dev/moq/actions/runs/41921320690) + and [41933525520](https://github.com/moq-dev/moq/actions/runs/41933525520) + (the first cancelled by Nightly's concurrency group after its bench + artifacts uploaded) confirmed no regression. Screen crossings of 10% either + way: shared 18 and 26 of 716, supplemental 10 and 15 of 460, io_uring 1 and + 1 of 41; at most 17 slower in one group, under the 20 confirmations. The + largest median among round sets that all agreed on a slowdown was +5.6%. +- Slowed-bench run + [41979277343](https://github.com/moq-dev/moq/actions/runs/41979277343) + compared `main` at `c150456928` against a base that collects 192 of + `pattern-algebra/collect-literal/256`'s 256 literals (branch + `bench-ci/slowed-base`, deleted). The shared job failed at its gate with + that case alone: screen +35.7%, rounds +34.6, +35.1, +35.2%. The + supplemental and io_uring groups passed. +- In that run, the rounds rejected the three `stats/tick/*_10pct` cases the + screen flagged, but each one's rounds agreed on a slowdown, at medians of + +9.3 to +9.7%, and every `100pct` case screened +6 to +9%, although `moq-stats` was source-identical on both sides. + The same cases sat within 6% either way in both A/A runs. A local build of + `producer_tick` at both commits, in separate trees and target directories, + gives byte-identical `.text`, `.rodata` and `.data.rel.ro`, so the measured + code is the same. That is the stable bias + [moq-dev/moq#7139](https://github.com/moq-dev/moq/pull/7139) found at +23% + and +46%, and it sits 0.3 points under the threshold. Find its source on the + ARM runner (for example, diff the two `producer_tick` builds there and rerun + the rounds against one binary in both slots), then fix it in `bench/run.sh`. + Lowering the threshold waits on that fix. +- The shared, supplemental and io_uring legs took 19 to 44 minutes of their + hour across the three runs. + +Decided 2026-11-05 (agent, recorded in +[moq-dev/moq#7928](https://github.com/moq-dev/moq/pull/7928)): `bench/run.sh` +does not pin glibc's malloc thresholds for every Criterion target. The largest +A/A screen outliers (`moq-mux/containers`' `ts_si_*`, +35 to +49% in both A/A +runs) were each rejected by rounds within a few percent, so a heap mode flip +costs confirmation budget but no false alarm, and a harness-wide pin would +shift every trend line and measure an allocator production does not run. A +bench whose rounds show a mode flip pins its own heap, as `ts_schedule` does. Decided 2026-10-23 (maintainer), when the paired mode landed: - The gate fails Nightly from day one rather than reporting only until A/A runs: Nightly already alerts for unrelated legs, so a false paired alarm adds no new channel, and this quest tunes it. -- Start at a 10% threshold, 20 confirmations per group, and 3 rounds; 10% - matches the trend table's floor, and three-round sign agreement filters - bimodal cases. -- `rust-shared` splits out of `rust-bench`, restarting the `rust/shared/*` - trend history (three nights lost), since `rust-bench` paired would exceed - its hour unsplit. - A target the base cannot build or run fails the gate rather than passing uncompared; it clears by itself the next night. @@ -74,40 +78,11 @@ Rejected hosts (decided 2026-10-22, maintainer): the merge queue runner, the moq.pro load nodes, and a dedicated bare-metal host; revisit the last only if A/A runs show the hosted noise floor hides regressions worth catching. -Decided 2026-10-28 (maintainer), superseding the 2026-10-27 plan to wait for -A/A: CodSpeed's CPU simulation runs now, on every `main` commit, as an extra -signal beside this gate rather than a replacement (see -[CodSpeed](/bench/README.md#codspeed)). It covers only the single-threaded -benches without I/O, so this calibration still decides the paired gate. - -Observed 2026-10-28, before any A/A run: the last two scheduled Nightlies' -gates failed with only slowdowns, and their rounds mostly agree within a few -points. `22808bbb2` against `8810d7365` confirmed 5 (net -retirement selection about +23%); `270395a33` against `22808bbb2` confirmed -20 across three groups (net `codec_messages/encode/lite` +67.7%, -`group_write_frames/batch32/*` about +45%). A one-signed, tightly agreeing -result is either a real regression or a systematic bias between the base and -head builds or run order, which noise would not produce; the A/A run tells -them apart, and CodSpeed's counts for the cases it covers are a second -opinion. If the A/A run also confirms regressions, fix that bias; if it is -clean, bisect the reported cases and fix the regressions. - -The A/A calibration includes moq-quic's windowed `benches/streams.rs` cases: -on a shared host the unchanged base alone varied 17 to 43% between -back-to-back runs ([moq-dev/moq#6950](https://github.com/moq-dev/moq/pull/6950)). - -Confirmed regressions with no extra work behind them, found by -[moq-dev/moq#7139](https://github.com/moq-dev/moq/pull/7139): every round of -`moq-net/group_relay_frames/plain/*` (+23%) and `moq-net/origin/serve_idle/3000r` -(+46%) agreed, yet callgrind counts the same instructions per frame read and per -poll on both builds. A paired round reruns the same two binaries, so a code or -heap layout shift between them reads as a stable regression that rounds cannot -filter. `codec_messages/*/lite` (+31%, +68%) moved only because a new bench group -in the same binary changed what LTO inlines. -[moq-dev/moq#7253](https://github.com/moq-dev/moq/pull/7253) found a third: -`server_admission/workers=4/0` (+10%) runs machine code identical apart from -addresses on both builds of the 2026-10-27 pair. The other four 2026-10-27 cases -were real, and were fixed or accepted there. +Decided 2026-10-28 (maintainer): CodSpeed's CPU simulation runs on every +`main` commit as an extra signal beside this gate rather than a replacement +(see [CodSpeed](/bench/README.md#codspeed)). It covers only the +single-threaded benches without I/O, so this calibration still decides the +paired gate. ## Related From 68c7abe7fc5de9616ddc6dbbfad9373b32666b4b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 00:15:40 -0800 Subject: [PATCH 04/10] bench: wrap the calibration paragraph Co-Authored-By: Claude Opus 5.5 --- bench/README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/bench/README.md b/bench/README.md index 8f0a5e88b93..2548bf60883 100644 --- a/bench/README.md +++ b/bench/README.md @@ -886,13 +886,13 @@ fails the job too, rather than passing uncompared; a bench target the base does not have yet is only reported as new. Three runs on the hosted runner set these numbers. In two A/A runs of the -shared, supplemental and io_uring groups, 2 to 4% of cases crossed 10% either -way in the screen, at most 17 of them slower, and the rounds rejected every -one; no round set that agreed on a slowdown had a median above 6%. A third run, -whose base cut one case's work by a quarter, failed with that case alone at -+35%. Its rounds also agreed, at medians of +9.3 to +9.7%, on `stats/tick` cases whose source -was identical on both sides, so 10% is a floor rather than a margin. The -moq-net and relay groups are not calibrated yet. +`rust-shared`, `rust-extra` and `rust-uring` groups, 2 to 4% of cases crossed +10% either way in the screen, at most 17 of them slower, and the rounds +rejected every one; no round set that agreed on a slowdown had a median above +6%. A third run, whose base cut one case's work by a quarter, failed with that +case alone at +35%. Its rounds also agreed, at medians of +9.3 to +9.7%, on +`stats/tick` cases whose source was identical on both sides, so 10% is a floor +rather than a margin. The moq-net and relay groups are not calibrated yet. The table page shows tonight's comparison above the trend: confirmed regressions, targets not compared, cases faster in the screen, slowdowns the From a8dbca814095e7aa2fa696f95e5cfb4adf208ebf Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 09:14:55 -0800 Subject: [PATCH 05/10] docs(bench): clarify calibration checks and record dispatch authorization Co-Authored-By: GPT-6.1 Sol --- bench/README.md | 9 +++++---- quest/m1/bench-ci.md | 26 +++++++++++++++----------- 2 files changed, 20 insertions(+), 15 deletions(-) diff --git a/bench/README.md b/bench/README.md index 2548bf60883..84fa18a3daa 100644 --- a/bench/README.md +++ b/bench/README.md @@ -892,7 +892,8 @@ rejected every one; no round set that agreed on a slowdown had a median above 6%. A third run, whose base cut one case's work by a quarter, failed with that case alone at +35%. Its rounds also agreed, at medians of +9.3 to +9.7%, on `stats/tick` cases whose source was identical on both sides, so 10% is a floor -rather than a margin. The moq-net and relay groups are not calibrated yet. +rather than a margin. The `rust-bench` and `rust-relay` groups are not +calibrated yet. The table page shows tonight's comparison above the trend: confirmed regressions, targets not compared, cases faster in the screen, slowdowns the @@ -913,9 +914,9 @@ Each run's evidence directory keeps both builds' estimates (`criterion` and `paired.json`. Each revision builds in its own Cargo target directory (`base` and `current` -under `CARGO_TARGET_DIR`, default `target/bench`). An A/B outside `run.sh` -uses its base mode or a target directory per source tree: two trees sharing -one can leave a bench linked against the other tree's library. +under `CARGO_TARGET_DIR`, default `target/bench`). For an A/B comparison, use +`run.sh`'s `BENCH_BASE` mode or a target directory per source tree: two trees +sharing one can leave a bench linked against the other tree's library. ### CodSpeed diff --git a/quest/m1/bench-ci.md b/quest/m1/bench-ci.md index f8d519aefb0..31203eb7f0c 100644 --- a/quest/m1/bench-ci.md +++ b/quest/m1/bench-ci.md @@ -5,25 +5,29 @@ The paired Rust Criterion comparison each Nightly runs (see [Paired comparison](/bench/README.md#paired-comparison)) fails only on real regressions in every paired group: A/A runs on the hosted ARM runners hold its -threshold, confirmation budget, and round count for moq-net (`rust-bench`) +threshold, confirmation budget, and round count for `rust-bench` and `rust-relay` too, and the rounds stop agreeing on slowdowns in code that did not change. ## Waiting -- Maintainer: authorize one agent dispatch of an A/A Nightly run on `main` - (`bench-base` set to its own commit) once - [moq-net's benches run to completion](/quest/m1/bench-test-mode.md) lands. - Agents dispatch Nightly only under a per-run authorization, since a dispatch - cancels any in-progress Nightly through its concurrency group. Neither - `rust-bench` nor `rust-relay` produced a `paired.json` in any run so far: - moq-net's `memory`, `origin` and `session` benches panic, and `rust-relay`'s - session bench includes moq-net's `benches/session/mod.rs` and panics at the - same `Evicted`. Check: that run's `bench-rust` and `bench-rust-relay` - artifacts each hold a `paired.json`. +- moq-dev/moq: once [moq-dev/moq#7940](https://github.com/moq-dev/moq/pull/7940) + merges, an agent may dispatch exactly one A/A Nightly run on `main`, with + `bench-base` set to main's head, under the authorization recorded below. + Check: that run's `bench-rust` and `bench-rust-relay` artifacts each hold + `paired.json` at the artifact root (the gate's input), rather than only + partial `runs/run.*/paired.json` files. ## Plan +Decided 2026-11-05 (maintainer, /quest-plan): authorize exactly one agent +dispatch of an A/A Nightly run on `main` once +[moq-dev/moq#7940](https://github.com/moq-dev/moq/pull/7940) merges: +`gh workflow run nightly.yml --ref main -f bench-base=`. +Check that `bench-rust` and `bench-rust-relay` each produce an artifact-root +`paired.json`. A dispatch cancels any in-progress Nightly through its +concurrency group. + Calibrated 2026-11-05 for the shared, supplemental (`rust-extra`) and io_uring groups; `bench/README.md` records the numbers. Threshold 10%, 20 confirmations, and 3 rounds stay as they were: From 4452ad2be999317208afc8d909cc0a85683d5044 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 09:19:36 -0800 Subject: [PATCH 06/10] docs(bench): retain calibration context and validate A/A evidence Co-Authored-By: GPT-6.1 Sol --- bench/compare.py | 3 ++- quest/m1/bench-ci.md | 22 ++++++++++++++++++++-- 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/bench/compare.py b/bench/compare.py index 00de468d467..3b22f7c7996 100644 --- a/bench/compare.py +++ b/bench/compare.py @@ -157,7 +157,8 @@ def criterion_baseline(directory, discovery, benchmark): # A paired case beyond this change is flagged, and fails the nightly only when its rounds agree. -# Rounds on unchanged code have agreed at up to +9.7% on the hosted runner, so lower is unsafe. +# Calibrated groups reached +9.7% on unchanged code; rust-bench and rust-relay still need A/A. +# Lowering this waits on fixing the stable bias recorded in quest/m1/bench-ci.md. THRESHOLD_PCT = 10 # Each alternating round costs seconds per case, so a noisy night confirms only its worst slowdowns. CONFIRMATIONS = 20 diff --git a/quest/m1/bench-ci.md b/quest/m1/bench-ci.md index 31203eb7f0c..adc61555ae7 100644 --- a/quest/m1/bench-ci.md +++ b/quest/m1/bench-ci.md @@ -26,7 +26,11 @@ dispatch of an A/A Nightly run on `main` once `gh workflow run nightly.yml --ref main -f bench-base=`. Check that `bench-rust` and `bench-rust-relay` each produce an artifact-root `paired.json`. A dispatch cancels any in-progress Nightly through its -concurrency group. +concurrency group; the 07:00 UTC scheduled run can also cancel the dispatch. +Dispatch clear of that cron window so the benchmark legs can finish. Verify +that the dispatched run's `head_sha` equals `bench-base` before treating it +as A/A evidence; a concurrent merge can move `main` between reading and +dispatching it. Calibrated 2026-11-05 for the shared, supplemental (`rust-extra`) and io_uring groups; `bench/README.md` records the numbers. Threshold 10%, 20 @@ -59,7 +63,21 @@ confirmations, and 3 rounds stay as they were: the rounds against one binary in both slots), then fix it in `bench/run.sh`. Lowering the threshold waits on that fix. - The shared, supplemental and io_uring legs took 19 to 44 minutes of their - hour across the three runs. + hour across the three runs. Check `rust-bench` and `rust-relay` against + their hour too; a cold mbx cache roughly doubles build time. A timeout + leaves calibration incomplete, rather than demonstrating bias. + +Retain the earlier bias cases when reading the remaining groups' A/A results: +[moq-dev/moq#7139](https://github.com/moq-dev/moq/pull/7139) found +`moq-net/group_relay_frames/plain/*` (+23%) and +`moq-net/origin/serve_idle/3000r` (+46%) agreed in every round despite equal +callgrind instruction counts. `codec_messages/*/lite` (+31%, +68%) changed +because a new bench group shifted LTO inlining. +[moq-dev/moq#7253](https://github.com/moq-dev/moq/pull/7253) found +`server_admission/workers=4/0` (+10%) had identical machine code apart from +addresses. Include moq-quic's windowed `benches/streams.rs` cases too: an +unchanged base varied 17 to 43% between back-to-back runs on a shared host +([moq-dev/moq#6950](https://github.com/moq-dev/moq/pull/6950)). Decided 2026-11-05 (agent, recorded in [moq-dev/moq#7928](https://github.com/moq-dev/moq/pull/7928)): `bench/run.sh` From 07ddc55eafe597451978820759ce06f0ae8a231f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 10:55:31 -0800 Subject: [PATCH 07/10] quest(bench-ci): CodSpeed runs only on main pushes touching Rust, Cargo, or Nix Co-Authored-By: Claude Opus 5.5 --- quest/m1/bench-ci.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/quest/m1/bench-ci.md b/quest/m1/bench-ci.md index adc61555ae7..8ee264eacaf 100644 --- a/quest/m1/bench-ci.md +++ b/quest/m1/bench-ci.md @@ -101,7 +101,8 @@ moq.pro load nodes, and a dedicated bare-metal host; revisit the last only if A/A runs show the hosted noise floor hides regressions worth catching. Decided 2026-10-28 (maintainer): CodSpeed's CPU simulation runs on every -`main` commit as an extra signal beside this gate rather than a replacement +`main` push that touches Rust, Cargo, or Nix files, as an extra signal beside +this gate rather than a replacement (see [CodSpeed](/bench/README.md#codspeed)). It covers only the single-threaded benches without I/O, so this calibration still decides the paired gate. From 3e6511ba61b3cfa9b2ae1749b8dec683c94c402e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 15:42:13 -0800 Subject: [PATCH 08/10] quest(bench-ci): record the post-#7940 A/A runs Relay is calibrated; shared confirmed one false alarm at +12% on an identical commit, and rust-bench outlasts its hour. Co-Authored-By: Claude Opus 5.5 --- bench/README.md | 24 +++++----- bench/compare.py | 4 +- quest/m1/README.md | 2 +- quest/m1/bench-ci.md | 107 +++++++++++++++++++++++++------------------ 4 files changed, 78 insertions(+), 59 deletions(-) diff --git a/bench/README.md b/bench/README.md index 0435badb321..ddbdaa880f1 100644 --- a/bench/README.md +++ b/bench/README.md @@ -890,8 +890,8 @@ the table requires no new renderer or data branch. A source's `paired.json`, as ### Paired comparison -Each Nightly's Rust bench groups (`rust-bench`, `rust-shared`, `rust-extra` and -`rust-uring`) also build the previous scheduled Nightly's commit beside their +Each Nightly's Rust bench groups (`rust-bench`, `rust-shared`, `rust-relay`, +`rust-extra` and `rust-uring`) also build the previous scheduled Nightly's commit beside their own, on the same runner, so night-to-night runner variance drops out. Each built target runs once per side in the trend's quick mode, alternating which side goes first; the current side is the trend point. A case more than 10% @@ -903,15 +903,17 @@ Alert workflow posts the failed Nightly. A target the base cannot build or run fails the job too, rather than passing uncompared; a bench target the base does not have yet is only reported as new. -Three runs on the hosted runner set these numbers. In two A/A runs of the -`rust-shared`, `rust-extra` and `rust-uring` groups, 2 to 4% of cases crossed -10% either way in the screen, at most 17 of them slower, and the rounds -rejected every one; no round set that agreed on a slowdown had a median above -6%. A third run, whose base cut one case's work by a quarter, failed with that -case alone at +35%. Its rounds also agreed, at medians of +9.3 to +9.7%, on -`stats/tick` cases whose source was identical on both sides, so 10% is a floor -rather than a margin. The `rust-bench` and `rust-relay` groups are not -calibrated yet. +Runs on the hosted runner set these numbers. In four A/A runs of the +`rust-shared`, `rust-extra` and `rust-uring` groups, and two of `rust-relay`, +1 to 9% of cases crossed 10% either way in the screen, at most 17 of them +slower in one group. The rounds rejected all but one: +`wire_decode/padded/packets_64` at +12%, which read +2% in the next A/A run. +A run whose base cut one case's work by a quarter failed with that case alone +at +35%; its rounds also agreed, at medians of +9.3 to +9.7%, on `stats/tick` +cases whose source was identical on both sides. Separate builds of the same +source can measure up to 12% apart, so the threshold cannot go lower until +that bias is fixed. +`rust-bench` is not calibrated yet: its paired leg outlasts its hour. The table page shows tonight's comparison above the trend: confirmed regressions, targets not compared, cases faster in the screen, slowdowns the diff --git a/bench/compare.py b/bench/compare.py index 3b22f7c7996..82d2c634010 100644 --- a/bench/compare.py +++ b/bench/compare.py @@ -157,8 +157,8 @@ def criterion_baseline(directory, discovery, benchmark): # A paired case beyond this change is flagged, and fails the nightly only when its rounds agree. -# Calibrated groups reached +9.7% on unchanged code; rust-bench and rust-relay still need A/A. -# Lowering this waits on fixing the stable bias recorded in quest/m1/bench-ci.md. +# Separate builds of unchanged code measured up to 12% apart in A/A runs, so lowering this +# waits on fixing that bias (quest/m1/bench-ci.md). THRESHOLD_PCT = 10 # Each alternating round costs seconds per case, so a noisy night confirms only its worst slowdowns. CONFIRMATIONS = 20 diff --git a/quest/m1/README.md b/quest/m1/README.md index 6e7088a08aa..304526e2deb 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -169,7 +169,7 @@ their end report, so an embedder's drain alert pages on real cuts. - [crates.io trusts the workflow](/quest/m1/crates-trusted-publisher.md) - maintainer action: every published crate gets a trusted publisher for `release-rs.yml` - [Rust releases publish through OIDC](/quest/m1/release-rs-oidc.md) - release-plz publishes with trusted publishing and `CARGO_REGISTRY_TOKEN` is revoked - [Main is no slower than release](/quest/m1/release-perf/README.md) - every benchmark where main is slower than release is fixed or accepted with a reason, checked by a final paired run before the cut -- [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - the moq-net and relay groups get A/A runs and the bias that makes rounds agree on unchanged code is removed, so every paired regression alert can be trusted +- [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - moq-net's paired group fits its hour and gets A/A runs, and separate builds of unchanged code stop measuring as a slowdown, so every paired regression alert can be trusted - [moq-json's table/deltas bench measures the same work every iteration](/quest/m1/bench-json-deltas.md) - the `table/deltas/1%` cases stop flipping between snapshot and delta timings, so their trend history is usable - [Publish moq-bench to crates.io](/quest/m1/bench-publish.md) - `cargo install moq-bench` installs one binary, with the host sampler as `moq-bench host`, first as 0.1.0 - [Unbounded groups](/quest/m1/unbounded-groups/README.md) - groups and frames lose their size caps; a group keeps a 32 MB tail, and readers behind it replay through paced flights or refresh diff --git a/quest/m1/bench-ci.md b/quest/m1/bench-ci.md index 8ee264eacaf..4213c6a0aba 100644 --- a/quest/m1/bench-ci.md +++ b/quest/m1/bench-ci.md @@ -4,37 +4,16 @@ The paired Rust Criterion comparison each Nightly runs (see [Paired comparison](/bench/README.md#paired-comparison)) fails only on real -regressions in every paired group: A/A runs on the hosted ARM runners hold its -threshold, confirmation budget, and round count for `rust-bench` -and `rust-relay` too, and the rounds stop agreeing on slowdowns in code that -did not change. - -## Waiting - -- moq-dev/moq: once [moq-dev/moq#7940](https://github.com/moq-dev/moq/pull/7940) - merges, an agent may dispatch exactly one A/A Nightly run on `main`, with - `bench-base` set to main's head, under the authorization recorded below. - Check: that run's `bench-rust` and `bench-rust-relay` artifacts each hold - `paired.json` at the artifact root (the gate's input), rather than only - partial `runs/run.*/paired.json` files. +regressions in every paired group: `rust-bench` finishes its paired leg within +its hour and A/A runs on the hosted ARM runners hold its threshold, +confirmation budget, and round count, and separate builds of unchanged code +stop measuring as a stable slowdown. ## Plan -Decided 2026-11-05 (maintainer, /quest-plan): authorize exactly one agent -dispatch of an A/A Nightly run on `main` once -[moq-dev/moq#7940](https://github.com/moq-dev/moq/pull/7940) merges: -`gh workflow run nightly.yml --ref main -f bench-base=`. -Check that `bench-rust` and `bench-rust-relay` each produce an artifact-root -`paired.json`. A dispatch cancels any in-progress Nightly through its -concurrency group; the 07:00 UTC scheduled run can also cancel the dispatch. -Dispatch clear of that cron window so the benchmark legs can finish. Verify -that the dispatched run's `head_sha` equals `bench-base` before treating it -as A/A evidence; a concurrent merge can move `main` between reading and -dispatching it. - -Calibrated 2026-11-05 for the shared, supplemental (`rust-extra`) and io_uring -groups; `bench/README.md` records the numbers. Threshold 10%, 20 -confirmations, and 3 rounds stay as they were: +Calibrated 2026-11-05 for the shared, supplemental (`rust-extra`), io_uring +and `rust-relay` groups; `bench/README.md` records the numbers. Threshold 10%, +20 confirmations, and 3 rounds stay as they were: - A/A runs [41921320690](https://github.com/moq-dev/moq/actions/runs/41921320690) and [41933525520](https://github.com/moq-dev/moq/actions/runs/41933525520) @@ -43,31 +22,62 @@ confirmations, and 3 rounds stay as they were: way: shared 18 and 26 of 716, supplemental 10 and 15 of 460, io_uring 1 and 1 of 41; at most 17 slower in one group, under the 20 confirmations. The largest median among round sets that all agreed on a slowdown was +5.6%. +- A/A runs [42102235726](https://github.com/moq-dev/moq/actions/runs/42102235726) + (`main` at `62ae716265`) and + [42119312487](https://github.com/moq-dev/moq/actions/runs/42119312487) + (`83d0bc46ba`), dispatched after + [moq-dev/moq#7940](https://github.com/moq-dev/moq/pull/7940) merged, each + verified with `head_sha` equal to `bench-base`. Crossings: shared 20 and 19 + of 728, supplemental 7 and 8 of 514 and 532, io_uring 3 and 2 of 41, relay + 15 and 37 of 410; at most 17 slower in one group. Relay confirmed nothing in + either run and is calibrated. The second run confirmed nothing. The first + failed the shared gate on `wire_decode/padded/packets_64`: screen +10.3%, + rounds +11.9, +12.9, +12.1%, with `padded/packets_1024` at +7.1%. The same + case read +2.1% in the second run and +5.4, -0.8 and +6.2% in the three + earlier runs. - Slowed-bench run [41979277343](https://github.com/moq-dev/moq/actions/runs/41979277343) compared `main` at `c150456928` against a base that collects 192 of `pattern-algebra/collect-literal/256`'s 256 literals (branch `bench-ci/slowed-base`, deleted). The shared job failed at its gate with that case alone: screen +35.7%, rounds +34.6, +35.1, +35.2%. The - supplemental and io_uring groups passed. + supplemental and io_uring groups passed. Every group runs the same gate + step, so no other group needs its own slowed run. - In that run, the rounds rejected the three `stats/tick/*_10pct` cases the screen flagged, but each one's rounds agreed on a slowdown, at medians of - +9.3 to +9.7%, and every `100pct` case screened +6 to +9%, although `moq-stats` was source-identical on both sides. - The same cases sat within 6% either way in both A/A runs. A local build of - `producer_tick` at both commits, in separate trees and target directories, - gives byte-identical `.text`, `.rodata` and `.data.rel.ro`, so the measured - code is the same. That is the stable bias - [moq-dev/moq#7139](https://github.com/moq-dev/moq/pull/7139) found at +23% - and +46%, and it sits 0.3 points under the threshold. Find its source on the - ARM runner (for example, diff the two `producer_tick` builds there and rerun - the rounds against one binary in both slots), then fix it in `bench/run.sh`. - Lowering the threshold waits on that fix. -- The shared, supplemental and io_uring legs took 19 to 44 minutes of their - hour across the three runs. Check `rust-bench` and `rust-relay` against - their hour too; a cold mbx cache roughly doubles build time. A timeout - leaves calibration incomplete, rather than demonstrating bias. - -Retain the earlier bias cases when reading the remaining groups' A/A results: + +9.3 to +9.7%, and every `100pct` case screened +6 to +9%, although + `moq-stats` was source-identical on both sides. The same cases sat within 6% + either way in both earlier A/A runs. A local build of `producer_tick` at both + commits, in separate trees and target directories, gives byte-identical + `.text`, `.rodata` and `.data.rel.ro`. + +Find the source of that bias, then fix it in `bench/run.sh`. It moves between +runs rather than following the source: an A/A run builds the same commit on +both sides, yet `wire_decode/padded/packets_64` agreed on +12% in one run and +not the next. A paired round reruns the same two binaries, so rounds cannot +filter it. Rerun the rounds against one binary in both slots on the ARM +runner to tell build differences from run placement. Lowering the threshold +waits on that fix. + +`rust-bench` is not calibrated: its leg hit its 60-minute timeout in both +runs, before uploading an artifact. Building both sides took 16 to 18 +minutes. The screen and rounds then ran past 40 minutes, because confirmation +rounds reran benches whose untimed work dominates their wall time. A round +side meant to take about 3 seconds took 2 minutes for +`interests/covered/1000w_1000s` and 8 for +`publisher_covered_before_start/wake/8192`: each builds its fixture in the +`bench_function` closure outside `b.iter`, and Criterion calls that closure +once per sample. `track_evict_fanout/512r/8` took about a minute, because its +untimed reads run inside every iteration. Make those benches pay their setup +once per benchmark, and bound the rounds by time if the leg still does not +fit. Its A/A run then needs a maintainer-authorized Nightly dispatch. + +A dispatch cancels any in-progress Nightly through its concurrency group, and +the 07:00 UTC scheduled run can cancel a dispatch. Dispatch clear of that +window, and verify that the run's `head_sha` equals `bench-base` before +treating it as A/A evidence. + +Retain the earlier bias cases when reading `rust-bench`'s A/A results: [moq-dev/moq#7139](https://github.com/moq-dev/moq/pull/7139) found `moq-net/group_relay_frames/plain/*` (+23%) and `moq-net/origin/serve_idle/3000r` (+46%) agreed in every round despite equal @@ -79,6 +89,13 @@ addresses. Include moq-quic's windowed `benches/streams.rs` cases too: an unchanged base varied 17 to 43% between back-to-back runs on a shared host ([moq-dev/moq#6950](https://github.com/moq-dev/moq/pull/6950)). +Decided 2026-11-05 (agent, recorded in +[moq-dev/moq#7928](https://github.com/moq-dev/moq/pull/7928)): the threshold +stays at 10% despite the A/A false alarm at +12.1%. The 2026-10-23 decision +already accepts false paired alarms while this quest tunes the gate. Raising +the threshold would hide the bias this quest fixes, along with real +regressions of 10 to 15%. + Decided 2026-11-05 (agent, recorded in [moq-dev/moq#7928](https://github.com/moq-dev/moq/pull/7928)): `bench/run.sh` does not pin glibc's malloc thresholds for every Criterion target. The largest From 7382ea2ec1aa9754ed408c7736fcb9f62786c6d3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 15:58:25 -0800 Subject: [PATCH 09/10] quest(bench-ci): keep the rust-bench dispatch as a maintainer wait Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 2 +- quest/m1/bench-ci.md | 20 +++++++++++++++++--- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/quest/m1/README.md b/quest/m1/README.md index 304526e2deb..e673ce4f099 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -169,7 +169,7 @@ their end report, so an embedder's drain alert pages on real cuts. - [crates.io trusts the workflow](/quest/m1/crates-trusted-publisher.md) - maintainer action: every published crate gets a trusted publisher for `release-rs.yml` - [Rust releases publish through OIDC](/quest/m1/release-rs-oidc.md) - release-plz publishes with trusted publishing and `CARGO_REGISTRY_TOKEN` is revoked - [Main is no slower than release](/quest/m1/release-perf/README.md) - every benchmark where main is slower than release is fixed or accepted with a reason, checked by a final paired run before the cut -- [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - moq-net's paired group fits its hour and gets A/A runs, and separate builds of unchanged code stop measuring as a slowdown, so every paired regression alert can be trusted +- [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - `rust-bench` fits its hour and gets A/A runs, and separate builds of unchanged code stop measuring as a slowdown, so every paired regression alert can be trusted - [moq-json's table/deltas bench measures the same work every iteration](/quest/m1/bench-json-deltas.md) - the `table/deltas/1%` cases stop flipping between snapshot and delta timings, so their trend history is usable - [Publish moq-bench to crates.io](/quest/m1/bench-publish.md) - `cargo install moq-bench` installs one binary, with the host sampler as `moq-bench host`, first as 0.1.0 - [Unbounded groups](/quest/m1/unbounded-groups/README.md) - groups and frames lose their size caps; a group keeps a 32 MB tail, and readers behind it replay through paced flights or refresh diff --git a/quest/m1/bench-ci.md b/quest/m1/bench-ci.md index 4213c6a0aba..520a017dd1e 100644 --- a/quest/m1/bench-ci.md +++ b/quest/m1/bench-ci.md @@ -9,8 +9,23 @@ its hour and A/A runs on the hosted ARM runners hold its threshold, confirmation budget, and round count, and separate builds of unchanged code stop measuring as a stable slowdown. +## Waiting + +- Maintainer: authorize one A/A Nightly dispatch for `rust-bench` once its + paired leg fits its hour, or now for an agent to dispatch then. Agents + dispatch Nightly only under a recorded authorization: a dispatch spends an + hour of every ARM leg and cancels any Nightly in progress. Check: a dated + authorization in this Plan. + ## Plan +Decided 2026-11-05 (maintainer, /quest-unblock): authorize two A/A Nightly +runs and one slowed-bench run on `main` after +[moq-dev/moq#7940](https://github.com/moq-dev/moq/pull/7940) merged, +replacing the earlier one-run authorization. The two A/A runs below spent it; +the slowed run was not needed, since every paired group runs the same gate +step. + Calibrated 2026-11-05 for the shared, supplemental (`rust-extra`), io_uring and `rust-relay` groups; `bench/README.md` records the numbers. Threshold 10%, 20 confirmations, and 3 rounds stay as they were: @@ -41,8 +56,7 @@ and `rust-relay` groups; `bench/README.md` records the numbers. Threshold 10%, `pattern-algebra/collect-literal/256`'s 256 literals (branch `bench-ci/slowed-base`, deleted). The shared job failed at its gate with that case alone: screen +35.7%, rounds +34.6, +35.1, +35.2%. The - supplemental and io_uring groups passed. Every group runs the same gate - step, so no other group needs its own slowed run. + supplemental and io_uring groups passed. - In that run, the rounds rejected the three `stats/tick/*_10pct` cases the screen flagged, but each one's rounds agreed on a slowdown, at medians of +9.3 to +9.7%, and every `100pct` case screened +6 to +9%, although @@ -70,7 +84,7 @@ side meant to take about 3 seconds took 2 minutes for once per sample. `track_evict_fanout/512r/8` took about a minute, because its untimed reads run inside every iteration. Make those benches pay their setup once per benchmark, and bound the rounds by time if the leg still does not -fit. Its A/A run then needs a maintainer-authorized Nightly dispatch. +fit. A dispatch cancels any in-progress Nightly through its concurrency group, and the 07:00 UTC scheduled run can cancel a dispatch. Dispatch clear of that From 0e87d9263c57bea4067654256a5f619f83d5c6d9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 16:21:46 -0800 Subject: [PATCH 10/10] bench: make the rust-shared paired gate report-only until the build bias is fixed Records the maintainer's 2026-11-05 decisions: rust-shared reports its paired regressions without failing Nightly, and one rust-bench A/A dispatch is authorized once its leg fits its hour. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/nightly.yml | 5 ++++- bench/README.md | 4 +++- quest/m1/bench-ci.md | 28 ++++++++++++++-------------- 3 files changed, 21 insertions(+), 16 deletions(-) diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 1e9495f7044..2f3113cae4a 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -120,12 +120,15 @@ jobs: artifact-path: target/bench/nightly/rust-bench benchmark: true paired: true + # Report-only until quest/m1/bench-ci.md fixes the build bias behind + # an A/A false alarm; the site still lists its confirmed regressions. - name: Rust shared bench targets recipe: gh nightly rust-shared artifact: bench-rust-shared artifact-path: target/bench/nightly/rust-shared benchmark: true paired: true + report-only: true - name: Relay session benchmarks recipe: gh nightly rust-relay artifact: bench-rust-relay @@ -235,7 +238,7 @@ jobs: # After the upload, so the benchmark site still records tonight's history and # publishes the comparison that failed this job before the alert links to it. - name: Fail on paired regressions - if: ${{ matrix.paired && steps.workload.outcome == 'success' }} + if: ${{ matrix.paired && !matrix.report-only && steps.workload.outcome == 'success' }} run: nix develop --command just gh benchmark-report gate "${{ matrix.artifact-path }}" # gh-pages is the one Pages site, and each directory has one writer: this job owns diff --git a/bench/README.md b/bench/README.md index ddbdaa880f1..bfd2e52b806 100644 --- a/bench/README.md +++ b/bench/README.md @@ -901,7 +901,9 @@ measurement, 30 samples). A case whose rounds are all slower with a median beyond 10% is a confirmed regression and fails its job after the upload, so the Alert workflow posts the failed Nightly. A target the base cannot build or run fails the job too, rather than passing uncompared; a bench target the base does -not have yet is only reported as new. +not have yet is only reported as new. `rust-shared` is report-only for now: its +confirmed regressions show on the site but do not fail the job, until the +build bias below is fixed. Runs on the hosted runner set these numbers. In four A/A runs of the `rust-shared`, `rust-extra` and `rust-uring` groups, and two of `rust-relay`, diff --git a/quest/m1/bench-ci.md b/quest/m1/bench-ci.md index 520a017dd1e..d929625a8b6 100644 --- a/quest/m1/bench-ci.md +++ b/quest/m1/bench-ci.md @@ -7,18 +7,17 @@ The paired Rust Criterion comparison each Nightly runs (see regressions in every paired group: `rust-bench` finishes its paired leg within its hour and A/A runs on the hosted ARM runners hold its threshold, confirmation budget, and round count, and separate builds of unchanged code -stop measuring as a stable slowdown. - -## Waiting - -- Maintainer: authorize one A/A Nightly dispatch for `rust-bench` once its - paired leg fits its hour, or now for an agent to dispatch then. Agents - dispatch Nightly only under a recorded authorization: a dispatch spends an - hour of every ARM leg and cancels any Nightly in progress. Check: a dated - authorization in this Plan. +stop measuring as a stable slowdown, so `rust-shared`'s gate fails Nightly +again. ## Plan +Decided 2026-11-05 (maintainer, after +[moq-dev/moq#7928](https://github.com/moq-dev/moq/pull/7928)'s A/A runs): +authorize one A/A Nightly dispatch for `rust-bench` once its paired leg fits +its hour, for the agent that fixes its benches to dispatch straight after. +Not chosen: asking again once it fits. + Decided 2026-11-05 (maintainer, /quest-unblock): authorize two A/A Nightly runs and one slowed-bench run on `main` after [moq-dev/moq#7940](https://github.com/moq-dev/moq/pull/7940) merged, @@ -103,11 +102,12 @@ addresses. Include moq-quic's windowed `benches/streams.rs` cases too: an unchanged base varied 17 to 43% between back-to-back runs on a shared host ([moq-dev/moq#6950](https://github.com/moq-dev/moq/pull/6950)). -Decided 2026-11-05 (agent, recorded in -[moq-dev/moq#7928](https://github.com/moq-dev/moq/pull/7928)): the threshold -stays at 10% despite the A/A false alarm at +12.1%. The 2026-10-23 decision -already accepts false paired alarms while this quest tunes the gate. Raising -the threshold would hide the bias this quest fixes, along with real +Decided 2026-11-05 (maintainer, after the A/A false alarm at +12.1%): +`rust-shared`'s gate is report-only until the build bias is fixed, and the +threshold stays at 10%; Nightly's `report-only` matrix flag skips its gate +step. Restore the gate once an A/A run of the fixed harness confirms nothing. +Not chosen: keeping the gate failing at 10% (the 2026-10-23 decision's +default); raising the threshold to about 15%, which would hide real regressions of 10 to 15%. Decided 2026-11-05 (agent, recorded in