diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 1e9495f7044..2f3113cae4a 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -120,12 +120,15 @@ jobs: artifact-path: target/bench/nightly/rust-bench benchmark: true paired: true + # Report-only until quest/m1/bench-ci.md fixes the build bias behind + # an A/A false alarm; the site still lists its confirmed regressions. - name: Rust shared bench targets recipe: gh nightly rust-shared artifact: bench-rust-shared artifact-path: target/bench/nightly/rust-shared benchmark: true paired: true + report-only: true - name: Relay session benchmarks recipe: gh nightly rust-relay artifact: bench-rust-relay @@ -235,7 +238,7 @@ jobs: # After the upload, so the benchmark site still records tonight's history and # publishes the comparison that failed this job before the alert links to it. - name: Fail on paired regressions - if: ${{ matrix.paired && steps.workload.outcome == 'success' }} + if: ${{ matrix.paired && !matrix.report-only && steps.workload.outcome == 'success' }} run: nix develop --command just gh benchmark-report gate "${{ matrix.artifact-path }}" # gh-pages is the one Pages site, and each directory has one writer: this job owns diff --git a/bench/README.md b/bench/README.md index 25f0f1a07cb..bfd2e52b806 100644 --- a/bench/README.md +++ b/bench/README.md @@ -890,8 +890,8 @@ the table requires no new renderer or data branch. A source's `paired.json`, as ### Paired comparison -Each Nightly's Rust bench groups (`rust-bench`, `rust-shared`, `rust-extra` and -`rust-uring`) also build the previous scheduled Nightly's commit beside their +Each Nightly's Rust bench groups (`rust-bench`, `rust-shared`, `rust-relay`, +`rust-extra` and `rust-uring`) also build the previous scheduled Nightly's commit beside their own, on the same runner, so night-to-night runner variance drops out. Each built target runs once per side in the trend's quick mode, alternating which side goes first; the current side is the trend point. A case more than 10% @@ -901,8 +901,21 @@ measurement, 30 samples). A case whose rounds are all slower with a median beyond 10% is a confirmed regression and fails its job after the upload, so the Alert workflow posts the failed Nightly. A target the base cannot build or run fails the job too, rather than passing uncompared; a bench target the base does -not have yet is only reported as new. The threshold is a starting point until -A/A runs on the hosted runner measure its noise floor. +not have yet is only reported as new. `rust-shared` is report-only for now: its +confirmed regressions show on the site but do not fail the job, until the +build bias below is fixed. + +Runs on the hosted runner set these numbers. In four A/A runs of the +`rust-shared`, `rust-extra` and `rust-uring` groups, and two of `rust-relay`, +1 to 9% of cases crossed 10% either way in the screen, at most 17 of them +slower in one group. The rounds rejected all but one: +`wire_decode/padded/packets_64` at +12%, which read +2% in the next A/A run. +A run whose base cut one case's work by a quarter failed with that case alone +at +35%; its rounds also agreed, at medians of +9.3 to +9.7%, on `stats/tick` +cases whose source was identical on both sides. Separate builds of the same +source can measure up to 12% apart, so the threshold cannot go lower until +that bias is fixed. +`rust-bench` is not calibrated yet: its paired leg outlasts its hour. The table page shows tonight's comparison above the trend: confirmed regressions, targets not compared, cases faster in the screen, slowdowns the @@ -923,6 +936,11 @@ Each run's evidence directory keeps both builds' estimates (`criterion` and `paired.json`. The base build streams its diagnostics and retains them in `base-build.log`, including when it fails. +Each revision builds in its own Cargo target directory (`base` and `current` +under `CARGO_TARGET_DIR`, default `target/bench`). For an A/B comparison, use +`run.sh`'s `BENCH_BASE` mode or a target directory per source tree: two trees +sharing one can leave a bench linked against the other tree's library. + ### CodSpeed The CodSpeed workflow adds instruction counts for every `main` commit that diff --git a/bench/compare.py b/bench/compare.py index 9f5b66f6786..82d2c634010 100644 --- a/bench/compare.py +++ b/bench/compare.py @@ -157,7 +157,8 @@ def criterion_baseline(directory, discovery, benchmark): # A paired case beyond this change is flagged, and fails the nightly only when its rounds agree. -# Loose until A/A runs on the hosted runner measure its noise floor. +# Separate builds of unchanged code measured up to 12% apart in A/A runs, so lowering this +# waits on fixing that bias (quest/m1/bench-ci.md). THRESHOLD_PCT = 10 # Each alternating round costs seconds per case, so a noisy night confirms only its worst slowdowns. CONFIRMATIONS = 20 diff --git a/quest/m1/README.md b/quest/m1/README.md index 382b920ece0..e673ce4f099 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -169,7 +169,7 @@ their end report, so an embedder's drain alert pages on real cuts. - [crates.io trusts the workflow](/quest/m1/crates-trusted-publisher.md) - maintainer action: every published crate gets a trusted publisher for `release-rs.yml` - [Rust releases publish through OIDC](/quest/m1/release-rs-oidc.md) - release-plz publishes with trusted publishing and `CARGO_REGISTRY_TOKEN` is revoked - [Main is no slower than release](/quest/m1/release-perf/README.md) - every benchmark where main is slower than release is fixed or accepted with a reason, checked by a final paired run before the cut -- [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - A/A and deliberately slowed Nightly runs set the paired comparison's threshold, so its regression alerts can be trusted +- [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - `rust-bench` fits its hour and gets A/A runs, and separate builds of unchanged code stop measuring as a slowdown, so every paired regression alert can be trusted - [moq-json's table/deltas bench measures the same work every iteration](/quest/m1/bench-json-deltas.md) - the `table/deltas/1%` cases stop flipping between snapshot and delta timings, so their trend history is usable - [Publish moq-bench to crates.io](/quest/m1/bench-publish.md) - `cargo install moq-bench` installs one binary, with the host sampler as `moq-bench host`, first as 0.1.0 - [Unbounded groups](/quest/m1/unbounded-groups/README.md) - groups and frames lose their size caps; a group keeps a 32 MB tail, and readers behind it replay through paced flights or refresh diff --git a/quest/m1/bench-ci.md b/quest/m1/bench-ci.md index 79a1e73bfb6..d929625a8b6 100644 --- a/quest/m1/bench-ci.md +++ b/quest/m1/bench-ci.md @@ -4,79 +4,126 @@ The paired Rust Criterion comparison each Nightly runs (see [Paired comparison](/bench/README.md#paired-comparison)) fails only on real -regressions: its threshold, confirmation budget, and round count come from A/A -runs on the hosted ARM runners, and a deliberately slowed bench fails the job. - -## Waiting - -- Maintainer: dispatch the slowed-bench Nightly run below. Agents never - dispatch Nightly; the 2026-11-04 authorization covered the two A/A runs only. -- Maintainer: authorize one more A/A Nightly run for the moq-net - (`bench-rust`) and `bench-rust-relay` groups, now that - [moq-dev/moq#7940](https://github.com/moq-dev/moq/pull/7940) lets moq-net's - benches run to completion. - Neither group produced a `paired.json` in either 2026-11-04 run, because - their benches panic. Check: both groups' `paired.json` exist in that run. +regressions in every paired group: `rust-bench` finishes its paired leg within +its hour and A/A runs on the hosted ARM runners hold its threshold, +confirmation budget, and round count, and separate builds of unchanged code +stop measuring as a stable slowdown, so `rust-shared`'s gate fails Nightly +again. ## Plan -A/A results, 2026-11-04 (/quest-unblock): runs -[41921320690](https://github.com/moq-dev/moq/actions/runs/41921320690) and -[41933525520](https://github.com/moq-dev/moq/actions/runs/41933525520) both -produced `paired.json` for the shared, extra, and uring groups. The first was -cancelled by Nightly's concurrency group after its bench artifacts uploaded. -At a 10% threshold over 3 rounds, neither run confirmed a regression: every -case the 20-case confirmation budget (`CONFIRMATIONS`) reached came back as -noise. Single-pass crossings: shared 18 and 26 of 716, extra 10 and 15 of 460, -uring 1 and 1 of 41, so the second run left 6 shared cases `unconfirmed`. |delta| p95 was -3.7-8.3%, p99 10-23%. Tune the threshold for those groups from these. - -Decided 2026-11-04 (maintainer, /quest-unblock): ✅ authorize one-time agent -dispatch of the two A/A Nightly runs on `main`, with `bench-base` set to each -run's own commit. Not chosen: the maintainer dispatching them, or skipping. -Nightly's concurrency group cancels an in-progress run, so the second is -dispatched only after the first finishes. - -- A/A: dispatch Nightly on `main` with `bench-base` set to the run's own - commit, at least twice. Read each paired group's `paired.json` (the - `bench-rust*` artifacts, or `bench/paired.json` on the site): the spread of - screen deltas sets the screen threshold, and any confirmed regression in an - A/A run means the rounds or threshold are too loose. -- Slowed bench: dispatch with `bench-base` on a branch commit that speeds up - one cheap bench (so tonight's `main` reads as slower), and check the job fails - with that case alone. -- Tune `THRESHOLD_PCT`, `CONFIRMATIONS` (`bench/compare.py`) and the round count - (`BENCH_RUNTIME_ROUNDS`, default three) from what the runs show, and - update the README's numbers. Bimodal series (about 50% up and back across - nights) are the expected false positives; three alternating rounds pass one - only when all three land on the slow mode. -- Check the paired legs' durations against their hour; the base build relies - on the mbx cache for the previous night's artifacts, and a cold one roughly - doubles each leg's build time. -- Each revision in a paired comparison needs its own `CARGO_TARGET_DIR`. - `bench/run.sh` already splits them (`target/bench/current` and - `target/bench/base`); the stale artifacts while attributing - [moq-dev/moq#7141](https://github.com/moq-dev/moq/pull/7141), such as a bench - linked against the other tree's library, came from an ad-hoc A/B of two - source trees sharing one. Say so in `bench/README.md`'s paired section, so an - A/B outside `run.sh` uses its base mode or a target per tree (decided - 2026-10-28). - -- Decide whether `bench/run.sh` pins glibc's malloc thresholds for every - Criterion target, as [moq-dev/moq#6727](https://github.com/moq-dev/moq/pull/6727) - did for `ts_schedule`: some small paired deltas may be the same heap mode flip. +Decided 2026-11-05 (maintainer, after +[moq-dev/moq#7928](https://github.com/moq-dev/moq/pull/7928)'s A/A runs): +authorize one A/A Nightly dispatch for `rust-bench` once its paired leg fits +its hour, for the agent that fixes its benches to dispatch straight after. +Not chosen: asking again once it fits. + +Decided 2026-11-05 (maintainer, /quest-unblock): authorize two A/A Nightly +runs and one slowed-bench run on `main` after +[moq-dev/moq#7940](https://github.com/moq-dev/moq/pull/7940) merged, +replacing the earlier one-run authorization. The two A/A runs below spent it; +the slowed run was not needed, since every paired group runs the same gate +step. + +Calibrated 2026-11-05 for the shared, supplemental (`rust-extra`), io_uring +and `rust-relay` groups; `bench/README.md` records the numbers. Threshold 10%, +20 confirmations, and 3 rounds stay as they were: + +- A/A runs [41921320690](https://github.com/moq-dev/moq/actions/runs/41921320690) + and [41933525520](https://github.com/moq-dev/moq/actions/runs/41933525520) + (the first cancelled by Nightly's concurrency group after its bench + artifacts uploaded) confirmed no regression. Screen crossings of 10% either + way: shared 18 and 26 of 716, supplemental 10 and 15 of 460, io_uring 1 and + 1 of 41; at most 17 slower in one group, under the 20 confirmations. The + largest median among round sets that all agreed on a slowdown was +5.6%. +- A/A runs [42102235726](https://github.com/moq-dev/moq/actions/runs/42102235726) + (`main` at `62ae716265`) and + [42119312487](https://github.com/moq-dev/moq/actions/runs/42119312487) + (`83d0bc46ba`), dispatched after + [moq-dev/moq#7940](https://github.com/moq-dev/moq/pull/7940) merged, each + verified with `head_sha` equal to `bench-base`. Crossings: shared 20 and 19 + of 728, supplemental 7 and 8 of 514 and 532, io_uring 3 and 2 of 41, relay + 15 and 37 of 410; at most 17 slower in one group. Relay confirmed nothing in + either run and is calibrated. The second run confirmed nothing. The first + failed the shared gate on `wire_decode/padded/packets_64`: screen +10.3%, + rounds +11.9, +12.9, +12.1%, with `padded/packets_1024` at +7.1%. The same + case read +2.1% in the second run and +5.4, -0.8 and +6.2% in the three + earlier runs. +- Slowed-bench run + [41979277343](https://github.com/moq-dev/moq/actions/runs/41979277343) + compared `main` at `c150456928` against a base that collects 192 of + `pattern-algebra/collect-literal/256`'s 256 literals (branch + `bench-ci/slowed-base`, deleted). The shared job failed at its gate with + that case alone: screen +35.7%, rounds +34.6, +35.1, +35.2%. The + supplemental and io_uring groups passed. +- In that run, the rounds rejected the three `stats/tick/*_10pct` cases the + screen flagged, but each one's rounds agreed on a slowdown, at medians of + +9.3 to +9.7%, and every `100pct` case screened +6 to +9%, although + `moq-stats` was source-identical on both sides. The same cases sat within 6% + either way in both earlier A/A runs. A local build of `producer_tick` at both + commits, in separate trees and target directories, gives byte-identical + `.text`, `.rodata` and `.data.rel.ro`. + +Find the source of that bias, then fix it in `bench/run.sh`. It moves between +runs rather than following the source: an A/A run builds the same commit on +both sides, yet `wire_decode/padded/packets_64` agreed on +12% in one run and +not the next. A paired round reruns the same two binaries, so rounds cannot +filter it. Rerun the rounds against one binary in both slots on the ARM +runner to tell build differences from run placement. Lowering the threshold +waits on that fix. + +`rust-bench` is not calibrated: its leg hit its 60-minute timeout in both +runs, before uploading an artifact. Building both sides took 16 to 18 +minutes. The screen and rounds then ran past 40 minutes, because confirmation +rounds reran benches whose untimed work dominates their wall time. A round +side meant to take about 3 seconds took 2 minutes for +`interests/covered/1000w_1000s` and 8 for +`publisher_covered_before_start/wake/8192`: each builds its fixture in the +`bench_function` closure outside `b.iter`, and Criterion calls that closure +once per sample. `track_evict_fanout/512r/8` took about a minute, because its +untimed reads run inside every iteration. Make those benches pay their setup +once per benchmark, and bound the rounds by time if the leg still does not +fit. + +A dispatch cancels any in-progress Nightly through its concurrency group, and +the 07:00 UTC scheduled run can cancel a dispatch. Dispatch clear of that +window, and verify that the run's `head_sha` equals `bench-base` before +treating it as A/A evidence. + +Retain the earlier bias cases when reading `rust-bench`'s A/A results: +[moq-dev/moq#7139](https://github.com/moq-dev/moq/pull/7139) found +`moq-net/group_relay_frames/plain/*` (+23%) and +`moq-net/origin/serve_idle/3000r` (+46%) agreed in every round despite equal +callgrind instruction counts. `codec_messages/*/lite` (+31%, +68%) changed +because a new bench group shifted LTO inlining. +[moq-dev/moq#7253](https://github.com/moq-dev/moq/pull/7253) found +`server_admission/workers=4/0` (+10%) had identical machine code apart from +addresses. Include moq-quic's windowed `benches/streams.rs` cases too: an +unchanged base varied 17 to 43% between back-to-back runs on a shared host +([moq-dev/moq#6950](https://github.com/moq-dev/moq/pull/6950)). + +Decided 2026-11-05 (maintainer, after the A/A false alarm at +12.1%): +`rust-shared`'s gate is report-only until the build bias is fixed, and the +threshold stays at 10%; Nightly's `report-only` matrix flag skips its gate +step. Restore the gate once an A/A run of the fixed harness confirms nothing. +Not chosen: keeping the gate failing at 10% (the 2026-10-23 decision's +default); raising the threshold to about 15%, which would hide real +regressions of 10 to 15%. + +Decided 2026-11-05 (agent, recorded in +[moq-dev/moq#7928](https://github.com/moq-dev/moq/pull/7928)): `bench/run.sh` +does not pin glibc's malloc thresholds for every Criterion target. The largest +A/A screen outliers (`moq-mux/containers`' `ts_si_*`, +35 to +49% in both A/A +runs) were each rejected by rounds within a few percent, so a heap mode flip +costs confirmation budget but no false alarm, and a harness-wide pin would +shift every trend line and measure an allocator production does not run. A +bench whose rounds show a mode flip pins its own heap, as `ts_schedule` does. Decided 2026-10-23 (maintainer), when the paired mode landed: - The gate fails Nightly from day one rather than reporting only until A/A runs: Nightly already alerts for unrelated legs, so a false paired alarm adds no new channel, and this quest tunes it. -- Start at a 10% threshold, 20 confirmations per group, and 3 rounds; 10% - matches the trend table's floor, and three-round sign agreement filters - bimodal cases. -- `rust-shared` splits out of `rust-bench`, restarting the `rust/shared/*` - trend history (three nights lost), since `rust-bench` paired would exceed - its hour unsplit. - A target the base cannot build or run fails the gate rather than passing uncompared; it clears by itself the next night. @@ -84,40 +131,12 @@ Rejected hosts (decided 2026-10-22, maintainer): the merge queue runner, the moq.pro load nodes, and a dedicated bare-metal host; revisit the last only if A/A runs show the hosted noise floor hides regressions worth catching. -Decided 2026-10-28 (maintainer), superseding the 2026-10-27 plan to wait for -A/A: CodSpeed's CPU simulation runs now, on every `main` commit, as an extra -signal beside this gate rather than a replacement (see -[CodSpeed](/bench/README.md#codspeed)). It covers only the single-threaded -benches without I/O, so this calibration still decides the paired gate. - -Observed 2026-10-28, before any A/A run: the last two scheduled Nightlies' -gates failed with only slowdowns, and their rounds mostly agree within a few -points. `22808bbb2` against `8810d7365` confirmed 5 (net -retirement selection about +23%); `270395a33` against `22808bbb2` confirmed -20 across three groups (net `codec_messages/encode/lite` +67.7%, -`group_write_frames/batch32/*` about +45%). A one-signed, tightly agreeing -result is either a real regression or a systematic bias between the base and -head builds or run order, which noise would not produce; the A/A run tells -them apart, and CodSpeed's counts for the cases it covers are a second -opinion. If the A/A run also confirms regressions, fix that bias; if it is -clean, bisect the reported cases and fix the regressions. - -The A/A calibration includes moq-quic's windowed `benches/streams.rs` cases: -on a shared host the unchanged base alone varied 17 to 43% between -back-to-back runs ([moq-dev/moq#6950](https://github.com/moq-dev/moq/pull/6950)). - -Confirmed regressions with no extra work behind them, found by -[moq-dev/moq#7139](https://github.com/moq-dev/moq/pull/7139): every round of -`moq-net/group_relay_frames/plain/*` (+23%) and `moq-net/origin/serve_idle/3000r` -(+46%) agreed, yet callgrind counts the same instructions per frame read and per -poll on both builds. A paired round reruns the same two binaries, so a code or -heap layout shift between them reads as a stable regression that rounds cannot -filter. `codec_messages/*/lite` (+31%, +68%) moved only because a new bench group -in the same binary changed what LTO inlines. -[moq-dev/moq#7253](https://github.com/moq-dev/moq/pull/7253) found a third: -`server_admission/workers=4/0` (+10%) runs machine code identical apart from -addresses on both builds of the 2026-10-27 pair. The other four 2026-10-27 cases -were real, and were fixed or accepted there. +Decided 2026-10-28 (maintainer): CodSpeed's CPU simulation runs on every +`main` push that touches Rust, Cargo, or Nix files, as an extra signal beside +this gate rather than a replacement +(see [CodSpeed](/bench/README.md#codspeed)). It covers only the +single-threaded benches without I/O, so this calibration still decides the +paired gate. ## Related