bench: calibrate the paired Nightly comparison from A/A and slowed-bench runs

.github/workflows/nightly.yml

@@ -120,12 +120,15 @@ jobs:
             artifact-path: target/bench/nightly/rust-bench
             benchmark: true
             paired: true
+          # Report-only until quest/m1/bench-ci.md fixes the build bias behind
+          # an A/A false alarm; the site still lists its confirmed regressions.
           - name: Rust shared bench targets
             recipe: gh nightly rust-shared
             artifact: bench-rust-shared
             artifact-path: target/bench/nightly/rust-shared
             benchmark: true
             paired: true
+            report-only: true
           - name: Relay session benchmarks
             recipe: gh nightly rust-relay
             artifact: bench-rust-relay
@@ -235,7 +238,7 @@ jobs:
       # After the upload, so the benchmark site still records tonight's history and
       # publishes the comparison that failed this job before the alert links to it.
       - name: Fail on paired regressions
-        if: ${{ matrix.paired && steps.workload.outcome == 'success' }}
+        if: ${{ matrix.paired && !matrix.report-only && steps.workload.outcome == 'success' }}
         run: nix develop --command just gh benchmark-report gate "${{ matrix.artifact-path }}"
 
   # gh-pages is the one Pages site, and each directory has one writer: this job owns

bench/README.md

@@ -890,8 +890,8 @@ the table requires no new renderer or data branch. A source's `paired.json`, as
 
 ### Paired comparison
 
-Each Nightly's Rust bench groups (`rust-bench`, `rust-shared`, `rust-extra` and
-`rust-uring`) also build the previous scheduled Nightly's commit beside their
+Each Nightly's Rust bench groups (`rust-bench`, `rust-shared`, `rust-relay`,
+`rust-extra` and `rust-uring`) also build the previous scheduled Nightly's commit beside their
 own, on the same runner, so night-to-night runner variance drops out. Each
 built target runs once per side in the trend's quick mode, alternating which
 side goes first; the current side is the trend point. A case more than 10%
@@ -901,8 +901,21 @@ measurement, 30 samples). A case whose rounds are all slower with a median
 beyond 10% is a confirmed regression and fails its job after the upload, so the
 Alert workflow posts the failed Nightly. A target the base cannot build or run
 fails the job too, rather than passing uncompared; a bench target the base does
-not have yet is only reported as new. The threshold is a starting point until
-A/A runs on the hosted runner measure its noise floor.
+not have yet is only reported as new. `rust-shared` is report-only for now: its
+confirmed regressions show on the site but do not fail the job, until the
+build bias below is fixed.
+
+Runs on the hosted runner set these numbers. In four A/A runs of the
+`rust-shared`, `rust-extra` and `rust-uring` groups, and two of `rust-relay`,
+1 to 9% of cases crossed 10% either way in the screen, at most 17 of them
+slower in one group. The rounds rejected all but one:
+`wire_decode/padded/packets_64` at +12%, which read +2% in the next A/A run.
+A run whose base cut one case's work by a quarter failed with that case alone
+at +35%; its rounds also agreed, at medians of +9.3 to +9.7%, on `stats/tick`
+cases whose source was identical on both sides. Separate builds of the same
+source can measure up to 12% apart, so the threshold cannot go lower until
+that bias is fixed.
+`rust-bench` is not calibrated yet: its paired leg outlasts its hour.
 
 The table page shows tonight's comparison above the trend: confirmed
 regressions, targets not compared, cases faster in the screen, slowdowns the
@@ -923,6 +936,11 @@ Each run's evidence directory keeps both builds' estimates (`criterion` and
 `paired.json`. The base build streams its diagnostics and retains them in
 `base-build.log`, including when it fails.
 
+Each revision builds in its own Cargo target directory (`base` and `current`
+under `CARGO_TARGET_DIR`, default `target/bench`). For an A/B comparison, use
+`run.sh`'s `BENCH_BASE` mode or a target directory per source tree: two trees
+sharing one can leave a bench linked against the other tree's library.
+
 ### CodSpeed
 
 The CodSpeed workflow adds instruction counts for every `main` commit that

bench/compare.py

@@ -157,7 +157,8 @@ def criterion_baseline(directory, discovery, benchmark):
 
 
 # A paired case beyond this change is flagged, and fails the nightly only when its rounds agree.
-# Loose until A/A runs on the hosted runner measure its noise floor.
+# Separate builds of unchanged code measured up to 12% apart in A/A runs, so lowering this
+# waits on fixing that bias (quest/m1/bench-ci.md).
 THRESHOLD_PCT = 10
 # Each alternating round costs seconds per case, so a noisy night confirms only its worst slowdowns.
 CONFIRMATIONS = 20

quest/m1/README.md

@@ -169,7 +169,7 @@ their end report, so an embedder's drain alert pages on real cuts.
 - [crates.io trusts the workflow](/quest/m1/crates-trusted-publisher.md) - maintainer action: every published crate gets a trusted publisher for `release-rs.yml`
 - [Rust releases publish through OIDC](/quest/m1/release-rs-oidc.md) - release-plz publishes with trusted publishing and `CARGO_REGISTRY_TOKEN` is revoked
 - [Main is no slower than release](/quest/m1/release-perf/README.md) - every benchmark where main is slower than release is fixed or accepted with a reason, checked by a final paired run before the cut
-- [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - A/A and deliberately slowed Nightly runs set the paired comparison's threshold, so its regression alerts can be trusted
+- [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - `rust-bench` fits its hour and gets A/A runs, and separate builds of unchanged code stop measuring as a slowdown, so every paired regression alert can be trusted
 - [moq-json's table/deltas bench measures the same work every iteration](/quest/m1/bench-json-deltas.md) - the `table/deltas/1%` cases stop flipping between snapshot and delta timings, so their trend history is usable
 - [Publish moq-bench to crates.io](/quest/m1/bench-publish.md) - `cargo install moq-bench` installs one binary, with the host sampler as `moq-bench host`, first as 0.1.0
 - [Unbounded groups](/quest/m1/unbounded-groups/README.md) - groups and frames lose their size caps; a group keeps a 32 MB tail, and readers behind it replay through paced flights or refresh