Skip to content

ci(nightly): split the Rust benchmarks into two legs - #5823

Merged
kixelated merged 2 commits into
mainfrom
quest/m0/ci/nightly-rust-bench
Oct 17, 2026
Merged

kixelated merged 2 commits into
mainfrom
quest/m0/ci/nightly-rust-bench

Conversation

@kixelated

@kixelated kixelated commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Completes quest/m0/ci/nightly-rust-bench.md.

Summary

Nightly's Rust benchmarks job was cancelled at its 60-minute limit on 2026-10-16 (run 38897397428) before bench-gst-catalog finished and moq-cli's stream-import benchmark ran. This splits just gh nightly rust into two matrix legs, each with its own hour:

  • rust-test ("Rust benchmark tests"): nextest's ignored benchmark tests, release builds first, then test-profile ones.
  • rust-bench ("Rust bench targets"): every cargo bench target.

just gh nightly rust now fails with the usage message. sh/gh/nightly.test.py covers both groups.

Sizing (from that run's log, cold mbx cache)

Step kind Minutes
Dev shell entry, per leg 13.5
nextest --release (latency, readiness, playback, ack cleanup, feed fetch) 16.1
nextest test profile (STUN, archive/mux/hls, ffi; moq-cli never ran) 7.7 + moq-cli
cargo bench (moq-net + CPU clock, kio, shared build) 19.2
bench-gst-catalog 4.2+ (cancelled)

Estimated per leg: rust-test about 39 minutes, rust-bench about 44 minutes. That leaves about 15 minutes of headroom each, and the Nix cache should take 13.5 off each leg. These are estimates from one log. Decision 2 has the measured numbers.

Decisions

Settled with the maintainer:

  1. Where to split: by harness, nextest vs cargo bench, not the quest's release vs test profile. By run 38897397428's numbers, the release side would take 53+ minutes and still not fit. A harness split balances the two legs and keeps every Criterion run in one leg, so ci(bench): publish measured nightly benchmark trends #5759's switch to measuring mode only adds time there.

  2. Verify with a dispatch before merging: run 38995041213.

    Every step passed in both legs:

    Leg Job Benchmarks Shell entry
    Rust benchmark tests 25 min 21 min 4 min
    Rust bench targets 33 min 29 min 4 min

    The dev shell was mostly substituted this time. With the 13.5-minute uncached shell from the cancelled run, the legs would take about 35 and 42 minutes, still inside the hour.

  3. Job names: "Rust benchmark tests" (rust-test) and "Rust bench targets" (rust-bench). ci(bench): publish measured nightly benchmark trends #5759 will need one artifact per leg.

Public API / wire impact

None. CI only.

Checks

  • just check passes.
  • sh/gh/nightly.test.py passes locally from a scratch copy with #!/usr/bin/env bash stubs. The checked-in stubs use #!/bin/bash, which doesn't exist on a NixOS host; CI runs them on Ubuntu.

Follow-ups

  • Make sh/gh/nightly.test.py's stubs use /usr/bin/env bash so it runs on NixOS hosts.
  • Release nextest steps spend about 16 minutes compiling across five invocations, mostly because dev-dependency feature unification differs per package set. Merging them into one nextest invocation could cut that a lot.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 2 commits October 16, 2026 19:21
The Rust benchmark group was cancelled at its 60-minute limit on
2026-10-16 before moq-gst's catalog bench and moq-cli's stream import ran.
Split it by harness so each gets its own hour: nextest's ignored benchmark
tests (release builds, then test-profile ones) and cargo's bench targets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Outcome: the Rust benchmark group is split into rust-test (nextest ignored benchmarks) and rust-bench (cargo bench targets), each with its own hour. I left this as a draft because the split differs from the quest's recorded profile split (decision 1). The leg runtimes are estimates from run 38897397428 and haven't been measured; a dispatch would measure them (decision 2).

(Written by Claude Opus 5.5)

@kixelated
kixelated marked this pull request as ready for review October 17, 2026 02:28
@coderabbitai

coderabbitai Bot commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 17 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 14c28e61-5412-42f2-9431-9fef473fbce9
📥 Commits

Reviewing files that changed from the base of the PR and between 3789ee2 and 546dd5f.

📒 Files selected for processing: 7
  • .github/workflows/nightly.yml
  • quest/m0/ci/README.md
  • quest/m0/ci/nightly-green.md
  • quest/m0/ci/nightly-rust-bench.md
  • quest/m2/bench-site.md
  • sh/gh/nightly.sh
  • sh/gh/nightly.test.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-17T02:30:41.504233Z 546dd5f Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated

Copy link
Copy Markdown
Collaborator Author

Decisions settled with the maintainer: split by harness (nextest vs cargo bench), keep the job names, and verify with a dispatch before merging. The dispatched Nightly passed both legs and ran every step: "Rust benchmark tests" in 25 minutes and "Rust bench targets" in 33. PR checks pass, and Codex reported no findings. Ready to merge.

(Written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Merging. This splits Nightly's Rust benchmark group into "Rust benchmark tests" (nextest's ignored benchmarks) and "Rust bench targets" (cargo bench), each with its own hour. The maintainer chose the harness split over the quest's profile split and kept these job names. The dispatched Nightly passed both legs in 25 and 33 minutes. PR checks pass, and Codex's review of 546dd5f had no findings.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 17, 2026
Merged via the queue into main with commit 1616287 Oct 17, 2026
98 of 119 checks passed
@kixelated
kixelated deleted the quest/m0/ci/nightly-rust-bench branch October 17, 2026 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant