Skip to content

ci(bench): publish measured nightly benchmark trends - #5759

Merged
kixelated merged 6 commits into
mainfrom
quest/m1/bench-site
Oct 18, 2026
Merged

kixelated merged 6 commits into
mainfrom
quest/m1/bench-site

Conversation

@kixelated

@kixelated kixelated commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Important

Maintainer prerequisite before merge: enable repository Pages with Settings > Pages > Source: GitHub Actions. Merging does not change repository settings.

Without it, the main-branch Nightly's Benchmark site job still validates the reports and pushes the history to gh-pages, then fails at actions/deploy-pages with "Ensure GitHub Pages has been enabled", so the Nightly run goes red every night. It never skips silently.

Problem

Nightly retained binding results, but most Rust Criterion targets only smoke-tested with --test, and the JS measurements had no durable charts.

Approach

  • The Rust bench targets group measures its Criterion targets in quick mode through the existing evidence runner (just gh criterion, bench/run.sh --criterion). The JS group tees each benchmark's CSV. Both write a validated report.json; the binding benchmark already emits one.
  • A main-only Benchmark site job validates every bench-* artifact that records an outcome, then stores cost and throughput charts on one serialized gh-pages branch with github-action-benchmark and deploys them explicitly to Pages. Alerts and comments stay off.
  • Malformed, empty, nonfinite, or duplicate measurements fail validation. A failed suite contributes no point.
  • Fixes stale benchmark callers: the JS catalog bench uses the moq-clock Manual clock, protocol diagnostics stay off CSV stdout, and moq-net's announce_duplicate and reprice_duplicate benches run with a zero update_hold. Both currently fail Nightly on main.

Impact

  • Public API: none. Wire: none.
  • Nightly measures the selected Rust targets, including the new moq-quic and e2ee benches, and retains Rust and JS evidence for 45 days.
  • The "Rust benchmark tests" group (nextest) is not charted; its output is not structured.

Validation

just check and the tooling regressions (sh/gh/nightly.test.py, sh/gh/bench-report.test.py) pass. A measured local just gh nightly js produced 567 valid entries across all 16 suites, including the new e2ee bench (forward stopped early, see Follow-ups). A measured moq-quic wire + receive run through just gh criterion produced 32 Criterion estimates. The moq-net origin bench passes --test again. The pinned upstream action generated both chart families twice in a temporary repository and kept both history points. The full measured Rust nightly suite was not run locally.

Alternatives

Smoke mode supplies no Criterion estimates. Pushing a Pages branch with GITHUB_TOKEN alone does not deploy a site, so the publisher deploys its generated artifact explicitly.

Follow-ups

  • After Pages is enabled, verify the first main-branch Nightly deployment at https://moq-dev.github.io/moq/bench/.
  • Watch the Rust bench targets job time against its 60-minute limit; quick mode adds measurement to every case.
  • Paired regression alerts remain with bench-ci.
  • Charting the nextest benchmark tests needs a structured output first.
  • Regression on main, outside this PR: js/net/bench/forward.ts hangs at 100% CPU at 2048 held watches since feat(net): announce consumers report when they have synced #5724 (bisected: 866e72783~1 finishes in about 2 s, 866e72783 never finishes). Nightly's JS group will stall on it until fixed, and a failed group publishes no JS point.

(Written by Claude Opus 5.5, building on GPT-6's draft)

🤖 Generated with Claude Code

kixelated and others added 2 commits October 16, 2026 12:01
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Implementation and local checks are complete. This stays a draft for the maintainer's Pages choice: enable Source: GitHub Actions (recommended), then verify the first main-branch Nightly deployment; or keep the site disabled. Repository settings have not changed.

(written by GPT-6)

kixelated and others added 4 commits October 17, 2026 20:57
Chart the split Rust bench targets job and the new moq-quic and e2ee benchmarks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench-browser and bench-ffi-gen also match the bench-* download pattern; the browser report has its own schema.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The origin now holds route updates and restarts for 300 ms against its clock, so announce_duplicate and reprice_duplicate saw nothing pending and panicked. Both now run with a zero hold, as the auth renewal bench does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The moq-quic targets time thread CPU through their own Measurement, which Criterion does not record, so the report labelled them elapsed time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review October 18, 2026 04:22
@coderabbitai

coderabbitai Bot commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 10 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 24eb2fc8-ff88-40fd-8ce7-b103224d3afb
📥 Commits

Reviewing files that changed from the base of the PR and between 8bf21d3 and 1bfcd7e.

📒 Files selected for processing: 15
  • .github/justfile
  • .github/workflows/nightly.yml
  • bench/README.md
  • bench/metadata.py
  • bench/run.sh
  • js/net/bench/frames.ts
  • js/net/bench/requests.ts
  • js/publish/bench/catalog.ts
  • quest/m2/bench-site.md
  • rs/moq-net/benches/origin.rs
  • sh/gh/bench-publish.sh
  • sh/gh/bench-report.py
  • sh/gh/bench-report.test.py
  • sh/gh/nightly.sh
  • sh/gh/nightly.test.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-18T04:26:27.896711Z 1bfcd7e Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated

Copy link
Copy Markdown
Collaborator Author

Merging at 1bfcd7e. The maintainer has enabled repository Pages (Source: GitHub Actions), the prerequisite in the description. Codex thumbs-up on this head; CI green. Check the first main deployment at https://moq-dev.github.io/moq/bench/ after the next Nightly.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 18, 2026
Merged via the queue into main with commit 9fcb3f3 Oct 18, 2026
16 checks passed
@kixelated
kixelated deleted the quest/m1/bench-site branch October 18, 2026 05:19
kixelated added a commit that referenced this pull request Oct 19, 2026
Resolves sh/gh/nightly.sh against #5759 (resume joins the measured JS benches) and ports #5725's streamed largest to resumed JS readers: the serving copy's largest, never moving back, ending only with the route, as Rust's Consumer::poll_largest does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant