Skip to content

fix(ci): get the Nightly workloads green - #5841

Merged
kixelated merged 6 commits into
mainfrom
quest/m0/ci/nightly-green
Oct 17, 2026
Merged

kixelated merged 6 commits into
mainfrom
quest/m0/ci/nightly-green

Conversation

@kixelated

@kixelated kixelated commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Repairs every workload that failed in Nightly run 38897397428 (b4eab0d), except those owned by quest/m0/release-link.md (Linux release builds, gst .deb/.rpm) and quest/m0/ci/platform-windows-drill.md. Also covers two that broke later.

Changes

Workload Root cause Fix
Audit rustls 0.23.45 has RUSTSEC-2026-0346..0349, and deny.toml didn't allow the kixelated/uniffi-rs git source Bump rustls to 0.23.46 and allow that one git source. Also drop the ignore for RUSTSEC-2024-0320, which no longer matches any crate.
Features moq-hls build_inits was #[cfg(feature = "server")] but tests call it Use the cfg_attr(..., allow(dead_code)) its siblings use
Features (would fail next) --all-features clippy: large_enum_variant on moq-mux catalog State allow with a reason, matching the crate's other state enums
Features (new on main) #5711 made moq-uring's transport name quic types unconditionally, so --no-default-features stopped compiling Gate those impls on quic; also silence the pre-existing dead-code warnings with the same cfg_attr
JS benchmarks lite-varint.ts missed Datagram.decode's timed argument. js/net/bench/tsconfig.json listed only 7 of 18 benches, so the PR that added the argument never typechecked this file Pass timed, and typecheck bench/*.ts. That also caught forward.ts missing goaway.
Browser backend baseline harness_begin kept a relative MOQ_TEST_RUNS, and wasm/run.sh then changes directory Resolve the root to an absolute path in harness_begin (fixes every harness). Added a regression test to harness.test.sh, which just test harness runs in interop CI. It fails without the fix.
moq-gst macOS formula gst-inspect-1.0 hangs at startup inside brew test's sandbox on the headless runner (GStreamer 1.28.8), with or without our plugin The formula test asserts the plugin exists and links libgstreamer-1.0, instead of running gst-inspect-1.0
Relay lock-wait profile perf's --call-graph dwarf requests the SVE VG register (sample_regs_user 0x4001ffffffff) on the SVE-capable ARM runner. The kernel refuses it for the software context-switches event (EOPNOTSUPP). Run that entry on ubuntu-24.04 (x86). A new per-entry runner matrix key defaults to ARM. Note added to bench/README.md.
Media (Firefox/WebKit player), new Playwright validates the browser's libraries with the ldd on PATH. In the dev shell that is Nix glibc's ldd, which can't see the apt-installed libraries, so it reports even libglib as missing. Set PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=1 on that step

This completes and deletes quest/m0/ci/nightly-green.md.

Evidence

  • Local: cargo deny check and just rs features (the full recipe) pass, as do just check, bun run check in js/net, and the lite-varint and forward benches. harness.test.sh passes, and fails against main's harness.sh.
  • just test wasm-baseline locally: the run directory now resolves to target/test/wasm-backend/..., and the relay logs land there. Chromium can't launch on this NixOS host, so the browser half of the run is unverified until Nightly.
  • moq-gst dry run dispatched on this branch (run 39000627149): Build tarball (aarch64-apple-darwin) passes, including Test Homebrew formula against tarball with the new formula test. The .deb/.rpm jobs fail on the known zigbuild link error that quest/m0/release-link.md owns.
  • Hosted-runner probes (throwaway branches, since deleted):
    • perf on ubuntu-24.04-arm: plain -c 1, --switch-events, -k monotonic, -m 2M, and --call-graph fp all open. Any --call-graph dwarf fails.
    • perf on ubuntu-24.04: the full bench command records.
    • Playwright 1.63 on ubuntu-24.04-arm: with system ldd, Chromium and WebKit launch. With Nix's ldd, WebKit is refused. With Nix's ldd and validation skipped, WebKit 26.6 launches.

Decisions

Settled by the maintainer (2026-10-17, recommended options).

  1. moq-gst macOS formula: brew test hangs.
    • The cause is not our plugin: a probe formula with no moq plugin hangs the same way on gst-inspect-1.0 --version.
    • gst-inspect-1.0 hangs at startup inside Homebrew's test sandbox. The last green run had GStreamer 1.28.6 on macOS 26 image 20260907; the first red one had 1.28.8 on image 20261013.
    • Outside the sandbox the same binary runs instantly, including under sandbox-exec with only LaunchServices denied.
    • Options:
      • ✅ A: drop the gst-inspect-1.0 call from the formula's test do and assert the plugin exists and links libgstreamer-1.0. The build job's unsandboxed smoke step already loads moqsink and moqsrc.
      • B: skip brew test only in sh/gh/gst-formula.py, leaving the formula unchanged.
      • C: wait for an upstream fix, leaving this Nightly job red until then.
  2. Relay lock-wait profile.
    • ✅ Run it on ubuntu-24.04 (x86). This is the smallest change that keeps the smoke test. Its Rust cache is keyed by arch, so this job builds cold.
    • Use the sched:sched_switch tracepoint, which needs tracefs permissions the runner doesn't grant.
    • Drop the profile from hosted Nightly.

Impact

Public API: none. Wire: none. CI-only, plus test, bench, and cfg changes.

Follow-ups

  • Confirm on the next scheduled Nightly. Triage the workloads that haven't run on main yet (relay shard sweep, rs send-depth). No quest tracks this once this PR merges.
  • Possibly report upstream: perf's arm64 user-register probe adds VG based on a hardware event, which then breaks software-event DWARF sampling.
  • harness_env still records a relative MOQ_TEST_RUNS in the printed rerun command.

Quest: quest/m0/ci/nightly-green.md (deleted).

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

@kixelated

Copy link
Copy Markdown
Collaborator Author

Outcome: draft. Nightly workloads that failed in run 38897397428 are fixed locally where reproducible and probed on hosted runners where not, plus two later breaks (moq-uring no-default-features, Firefox/WebKit media). Open: the moq-gst macOS brew test hang (decision 1, recommend A) and confirming the x86 move for the lock-wait profile (decision 2). Verification of the full set waits on the next scheduled Nightly.

(Written by Claude Opus 5.5)

@kixelated
kixelated marked this pull request as ready for review October 17, 2026 06:46
@kixelated

Copy link
Copy Markdown
Collaborator Author

Iterated with the maintainer's recommended choices: the formula test now checks that the plugin exists and links libgstreamer-1.0 instead of running gst-inspect-1.0 (verified by a moq-gst dry run on macOS, run 39000627149), and the lock-wait profile stays on x86. The quest is complete and deleted. All PR checks pass; no decisions remain. Full confirmation comes from the next scheduled Nightly.

(Written by Claude Opus 5.5)

@coderabbitai

coderabbitai Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 9 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 94a12516-3203-4406-9c3d-14747ea974bd
📥 Commits

Reviewing files that changed from the base of the PR and between a3870e0 and c48229e.

⛔ Files ignored due to path filters: 1
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing: 17
  • .github/homebrew/Formula/moq-gst.rb.tmpl
  • .github/workflows/nightly.yml
  • bench/README.md
  • deny.toml
  • js/net/bench/forward.ts
  • js/net/bench/lite-varint.ts
  • js/net/bench/tsconfig.json
  • quest/m0/ci/README.md
  • quest/m0/ci/nightly-ffi-reuse.md
  • quest/m0/ci/nightly-green.md
  • rs/moq-hls/src/export/mod.rs
  • rs/moq-mux/src/catalog/consumer.rs
  • rs/moq-uring/src/transport.rs
  • rs/moq-uring/src/udp.rs
  • rs/moq-uring/src/worker.rs
  • test/lib/harness.sh
  • test/lib/harness.test.sh
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-17T07:43:29.538167Z c48229e New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated
kixelated force-pushed the quest/m0/ci/nightly-green branch from 3c6010f to 06869a8 Compare October 17, 2026 06:47

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3c6010fe5a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread test/lib/harness.sh Outdated
@@ -81,6 +81,8 @@ harness_begin() {
root="${root%/}"
[[ -n "${MOQ_TEST_RUNS:-}" ]] || root="$root/moq-test-$(id -u)"
mkdir -p "$root"
# Harnesses change directory after this, so a relative root would move under them.
root=$(cd "$root" && pwd)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bypass CDPATH when absolutizing the run root

When MOQ_TEST_RUNS is relative and the caller exports a CDPATH containing a directory with the same name, cd "$root" resolves to that directory instead of the one just created by mkdir; Bash also prints the resolved directory, so the command substitution captures both that output and pwd, leaving root with two newline-separated paths and causing mktemp to fail. Resolve the explicitly created path without applying CDPATH.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in c48229e: CDPATH='' cd --. Checked with a CDPATH holding a same-named directory.

(Written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Rebased onto main (conflicts in nightly.yml with the Rust bench split #5823, and in quest files deleted upstream). After the rebase, Test and TS validation each failed once on known flakes unrelated to this diff: bursts_cross_a_transport_peer::impaired (quest/m1/test-flakes-2/relay-burst-impaired.md), and a 24 ms mp2 hole in TS compliance that also fails other branches today (#5854/#5857 address the live export burst loss). Both passed on rerun; all checks are green.

(Written by Claude Opus 5.5)

kixelated and others added 6 commits October 17, 2026 00:40
…seline workloads

- Bump rustls to 0.23.46 (RUSTSEC-2026-0346..0349) and allow uniffi-bindgen's
  UniFFI fork as a git source; drop the ignore for an advisory no crate matches.
- Gate moq-hls's build_inits like its siblings so the no-default-features test
  build compiles, and allow moq-mux's catalog State variant gap that only
  --all-features clippy sees.
- Typecheck every js/net bench, which caught lite-varint and forward drifting
  from their APIs.
- Resolve a relative MOQ_TEST_RUNS once in harness_begin, before a harness
  changes directory, so wasm-baseline's relay log lands where it reads it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The send-group adapters name quic types unconditionally, so the
no-default-features check in the Nightly features workload fails. Gate them
on the feature, and allow the worker and socket helpers only quic uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n media QA

- perf's DWARF unwinding on the SVE-capable ARM runner requests the SVE VG
  register, which the kernel refuses for the software context-switches event.
  The same recording opens on the x86 runner.
- Playwright validates Firefox and WebKit libraries with the dev shell's Nix
  ldd, which can't see the apt-installed ones, and refuses browsers that
  launch fine. Skip that check in the media-engines job.
- Narrow the quest to the moq-gst formula hang, which needs a decision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ing it

gst-inspect-1.0 1.28.8 hangs at startup inside brew test's sandbox on the
headless macOS runner, even for --version with no plugin involved. Assert
the plugin exists and links libgstreamer-1.0; the release build's smoke step
already loads it into GStreamer outside the sandbox.

The quest is complete, so delete it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated force-pushed the quest/m0/ci/nightly-green branch from 06869a8 to c48229e Compare October 17, 2026 07:40
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merge summary: repairs the Nightly workloads that failed on 2026-10-16 (Audit, Features plus the later moq-uring break, JS benchmarks, browser backend baseline, relay lock-wait profile, moq-gst macOS formula) and the new Firefox/WebKit media rows. Decisions settled by the maintainer: the formula test checks the plugin's GStreamer link instead of running gst-inspect-1.0 (✅ A), and the lock-wait profile runs on x86 (✅). Codex's CDPATH finding is fixed in c48229e. The quest is deleted; the next scheduled Nightly is the end-to-end confirmation.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 17, 2026
Merged via the queue into main with commit 87e8e3d Oct 17, 2026
17 checks passed
@kixelated
kixelated deleted the quest/m0/ci/nightly-green branch October 17, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant