Skip to content

fix(scene): the nightly and the browser suite pass on every path - #78

Merged
kixelated merged 6 commits into
mainfrom
quest/m0/nightly-green
Nov 7, 2026
Merged

kixelated merged 6 commits into
mainfrom
quest/m0/nightly-green

Conversation

@kixelated

@kixelated kixelated commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem

quest/m0/nightly-green.md: the nightly has failed every night since 2026-11-04, and the browser suite (rs/moq-studio/tests/browser.rs) did not pass on every path. The latest nightly (run 42235026329, 2026-11-06) still had one failure, the same one as the 11-04 and 11-05 runs. The new v4l2loopback camera job had not run yet. Three causes, one per known failure:

  1. render-sw on GitHub's runner: every browser helper aborts with SIGTRAP. The suite first ran in the nightly on 11-04, when moq-dev/studio#8 landed it, and it has never passed there. Chromium's log says: FATAL:setuid_sandbox_host.cc:172 The SUID sandbox helper binary was found, but is not configured correctly. Chromium sandboxes its processes in unprivileged user namespaces. Ubuntu 24.04's AppArmor (kernel.apparmor_restrict_unprivileged_userns=1) lets only programs with a profile create them. So Chromium falls back to the setuid chrome-sandbox that CEF ships, which is not setuid root, and aborts instead of running unsandboxed. Reproduced on nixos by running the suite in a user namespace that may create no more (max_user_namespaces capped in a parent namespace): same FATAL line, same SIGTRAP.
  2. cookies_survive_a_restart on nixos2: no page served over HTTP ever paints. This is not about cookies at all. A plain http://127.0.0.1 page with a static background fails the same way, while data: pages paint. The net log shows the main-frame request entering URL_REQUEST_START_JOB and never reaching the HTTP cache, which is where Chromium waits for the cookie store to load. Chromium encrypts the profile's cookies with a key it reads from the desktop keyring (org.freedesktop.secrets). nixos2's GNOME keyring is locked (Locked = true; nixos's is unlocked), and its unlock prompt goes to a session with no monitor. So the cookie store never loads, and every network page stays Starting until the test's 20 s wait gives up. With --password-store=basic the same tests pass at once.
  3. copy_and_cut_reach_the_clipboard on the X11 GPU path. The test read only the helper's relay (Helper::copied()). Through an X display, CEF copies to the X clipboard itself (moq-dev/studio#59 kept that so page copy handlers apply), so the relay stays empty by design.

Approach

  • Sandbox: the helper passes --no-sandbox on Linux, so Chromium never needs user namespaces or a setuid-root chrome-sandbox. It runs every page unsandboxed, as OBS's browser source does. macOS and Windows have no browser sources yet and are unchanged.
  • Keyring: the helper passes --password-store=basic, so the profile's cookies never wait on the desktop keyring. This is the store Chromium already uses on a desktop without a keyring, which includes the CI runner.
  • X11 clipboard test: Helper::relays_copies() says whether copies come back through copied(). The test reads the X clipboard (a ConvertSelection of CLIPBOARD as UTF8_STRING, through x11rb, which the studio already depends on) where CEF copies itself, and the relay elsewhere. Each step now copies a different text, tagged with the test's process id, so a copy is told apart from whatever the X clipboard held before. The cut now targets a second text box. Raster::native_copy() is the one place that decides it, for the helper and for relays_copies().
  • The quest is finished and deleted.

Impact

  • Browser sources start on every Linux host, including stock Ubuntu 24.04+ (AppArmor) and the planned Flatpak, where every helper used to abort with SIGTRAP. In exchange, pages run without Chromium's sandbox: a page that exploits a browser bug can do anything the user can. README says so and to add only pages you trust.
  • Browser sources that load over the network no longer hang in Starting behind a locked keyring, for example under auto-login or with no monitor attached. In exchange, the profile's cookies are no longer encrypted with the keyring's key. Chromium's basic store uses a fixed key, so anything that can read the profile directory can read them. README says so. Cookies an existing profile saved under the keyring's key can no longer be decrypted, so those logins need signing in again once.
  • Nightly: the browser suite runs on stock Ubuntu 24.04, with no runner change.
  • Public API (moq-scene, unpublished, used only by moq-studio): adds browser::Helper::relays_copies() (Linux). Wire: none. The helper protocol is unchanged.

Decisions

Settled by the maintainer in #78 (comment), which replaced this PR's earlier choice to keep the sandbox and lift kernel.apparmor_restrict_unprivileged_userns on the CI runner (recorded in #78 (comment)).

  1. Chromium's sandbox, which aborts with SIGTRAP on Ubuntu 24.04+ where unprivileged user namespaces are blocked:
    • Keep the sandbox, and lift the restriction only on the CI runner. Stock Ubuntu users would still get SIGTRAP.
    • ✅ Disable the sandbox on Linux, as OBS does. It works everywhere, the planned Flatpak included, and the CI sysctl is no longer needed.
  2. Cookie storage, where pages blocked behind a locked desktop keyring:
    • ✅ --password-store=basic: never block on the keyring. The README says cookies in the studio's browser profile are not keyring-encrypted. A studio profile holds overlay logins, not saved passwords, and the keyring's protection against a process of the same user is nil once it is unlocked.
    • Keep the keyring. A locked keyring shows an unlock prompt that browser sources wait behind, possibly mid-stream, and with no monitor they wait forever.

Choices made without asking

Alternatives

  • Make the suite hermetic (no session bus, so Chromium picks the basic store by itself). That would hide the hang from the tests while users keep it.
  • Unlock nixos2's keyring. That fixes the host, not the studio.

Validation

Hosts: nixos (RTX 3070 Ti, GNOME with a monitor, under heavy load from other runners: load 30 to 65, about 37% iowait) and nixos2 (Intel UHD 630 running GNOME with no monitor, plus an RTX 2070), over ssh. Every GPU run used the logged-in GNOME session (XDG_RUNTIME_DIR=/run/user/1000): Wayland with WAYLAND_DISPLAY=wayland-0, X11 with DISPLAY=:0 and the session's XAUTHORITY. Every CPU run unset both, with Mesa's lavapipe as just render-sw sets it. "User namespaces blocked" ran the suite as the same uid in a user namespace whose parent caps user.max_user_namespaces at 1, so it can create none (unshare --user true fails with ENOSPC), as Ubuntu's AppArmor restriction does to Chromium.

Host Head Path binary(browser)
GitHub ubuntu-24.04 main (run 42235026329) CPU 0/4 run, SIGTRAP (setuid sandbox FATAL)
nixos, user namespaces blocked claim CPU same FATAL, SIGTRAP
nixos2 claim CPU 8/9, cookies: center stays black
nixos2 claim + --password-store=basic CPU cookies and two plain-HTTP pages pass
nixos claim X11 8/9, copy and cut: "the copy never came back"
nixos claim Wayland 9/9
GitHub ubuntu-24.04 63665fa (run 42345881655, dispatched, with the since-dropped sysctl) CPU 9/9; whole nightly green: Software render, v4l2loopback camera, macOS
nixos2 63665fa CPU 9/9
nixos2 63665fa X11 9/9; copies went to the X clipboard (relays_copies() == false, printed by a temporary eprintln!, not committed)
nixos 63665fa X11 9/9; copies went to the X clipboard (same check)
nixos 63665fa Wayland 9/9
nixos 63665fa + #76 merged (scratch) CPU, X11, Wayland 10/10 each
nixos2 63665fa + #76 merged (scratch) CPU, X11 10/10 each
nixos2, user namespaces blocked 6d78325 (sandbox on, control) CPU 0/9, every helper: setuid sandbox FATAL, SIGTRAP
nixos2, user namespaces blocked 03c845b CPU, X11 10/10 each
nixos, user namespaces blocked 03c845b CPU 10/10 (one earlier run: 0/10, every wait timed out at 20 s, no FATAL or SIGTRAP; see below)
nixos, user namespaces blocked 03c845b X11, Wayland 10/10 each
nixos 03c845b CPU 10/10
GitHub ubuntu-24.04 03c845b (run 42350943682, dispatched, no sysctl) CPU 10/10; whole nightly green on attempt 2: Software render, v4l2loopback camera, macOS. Attempt 1's Software render failed before the browser suite, in engine::tests::the_program_reaches_a_relay (decode: Mux(Moq(Stream(NotFound)))), which this PR does not touch; see below

Other checks:

  • 03c845b's parent merges origin/main at 3ff7dc9 (#76) without conflicts.
  • nix develop --command quest gate just render-sw on nixos2 at 03c845b, as CI runs it: 49 + 4 + 10 + 1 tests pass.
  • nix develop --command quest gate just ci on nixos2 (Linux) at 03c845b: passes, 278 tests, including both clippy runs (--features cef covers the test).
  • engine::tests::the_program_reaches_a_relay failed once on the runner (attempt 1 of run 42350943682): its decoder read Stream(NotFound) from the relay 0.6 s in. The PR changes nothing it builds or runs (it runs without cef). It passed on attempt 2, in just render-sw on nixos2, and 30 of 30 times under cargo nextest run --stress-count 30 on nixos2 at 03c845b, so it is a runner-only flake not reproduced here.
  • At 7533590 (the merge of feat(studio)!: audio scenes replace mixes #77 and fix(scene): GPU tests exit cleanly on NVIDIA #79), on nixos2: nix develop --command quest gate just ci passes (278 tests), and nix develop --command quest gate just render-sw passes (49 + 4 + 10 + 1).
  • quest check and quest check --base origin/main on macOS: pass. Nothing was built on macOS (disk space).
  • Wayland on nixos2 still hangs in CEF's initialize with no monitor (every test hits nextest's 60 s limit). That is the known no-wl_output hang that quest/m0/cef-headless.md waits on a dummy plug for, not this quest's. No helpers were left behind afterwards.
  • Timing on nixos depends on its load. Under 30 to 65 load and heavy iowait, two Wayland runs failed every wait at 20 s. A timed run showed 12.8 s inside Studio::new (wgpu device setup, before any CEF code), and the same single test took 2.9 s once the host settled. The full Wayland suite passed in two other runs on this head and in one with fix(scene): browser sources under X11 paint only on the X server's GPU #76 merged. At 03c845b, the first blocked CPU run on nixos (load 37, while a build held the cargo lock) timed out every wait at once with no helper failure; the next blocked run, an unblocked run, and a single blocked test all passed within 6 s. No retry or timeout was changed.

Follow-ups

  • Worth a quest: engine::tests::the_program_reaches_a_relay flaked on GitHub's runner with Stream(NotFound) (see Validation), which can fail the nightly before the browser suite runs. No open PR fixes it.
  • Worth a quest: the helper contacts Google in the background: the component updater (update.googleapis.com, edgedl.me.gvt1.com), the spellcheck dictionary (redirector.gvt1.com), network time (clients2.google.com), and www.google.com/async/folae. Each one is a privacy leak and a network dependency nobody asked for. Chromium's switches for these, such as --disable-component-update and --disable-background-networking, plus turning spellcheck off, would stop them, and the suite could then assert that no outside request is made.

Review

  • Reviewer: Codex (quest review --new, after the design change), gpt-6.1-sol, medium effort.
  • Round 1 of the new session, at 03c845b: no findings.
  • Reviewed head: 03c845b.
  • Head 7533590 adds only a merge of origin/main at 167cdca (#77, #79). The local merge had no conflicts (GitHub flagged quest link lists its merge cannot resolve), and the changed lines of git diff origin/main...7533590 match those of git diff 3ff7dc9...03c845b exactly, so the review above covers it.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

Quest-Claim: e5bfe46250aa1a7f
@pullfrog

pullfrog Bot commented Nov 7, 2026

Copy link
Copy Markdown

Pullfrog  | Review this ➔

kixelated and others added 2 commits November 6, 2026 22:11
…ly runs the browser suite

- The helper passes `--password-store=basic`: Chromium otherwise reads the
  profile's cookies only once the desktop keyring hands over its key, and a
  locked keyring with no one to answer its prompt held every network page
  in Starting (cookies_survive_a_restart on nixos2).
- copy_and_cut_reach_the_clipboard reads the X clipboard where CEF copies
  natively (the X11 GPU path), and the helper's relay elsewhere, through
  the new Helper::relays_copies.
- The nightly render job lets the runner create unprivileged user
  namespaces, which Chromium's sandbox needs; Ubuntu's AppArmor forbade
  them, and every helper aborted with SIGTRAP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Nightly and the browser suite pass fix(scene): the nightly and the browser suite pass on every path Nov 7, 2026
@kixelated
kixelated marked this pull request as ready for review November 7, 2026 06:55
@coderabbitai

coderabbitai Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 51 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: caaa451b-5fb0-4a92-bbac-19476eefa349
Commits

Reviewing files that changed from the base of the PR and between 167cdca and 7533590.

Files selected for processing: 7

  • README.md
  • quest/m0/README.md
  • quest/m0/nightly-green.md
  • rs/moq-scene/src/browser/host.rs
  • rs/moq-scene/src/browser/mod.rs
  • rs/moq-scene/src/browser/protocol.rs
  • rs/moq-studio/tests/browser.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated kixelated left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 6d78325
Base: 2bbdd5b

No actionable findings.

Direction: the clipboard test now observes the actual X11 path while sharing its routing predicate with the helper, and distinct copy/cut payloads avoid accepting an old clipboard value. The runner change permits Chromium's namespace sandbox rather than disabling Chromium sandboxing. The basic cookie store removes the locked-keyring dependency, with an explicit security tradeoff: profile cookies lose keyring-backed encryption and existing encrypted logins need signing in again. That is product behavior, not just a test fix; the README and PR disclose it. Stock Ubuntu sandbox setup and background networking remain identified follow-ups, not fixes delivered here.

Verification: GitHub-only static review of the changes and browser startup/copy paths. GitHub reports Software render, macOS and v4l2loopback camera successful in run 42345881655, at earlier head 63665fa. I did not run the final head's tests, inspect keyring/network traces, exercise an existing profile's migration, or reproduce the X11/Wayland hardware results. COMMENT only, not approval or security certification.

(Written by review (OpenAI))

@kixelated

Copy link
Copy Markdown
Contributor Author

Decisions (maintainer, /quest-complete, 2026-11-07):

  1. Chromium's sandbox, which aborts with SIGTRAP on Ubuntu 24.04+ where unprivileged user namespaces are blocked:
    • Keep the sandbox, and lift the restriction only on the CI runner.
    • ✅ Disable the sandbox on Linux, as OBS does. It works everywhere, the planned Flatpak included, and the CI sysctl is no longer needed.
  2. Cookie storage, where pages blocked behind a locked desktop keyring:
    • ✅ --password-store=basic: never block on the keyring. The README says cookies in the studio's browser profile are not keyring-encrypted.
    • Keep the keyring.

(Written by Claude Opus 5.5)

kixelated and others added 2 commits November 6, 2026 22:59
Chromium's sandbox needs unprivileged user namespaces, which Ubuntu 24.04's
AppArmor and a Flatpak deny, so every helper aborted with SIGTRAP there.
The maintainer chose to disable it, as OBS does, over lifting the restriction
on the CI runner, so the nightly's sysctl step goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Contributor Author

Design change: per the maintainer's decision in #78 (comment), the browser helper now runs without Chromium's sandbox on Linux (--no-sandbox, as OBS does) instead of the nightly lifting kernel.apparmor_restrict_unprivileged_userns on the runner. The nightly's sysctl step is gone, the README says pages run unsandboxed on Linux, and the "decide the sandbox at startup" follow-up is dropped. --password-store=basic stays. The next local review starts a new session.

(Written by Claude Opus 5.5)

@kixelated kixelated left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Incremental review of 03c845b against base 3ff7dc9, following review 5810654195 at 6d78325.

No new actionable implementation findings in the focused change. The helper now appends --no-sandbox, the nightly sysctl change is removed, and the README explicitly warns about unsandboxed pages. The host module is gated to Linux with the cef feature. This matches the subsequently recorded design decision and supersedes my previous review's statement that Chromium's sandbox remains enabled.

Direction: this is a material product security tradeoff, not just a CI adjustment. Chromium renderer isolation is disabled on every supported Linux host, including hosts where it would otherwise work; a renderer exploit is no longer contained by Chromium's sandbox. Keep the warning visible and treat all loaded page content and its subresources accordingly. The PR description still says the sandbox is kept and lists setup selection as a future decision; update those stale sections to match the committed code and the later design-change comment.

Verification limits: GitHub-only static comparison of the old/current base-relative changes, startup callback, Linux module gating and recorded decision comments. No browser launch, sandbox-status inspection, Ubuntu/Flatpak test or final-head CI execution was performed. The earlier successful sandboxed nightly is not evidence for this new configuration. No security certification or merge approval is implied.

(Written by review (OpenAI))

@kixelated
kixelated enabled auto-merge (squash) November 7, 2026 08:11
@kixelated

Copy link
Copy Markdown
Contributor Author

Pick (/quest-complete, maintainer away under /afk, recommended option): ✅ /quest-merge on head 7533590. Both maintainer decisions are applied, and the Codex review counts for this head, since the later merge changed none of the PR's own lines.

(Written by Claude Opus 5.5)

@kixelated
kixelated merged commit 2e47682 into main Nov 7, 2026
3 checks passed
@kixelated
kixelated deleted the quest/m0/nightly-green branch November 7, 2026 08:27
kixelated added a commit that referenced this pull request Nov 7, 2026
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant