Skip to content

fix(obs): stop-drain test fits the drain deadline; TSan suite runs again - #6277

Merged
kixelated merged 6 commits into
mainfrom
quest/m1/test-flakes-2/three-flakes
Oct 22, 2026
Merged

kixelated merged 6 commits into
mainfrom
quest/m1/test-flakes-2/three-flakes

Conversation

@kixelated

@kixelated kixelated commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m1/test-flakes-2/three-flakes.md.

Problem

Reproduction

  • just obs test on main failed deterministically before reaching any assertion: a tokio task-cell handoff reported as a race with only demangled Rust frames (from inlined debug info), which race:^_R misses.
  • With that suppressed, 4 of 8 loaded TSan runs (4 in parallel) failed CHECK(ended). Timing Stop() plus the destructor showed the 16 MiB frame took 740-810 ms of moq-net's one-second drain deadline on a single run and 930-1220 ms with 4 in parallel; every run over about 1050 ms failed. The deadline cutting the track off is intended, so the frame size, not the drain, was wrong.
  • One loaded run also failed o.GetConnectTime() > 0 right after IsLiveSession(): OnConnect published live before storing connect_time_ms.

Changes

  • cpp/obs/test/tsan.supp: also match Rust frames by their .rs source file, since a frame symbolized from inlined debug info is demangled and has no _R prefix.
  • cpp/obs/src/moq-output.cpp: store the connect time before the session reads as live, so OBS never sees a live session with the "never connected" sentinel.
  • cpp/obs/test/moq-output-test.cpp: the stop-drain frame is 2 MiB instead of 16 MiB. All lifecycle assertions are unchanged.
  • Quest: deletes the finished child; the epic keeps its own loaded just check --all runs, now sized [S], and its 2026-10-21 note names the two flakes.

Choosing the size

Mutation-checked under TSan with Stop() calling cancel() instead of shutdown():

frame cancelling Stop() caught loaded drain (8 parallel)
0 (keyframe only) 0/4 -
256 KiB 3/4 -
1 MiB 32/32 90-236 ms
2 MiB 32/32 93-266 ms
16 MiB - 930-1220 ms (4 parallel), half failed

2 MiB sits 8x above the size where detection gets unreliable and drains in about a quarter of the deadline under 8 parallel TSan runs.

Verification

  • Final build, 24 TSan runs of moq-output-test, 8 in parallel, on a host already at load 20-35: 24/24 pass (the unchanged suite failed 4 of 8 at 4 parallel).
  • js/net under load (host at load 22-33 during a cargo build): src/path.test.ts 30/30, whole @moq/net suite 3/3 (2494 pass each). No RSS-based test remains in js/.
  • just check: pass (obs compile and lint, quest, shell, markdown, gh).
  • Full just obs test (every TSan binary): pass; on main it fails at the first relay.
  • Not run: just obs ci (the plain build CI runs, which needs the .#obs shell); the same assertions passed under TSan.
  • quest check: ok.

Not covered

  • A destructor that drops the drain future instead of waiting is caught only about 7 runs in 8, at 16 MiB as well as 2 MiB: moq-net's drain outlives the session handle, so dropping the future aborts it only if the abort beats the spawned task's first poll. See follow-ups.

Decisions

Settled by the maintainer's go-ahead on the recommended course:

  • ✅ Scope: the TSan suppression and the connect-time ordering fix stay in this PR. Both block a loaded just obs test of this file (the first deterministically), so splitting them would leave this PR unverifiable.
  • ✅ Frame size 2 MiB rather than 1 MiB: both caught a cancelling Stop() 32/32 with similar drain times; 2 MiB doubles the detection margin.
  • ✅ The epic README, now childless, is sized [S] for its remaining loaded just check --all runs.

Impact

Public API: none. Wire: none. The connect-time fix changes only the order OBS observes two values.

Follow-ups

  • moq-ffi Session::shutdown drain request: it spawns the close detached with abort-on-drop, so a future dropped before its first poll never requests the drain and the session falls back to the implicit cancel. Requesting the drain synchronously would make "keep the future or the drain aborts" (the MoQOutput::Stop comment) unnecessary, and make a destructor that drops the drain reliably detectable.
  • just obs test nightly coverage: the TSan suite is manual, which is how it stayed broken on main unnoticed. Run it in a nightly.
  • Slow loaded TSan runs: some loaded runs of moq-output-test take 60-90 s instead of about 8 s, CPU-bound on one thread; not investigated.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 5 commits October 21, 2026 22:35
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
just obs test failed on its first relay: a tokio task-cell handoff reported
with only debug-info frames, which are demangled, so race:^_R missed it.
Match frames by their .rs source file as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reader that saw IsLiveSession() could still read connect_time_ms as 0,
OBS's never-connected sentinel. Seen as a loaded TSan failure of the
real-session publish test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 16 MiB frame took 750-800 ms of moq-net's one-second drain deadline
under TSan and overran it on a loaded host, so the deadline cut the track
off by design. 2 MiB still catches a Stop() that cancels instead of
draining, and drains in under 300 ms under the same load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Outcome: quest complete, opened as a draft only so the maintainer can settle the three choices under Decisions (scope of the two ride-along fixes, 2 MiB frame, [S] epic size); each is recommended as-is. Reproduced the drain failure (4/8 loaded TSan runs, every one past about 1.05 s of drain), fixed by sizing the frame inside the deadline with mutation-checked detection, and verified the merged #6188 RSS replacement under load. After settling, run /quest-ready.

(Written by Claude Opus 5.5)

@kixelated
kixelated marked this pull request as ready for review October 22, 2026 06:46
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-22T15:56:02.900347Z 5b202ba New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 12 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 128aab2f-8c40-46ab-8b89-275ac5dfc995
📥 Commits

Reviewing files that changed from the base of the PR and between d44813c and 5b202ba.

📒 Files selected for processing: 5
  • cpp/obs/src/moq-output.cpp
  • cpp/obs/test/moq-output-test.cpp
  • cpp/obs/test/tsan.supp
  • quest/m1/test-flakes-2/README.md
  • quest/m1/test-flakes-2/three-flakes.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merged origin/main to clear the conflict (mechanical). Only quest docs conflicted: main had narrowed the test-flakes-2 child to the OBS stop drain, and this PR finishes that child, so three-flakes.md stays deleted and its README entry is dropped. quest check passed both plain and with --base origin/main. just obs test (TSan) and just obs ci both pass on the merge, including "stop drains the session".

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 22, 2026
Merged via the queue into main with commit d4cdfe4 Oct 22, 2026
11 checks passed
@kixelated
kixelated deleted the quest/m1/test-flakes-2/three-flakes branch October 22, 2026 17:28
kixelated added a commit that referenced this pull request Oct 23, 2026
Nothing ran `just obs test`, so it failed on main unnoticed until #6277.
A Nightly lane now runs it in the plain dev shell: the tests stub libobs,
so the `.#obs` shell's obs-studio is not needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant