Skip to content

test(ts): audit random-loss decisions with bufferbloat - #6000

Merged
kixelated merged 4 commits into
mainfrom
quest/m1/random-loss-audit
Oct 19, 2026
Merged

kixelated merged 4 commits into
mainfrom
quest/m1/random-loss-audit

Conversation

@kixelated

@kixelated kixelated commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

holes.py --shed rejected every legacy audio frame in a shaped headroom capture: each MP2 PID's 4,912 frames were called out of order. Hang's legacy container converts their timestamps to microseconds before TS floors them back to 90 kHz. The grader compared raw source timestamps instead of that documented conversion.

The random-loss decision audit also found stale performance inputs after #5918: the BBR study requested a 10% random-loss target, and cut-through swept random loss for its go/no-go.

Approach

  • Compare legacy audio against its exact wire timestamp conversion. Regression controls reject larger shifts, missing tails, and shifted raw video. They run in existing TS CI steps.
  • Use the measured four-audio-PID headroom input for both semantic nightly congestion lanes. Run passthrough behind the measured overflowing CUBIC queue.
  • Refuse --loss with --rate or --live before build/network startup. Keep standalone random-loss correctness checks, guarded by a positive control in existing ts-controls CI.
  • Rework future impairment performance inputs to finite queues below the media rate with verified overflow. Delete the completed audit quest and its references.

Verdicts

Decision / origin Verdict Evidence and action
Fixed-delay jitter, half-delay skip budget, bounded acquisition (#4613, #4645) Keep HRD and multi-audio shaped captures below preserve T-STD, continuity, PCR, and complete tails while shedding. Correct stale with_delay docs to describe the existing two-delay acquisition bound and latest-sent-track anchor. No claim that either bound is optimal.
BBR fixes accepted using 2% random loss (#4193), ported in #4921 Keep moq-dev/noq#3-#9 correct packet identity, once-only feedback, starvation notification, initial RTT pacing, ProbeRTT samples, and recovery snapshots. Their regressions remain correctness assertions; the shaped BBR drop control below rechecks current end-to-end behavior. Individual fixes' performance contributions were not isolated.
BBR defaults (#1706, #2468, #4921), 2% LOSS_THRESH Keep #2468 already recorded bufferbloat RTT evidence; #4921 explicitly settled the current default. Current shaped runs pass without changing either policy. The uncommitted 20% experiment from #5805 is not adopted.
BBR study's 10% target and loss discrimination (#5805) Rework Remove the random-loss target and obsolete ceiling table. Retain independent natural-drain, growth-recognition, and probe-stop studies, each allowed to keep the baseline. Deep BBR runs without drops cannot judge its loss response; CUBIC success cannot establish a BBR improvement.
Former GCC premise, replaced by realtime uplink (#5889) Keep approved scope Current goals measure uplink queue delay with an explicit opt-in controller. No random-loss performance target survives there. This audit neither implements nor selects a controller, and preserves #5863's approved probing goals.
Nightly HRD selection and random-loss quality lanes (#5805, #5856) Rework Retain #5918's shaping migration; switch semantic lanes to measured headroom after fixing its false audio rejection. Passthrough now also requires actual queue overflow. No random-loss nightly gate remains.
Exact source-PTS matching under shedding (#5918) Rework Legacy MP2/AC-3/E-AC-3 uses the exact microsecond-prefix conversion back to 90 kHz. The same captured failure passes every grader after correction; raw video remains exact. No arbitrary timestamp tolerance.
Disable WebSocket fallback; verify impairment activation (#5805) Keep TCP fallback would measure another path. Reports below prove shaping acted; CUBIC lanes require actual overflow. Keep the deep BBR lane's relay-shedding evidence separate. Refuse mixed random-loss/congestion or random-loss/live-timing grading.
passthrough.py --loss-ok (#5856) Keep shaped-only use Packet identity and the complete source tail still gate. Actual omissions preserve source continuity counters, so capture compliance is report-only. Standalone random loss still requires complete content.
Relay drills; QUIC simulated/scripted loss (#5918, #5780, #4921) Keep correctness scope Drills assert cleanup, abort/rejoin and eventual delivery. Simulation asserts delivery/no deadlock and checks priority only on clean paths. BBR scripted drops exercise transitions; its capacity-growth measurement starts after seeding loss stops. None sets a random-loss media-quality target.
Probe, sim, deadlines, resume, cut-through, TS recovery inputs Rework Preserve their goals; require shaped impairment acceptance with verified overflow. Keep random/scripted loss only for recovery correctness. Preserve probing's 3→12→3 Mb/s capacity scenario and add a separate below-media congestion arm.

Measurements

Runtime baseline: 6c357679a. Production behavior is unchanged on this branch. Commands used pinned Nix, debug builds, CARGO_BUILD_JOBS=2, NEXTEST_TEST_THREADS=4, seed 1, 25 ms each way, and the existing 1 s export delay on a shared host. No random loss was configured. CUBIC means MOQ_QUIC_CONGESTION_CONTROL=loss.

All rows use just test ts, --rate 7000000, and --duration 120, except the BBR drop control's 60 s. HRD is 9 Mb/s video in a 10 Mb/s multiplex; headroom adds four audio PIDs in 12.5 Mb/s; passthrough carries the whole 10 Mb/s multiplex.

Input / controller / queue Measured result
--hrd, BBR3, default 2 s PASS: 83,852 throttled, 0 overflowed; 37.9 s video shed; dropped=0/gaps=117. Exact tail, T-STD/continuity/PCR pass.
--hrd, CUBIC, --queue 250ms PASS: 86,360 throttled, 427 overflowed; 36.7 s video shed; dropped=25/gaps=117. Exact tail, T-STD/continuity/PCR pass.
--hrd, BBR3, --queue 50ms --duration 60 PASS: 40,342 throttled, 136 overflowed; 17.7 s video shed; dropped=0/gaps=57. Exact tail and all graders pass. This one-RTT drop-response control supplements the deep-queue runs.
--headroom, BBR3, default 2 s Original invocation FAIL at the false audio-PTS comparison. Same capture replay PASS after the fix, including all remaining T-STD/continuity/PCR/accuracy graders: 83,870 throttled, 0 overflowed; 62.5 s video and 3 AC-3 frames shed; no MP2 holes; dropped=5/gaps=118; every tail complete.
--headroom, CUBIC, --queue 250ms Fresh full invocation PASS: 86,463 throttled, 427 overflowed; 64.5 s video and 3 audio frames shed; dropped=74/gaps=120. Every tail and all graders pass.
--passthrough, CUBIC, --queue 250ms PASS under its raw-carriage contract: 85,444 throttled, 428 overflowed; 515,284 exact-position packets, 116 gaps, 0 missing tail packets. Its 458 continuity errors and source AAC T-STD failures are report-only, not a semantic-conformance pass.

File captures do not measure live release timing. These are acceptance outcomes, not a controller bake-off, fairness study, or natural-drain policy comparison. Passing local evidence remains under the worktree's ignored .scratch/runs/; every process finished and was reaped.

Impact

  • Production public API and wire: none; existing with_delay documentation corrected.
  • Private test harness: --loss combined with --rate or --live is now refused.
  • Nightly fixtures cover four audio tracks and real passthrough queue drops, without another nightly step.
  • No production controller, default, congestion threshold, or timestamp changes.

Alternatives

An arbitrary ±1-tick tolerance would accept shifts the exact conversion rejects. Changing production timestamps would alter carriage behavior instead of fixing the oracle. Raising LOSS_THRESH for 10% random loss or abandoning independently approved probing/realtime work does not follow from these results.

Follow-ups

Existing BBR/probe/realtime quests retain their independent studies and adoption decisions. No new maintainer decision is pending in this audit. The required full local check passed at frozen head 62730e278d56fda0c8e435adefc3e447e0ffbdc0. The PR is ready for review; no merge or auto-merge was requested.

Validation

  • Legacy timestamp regression: two captured codec cases fail without the fix; all 24 controls pass after it.
  • CLI mode control: the old combined mode reaches the protected build; both unsupported combinations now refuse early and standalone loss reaches the guarded build.
  • All six measured captures and outcomes are listed above; the original headroom failure is retained and replayed rather than relabeled as a passing invocation.
  • Nix shellcheck -x and shfmt --diff for both changed shell scripts: pass.
  • quest check: 436 documents pass. git diff --check: pass.
  • Full CARGO_BUILD_JOBS=2 NEXTEST_TEST_THREADS=4 nix develop --command just check: passed at 62730e278d56fda0c8e435adefc3e447e0ffbdc0 (formal exit 0).

(written by GPT-6.1-Sol)

kixelated and others added 4 commits October 19, 2026 13:10
Co-Authored-By: GPT-6.1-Sol <noreply@openai.com>
Keep the current BBR policy and approved probing and realtime goals. Rework future impairment acceptance to require shaped finite queues and actual overflow, while retaining scripted recovery correctness checks. Bufferbloat measurements and final audit verdicts remain in progress; quest check and git diff --check pass.

Co-Authored-By: GPT-6.1-Sol <noreply@openai.com>
Reproduce the false headroom rejection with MP2 and AC-3 timestamps, then compare their exact legacy microsecond conversion back to 90 kHz. Keep raw media matching and reject larger shifts and missing tails. Correct the existing jitter acquisition documentation. The regression fails twice without the fix; all 24 controls and all graders on the saved headroom capture pass. Fresh overflowing headroom and passthrough measurements remain in progress; full just check is reserved with the coordinator.

Co-Authored-By: GPT-6.1-Sol <noreply@openai.com>
Replace the semantic nightly HRD fixtures with measured headroom inputs and run passthrough behind the measured CUBIC overflow queue. Refuse random loss combined with rate shaping or live timing, with CI-wired refusal and positive controls. Delete the completed random-loss decision audit and its references; verdicts and exact measurements are recorded in the PR. All focused measurements have finished, shell checks and quest check pass, and the full just check remains reserved with the coordinator.

Co-Authored-By: GPT-6.1-Sol <noreply@openai.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Audit checkpoint: 62730e278d56fda0c8e435adefc3e447e0ffbdc0. Keep the current BBR policy and 2% threshold; use the measured headroom and overflowing passthrough fixtures. The original headroom invocation failed at the grader, and its exact saved-capture replay passes after the documented timestamp conversion fix. Full local just check remains pending; the verdict table records every measured outcome.

(written by GPT-6.1-Sol)

@kixelated
kixelated marked this pull request as ready for review October 19, 2026 22:21
@coderabbitai

coderabbitai Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 23 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7690f5fa-39c7-45e6-a890-fcaaf68acf83
📥 Commits

Reviewing files that changed from the base of the PR and between d0358f6 and 62730e2.

📒 Files selected for processing: 17
  • .github/workflows/ts.yml
  • quest/m1/README.md
  • quest/m1/random-loss-audit.md
  • quest/m1/ts-reliability/recovery.md
  • quest/m2/probe/native.md
  • quest/m2/quic-bbr-natural-drain.md
  • quest/m2/quic-deadline.md
  • quest/m2/quic-outage-probe.md
  • quest/m2/quic-sim.md
  • quest/m3/cut-through/bench.md
  • quest/m3/quic-careful-resume.md
  • rs/moq-mux/src/container/ts/export.rs
  • test/ts/README.md
  • test/ts/holes.py
  • test/ts/holes.test.py
  • test/ts/process.test.sh
  • test/ts/run.sh
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T22:24:16.252155Z 62730e2 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated
kixelated added this pull request to the merge queue Oct 19, 2026
Merged via the queue into main with commit a846cc5 Oct 19, 2026
13 checks passed
@kixelated
kixelated deleted the quest/m1/random-loss-audit branch October 19, 2026 23:50
kixelated added a commit that referenced this pull request Oct 20, 2026
Take main's random-loss audit verdicts (#6000) for the BBR study and drop
the deleted audit quest. Drop the finished 0-RTT window raise and the
retired datagram priority entries from the QUIC epic, take main's auth
epic, and recheck the audit checkpoint against current main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant