[kixelated]

Finishes QUIC scheduling churn from the performance survey.

Summary

The QUIC send scheduler kept each priority level's round robin of send groups in a VecDeque. Moving a group to another priority (Groups::set_priority, used by moq-lite subscription updates) or freeing a group that still held a round-robin place searched that deque linearly. On a session with many queued subscriptions at one level, each update or cancel cost about 5 instructions per queued group ahead of it.

Each level's round robin is now a circular doubly linked list threaded through the group slab (Ring { prev, next } on each group, with the level map holding its head). Leaving the round robin from anywhere is O(1). Push-back, pop-front, and the deficit rotation keep the same order. Groups that run dry away from the front still keep their place until the round robin reaches them. No scheduling policy, public API, or wire change.

• rs/moq-quic/src/connection/streams/schedule.rs: the linked ring, plus a unit test covering removal from the middle of a ring and a reprioritized group going back to the tail.
• rs/moq-quic/benches/streams.rs: schedule/group_priority and schedule/group_retire (queued × changed groups, front/tail/mixed), and group_churn (send groups churned through the public API under real packet exchange). streams already runs nightly in just gh nightly rust-shared and on CodSpeed, so no workflow change is needed.

Measurements

All measurements are on an Intel Core Ultra 7 270K Plus (Linux 7.0), Rust from nix develop. Each process is pinned to P-core 6 (taskset -c 6), and user-space instruction counts come from perf_event_open / perf stat -e instructions:u. Base is d42abb2a86. Base and candidate runs alternated, two rounds each. The A/A control (base against itself) matched exactly in every scheduler case.

Scheduler alone. A scratch harness compiled both schedule.rs versions from source into one binary. N groups at one priority each had a queued stream. Each run moved M of them to another level, or retired M of them (close the group, then its last stream leaves). Instructions per operation:

N queued	M changed	order	priority base → cand	retire base → cand
16	16	tail	378 → 235	723 → 622
256	16	tail	1,569 → 228	1,915 → 615
256	16	mixed	1,071 → 228	1,424 → 615
4,096	16	middle	12,110 → 228	12,459 → 615
4,096	16	tail	20,765 → 228	21,115 → 615
4,096	256	mixed	10,540 → 224	11,028 → 704
16,384	256	tail	81,556 → 224	82,078 → 739
16,384	4,096	mixed	38,460 → 224	38,968 → 717
any	any	front	272–342 → 225–235	568–714 → 611–762

• Cost no longer grows with N or with position.
• Retiring from the front, already the cheap case, got 43–53 instructions slower (+6–8%), because unlinking does a bit more than pop_front.
• Flipping one tail group between two levels (repeat, 256 flips) went from 10,701 to 288 instructions per flip at N = 4,096. It also went from 128 allocations to 0, because a level no longer allocates a deque.
• Moving groups to a new level allocated 3–11 times per run before and 0 now.

Dispatch during churn. N groups were kept backlogged, with 100,000 pop/sent dispatches. Every 8th dispatch also cancelled a random group and opened a replacement, and every 8th (offset) moved a random group to the other level. Instructions per dispatch, churn included:

N	base p50 / p99 / max	cand p50 / p99 / max
16	481 / 1,618 / 2,223	384 / 1,392 / 2,020
256	481 / 2,373 / 3,077	384 / 1,392 / 2,107
4,096	481 / 16,267 / 21,383	384 / 1,392 / 1,609

Connection, real packets. The group_churn bench gives one send group per subscription on one connection. Each round writes a 256-byte stream in every group. With that data queued, it moves churn groups to another priority and back and cancels churn others (reset, close, reopen), then drives until the server has seen every stream. The bench was built twice (base and candidate schedule.rs). Each figure is one --test pass of the case minus a pass whose filter matches nothing, so setup cancels out. Instructions per round:

subs × churn	base	cand	change
16 × 4	299,417	295,008	−1.5%
256 × 16	2,508,719	2,489,500	−0.8%
4,096 × 16	38,037,508	37,305,239	−1.9%
4,096 × 256	40,183,158	36,813,862	−8.4%

(Rounds agree within about 1.5k instructions. With an unlimited controller the backlog drains every transmit, so only groups queued between writes and the next transmit get scanned. A congested session keeps groups backlogged across transmits, which makes the scheduler-alone case the closer model.)

Thread-CPU timing of the same cases on this shared host (load average about 13) stayed within criterion's noise for 16 churned groups. The instruction counts are the evidence here.

Equivalent work. A differential trace ran 200 seeds × 50,000 random operations on both versions: open, close, and reprioritize groups; join, leave, and reprioritize streams; push, evict, and send datagrams; pop/sent with fair and unfair requeue. Every seed produced an identical hash of every pop result and eviction outcome. A temporary counter confirmed each trace removes a group from mid-ring about 2,300 times.

Memory. Group grows from 96 to 120 bytes. Each level's VecDeque (a 24-byte value, plus a heap buffer of 8–16 bytes per ringed group, allocated per level) becomes one usize head. The net is about +8 to +16 bytes per group.

Nightly. The new cases run in just gh nightly rust-shared (--bench streams). The new benches add 24 scheduler cases and 4 connection cases. Locally, cargo bench -p moq-quic --features fuzz --bench streams -- --test group_ runs each case once, and all of them pass.

Checks

just check origin/main passed rs (clippy, then nextest over moq-quic and its dependents), gh, drill, shell, toml, nix, and justfile. Markdown passed on a re-run: the first run raced the quest deletion. just rs check skips the fuzz-gated benches, so I also ran cargo clippy -p moq-quic --features fuzz --all-targets -D warnings and cargo clippy -p moq-net --features async --bench schedule -D warnings (that bench compiles schedule.rs from source), plus a --test pass of the moq-net schedule bench (60 cases).

Choices made without asking

• Landed the optimization. By the survey's contract, a measured consumer benefit has to justify the complexity. Here the per-operation cost is now flat in queued groups (16–47× fewer instructions at 4,096 groups in mixed order, and up to 370× at the tail of 16,384), dispatch p99 under churn at 4,096 groups drops 11×, and a connection churning 256 of 4,096 subscriptions per round spends 8.4% fewer instructions. The cost is about 50 lines of linked-ring code. Order is unchanged, as the differential traces show.
• Kept lazy removal of groups that run dry away from the front. Removing them eagerly is now cheap, but a group that refills before its turn would then go to the back. That changes scheduling policy, which the quest rules out. The comment on deactivate now gives that as the reason instead of the old scan cost.
• Dropped the "domains × retired domains" axis. Scheduling domains were removed in moq-dev/moq#6809, after the quest was written.
• Confirmed at the moq-quic session level, not through moq-net/relay. The scheduler is per connection. Subscribers on separate sessions get independent schedulers, and moq-net maps a subscription's subscriber and publisher priority to one group priority (priority::group). So the publisher axis decides how groups spread across levels, and the scan only touched groups at the same level. The connection bench uses the worst case: every subscription at one level. A full relay run would add relay costs on top of the same scheduler path.
• Bench pairs. The nightly scheduler cases use (queued, changed) ∈ {(16,16), (256,16), (4096,16), (4096,256)} × {front, tail, mixed}, to keep nightly time bounded. Middle and the 16,384-group sizes were measured once here (tables above). Middle cost sits between front and tail.

Coordination

moq-dev/moq#7490 (draft, quic-api/waits) touches moq-quic's poll_transmit, streams/{mod,send,state}.rs, and benches/common/mod.rs. This PR changes only streams/schedule.rs and benches/streams.rs, so the two don't overlap. moq-dev/moq#7388 touches streams/state.rs only.

Measured no-win

I tried trimming the front-retirement regression by unlinking through a single BTreeMap::entry lookup. It saved 9 instructions there but cost 24 per priority move, 34 per mid-ring retirement, and 10 per dispatch, so I reverted it.

Follow-ups

None.

Review

Round 1: Codex (quest review), gpt-6.1-sol, medium effort, against origin/main ba421c30d3. Reviewed head 1077a17ef620323b2c55627778ea0942b9114001. No findings.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 2 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 203e6925-02dd-4e12-bd3f-6da8e90d2ad3

Commits

Reviewing files that changed from the base of the PR and between ba421c3 and 1077a17.

Files selected for processing: 4

• quest/m2/performance-survey/README.md
• quest/m2/performance-survey/quic-schedule-churn.md
• rs/moq-quic/benches/streams.rs
• rs/moq-quic/src/connection/streams/schedule.rs

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[kixelated]

Automated review by review (OpenAI)

Reviewed commit: 1077a17ef620323b2c55627778ea0942b9114001.

No actionable bugs found. Checked ring insertion/removal (including singleton and head removal), deficit rotation, reprioritization, lazy dry-group retention, slab retirement, and the new scheduler/connection benchmarks. The slab-linked ring directly removes the linear scan while preserving queue order; its localized complexity is proportionate, with no public API or wire change.

Verification limits: static source review, including an independent scheduler pass. I did not run tests, benchmarks, or the reported differential trace, and have not independently reproduced the performance figures.

(Written by OpenAI)

[pullfrog]

✅ No new issues found.

Reviewed changes

• O(1) round-robin removal: schedule.rs replaces each priority level's VecDeque<usize> with a circular doubly linked Ring { prev, next } on each group, with levels: BTreeMap<i64, usize> holding the head. unring splices a group out from anywhere instead of scanning, so set_group_priority and retirement no longer cost O(queued).
• Preserved policy: deactivate still only unlinks the level head, so a group that runs dry away from the front keeps its place; free_group unrings before groups.remove, and set_group_priority unrings before mutating priority, keeping the ring and head map consistent.
• Tests: new groups_leave_the_round_robin_from_anywhere covers mid-ring removal and a reprioritized group returning to the tail.
• Benchmarks: schedule/group_priority, schedule/group_retire (queued × changed × front/tail/mixed), and real-packet group_churn.
• Quest docs: deletes quic-schedule-churn.md and its README line.

  ^{｜ View workflow run ｜ Using deepseek-v4.1-flash (free via Pullfrog for OSS) ｜ 𝕏}

[moq-bot]

codex / Codex land fixer note:

The landing log reports callback timeouts in moq-c tests test::video_raw_decode_resize, test::video_raw_decode, test::video_raw_publish_consume, and test::video_raw_decode_rgba. These tests use local broadcasts rather than QUIC connections. The complete moq-quic and moq-c test suites passed on the unchanged batch: 873 passed, 16 skipped. The tests listed above all passed. No evidence ties the timeouts to this PR's scheduler changes. Requeue unchanged; follow up the intermittent callback timeouts separately.

(written by codex / Codex)