Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions quest/m0/test-flakes-2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,12 @@ Added 2026-11-08 (maintainer, /quest-complete): `js/net`'s widened max-age
re-drop test, which failed on [moq-dev/moq#8230](https://github.com/moq-dev/moq/pull/8230)
with the `Cancel` reset [moq-dev/moq#8195](https://github.com/moq-dev/moq/pull/8195) intends for a reply still pending at the stream's end; the test races its finish against that reply.

Added 2026-11-08 (maintainer's standing rule that flakes join this epic): moq-ffi's
raw media warmup test, which failed in the merge queue for
[moq-dev/moq#8272](https://github.com/moq-dev/moq/pull/8272) when its reader skipped one of three
buffered groups on the FFI runtime's real clock. It is quarantined with `#[ignore]` until its
child fixes it.

Public API: none. Wire: none.

## Required
Expand All @@ -219,3 +225,4 @@ Public API: none. Wire: none.
- [js/net's lite-07 datagram queue test holds up under load](/quest/m0/test-flakes-2/datagram-queues.md) - the createWritable queue test waits on the writes it asserts, and its `test.skip` is removed
- [js/net's lite-06 session-close test keeps the end under load](/quest/m0/test-flakes-2/session-close-end.md) - a session close right after a datagram track's end returns the end, not a `remote error: 0`, and its lite-06 skip is removed
- [js/net's widened max-age re-drop test ends cleanly, not with a reset](/quest/m0/test-flakes-2/widened-max-age-reset.md) - the test finishes the track only after the re-drop reply arrives, so the end is a FIN, and drops its wall-clock assert
- [moq-ffi's warmup test keeps every group under load](/quest/m0/test-flakes-2/ffi-warmup-wall-clock.md) - `raw_media_catalog_warmup_reaches_the_subscription` keeps the group between its warmup start and the live edge, and its `#[ignore]` is removed
41 changes: 41 additions & 0 deletions quest/m0/test-flakes-2/ffi-warmup-wall-clock.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# [S] moq-ffi's warmup test keeps every group under load

## Goal

moq-ffi's `media::read_tests::raw_media_catalog_warmup_reaches_the_subscription` passes under load
and runs again. The reader keeps the 4.7 s group that sits between the start the warmup reaches and
the live edge.

## Plan

Planned 2026-11-08 (maintainer's standing rule that flakes join this epic). The test failed in the
merge queue for [moq-dev/moq#8272](https://github.com/moq-dev/moq/pull/8272), which touches no
moq-ffi read path ([job 128797584818](https://github.com/moq-dev/moq/actions/runs/42502335110/job/128797584818)).
It panicked at `rs/moq-ffi/src/media.rs:1635` with `video=false; left: [4400000, 5000000]; right:
[4400000, 4700000, 5000000]`: the audio pass started at group 3 and ended at group 5, but skipped
group 4. It is quarantined with `#[ignore]` and a `sh/gh/test-inventory.allow` entry until this quest
fixes it.
Its subscription assertions moved to `raw_media_catalog_warmup_widens_the_subscription`,
which stays in the gate.

The test writes groups 5, 4, 3 newest first, finishes the track, then reads. The read polls on the
FFI runtime, whose clock is real time, so every wall-clock rule in the read path runs on a loaded
host's clock. Group 4 lands before group 3, so a rule keyed on arrival ages group 4 out first, while
group 5 survives as the newest and group 3 as the start. Candidates in `moq-mux`'s
`container::Consumer`: the missing-sequence walk (`front.late.elapsed() >= max_delay` and its hole
timer, with a 200 ms max delay), and the start's `Startup::Earliest` arrival anchor. Also check
`moq-net`'s subscription delivery under the widened 500 ms max delay.

Reproduce first: run the test repeatedly under a loaded `just check`, with tracing on
`skip_to` and the hole walk, to name the rule that drops group 4. Then fix the cause:

- If the rule drops an arrived, finished group whose media lies inside the budget, that is a
`moq-mux` bug a real viewer on a slow host hits too. Fix it there, with a test on a controlled
clock beside the start-reach tests.
- If the skip is correct for that much elapsed time, drive the test's clock instead of the host's
(`kio::clock::Manual`, or build the consumer on the test runtime's clock), so elapsed time is
zero by construction.

Then remove the `#[ignore]` and the allowlist entry, and delete this quest.

Public API: none. Wire: none.
125 changes: 71 additions & 54 deletions rs/moq-ffi/src/media.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1546,65 +1546,82 @@ mod read_tests {
use moq_mux::container::Container as _;
use std::time::Duration;

/// A catalog rendition's warmup survives conversion, widens the raw subscription past the
/// caller's max delay, keeping the min delay as asked, and starts the reader that much further
/// back.
///
/// Reads no clock: the reader polls on the FFI runtime, whose clock is not a paused test
/// runtime's, so a wait for the reach would measure real time. moq-mux's start-reach tests
/// cover that wait on one clock.
#[tokio::test]
async fn raw_media_catalog_warmup_reaches_the_subscription() {
const WARMUP: Duration = Duration::from_millis(300);
const DELAY: Duration = Duration::from_millis(200);

/// Subscribes to a rendition whose catalog warmup is [`WARMUP`], at a [`DELAY`] min and max delay.
async fn subscribe_warmup(
video: bool,
) -> (
moq_net::broadcast::Producer,
moq_net::track::Producer,
Arc<MoqMediaTrackConsumer>,
moq_mux::container::Kind,
) {
use crate::consumer::{MoqDelay, MoqSubscription, moq_subscription_defaults};

let warmup = Duration::from_millis(300);
let delay = Duration::from_millis(200);
for video in [false, true] {
let broadcast = moq_net::broadcast::Info::new().produce();
let track = broadcast.create_track("media", hang::container::track_info(0)).unwrap();
let mut catalog = moq_mux::catalog::hang::Catalog::default();
let mut audio = hang::catalog::AudioConfig::new(hang::catalog::AudioCodec::Opus, 48_000, 2);
audio.warmup = Some(warmup);
catalog.audio.renditions.insert("media".into(), audio);
let mut config = hang::catalog::VideoConfig::new(hang::catalog::VideoCodec::VP8);
config.warmup = Some(warmup);
catalog.video.renditions.insert("media".into(), config);
let catalog = convert_catalog(&catalog);
let (rendition, kind) = match video {
true => (
MoqRendition::Video {
id: "media".into(),
config: catalog.video["media"].clone(),
},
moq_mux::container::Kind::Video,
),
false => (
MoqRendition::Audio {
id: "media".into(),
config: catalog.audio["media"].clone(),
},
moq_mux::container::Kind::Audio,
),
};
let reader = MoqMediaTrackConsumer::subscribe(
&MoqBroadcastConsumer::new(broadcast.consume()),
MoqMediaTrackConfig {
rendition,
subscription: Some(MoqSubscription {
delay: MoqDelay {
min: Some(delay),
max: Some(delay),
},
..moq_subscription_defaults()
}),
let broadcast = moq_net::broadcast::Info::new().produce();
let track = broadcast.create_track("media", hang::container::track_info(0)).unwrap();
let mut catalog = moq_mux::catalog::hang::Catalog::default();
let mut audio = hang::catalog::AudioConfig::new(hang::catalog::AudioCodec::Opus, 48_000, 2);
audio.warmup = Some(WARMUP);
catalog.audio.renditions.insert("media".into(), audio);
let mut config = hang::catalog::VideoConfig::new(hang::catalog::VideoCodec::VP8);
config.warmup = Some(WARMUP);
catalog.video.renditions.insert("media".into(), config);
let catalog = convert_catalog(&catalog);
let (rendition, kind) = match video {
true => (
MoqRendition::Video {
id: "media".into(),
config: catalog.video["media"].clone(),
},
)
.await
.unwrap();
moq_mux::container::Kind::Video,
),
false => (
MoqRendition::Audio {
id: "media".into(),
config: catalog.audio["media"].clone(),
},
moq_mux::container::Kind::Audio,
),
};
let reader = MoqMediaTrackConsumer::subscribe(
&MoqBroadcastConsumer::new(broadcast.consume()),
MoqMediaTrackConfig {
rendition,
subscription: Some(MoqSubscription {
delay: MoqDelay {
min: Some(DELAY),
max: Some(DELAY),
},
..moq_subscription_defaults()
}),
},
)
.await
.unwrap();
(broadcast, track, reader, kind)
}

/// A catalog rendition's warmup survives conversion and widens the raw subscription past the
/// caller's max delay, keeping the min delay as asked.
#[tokio::test]
async fn raw_media_catalog_warmup_widens_the_subscription() {
for video in [false, true] {
let (_broadcast, track, _reader, _) = subscribe_warmup(video).await;
let subscription = track.subscription().unwrap();
assert_eq!(subscription.delay.min, Some(delay), "video={video}");
assert_eq!(subscription.delay.max, Some(delay + warmup), "video={video}");
assert_eq!(subscription.delay.min, Some(DELAY), "video={video}");
assert_eq!(subscription.delay.max, Some(DELAY + WARMUP), "video={video}");
}
}

/// The warmup starts the reader that much further back than the min delay.
#[tokio::test]
#[ignore = "flaky: the 4.7 s group is sometimes skipped under load; quest/m0/test-flakes-2/ffi-warmup-wall-clock.md"]
async fn raw_media_catalog_warmup_reaches_the_subscription() {
for video in [false, true] {
let (_broadcast, track, reader, kind) = subscribe_warmup(video).await;
let write = |sequence: u64, micros| {
let mut group = track.create_group(sequence.into()).unwrap();
moq_mux::catalog::hang::Container::Legacy(kind)
Expand Down
5 changes: 1 addition & 4 deletions sh/gh/test-inventory.allow
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,7 @@ ignore moq-video::render::export::dx12::tests::exports_an_nv12_texture /quest/m5
ignore moq-video::render::export::dx12::tests::media_foundation_encodes_the_export_in_place /quest/m5/video-hardware.md

# Quarantined flakes, re-enabled by the quest that fixes them.

# Flaky, disabled until its quest fixes the cause.
ignore moq-ffi::media::read_tests::raw_media_catalog_warmup_reaches_the_subscription /quest/m0/test-flakes-2/ffi-warmup-wall-clock.md

# Need another implementation to speak the route wire; the quest ports it.
ignore moq-net::test_interop::lite_tracks_interop /quest/m2/cluster-routing/routes/js.md

# Quarantined flakes, until the quest that owns the flake fixes it at its cause.
Loading