diff --git a/quest/m1/README.md b/quest/m1/README.md index da300e96d79..8698a16b258 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -58,6 +58,8 @@ moves what remains of them once their claims merge. - [Session auth grants and replies stay consistent and bounded](/quest/m1/auth-stream-bounds.md) - an AUTH_ERROR revokes its grant, queued grants and retiring tokens are bounded, and dropping queued requests cannot deadlock - [Relay settings parse once into a validated configuration](/quest/m1/relay-config-checks.md) - settings that cannot work together are refused at load instead of accepted and ignored - [Track drop runs are forgotten below the floor](/quest/m1/drop-runs-prune.md) - a track's dropped-group record stays proportional to its window in Rust and JS +- [A macOS camera delivers frames the renderer imports](/quest/m1/camera-nv12-macos.md) - AVFoundation capture asks for NV12, so a camera whose native format is `2vuy` reaches the renderer +- [Encoders keyframe by time, not frame count](/quest/m1/keyframe-by-time.md) - groups stay within the declared join interval when a source's frame rate drifts from the configured one ## Related diff --git a/quest/m1/camera-nv12-macos.md b/quest/m1/camera-nv12-macos.md new file mode 100644 index 00000000000..55ed9179836 --- /dev/null +++ b/quest/m1/camera-nv12-macos.md @@ -0,0 +1,30 @@ +# [S] A macOS camera delivers frames the renderer imports + +## Goal + +A camera opened through `moq_video::capture` on macOS delivers NV12 pixel +buffers, which the Metal import and the CPU download both read, instead of +the device's native format. Today a MacBook Air camera delivers `2vuy`, and +`Output::Render` fails at the first frame with `cannot download pixel format +0x32767579; expected NV12`. + +## Plan + +Found by moq-dev/studio's first Mac capture run +([moq-dev/studio#70](https://github.com/moq-dev/studio/pull/70)). Decided +2026-11-07 (maintainer, moq-dev/studio's /quest-unblock): ✅ m1 (rejected: m3), +since it is an additive fix to shipped code and blocks the studio's macOS +camera. + +`open` in `rs/moq-video/src/capture/avfoundation.rs` builds its +`AVCaptureVideoDataOutput` without `videoSettings`, so AVFoundation passes the +device's format through. Request `kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange` +through `kCVPixelBufferPixelFormatTypeKey`, as the display source already does +(`rs/moq-video/src/capture/screencapture.rs`). A local patch doing this +brought the camera to the studio's program and broadcast (1280x720 and +640x360 H.264 at 30 fps, correct colors). + +CI has no camera: test that the output requests NV12 by reading its +`videoSettings` back, and record a run on a Mac with a camera in the PR. + +Once it lands, moq-dev/studio bumps its `moq` pin. diff --git a/quest/m1/keyframe-by-time.md b/quest/m1/keyframe-by-time.md new file mode 100644 index 00000000000..0f919badb21 --- /dev/null +++ b/quest/m1/keyframe-by-time.md @@ -0,0 +1,31 @@ +# [M] Encoders keyframe by time, not frame count + +## Goal + +A video encoder's groups stay within its declared join interval +(`encode::Config::join`) when the source runs slower or faster than the +configured `framerate`, such as a camera dropping to 20 fps in low light. Today +a 2 s join at 30 fps keyframes every 60 frames, which a 20 fps source +stretches to 3 s, so moq-mux warns that the track's groups are longer than the +declared interval and HLS lists them as overruns. + +## Plan + +Found by moq-dev/studio's first Mac capture run +([moq-dev/studio#70](https://github.com/moq-dev/studio/pull/70)). Decided +2026-11-07 (maintainer, moq-dev/studio's /quest-unblock): ✅ keyframe by +elapsed time (rejected: have AVFoundation report the device's frame rate, +which stays 30 when auto-exposure slows the camera; drop it); ✅ m1 (rejected: +m3), since it changes no public API and fixes every variable-rate source. + +`Config::keyframe_interval()` (`rs/moq-video/src/encode/encoder.rs`, +crate-private) converts `join` to frames at the configured rate, and each +backend sets that as its GOP length. Instead, force a keyframe on the first +frame whose timestamp reaches the last keyframe's plus `join`, through the +forced-keyframe path that `Cuts` (`rs/moq-video/src/encode/cuts.rs`) already +drives on every backend. VideoToolbox can also take +`kVTCompressionPropertyKey_MaxKeyFrameIntervalDuration`. Decide in the PR +whether a backend's own GOP stays as a ceiling beyond `join` or goes. + +Test with a mocked source whose frame rate varies: no group exceeds `join` by +more than one frame.