Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions quest/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ there first. A quest here that needs it waits on a pin quest, which

## Required

- [m0: the studio](/quest/m0/README.md) - the native and wasm studio for creators who rely on OBS today
- [m1: after the studio ships](/quest/m1/README.md) - viewers and a server compose a broadcast from its sources and a layout track, plus studio features held past m0
- [m0: the studio](/quest/m0/README.md) - the native studio for creators who rely on OBS today
- [m1: after the studio ships](/quest/m1/README.md) - studio features held past m0; empty now
- [m2: later](/quest/m2/README.md) - work with no consumer yet
- [m3: someday](/quest/m3/README.md) - work no customer asks for, kept so its plans are not lost
37 changes: 22 additions & 15 deletions quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
## Goal

A native, cross-platform streaming studio (`rs/moq-studio`) for creators and
teams who rely on OBS today, plus the same app built to wasm. It composites
teams who rely on OBS today. It composites
cameras, screens, media, browser sources, and remote MoQ guests into scenes
on the GPU with moq-video and moq-audio, publishing several scenes at once
(landscape and portrait), and goes live to a MoQ relay and directly to RTMP,
Expand All @@ -29,9 +29,8 @@ Decided 2026-10-22 with the maintainer (/quest-plan):
feature, gpui has no wgpu on macOS or Windows, and Tauri is a webview. The
app owns the wgpu device.
- **Compositor:** a UI-free crate, `rs/moq-scene`: scene graph, sources,
transforms, transitions, and canvases on wgpu. The program audio bus goes
in moq-audio, whose only mixer today is output-only
(`rs/moq-audio/src/playback/mixer.rs`). Both grow slice by slice with the
transforms, transitions, and canvases on wgpu. The program audio bus is
moq-audio's `bus` module. Both grow slice by slice with the
studio UI that consumes them, never ahead of it.
- **Outputs:** the studio publishes to MoQ and pushes every RTMP(S) and WHIP
destination itself. It remuxes its own broadcast through a UI-free egress
Expand All @@ -47,6 +46,12 @@ Decided 2026-10-22 with the maintainer (/quest-plan):
control. Its media runs on new browser backends in moq-video and moq-audio
(getUserMedia, getDisplayMedia, WebCodecs) rather than a bridge to
js/publish and js/watch, so one Rust media path drives both builds.
Changed 2026-11-09 (maintainer, /quest-audit): ✅ the
[web studio](/quest/m2/web.md) and its two pins moved to m2 (rejected:
keeping them in m0), so m0 ships the native studio alone. moq removed all
WASM support on 2026-10-31, its browser-backend quests are deferred with
no other consumer, and nothing plans a browser WebTransport session for
moq-net.
- **Guests:** guests and phone cameras join a moq-room prefix the studio
owns, through a web join link or QR code.
- **Naming:** quests name no streaming platforms; ingest requirements are
Expand Down Expand Up @@ -76,9 +81,10 @@ broadcast reaches viewers, not the fallback. Sources are not published to the
relay by default (decided 2026-11-03 on
[moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)), because
each encoding costs an encoder session and consumer GPUs have few, so the
[viewer-side composition](/quest/m2/compose/README.md) epic moved to m1.
Overlays and animation render into the program here, and travel to viewers
in the layout track only once that epic lands.
[viewer-side composition](/quest/m2/compose/README.md) epic left m0 (now m2,
replanned 2026-11-08 around independently published sources layered over the
program). Overlays and animation render into the program here; that epic's
layout track carries only the layers of independently published sources.

Dropped from the first scoping round: relay fan-out ("encode once, the relay
restreams") and its upload savings. Direct push with shared encoders matches
Expand All @@ -89,24 +95,25 @@ repository.

- [Studio on macOS with capture granted](/quest/m0/app-macos.md) - camera, screen, and audio capture run on a Mac, not only Linux
- [nixos2 boots Windows for agents](/quest/m0/windows-host.md) - a Windows machine agents reach over ssh, with a camera and microphone
- [The studio builds for Windows in CI](/quest/m0/windows-build.md) - a nightly `windows-latest` build, which needs no Windows host
- [Studio on Windows](/quest/m0/app-windows.md) - the studio runs on Windows, not only Linux
- [Wayland display and window sources](/quest/m0/wayland-sources.md) - portal-picked sources that reopen without the picker
- [Destinations](/quest/m0/destinations.md) - RTMP, RTMPS, and WHIP push with per-destination codecs and shared encoders, on a UI-free egress library a hosted service can reuse
- [AAC for destinations](/quest/m0/destinations-aac.md) - AAC for the ingests that require it, from the platform encoders
- [Clap-to-measure sync delay](/quest/m0/sync-clap.md) - a clap sets a microphone's delay against its camera
- [Audio filters](/quest/m0/audio-filters.md) - per-source noise suppression, gate, compressor, limiter, and plugins, configured from the studio
- [Browser sources on macOS and Windows](/quest/m0/browser-source.md) - the out-of-process CEF pages Linux has, with shared GPU textures on the other desktops
- [Web studio](/quest/m0/web.md) - the full studio in wasm on WebGPU and WebCodecs, served from this repository
- [Audio filters](/quest/m0/audio-filters.md) - noise suppression, gate, compressor, limiter, and plugins on each microphone and system audio source, configured from the studio
- [Browser sources on macOS and Windows](/quest/m0/browser-source/README.md) - the out-of-process CEF pages Linux has, with shared GPU textures on the other desktops
- [Virtual camera](/quest/m0/vcam/README.md) - the program or a published scene exposed as a system camera on Linux, Windows, and macOS
- [The Apple signing identity is available](/quest/m0/signing.md) - the Apple credentials packaging and the macOS camera need; Windows ships unsigned
- [Packaging](/quest/m0/packaging.md) - a signed macOS app, a Windows installer, and a Flatpak for Linux
- [The Apple signing identity is available](/quest/m0/signing.md) - the Apple credentials the macOS app and its camera need; Windows ships unsigned
- [Packaging](/quest/m0/packaging/README.md) - a signed macOS app, a Windows installer, and a Flatpak for Linux, each its own quest
- [The persistable portal restore token is pinned](/quest/m0/pin-portal-token.md) - moq-video's restore token, for Wayland sources
- [Platform AAC encoders are pinned](/quest/m0/pin-aac-encode.md) - moq-audio's AAC encoders, for destinations
- [Platform AAC encoders are pinned](/quest/m0/pin-aac-encode.md) - moq-audio's AAC encoders, for AAC destinations
- [moq-audio's filter chain is pinned](/quest/m0/pin-audio-filters.md) - the chain audio filters configure
- [moq-video's browser backends are pinned](/quest/m0/pin-browser-video.md) - the web studio's video slice and browser composition
- [moq-audio's browser backends are pinned](/quest/m0/pin-browser-audio.md) - the web studio's audio slice
- [moq's egress dial changes are pinned](/quest/m0/pin-egress-dial.md) - RTMPS that keeps `tcUrl` and a caller-supplied dial policy, for destinations
- [moq-audio's source tap is pinned](/quest/m0/pin-audio-tap.md) - the samples clap-to-measure sync times a clap from
- [Playout writes land on positioned bus channels](/quest/m0/playout-positioned.md) - broadcast, media, and replay sound at the bus position its pictures are timed for
- [moq-audio's positioned bus channel is pinned](/quest/m0/pin-bus-positioned.md) - `Bus::positioned`, which playout writes through
- [moq-mux's decoupled playout clock is pinned](/quest/m0/pin-sync.md) - `moq_mux::sync` in place of `moq_mux::timeline`, which every later pin crosses

## Related

Expand Down
13 changes: 11 additions & 2 deletions quest/m0/app-windows.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,25 @@ the studio landed.
- System audio is `Unsupported` there until
[Windows capture parity](https://github.com/moq-dev/moq/blob/main/quest/m3/capture-windows.md)
adds WASAPI loopback, so that quest runs it.
- Build with `--features capture`, since a default build has image, color,
and broadcast sources only.
- Build with `--features capture`, since a default build has no camera,
display, window, microphone, or system-audio sources.
- Hotkeys, from [moq-dev/studio#84](https://github.com/moq-dev/studio/pull/84),
only compiled for Windows so far: run the keypad keys, the refusal of
keypad Enter (Windows' hotkey API cannot tell it from Enter), and a
keyboard layout switch while a game has focus, which re-registers every
binding on its physical key.
- The effects setting's read of Windows' animation setting
(`rs/moq-studio/src/fx/motion.rs`, from
[moq-dev/studio#85](https://github.com/moq-dev/studio/pull/85)) has not run
there either.

Decided 2026-11-09 (maintainer, /quest-audit): ✅ compiling for Windows is
[its own quest](/quest/m0/windows-build.md) (rejected: folding it in here),
since a hosted runner does it without the host this quest waits on.

## Required

- [The studio builds for Windows in CI](/quest/m0/windows-build.md) - the crate compiling for Windows before it is run there
- [nixos2 boots Windows for agents](/quest/m0/windows-host.md) - the Windows machine this runs on

## Related
Expand Down
24 changes: 16 additions & 8 deletions quest/m0/audio-filters.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,30 +2,38 @@

## Goal

Each audio source's filter chain (gain, noise suppression, noise gate,
compressor, limiter, ducking, and VST3 and CLAP plugins) is configured from
the studio and saved with the scene collection. Ducking lowers one source
(music) when another (the microphone) is active. A plugin that fails or keeps
missing its deadline shows as failed on its source.
Each microphone's and system audio source's filter chain (noise suppression,
noise gate, compressor, limiter, ducking, and VST3 and CLAP plugins) is
configured from the studio and saved with the scene collection. Ducking
lowers one source (music) when another (the microphone) is active. A plugin
that fails or keeps missing its deadline shows as failed on its source.

Guest, broadcast, media, browser, and replay sound stays unfiltered.

## Plan

Changed 2026-11-02 (maintainer), when the studio moved out of moq: the chain
itself, its filters, and the out-of-process plugin host are moq-audio work in
[moq-audio filter chain](https://github.com/moq-dev/moq/blob/main/quest/m5/audio-filters.md).
[moq-audio filter chain](https://github.com/moq-dev/moq/blob/main/quest/m3/audio-filters.md),
which moves to moq's m3 in [moq-dev/moq#8421](https://github.com/moq-dev/moq/pull/8421).
This quest is the studio's side: the panel, the scene-file shape, and wiring
each source's chain into the program bus.

Each microphone's and system audio's own track encodes its capture, before
the bus applies gain and mute (`Driver::listen` in
`rs/moq-scene/src/source.rs`). The chain goes before that tap, so the track
carries filtered sound, as video bakes in its ML effects (decided 2026-11-02,
maintainer).
carries filtered sound (decided 2026-11-02, maintainer), unlike a picture's
track, which is its capture before any filter.

The scene file names filters and their settings, never a plugin's absolute
path, so a collection stays portable; a missing plugin loads as failed rather
than refusing the file.

Decided 2026-11-09 (maintainer, /quest-audit): ✅ the Goal covers
microphones and system audio only (rejected: widening moq's chain to bus
channels). moq's chain runs on the capture producer, so only captured sources
have one; sound that reaches the bus another way never passes through it.

## Required

- [moq-audio's filter chain is pinned](/quest/m0/pin-audio-filters.md) - the chain this configures
51 changes: 0 additions & 51 deletions quest/m0/browser-source.md

This file was deleted.

41 changes: 41 additions & 0 deletions quest/m0/browser-source/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Browser sources on macOS and Windows

## Goal

Browser sources run on macOS and Windows as they do on Linux: a URL (an
overlay, alert box, or chat widget) renders into the compositor at the canvas
frame rate, the creator can interact with it, its audio joins the bus, and a
crash drops browser sources, never the stream: a page's renderer only its own
source, the shared browser process every browser source. Each platform lands
on its own.

## Plan

Decided 2026-10-22: use CEF through `cef-rs`, desktop only.
- **Rejected engines:** Servo's site compatibility is not yet enough for
common overlay sites. Blitz runs no JavaScript.
- **wasm build:** omits browser sources until a web API can draw HTML into a
canvas.
- **Rendering:** CEF offscreen rendering with a shared texture
(`OnAcceleratedPaint`), imported into wgpu as a D3D11 shared handle on
Windows, an IOSurface on macOS, and a dmabuf on Linux. The CPU `OnPaint`
path is the fallback and shows in stats.
- **Process:** CEF runs out of process.

Linux landed first (`moq_scene::browser`): one helper process every page
shares, on one persistent profile, which is the studio's executable started
again, talking over a Unix socket on its stdin that carries the DMA-BUF
descriptors, and `cef-rs`'s `osr_texture_import` copies each texture on the
studio's device. The profile lives in the XDG data directory
(`engine::browser_profile`); each platform keeps it in its own per-user data
directory instead.

The studio holds the channel open until the helper exits and stops it with
`Command::Stop`, so the helper reads the channel's end, from before CEF
starts, as the studio dying, and kills the process group it leads with CEF's
processes. Each platform's quest says what it needs on top.

Decided 2026-11-09 (maintainer, /quest-audit): ✅ one quest per platform
(rejected: Related links only; leaving one quest). One quest held macOS
behind the Windows host, and shaped the macOS app bundle with no link to the
[quest that owns it](/quest/m0/packaging/macos.md).
28 changes: 28 additions & 0 deletions quest/m0/browser-source/macos.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# [M] Browser sources on macOS

## Goal

Browser sources run on macOS as the
[epic](/quest/m0/browser-source/README.md) describes, each page's shared
texture imported into wgpu as an IOSurface.

## Plan

On top of what Linux landed:

- The helper is CEF's browser process, so it needs an app bundle with CEF's
helper apps (`cef-rs`'s `bundle-cef-app`) and its own NSApplication. They
go into the bundle [the macOS app](/quest/m0/packaging/macos.md) owns, and
through its signing, rather than a second bundle.
- An IOSurface crosses processes only as a Mach port
(`IOSurfaceCreateMachPort`), so the channel needs a Mach port rendezvous.
- Exiting with the studio stays as on Linux: the helper kills the process
group it leads.

## Required

- [macOS app](/quest/m0/packaging/macos.md) - the app bundle CEF's helper apps join

## Related

- [Studio on macOS with capture granted](/quest/m0/app-macos.md) - the rest of the macOS app, which pages do not need
22 changes: 22 additions & 0 deletions quest/m0/browser-source/windows.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# [M] Browser sources on Windows

## Goal

Browser sources run on Windows as the
[epic](/quest/m0/browser-source/README.md) describes, each page's shared
texture imported into wgpu as a D3D11 shared handle.

## Plan

On top of what Linux landed:

- A channel without Unix sockets, such as the helper's stdio pipes. The
studio duplicates CEF's NT handle out of the helper with `DuplicateHandle`
before `cef-rs` imports it.
- Windows has no process groups: start the helper in a job object with
`JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE`, which the studio's handle keeps
alive.

## Required

- [Studio on Windows](/quest/m0/app-windows.md) - the studio runs there at all
29 changes: 29 additions & 0 deletions quest/m0/destinations-aac.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# [S] AAC for destinations

## Goal

A destination can take AAC: the creator, or a preset, picks it for a
destination whose ingest requires it, and the studio encodes it with the
platform's encoder on macOS and Windows. Destinations that share a picture
scene, an audio scene, and encode settings still share one encoder. On Linux,
choosing AAC fails loudly when no encoder is available.

## Plan

Decided 2026-11-09 (maintainer, /quest-audit): ✅ split from
[Destinations](/quest/m0/destinations.md) (rejected: leaving that quest
blocked on AAC), which lands with Opus first.

Decided 2026-10-22 with the maintainer, carried from that quest:

- **AAC for RTMP ingests,** which commonly require H.264 and AAC. Opus stays
the default.
- **AAC encoders:** platform encoders only, from the AudioToolbox and Media
Foundation quests. moq-audio has no software AAC tier, and none is
planned. On Linux, choosing AAC for a destination fails loudly when no
encoder is available. It never silently falls back to another codec.

## Required

- [Destinations](/quest/m0/destinations.md) - the egress library and the per-destination codec choice AAC joins
- [Platform AAC encoders are pinned](/quest/m0/pin-aac-encode.md) - AAC on macOS and Windows
Loading