From 315b644798f9da6d1de5294da41284b1fa16b78a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 13:27:30 -0800 Subject: [PATCH 1/7] chore(quest): claim quest/m0/animation.md Quest-Claim: e94fafb24b1b4ee7 From bd27dde1678907c492501b9cdf2033b7ecfd82bd Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 15:06:11 -0800 Subject: [PATCH 2/7] feat(scene): keyframes that play from when an item's scene goes on Co-Authored-By: Claude Opus 5.5 --- README.md | 35 +++ quest/m0/README.md | 1 - quest/m0/animation.md | 35 --- quest/m0/stinger.md | 2 +- quest/m1/compose/layout.md | 4 + rs/moq-scene/src/collection.rs | 397 ++++++++++++++++++++++++++++++++- rs/moq-scene/src/compositor.rs | 209 +++++++++++++---- rs/moq-scene/src/lib.rs | 7 +- rs/moq-studio/src/ui.rs | 252 +++++++++++++++++++-- 9 files changed, 835 insertions(+), 107 deletions(-) delete mode 100644 quest/m0/animation.md diff --git a/README.md b/README.md index 0353a079..5be0f757 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,7 @@ Every panel is a tab you can drag, split, or stack. | Devices | List this machine's cameras, displays, windows, and audio inputs, and add one as a source. | | Guests | Invite a guest, show their join link as a QR code, see who joined, and take a guest out of every scene. | | Filters | Crop the selected source or item, add, order, and set its filters, set an item's blend mode, and copy or paste filters. | +| Keyframes | Animate the selected item: add, time, ease, and remove its keyframes, and set how far each moves, resizes, crops, and filters it (see [Keyframes](#keyframes)). **Play** plays the preview scene's keyframes again. | | Audio | Add, rename (double-click), and remove mixes, pick the one on program and the one to listen to and on which output, and set each audio source's gain, mute, delay, and the mixes it feeds, with a meter on every mix and source. | | Output | Start and stop recording, set the canvas size and frame rate, and see the broadcast path, whether the program is encoding, the encoder sessions open, and each rendition's encoder work. | | Health | See how the stream reaches viewers, relay by relay and track by track, and the cause of each problem. | @@ -122,6 +123,29 @@ the stream and a recording show the same frames of it, and a second **Transition** while one runs starts from the scene it was heading to. A transition set to 0 ms is a cut. +### Keyframes + +An item's keyframes animate it: an item can fly in, fade in, or blur into +focus, and a camera can push in. Each keyframe is a time after the item's +scene goes on and how far the item is from where it rests then: moved and +resized by canvas pixels, cropped further, and drawn through filters after its +own. Between two keyframes the item eases from one to the next, linearly or +easing in, out, or (by default) both. Filters of the same kinds, in order, +mix their settings, and others change halfway. A keyframe's filters cannot be +a LUT or a mask. + +Keyframes play once, from when the item's scene goes on program, and hold the +last keyframe; before the first, the item holds the first. A cut or transition +starts them again, even to the scene already on program, while a scene leaving +in a transition carries on with its own. The preview plays them from when the +scene went into preview; pick the scene again, or press **Play**, to watch them +again. They are drawn on the program clock, so the stream, a recording, and a +replay of the program show the same frames. Top-layer items have none, since +the top layer never goes on. + +Dragging an item in the preview grabs it where its keyframes have it and moves +where it rests, so its whole animation moves with it. + In the preview, click an item of whatever the Items panel shows to select it, drag it to move it, and drag its edges or corners to resize it. A drag snaps edges and centers to the canvas's edges and center, to the other items shown, @@ -395,6 +419,11 @@ items = [ name = "Break" items = [ { source = "camera-faded", rect = { x = 0, y = 0, width = 1920, height = 1080 } }, + # Flies in from the right, blurred, over half a second, landing where it rests. + { source = "guest", rect = { x = 1280, y = 720, width = 640, height = 360 }, keyframes = [ + { at = "0s", x = 700, filters = [{ kind = "blur", radius = 12 }] }, + { at = "500ms", ease = "out", filters = [{ kind = "blur", radius = 0 }] }, + ] }, ] [[top]] @@ -442,6 +471,12 @@ An item's `blend` mixes it with what is under it by a CSS `mix-blend-mode`: `saturation`, `color`, or `luminosity`. Removing a source removes the filters that name it. +An item's `keyframes`, in order of time, each have an `at`, how long after the +scene goes on, and optionally an `ease` into it from the one before (`linear`, +`in`, `out`, or `in_out`, the default), `x`, `y`, `width`, and `height` added +to the item's `rect`, a `crop` of what the item's crop leaves, and `filters` +applied after the item's own. + The studio writes the file a second after the last change, so a burst such as a slider drag is one write, and once more on quit. Edits that never pause for a second are written five seconds after the first, so a crash loses at most the diff --git a/quest/m0/README.md b/quest/m0/README.md index 48c63b8e..d389511d 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -93,7 +93,6 @@ repository. - [Audio source tracks](/quest/m0/audio-source-tracks.md) - microphones and system audio get their own tracks, for isolated recording and replay - [Broadcast sources play in sync](/quest/m0/broadcast-sync.md) - subscribed broadcasts and guests play audio and video on the publisher's timestamps, with `moq play`'s delay - [Wayland display and window sources](/quest/m0/wayland-sources.md) - portal-picked sources that reopen without the picker -- [Keyframes](/quest/m0/animation.md) - keyframed item properties with easing, starting when their scene goes on program - [Media file source](/quest/m0/media-file.md) - a local video or audio file plays as a source, looping or from the start on demand - [Stinger transitions](/quest/m0/stinger.md) - a clip played over the program that switches scenes at its cut point - [ML video effects](/quest/m0/effects.md) - background removal and other ML effects, baked into their source's track diff --git a/quest/m0/animation.md b/quest/m0/animation.md deleted file mode 100644 index a2c69ede..00000000 --- a/quest/m0/animation.md +++ /dev/null @@ -1,35 +0,0 @@ -# [M] Keyframes - -## Goal - -Any item property in `moq-studio` can be keyframed with easing, so an item -can fly in or a camera can push in, and the same animation plays in the -program, a recording, and a replay. - -## Plan - -Fade, slide, wipe, and move transitions are drawn by moq-scene's compositor -on the program clock -([moq-dev/studio#33](https://github.com/moq-dev/studio/pull/33)). Animate in -moq-scene on the media clock, so the program, a recording, and a replay match. -Carrying keyframes in the layout track, for viewers to compose, waits for the -m1 [composition epic](/quest/m1/compose/README.md) (changed 2026-11-04, -maintainer: the program comes first). Overlays animate through their own CSS. -No Lottie runtime for now (decided 2026-10-26): the open ones either lack text -(velato) or add a C++ build (ThorVG). Keyframes change through moq-scene's -`Edit`, so they undo like any edit. - -Decided 2026-11-04 (maintainer): - -- **Start:** an item's keyframes run on the time since its scene went on - program (and into preview, for the preview), play once, and hold the last - keyframe. Rejected: a loop on the program clock, and a trigger or hotkey. -- **Editing:** a drag in the preview moves the item's resting place, and - keyframes are offsets from it, so editing needs no timeline scrubber. - Rejected: setting the keyframe at the preview's time. - -Stingers are their own quest, [Stinger transitions](/quest/m0/stinger.md). - -## Related - -- [Layout track](/quest/m1/compose/layout.md) - where keyframes travel to viewers in m1 diff --git a/quest/m0/stinger.md b/quest/m0/stinger.md index 9c6847e5..3efb4f28 100644 --- a/quest/m0/stinger.md +++ b/quest/m0/stinger.md @@ -7,7 +7,7 @@ program that switches scenes at its cut point, as broadcast graphics do. ## Plan -Split from [Keyframes](/quest/m0/animation.md) on 2026-11-04 (maintainer). A +Split from the keyframes quest on 2026-11-04 (maintainer). A stinger is a transition kind beside fade, slide, wipe, and move ([moq-dev/studio#33](https://github.com/moq-dev/studio/pull/33)), drawn on the program clock, whose clip is a media file source. It must start its clip as diff --git a/quest/m1/compose/layout.md b/quest/m1/compose/layout.md index 3c725cbc..bb7203f2 100644 --- a/quest/m1/compose/layout.md +++ b/quest/m1/compose/layout.md @@ -45,6 +45,10 @@ themselves; off, viewers see the page only in the program. A replay source layout says when it is hidden and when its span has ended, rather than leaving a viewer on its last frame. +Items carry their keyframes (`moq_scene::Keyframe`), which play from when +their scene went on program, so the layout says when that was on the media +clock, and a viewer that joins late lands at the same point of them. + Only sources the creator publishes independently ([Published independently](/quest/m0/publish-independently.md)) have tracks a viewer can fetch; the rest reach viewers only in the program. diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 9c742264..9544cf80 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -1,5 +1,6 @@ //! The scene graph, and the TOML file a [`Collection`] saves to. +use std::borrow::Cow; use std::collections::{BTreeMap, BTreeSet}; use std::fmt; use std::path::PathBuf; @@ -177,6 +178,19 @@ impl Collection { .validate() .map_err(|err| Error::Invalid(format!("{on}: {err}")))?; self.validate_chain(&item.filters, &on)?; + if layer == Layer::Top && !item.keyframes.is_empty() { + return Err(Error::Invalid(format!( + "{on} has keyframes, but the top layer never goes on to start them" + ))); + } + if !item.keyframes.is_sorted_by_key(|keyframe| keyframe.at) { + return Err(Error::Invalid(format!("{on} has keyframes out of order"))); + } + for keyframe in &item.keyframes { + keyframe + .validate() + .map_err(|err| Error::Invalid(format!("{on}: {err}")))?; + } } } @@ -411,14 +425,48 @@ impl Transition { /// How far along it looks once `linear` of its time has passed, both from /// 0 to 1: as is for a fade, and eased in and out otherwise. pub(crate) fn progress(&self, linear: f32) -> f32 { + match self { + Self::Fade { .. } => Ease::Linear.apply(linear), + _ => Ease::InOut.apply(linear), + } + } +} + +/// How a change speeds up and slows down over its time. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum Ease { + /// At an even pace. + Linear, + /// Starting slowly (cubic). + In, + /// Landing slowly (cubic). + Out, + /// Starting and landing slowly (cubic). + #[default] + InOut, +} + +impl Ease { + /// Every curve. + pub const ALL: [Self; 4] = [Self::Linear, Self::In, Self::Out, Self::InOut]; + + /// How far along the change looks once `linear` of its time has passed, + /// both from 0 to 1. + pub(crate) fn apply(self, linear: f32) -> f32 { let linear = linear.clamp(0.0, 1.0); match self { - Self::Fade { .. } => linear, - // The cubic ease in and out. - _ if linear < 0.5 => 4.0 * linear.powi(3), - _ => 1.0 - (2.0 - 2.0 * linear).powi(3) / 2.0, + Self::Linear => linear, + Self::In => linear.powi(3), + Self::Out => 1.0 - (1.0 - linear).powi(3), + Self::InOut if linear < 0.5 => 4.0 * linear.powi(3), + Self::InOut => 1.0 - (2.0 - 2.0 * linear).powi(3) / 2.0, } } + + fn is_default(&self) -> bool { + *self == Self::default() + } } impl Default for Transition { @@ -512,10 +560,16 @@ pub struct Item { /// How the filtered picture mixes with what is under it. #[serde(default, skip_serializing_if = "Blend::is_normal")] pub blend: Blend, + /// How the item moves away from and back to where it rests, from when its + /// scene goes on, in order of time. It plays once: before the first + /// keyframe the item holds that one, and after the last it holds the last. + /// A top-layer item has none, since the top layer never goes on. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub keyframes: Vec, } impl Item { - /// `source` shown, uncropped and unfiltered, at `rect`. + /// `source` shown, uncropped, unfiltered, and still, at `rect`. pub fn new(source: impl Into, rect: Rect) -> Self { Self { source: source.into(), @@ -524,8 +578,145 @@ impl Item { crop: Crop::default(), filters: Vec::new(), blend: Blend::Normal, + keyframes: Vec::new(), + } + } + + /// The item as it is drawn `since` its scene went on: moved, resized, + /// cropped, and filtered by its keyframes, which the result has none of. + pub fn posed(&self, since: Duration) -> Cow<'_, Self> { + let Some(offset) = self.offset(since) else { + return Cow::Borrowed(self); + }; + let mut filters = self.filters.clone(); + filters.extend(offset.filters); + Cow::Owned(Self { + source: self.source.clone(), + rect: Rect { + x: self.rect.x + offset.x, + y: self.rect.y + offset.y, + // A size shrunk past nothing draws nothing. + width: (self.rect.width + offset.width).max(0.0), + height: (self.rect.height + offset.height).max(0.0), + }, + visible: self.visible, + crop: self.crop.then(&offset.crop), + filters, + blend: self.blend, + keyframes: Vec::new(), + }) + } + + /// `since` held within the keyframes' times, so two times this clamps to + /// the same pose the item the same. + pub(crate) fn settled(&self, since: Duration) -> Duration { + match (self.keyframes.first(), self.keyframes.last()) { + (Some(first), Some(last)) => since.clamp(first.at, last.at), + _ => Duration::ZERO, } } + + /// How far the keyframes take the item from where it rests `since` its + /// scene went on, or `None` if it has none. + fn offset(&self, since: Duration) -> Option { + let next = self.keyframes.iter().position(|keyframe| keyframe.at > since); + let (from, to) = match next { + None => return self.keyframes.last().cloned(), + Some(0) => return self.keyframes.first().cloned(), + Some(next) => (&self.keyframes[next - 1], &self.keyframes[next]), + }; + // `to` is later than `since`, and `from` no later, so the span is not empty. + let linear = (since - from.at).as_secs_f32() / (to.at - from.at).as_secs_f32(); + Some(from.toward(to, to.ease.apply(linear))) + } +} + +/// Where an item is at a moment after its scene goes on, as an offset from +/// where it rests, so moving the item moves its whole animation. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Keyframe { + /// How long after the item's scene goes on. + #[serde(with = "humantime_serde")] + pub at: Duration, + /// How the item eases into this keyframe from the one before. + #[serde(default, skip_serializing_if = "Ease::is_default")] + pub ease: Ease, + /// Canvas pixels added to the item's left edge. + #[serde(default, skip_serializing_if = "is_zero")] + pub x: f32, + /// Canvas pixels added to the item's top edge. + #[serde(default, skip_serializing_if = "is_zero")] + pub y: f32, + /// Canvas pixels added to the item's width. + #[serde(default, skip_serializing_if = "is_zero")] + pub width: f32, + /// Canvas pixels added to the item's height. + #[serde(default, skip_serializing_if = "is_zero")] + pub height: f32, + /// Trimmed from what the item's crop leaves, as a push-in does. + #[serde(default, skip_serializing_if = "Crop::is_none")] + pub crop: Crop, + /// Applied after the item's own. Each one's settings mix into the next + /// keyframe's when the two lists pair up by kind, in order, and otherwise + /// change halfway. None reads another source. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub filters: Vec, +} + +impl Keyframe { + /// The item where it rests, `at` after its scene goes on. + pub fn new(at: Duration) -> Self { + Self { + at, + ease: Ease::default(), + x: 0.0, + y: 0.0, + width: 0.0, + height: 0.0, + crop: Crop::default(), + filters: Vec::new(), + } + } + + /// `progress` of the way from this keyframe's offset to `to`'s. + fn toward(&self, to: &Self, progress: f32) -> Self { + let mix = |from: f32, to: f32| mix(from, to, progress); + let paired = self.filters.len() == to.filters.len(); + let pairs = self.filters.iter().zip(&to.filters); + let filters = paired.then(|| pairs.map(|(from, to)| from.toward(to, progress)).collect()); + Self { + at: to.at, + ease: to.ease, + x: mix(self.x, to.x), + y: mix(self.y, to.y), + width: mix(self.width, to.width), + height: mix(self.height, to.height), + crop: self.crop.toward(&to.crop, progress), + filters: filters + .flatten() + .unwrap_or_else(|| if progress < 0.5 { &self.filters } else { &to.filters }.clone()), + } + } + + fn validate(&self) -> Result<(), String> { + let offsets = [self.x, self.y, self.width, self.height]; + if !offsets.iter().all(|offset| offset.is_finite()) { + return Err(format!("a keyframe at {:?} offsets by {offsets:?}", self.at)); + } + self.crop.validate()?; + if let Some(source) = self.filters.iter().find_map(Filter::source) { + return Err(format!( + "a keyframe at {:?} has a filter reading `{source}`; a keyframe's filters read no other source", + self.at + )); + } + self.filters.iter().try_for_each(Filter::validate) + } +} + +fn is_zero(value: &f32) -> bool { + *value == 0.0 } fn visible() -> bool { @@ -899,6 +1090,18 @@ impl Crop { } } + /// `progress` of the way from this crop to `other`, which a mix of two + /// valid crops is. + pub(crate) fn toward(&self, other: &Self, progress: f32) -> Self { + let mix = |from: f32, to: f32| mix(from, to, progress); + Self { + left: mix(self.left, other.left), + top: mix(self.top, other.top), + right: mix(self.right, other.right), + bottom: mix(self.bottom, other.bottom), + } + } + fn validate(&self) -> Result<(), String> { let edges = [self.left, self.top, self.right, self.bottom]; if edges.iter().any(|edge| !(0.0..1.0).contains(edge)) @@ -1320,6 +1523,22 @@ mod tests { ..Crop::default() }, blend: Blend::Screen, + keyframes: vec![ + Keyframe { + x: -740.0, + crop: Crop { + right: 0.5, + ..Crop::default() + }, + filters: vec![Filter::Blur { radius: 8.0 }], + ..Keyframe::new(Duration::ZERO) + }, + Keyframe { + ease: Ease::Out, + filters: vec![Filter::Blur { radius: 0.0 }], + ..Keyframe::new(Duration::from_millis(400)) + }, + ], ..Item::new("camera", rect) }); collection @@ -1384,7 +1603,10 @@ path = "grade.cube" name = "Main" items = [ { source = "bg", rect = { x = 0, y = 0, width = 1280, height = 720 } }, - { source = "cam", rect = { x = 10, y = 10, width = 320, height = 180 }, visible = false }, + { source = "cam", rect = { x = 10, y = 10, width = 320, height = 180 }, visible = false, keyframes = [ + { at = "0s", x = -330, crop = { left = 0.2 }, filters = [{ kind = "opacity", amount = 0 }] }, + { at = "1.5s", ease = "out", filters = [{ kind = "opacity", amount = 1 }] }, + ] }, { source = "cam-faded", rect = { x = 0, y = 0, width = 640, height = 360 }, filters = [{ kind = "opacity", amount = 0.8 }] }, { source = "cam", rect = { x = 0, y = 0, width = 640, height = 360 }, crop = { left = 0.1, right = 0.1 }, blend = "soft_light", filters = [ { kind = "chroma_key", color = "#00ff00", similarity = 0.4, smoothness = 0.08, spill = 0.1 }, @@ -1427,6 +1649,25 @@ rect = { x = 0, y = 680, width = 1280, height = 40 } let main = collection.scene("Main").unwrap(); assert!(main.items[0].visible); assert!(!main.items[1].visible); + assert_eq!( + main.items[1].keyframes, + [ + Keyframe { + x: -330.0, + crop: Crop { + left: 0.2, + ..Crop::default() + }, + filters: vec![Filter::Opacity { amount: 0.0 }], + ..Keyframe::new(Duration::ZERO) + }, + Keyframe { + ease: Ease::Out, + filters: vec![Filter::Opacity { amount: 1.0 }], + ..Keyframe::new(Duration::from_millis(1500)) + }, + ] + ); assert_eq!(collection.capture("cam-faded"), "cam"); let filters: Vec<_> = collection.filters(&main.items[2]).collect(); assert_eq!( @@ -1715,6 +1956,150 @@ rect = { x = 0, y = 680, width = 1280, height = 40 } assert_eq!(Crop::default().then(&item), item); } + /// An item with keyframes `keyframes`, resting at (100, 50), 200x100. + fn animated(keyframes: Vec) -> Item { + let rect = Rect { + x: 100.0, + y: 50.0, + width: 200.0, + height: 100.0, + }; + Item { + filters: vec![Filter::Grayscale { amount: 1.0 }], + keyframes, + ..Item::new("camera", rect) + } + } + + /// Keyframes offset the item from where it rests, ease between each other, + /// and hold the first before it and the last after it. + #[test] + fn keyframes_offset_the_item_and_hold_at_the_ends() { + let item = animated(vec![ + Keyframe { + x: -300.0, + width: -250.0, + crop: Crop { + left: 0.5, + ..Crop::default() + }, + filters: vec![Filter::Opacity { amount: 0.0 }], + ..Keyframe::new(Duration::from_secs(1)) + }, + Keyframe { + ease: Ease::Linear, + filters: vec![Filter::Opacity { amount: 1.0 }], + ..Keyframe::new(Duration::from_secs(2)) + }, + ]); + let at = |millis| item.posed(Duration::from_millis(millis)).into_owned(); + + let first = at(0); + assert_eq!(first, at(1000), "held before the first keyframe"); + assert_eq!(first.rect.x, -200.0); + assert_eq!(first.rect.width, 0.0, "a size shrunk past nothing is nothing"); + assert_eq!(first.crop.left, 0.5); + assert_eq!( + first.filters, + [Filter::Grayscale { amount: 1.0 }, Filter::Opacity { amount: 0.0 }], + "after the item's own" + ); + assert!(first.keyframes.is_empty()); + + let half = at(1500); + assert_eq!(half.rect.x, -50.0); + assert_eq!(half.rect.width, 75.0); + assert_eq!(half.crop.left, 0.25); + assert_eq!(half.filters[1], Filter::Opacity { amount: 0.5 }); + + let last = at(5000); + assert_eq!(last, at(2000), "held after the last keyframe"); + assert_eq!(last.rect, item.rect); + assert_eq!(item.settled(Duration::from_secs(5)), Duration::from_secs(2)); + assert_eq!(item.settled(Duration::ZERO), Duration::from_secs(1)); + + let still = Item::new("camera", item.rect); + assert!(matches!(still.posed(Duration::from_secs(1)), Cow::Borrowed(_))); + } + + #[test] + fn eases_start_and_land_slowly() { + assert_eq!(Ease::Linear.apply(0.5), 0.5); + assert_eq!(Ease::In.apply(0.5), 0.125); + assert_eq!(Ease::Out.apply(0.5), 0.875); + assert_eq!(Ease::InOut.apply(0.25), 0.0625); + assert_eq!(Ease::InOut.apply(0.5), 0.5); + for ease in Ease::ALL { + assert_eq!((ease.apply(-1.0), ease.apply(2.0)), (0.0, 1.0), "{ease:?}"); + } + } + + /// Filters that do not pair up by kind change halfway rather than mixing. + #[test] + fn unpaired_keyframe_filters_change_halfway() { + let item = animated(vec![ + Keyframe { + filters: vec![Filter::Sepia { amount: 1.0 }], + ..Keyframe::new(Duration::ZERO) + }, + Keyframe { + ease: Ease::Linear, + filters: vec![Filter::Invert { amount: 1.0 }], + ..Keyframe::new(Duration::from_secs(1)) + }, + ]); + let filters = |millis| item.posed(Duration::from_millis(millis)).filters[1].clone(); + assert_eq!(filters(499), Filter::Sepia { amount: 1.0 }); + assert_eq!(filters(500), Filter::Invert { amount: 1.0 }); + } + + #[test] + fn refuses_keyframes_it_cannot_play() { + let at = |millis| Keyframe::new(Duration::from_millis(millis)); + let cases = [ + vec![at(500), at(100)], + vec![Keyframe { x: f32::NAN, ..at(0) }], + vec![Keyframe { + crop: Crop { + left: 0.6, + right: 0.6, + ..Crop::default() + }, + ..at(0) + }], + vec![Keyframe { + filters: vec![Filter::Opacity { amount: 2.0 }], + ..at(0) + }], + vec![Keyframe { + filters: vec![Filter::Lut { + source: "grade".into(), + amount: 1.0, + }], + ..at(0) + }], + ]; + for keyframes in cases { + let mut collection = example(); + collection.sources.insert( + "grade".into(), + Source::Lut { + path: "grade.cube".into(), + }, + ); + collection.scenes[0].items[0].keyframes = keyframes.clone(); + assert!(matches!(collection.validate(), Err(Error::Invalid(_))), "{keyframes:?}"); + } + + let mut collection = example(); + collection.top[0].keyframes = vec![at(0)]; + assert!(matches!(collection.validate(), Err(Error::Invalid(_))), "the top layer"); + // The same moment twice jumps. + collection.top[0].keyframes.clear(); + collection.scenes[0].items[0].keyframes = vec![at(100), at(100)]; + collection.validate().unwrap(); + } + #[test] fn refuses_duplicate_scenes() { let mut collection = example(); diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index 492c612b..c44f64c5 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -1,7 +1,9 @@ //! Draw scenes onto the canvas on the GPU: a preview, a program, and the //! transition between them. +use std::borrow::Cow; use std::collections::{BTreeMap, HashMap, HashSet}; +use std::time::Duration; use kio::clock::Instant; use moq_video::render::{Renderer, wgpu}; @@ -30,9 +32,11 @@ pub const FORMAT: wgpu::TextureFormat = wgpu::TextureFormat::Rgba8Unorm; /// broadcast switcher and CSS do, so a 50% fade is a 50% mix of the stored /// values. /// -/// A transition runs on the clock [`render`](Self::render) is given, so every -/// output drawn from the program, such as the encoder and a recording, sees the -/// same frame of it. +/// A transition and each item's keyframes run on the clock +/// [`render`](Self::render) is given, so every output drawn from the program, +/// such as the encoder and a recording, sees the same frame of them. An item's +/// keyframes start when its scene goes on program, and separately when it goes +/// into preview, for the preview. pub struct Compositor { device: wgpu::Device, queue: wgpu::Queue, @@ -52,12 +56,19 @@ pub struct Compositor { overlays: Option, preview_scene: Option, program_scene: Option, + /// When the preview scene went into preview, or `None` until the next + /// render, which starts it. + preview_start: Option, + /// When the program scene went on program, as `preview_start` is. + program_start: Option, switch: Option, } /// A transition under way from `from` to the program scene. struct Switch { from: Option, + /// When `from` went on program. + from_start: Instant, start: Instant, transition: Transition, } @@ -121,6 +132,8 @@ impl Compositor { overlays: None, preview_scene: None, program_scene: None, + preview_start: None, + program_start: None, switch: None, }) } @@ -190,9 +203,18 @@ impl Compositor { self.program_scene.as_deref().into_iter().chain(leaving) } - /// Put the scene called `scene` in preview, or nothing. + /// Put the scene called `scene` in preview, or nothing, playing its + /// keyframes from the start, even if it was already there. pub fn set_preview(&mut self, scene: Option) { self.preview_scene = scene; + self.preview_start = None; + } + + /// How long the preview scene has been in preview as of `now`: where its + /// items' keyframes are, for [`Item::posed`]. + pub fn preview_since(&self, now: Instant) -> Duration { + self.preview_start + .map_or(Duration::ZERO, |start| now.saturating_duration_since(start)) } /// Follow each renamed scene to its new name, wherever it is: in preview, @@ -232,6 +254,7 @@ impl Compositor { pub fn cut(&mut self) { self.switch = None; self.program_scene = self.preview_scene.clone(); + self.program_start = None; } /// Take the preview to program by `transition`, starting at `now`. @@ -240,8 +263,10 @@ impl Compositor { /// starts from what that one was heading to. pub fn transition(&mut self, transition: Transition, now: Instant) { let from = std::mem::replace(&mut self.program_scene, self.preview_scene.clone()); + let from_start = self.program_start.replace(now).unwrap_or(now); self.switch = (!transition.duration().is_zero()).then_some(Switch { from, + from_start, start: now, transition, }); @@ -257,6 +282,10 @@ impl Compositor { self.update(collection, sources, now); self.paint(collection, now); + let since = |start: &mut Option| now.saturating_duration_since(*start.get_or_insert(now)); + let preview_since = since(&mut self.preview_start); + let program_since = since(&mut self.program_start); + let linear = self.switch.as_ref().map(|switch| { let elapsed = now.saturating_duration_since(switch.start); (elapsed.as_secs_f32() / switch.transition.duration().as_secs_f32()).min(1.0) @@ -277,13 +306,13 @@ impl Compositor { // The top layer goes over both outputs, and over a transition rather // than through it, so it stays on across the switch. draw.target(&self.preview); - draw.scene(&self.preview, self.preview_scene.as_deref()); - draw.layer(&self.preview, Place::Top); + draw.scene(&self.preview, On(self.preview_scene.as_deref(), preview_since)); + draw.layer(&self.preview, Place::Top, Duration::ZERO); - let to = self.program_scene.as_deref(); + let to = On(self.program_scene.as_deref(), program_since); match (&self.switch, linear) { (Some(switch), Some(linear)) => { - let from = switch.from.as_deref(); + let from = On(switch.from.as_deref(), now.saturating_duration_since(switch.from_start)); let progress = switch.transition.progress(linear); let canvas = Rect::fill(&self.canvas); match switch.transition { @@ -324,7 +353,7 @@ impl Compositor { draw.scene(&self.program, to); } } - draw.layer(&self.program, Place::Top); + draw.layer(&self.program, Place::Top, Duration::ZERO); self.effects.submit(frame); Ok(()) @@ -526,12 +555,7 @@ impl Pose { width: mix(self.rect.width, other.rect.width), height: mix(self.rect.height, other.rect.height), }, - crop: Crop { - left: mix(self.crop.left, other.crop.left), - top: mix(self.crop.top, other.crop.top), - right: mix(self.crop.right, other.crop.right), - bottom: mix(self.crop.bottom, other.crop.bottom), - }, + crop: self.crop.toward(&other.crop, progress), opacity: mix(self.opacity, other.opacity), } } @@ -554,9 +578,9 @@ fn mixed(old: &Item, new: &Item, progress: f32) -> Option> { /// `other` goes just over the paired item it was over in `other`, or under /// every slot if it was under every paired item. fn stack( - lead: &[Item], + lead: &[Cow<'_, Item>], lead_pairs: &[Option], - other: &[Item], + other: &[Cow<'_, Item>], other_pairs: &[Option], ) -> Vec<(Option, Option)> { let mut over: HashMap, Vec> = HashMap::new(); @@ -597,14 +621,21 @@ fn fits(device: &wgpu::Device, canvas: Canvas) -> Result<(), Error> { Ok(()) } +/// A scene to draw, or none, and how long it has been on: where its items' +/// keyframes are. +#[derive(Clone, Copy)] +struct On<'a>(Option<&'a str>, Duration); + /// One render's draws onto its targets. struct Draw<'a> { collection: &'a Collection, effects: &'a mut Effects, library: Library<'a>, frame: &'a mut Frame, - /// Each item, filtered once though it is drawn onto several targets. - prepared: HashMap<(Place, usize), Option>, + /// Each item, filtered once though it is drawn onto several targets, by + /// where it is and its [`Item::settled`] time, as two outputs showing its + /// scene can be at different points of its keyframes. + prepared: HashMap<(Place, usize, Duration), Option>, } impl Draw<'_> { @@ -613,17 +644,17 @@ impl Draw<'_> { self.frame.target(&target.picture); } - /// Draw `scene`'s items onto `target`. A missing scene draws nothing. - fn scene(&mut self, target: &Target, scene: Option<&str>) { - if let Some(scene) = scene.filter(|name| self.collection.scene(name).is_some()) { - self.layer(target, Place::Scene(scene.to_string())); + /// Draw `on`'s items onto `target`. A missing scene draws nothing. + fn scene(&mut self, target: &Target, on: On) { + if let Some(scene) = on.0.filter(|name| self.collection.scene(name).is_some()) { + self.layer(target, Place::Scene(scene.to_string()), on.1); } } - /// Start drawing `target`, and draw `scene` onto it whole. - fn whole(&mut self, target: &Target, scene: Option<&str>) { + /// Start drawing `target`, and draw `on` onto it whole. + fn whole(&mut self, target: &Target, on: On) { self.target(target); - self.scene(target, scene); + self.scene(target, on); } /// Draw `picture`, a canvas-sized target drawn this render, onto `target` @@ -634,19 +665,20 @@ impl Draw<'_> { } } - /// Draw `place`'s items onto `target`, back to front, each through its crop, - /// filters, and blend mode. A hidden item, a source with no picture yet, or - /// a filter whose source has nothing yet draws nothing. - fn layer(&mut self, target: &Target, place: Place) { + /// Draw `place`'s items onto `target`, back to front, each posed `since` + /// `place` went on, through its crop, filters, and blend mode. A hidden + /// item, a source with no picture yet, or a filter whose source has nothing + /// yet draws nothing. + fn layer(&mut self, target: &Target, place: Place, since: Duration) { let items = self.collection.items(&place).map_or(&[][..], Vec::as_slice); for (index, item) in items.iter().enumerate().filter(|(_, item)| item.visible) { - self.item(target, &place, index, item); + self.item(target, (place.clone(), index, item.settled(since)), &item.posed(since)); } } - /// Draw `item`, at `index` of `place`, onto `target` where it is placed. - fn item(&mut self, target: &Target, place: &Place, index: usize, item: &Item) { - let key = (place.clone(), index); + /// Draw `item`, posed, where it is placed onto `target`, shared with every + /// other draw of it under `key`. + fn item(&mut self, target: &Target, key: (Place, usize, Duration), item: &Item) { let prepared = match self.prepared.get(&key) { Some(prepared) => prepared.clone(), None => { @@ -671,7 +703,7 @@ impl Draw<'_> { } /// Draw `progress` of a move from scene `from` to scene `to` onto - /// `target`. + /// `target`, each item posed by its keyframes as its scene is. /// /// Each visible item of `to` is paired with the first visible item of /// `from` showing the same source that no earlier one took, and glides @@ -682,13 +714,15 @@ impl Draw<'_> { /// unpaired item just over the paired item it was over, so the move /// starts from `from` and lands on `to` as each is drawn. A paired item /// whose blend mode or kinds of filter differ changes look halfway too. - fn moving(&mut self, target: &Target, from: Option<&str>, to: Option<&str>, progress: f32) { + fn moving(&mut self, target: &Target, from: On, to: On, progress: f32) { let collection = self.collection; let items = |scene: Option<&str>| { let scene = scene.and_then(|name| collection.scene(name)); scene.map_or(&[][..], |scene| scene.items.as_slice()) }; - let (leaving, entering) = (items(from), items(to)); + let posed = |on: On| -> Vec> { items(on.0).iter().map(|item| item.posed(on.1)).collect() }; + let resting = items(to.0); + let (leaving, entering) = (posed(from), posed(to)); // Which item of the other scene each visible item pairs with. let mut left = vec![None; leaving.len()]; @@ -704,13 +738,13 @@ impl Draw<'_> { // Each slot is an index into `leaving`, `entering`, or both. let slots: Vec<(Option, Option)> = if progress < 0.5 { - stack(leaving, &left, entering, &entered) + stack(&leaving, &left, &entering, &entered) } else { - let slots = stack(entering, &entered, leaving, &left); + let slots = stack(&entering, &entered, &leaving, &left); slots.into_iter().map(|(to, from)| (from, to)).collect() }; - let place = Place::Scene(to.unwrap_or_default().to_string()); + let place = Place::Scene(to.0.unwrap_or_default().to_string()); for slot in slots { match slot { (Some(at), None) => { @@ -730,13 +764,14 @@ impl Draw<'_> { ); } (Some(at), Some(index)) => { - let (old, new) = (&leaving[at], &entering[index]); + let (old, new) = (&*leaving[at], &*entering[index]); let own = Pose::of(collection, new); let pose = Pose::of(collection, old).toward(&own, progress); if old.filters == new.filters && old.blend == new.blend { if pose == own { - // Shared with the preview, which draws it the same. - self.item(target, &place, index, new); + // Shared with the preview when it draws the same. + let key = (place.clone(), index, resting[index].settled(to.1)); + self.item(target, key, new); } else { self.posed(target, new, pose); } @@ -1300,6 +1335,94 @@ mod tests { .await; } + /// Keyframes sliding an item in from off the canvas's left over a second. + fn slide_in() -> Vec { + vec![ + crate::Keyframe { + x: -8.0, + ..crate::Keyframe::new(Duration::ZERO) + }, + crate::Keyframe { + ease: crate::Ease::Linear, + ..crate::Keyframe::new(Duration::from_secs(1)) + }, + ] + } + + /// An item's keyframes play from when its scene went into preview, for the + /// preview, and from when it went on program, for the program, so one scene + /// can be at two points of them at once. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn keyframes_play_from_when_the_scene_goes_on() { + local(async { + let mut collection = collection(); + collection.scenes[1].items[0].keyframes = slide_in(); + let mut studio = Studio::new(collection).await; + let start = Instant::now(); + let at = |millis| start + Duration::from_millis(millis); + studio.compositor.set_preview(Some("red".into())); + studio.compositor.cut(); + studio.compositor.set_preview(Some("blue".into())); + + studio.render(at(0)); + assert_eq!(pixel(&studio.preview(), 1, 6), BLACK, "off the canvas"); + studio.render(at(500)); + let preview = studio.preview(); + assert_eq!(pixel(&preview, 1, 6), BLUE, "halfway in"); + assert_eq!(pixel(&preview, 6, 6), BLACK); + + studio.compositor.cut(); + studio.render(at(500)); + assert_eq!(pixel(&studio.program(), 1, 6), BLACK, "on program from the start"); + assert_eq!(pixel(&studio.preview(), 1, 6), BLUE, "the preview carries on"); + + studio.render(at(1000)); + let program = studio.program(); + assert_eq!(pixel(&program, 1, 6), BLUE); + assert_eq!(pixel(&program, 6, 6), BLACK); + assert_eq!(pixel(&studio.preview(), 6, 6), BLUE, "landed where it rests"); + studio.render(at(9000)); + assert_eq!(pixel(&studio.program(), 6, 6), BLUE, "holding the last keyframe"); + + // Picking the scene again plays it again. + studio.compositor.set_preview(Some("blue".into())); + studio.render(at(9000)); + assert_eq!(pixel(&studio.preview(), 1, 6), BLACK); + }) + .await; + } + + /// A transition starts the incoming scene's keyframes, however long it was + /// in preview, while the outgoing scene's carry on. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_transition_starts_the_incoming_keyframes() { + local(async { + let mut collection = collection(); + collection.scenes[0].items[0].keyframes = slide_in(); + collection.scenes[1].items[0].keyframes = slide_in(); + let mut studio = Studio::new(collection).await; + let start = Instant::now(); + let at = |millis| start + Duration::from_millis(millis); + studio.compositor.set_preview(Some("blue".into())); + studio.compositor.cut(); + studio.compositor.set_preview(Some("red".into())); + studio.render(at(0)); + + studio.compositor.transition(FADE, at(9000)); + studio.render(at(9500)); + let program = studio.program(); + // Red has slid halfway in over the left half, blue is in place, and + // the fade is halfway. + let actual = pixel(&program, 1, 6); + assert!(near(actual, [128, 0, 128, 255]), "red half over blue: {actual:?}"); + let actual = pixel(&program, 6, 6); + assert!(near(actual, [0, 0, 128, 255]), "nothing half over blue: {actual:?}"); + }) + .await; + } + /// A move starts from the outgoing scene's stacking and switches to the /// incoming scene's halfway, and an item's filter settings mix from one /// scene's to the other's as it goes. diff --git a/rs/moq-scene/src/lib.rs b/rs/moq-scene/src/lib.rs index f5d0d77f..b7665c5d 100644 --- a/rs/moq-scene/src/lib.rs +++ b/rs/moq-scene/src/lib.rs @@ -18,7 +18,8 @@ //! program mix. //! - [`Compositor`] draws the scene in preview and the scene on program into //! textures on the caller's `wgpu` device, cutting or running a -//! [`Transition`] between them. `moq_video::render::export` hands the +//! [`Transition`] between them, and plays each item's [`Keyframe`]s from +//! when its scene went on. `moq_video::render::export` hands the //! program to an encoder, on the GPU where the platform allows, and a //! [`Scaler`] draws it smaller for a smaller rendition. //! - [`publish`] gives each local source its own track, in a broadcast the @@ -44,8 +45,8 @@ pub mod source; pub use clip::Clip; pub use collection::{ - Audio, Blend, Canvas, Collection, Crop, Direction, Filter, Item, Layer, MaskMode, Mixes, Rect, Rgba, Scene, Source, - Transition, Video, + Audio, Blend, Canvas, Collection, Crop, Direction, Ease, Filter, Item, Keyframe, Layer, MaskMode, Mixes, Rect, + Rgba, Scene, Source, Transition, Video, }; pub use compositor::{Compositor, FORMAT}; pub use edit::{Change, Draft, Edit, History}; diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index f31be186..ff33e380 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -11,8 +11,8 @@ use egui_dock::{DockArea, DockState, NodeIndex, Style, TabViewer}; use moq_scene::browser::Paint; use moq_scene::source::{Feed, Status}; use moq_scene::{ - Blend, Canvas, Clip, Collection, Crop, Direction, Draft, Filter, Item, Layer, MaskMode, Rect, Scene, Source, - Transition, Video, wgpu, + Blend, Canvas, Clip, Collection, Crop, Direction, Draft, Ease, Filter, Item, Keyframe, Layer, MaskMode, Rect, + Scene, Source, Transition, Video, wgpu, }; use crate::autosave::Conflict; @@ -36,6 +36,7 @@ enum Tab { Devices, Guests, Filters, + Keyframes, Audio, Output, Health, @@ -54,6 +55,7 @@ impl Tab { Self::Devices => "Devices", Self::Guests => "Guests", Self::Filters => "Filters", + Self::Keyframes => "Keyframes", Self::Audio => "Audio", Self::Output => "Output", Self::Health => "Health", @@ -90,7 +92,14 @@ impl Studio { tree.split_right( sources, 0.5, - vec![Tab::Filters, Tab::Replay, Tab::Health, Tab::Audio, Tab::Output], + vec![ + Tab::Filters, + Tab::Keyframes, + Tab::Replay, + Tab::Health, + Tab::Audio, + Tab::Output, + ], ); Self { @@ -295,6 +304,7 @@ impl TabViewer for Panels { Tab::Devices => self.devices(ui), Tab::Guests => self.guests(ui), Tab::Filters => self.filters(ui), + Tab::Keyframes => self.keyframes(ui), Tab::Audio => self.audio(ui), Tab::Output => self.output(ui), Tab::Health => self.health(ui), @@ -756,7 +766,9 @@ impl Panels { } /// Select, move, and resize the items of the layer the Items panel shows - /// by dragging them in the preview, snapping as they go. + /// by dragging them in the preview, snapping as they go. An item is + /// grabbed where its keyframes have it in the preview, and a drag moves + /// where it rests, which carries its keyframes along. fn arrange(&mut self, ui: &Ui, state: &mut State, response: &egui::Response, canvas: Canvas) -> anyhow::Result<()> { let gesture = self.gestures.of(response); // The release still moves the item to where the pointer let go. @@ -774,9 +786,11 @@ impl Panels { } let Ok(layer) = self.layer(state) else { return Ok(()) }; - let Some(items) = state.collection().items(&layer).cloned() else { + let Some(resting) = state.collection().items(&layer).cloned() else { return Ok(()); }; + let since = state.compositor.preview_since(kio::clock::Instant::now()); + let items: Vec = resting.iter().map(|item| item.posed(since).into_owned()).collect(); let selected = match &self.selection { Some(Selection::Item(selected, index)) if *selected == layer => Some(*index), _ => None, @@ -799,7 +813,7 @@ impl Panels { layer: layer.clone(), index, grab, - rect: items[index].rect, + rect: resting[index].rect, origin: view.point(origin), gesture, hints: None, @@ -813,6 +827,21 @@ impl Panels { let point = view.point(pos); // Whole canvas pixels, so an unsnapped drag saves round numbers. let delta = [(point[0] - drag.origin[0]).round(), (point[1] - drag.origin[1]).round()]; + let item = state + .collection() + .items(&drag.layer) + .and_then(|items| items.get(drag.index)) + .cloned(); + // How far the keyframes have the item from where it rests, which + // snapping, done where the item shows, leaves alone. + let shown = item.as_ref().map_or(drag.rect, |item| item.posed(since).rect); + let rest = item.as_ref().map_or(drag.rect, |item| item.rect); + let away = |rect: Rect, sign: f32| Rect { + x: rect.x + sign * (shown.x - rest.x), + y: rect.y + sign * (shown.y - rest.y), + width: rect.width + sign * (shown.width - rest.width), + height: rect.height + sign * (shown.height - rest.height), + }; let rect = drag.grab.drag(drag.rect, delta); let free = ui.input(|input| input.modifiers.command); drag.hints = (self.snapping.on && !free).then(|| { @@ -820,14 +849,11 @@ impl Panels { reach, grid: self.snapping.grid.then_some(self.snapping.spacing as f32), }; - snap.apply(drag.grab, rect, &others(state, &drag.layer, drag.index), &canvas) + let others = others(state, &drag.layer, drag.index, since); + snap.apply(drag.grab, away(rect, 1.0), &others, &canvas) }); - let rect = drag.hints.as_ref().map_or(rect, |snapped| snapped.rect); - let item = state - .collection() - .items(&drag.layer) - .and_then(|items| items.get(drag.index)); - if let Some(item) = item.cloned() + let rect = drag.hints.as_ref().map_or(rect, |snapped| away(snapped.rect, -1.0)); + if let Some(item) = item && item.rect != rect { let verb = if drag.grab == Grab::Move { "Move" } else { "Resize" }; @@ -842,7 +868,8 @@ impl Panels { } // Drawn from the collection and selection as this frame left them. - let items = state.collection().items(&layer).cloned().unwrap_or_default(); + let items = state.collection().items(&layer).map_or(&[][..], Vec::as_slice); + let items: Vec = items.iter().map(|item| item.posed(since).into_owned()).collect(); let selected = match &self.selection { Some(Selection::Item(selected, index)) if *selected == layer => Some(*index), _ => None, @@ -1729,6 +1756,185 @@ impl Panels { self.check(result); } + /// The selected item's keyframes: when each is, how the item eases into + /// it, and how far it takes the item from where it rests. + fn keyframes(&mut self, ui: &mut Ui) { + let engine = self.engine.clone(); + let mut state = engine.state.lock().unwrap(); + self.follow_reloads(&state); + let selected = match &self.selection { + Some(Selection::Item(layer, index)) => { + let item = state.collection().items(layer).and_then(|items| items.get(*index)); + item.map(|item| (layer.clone(), *index, item.clone())) + } + _ => None, + }; + let Some((layer, index, item)) = selected else { + ui.label("Select an item to animate it."); + return; + }; + if layer == Layer::Top { + ui.label("The top layer never goes on program, so its items have no keyframes to play."); + return; + } + let what = format!("`{}`", item.source); + let mut play = false; + ui.horizontal(|ui| { + ui.strong(format!("Keyframes of {what}")); + play = ui + .button("Play") + .on_hover_text("Play the preview scene's keyframes again from the start") + .clicked(); + }); + ui.weak( + "Each keyframe moves the item away from where it rests, so dragging it in the preview \ + moves its whole animation. They play once, from when the item's scene goes on, and the \ + last one holds.", + ); + + let mut next = item.keyframes.clone(); + let mut action: Option<(String, Option)> = None; + let mut remove = None; + egui::ScrollArea::vertical().show(ui, |ui| { + for at in 0..next.len() { + // Between its neighbors, so the keyframes stay in order. + let earliest = at.checked_sub(1).map_or(0, |before| next[before].at.as_millis() as u64); + let latest = next.get(at + 1).map_or(u64::MAX, |after| after.at.as_millis() as u64); + let keyframe = &mut next[at]; + ui.push_id(at, |ui| { + ui.horizontal_wrapped(|ui| { + let mut millis = keyframe.at.as_millis() as u64; + let response = ui + .add(egui::DragValue::new(&mut millis).range(earliest..=latest).suffix(" ms")) + .on_hover_text("How long after the item's scene goes on"); + if let Some(gesture) = changed(&response, &mut self.gestures) { + keyframe.at = Duration::from_millis(millis); + action = Some((format!("Time a keyframe of {what}"), gesture)); + } + let before = keyframe.ease; + egui::ComboBox::from_id_salt("ease") + .selected_text(ease_name(keyframe.ease)) + .show_ui(ui, |ui| { + for ease in Ease::ALL { + ui.selectable_value(&mut keyframe.ease, ease, ease_name(ease)); + } + }) + .response + .on_hover_text("How the item eases into this keyframe from the one before"); + if keyframe.ease != before { + action = Some((format!("Ease a keyframe of {what}"), None)); + } + let offsets = [ + ("x", &mut keyframe.x), + ("y", &mut keyframe.y), + ("width", &mut keyframe.width), + ("height", &mut keyframe.height), + ]; + for (label, offset) in offsets { + ui.weak(label); + let response = ui.add(egui::DragValue::new(offset).suffix(" px")); + if let Some(gesture) = changed(&response, &mut self.gestures) { + action = Some((format!("Move a keyframe of {what}"), gesture)); + } + } + let crop = &mut keyframe.crop; + let edges = [ + ("crop left", &mut crop.left), + ("top", &mut crop.top), + ("right", &mut crop.right), + ("bottom", &mut crop.bottom), + ]; + for (label, edge) in edges { + ui.weak(label); + let response = ui.add(egui::DragValue::new(edge).range(0.0..=0.95).speed(0.002)); + if let Some(gesture) = changed(&response, &mut self.gestures) { + action = Some((format!("Crop a keyframe of {what}"), gesture)); + } + } + if ui.small_button("Remove").clicked() { + remove = Some(at); + } + }); + + let mut dropped = None; + for (index, filter) in keyframe.filters.iter_mut().enumerate() { + ui.push_id(index, |ui| { + ui.horizontal_wrapped(|ui| { + if let Some(gesture) = filter_ui(ui, filter, &[], &[], &mut self.gestures) { + let label = format!("Set {} on a keyframe of {what}", filter_name(filter)); + action = Some((label, gesture)); + } + if ui.small_button("Remove").clicked() { + dropped = Some(index); + } + }); + }); + } + if let Some(index) = dropped { + keyframe.filters.remove(index); + action = Some((format!("Remove a filter from a keyframe of {what}"), None)); + } + // A keyframe's filters read no other source, so no LUT or mask. + egui::ComboBox::from_id_salt("add filter") + .selected_text("Add filter") + .show_ui(ui, |ui| { + for filter in templates(&[], &[]) { + if ui.selectable_label(false, filter_name(&filter)).clicked() { + let label = format!("Add {} to a keyframe of {what}", filter_name(&filter)); + action = Some((label, None)); + keyframe.filters.push(filter); + } + } + }) + .response + .on_hover_text( + "Applied after the item's own. Settings mix into the next keyframe's when \ + its filters are of the same kinds, in order, and otherwise change halfway.", + ); + ui.separator(); + }); + } + }); + if let Some(at) = remove { + next.remove(at); + action = Some((format!("Remove a keyframe from {what}"), None)); + } + if ui + .button("Add keyframe") + .on_hover_text("Half a second after the last, where the item rests, with the last one's filters") + .clicked() + { + let keyframe = match next.last() { + Some(last) => Keyframe { + ease: last.ease, + filters: last.filters.clone(), + ..Keyframe::new(last.at + Duration::from_millis(500)) + }, + None => Keyframe::new(Duration::ZERO), + }; + next.push(keyframe); + action = Some((format!("Add a keyframe to {what}"), None)); + } + + if play { + let scene = state.compositor.preview_scene().map(str::to_owned); + state.compositor.set_preview(scene); + } + let Some((label, gesture)) = action else { return }; + let result = edit(&mut state, label, gesture, |draft| { + draft.set_item( + &layer, + index, + Item { + keyframes: next, + ..item + }, + ) + }); + drop(state); + self.check(result); + } + /// The mixes, which the program carries and which plays to the speakers, /// and each source's gain, mute, delay, and routing. fn audio(&mut self, ui: &mut Ui) { @@ -2526,6 +2732,15 @@ fn direction_name(direction: Direction) -> &'static str { } } +fn ease_name(ease: Ease) -> &'static str { + match ease { + Ease::Linear => "linear", + Ease::In => "ease in", + Ease::Out => "ease out", + Ease::InOut => "ease in and out", + } +} + fn blend_name(blend: Blend) -> &'static str { match blend { Blend::Normal => "normal", @@ -2849,9 +3064,10 @@ struct Drag { /// How far, in points on screen, the pointer reaches to an edge or a line. const REACH: f32 = 8.0; -/// The visible items of the scene in preview and the top layer, other than -/// the one at `index` of `layer`: what a dragged item snaps to. -fn others(state: &State, layer: &Layer, index: usize) -> Vec { +/// Where the visible items of the scene in preview and the top layer show +/// `since` the scene went into preview, other than the one at `index` of +/// `layer`: what a dragged item snaps to. +fn others(state: &State, layer: &Layer, index: usize, since: Duration) -> Vec { let scene = state .compositor .preview_scene() @@ -2862,7 +3078,7 @@ fn others(state: &State, layer: &Layer, index: usize) -> Vec { let items = collection.items(&shown).map_or(&[][..], Vec::as_slice); for (at, item) in items.iter().enumerate() { if item.visible && !(shown == *layer && at == index) { - rects.push(item.rect); + rects.push(item.posed(since).rect); } } } From 60999469e5801162b56a6cdf02597de9194e54ad Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 15:11:27 -0800 Subject: [PATCH 3/7] fix(scene): copy animated items, and take browser input where keyframes draw Co-Authored-By: Claude Opus 5.5 --- rs/moq-scene/src/clip.rs | 34 +++++++++++++++++++++++++++++--- rs/moq-studio/src/interact.rs | 37 +++++++++++++++++++++++++---------- rs/moq-studio/src/ui.rs | 3 ++- 3 files changed, 60 insertions(+), 14 deletions(-) diff --git a/rs/moq-scene/src/clip.rs b/rs/moq-scene/src/clip.rs index 3c6398d9..9da603ad 100644 --- a/rs/moq-scene/src/clip.rs +++ b/rs/moq-scene/src/clip.rs @@ -7,7 +7,9 @@ use std::path::Path; use serde::{Deserialize, Serialize}; use crate::Error; -use crate::collection::{Audio, Collection, Filter, Item, Layer, Mixes, Source, Video, validate_filters, versioned}; +use crate::collection::{ + Audio, Collection, Filter, Item, Layer, Mixes, Scene, Source, Video, validate_filters, versioned, +}; use crate::edit::Draft; /// What a copy holds: items, or a filter chain. @@ -113,8 +115,12 @@ impl Clip { }, audio: items.audio.clone(), video: items.video.clone(), - scenes: Vec::new(), - top: items.items.clone(), + // In a scene, where keyframes play; pasting into the top + // layer checks it holds none. + scenes: vec![Scene { + items: items.items.clone(), + ..Scene::new("clip") + }], ..Collection::new() } .validate()? @@ -293,6 +299,28 @@ mod tests { assert_eq!(pasted.top, collection.scenes[0].items); } + /// An animated item copies with its keyframes into a scene, and the top + /// layer, which never goes on to play them, refuses it. + #[test] + fn pastes_keyframes_into_a_scene_only() { + let mut collection = collection(); + collection.scenes[0].items[0].keyframes = vec![crate::Keyframe { + x: -100.0, + ..crate::Keyframe::new(std::time::Duration::ZERO) + }]; + let clip = Clip::items(&collection, collection.scenes[0].items.clone(), Path::new(DIR)); + let text = clip.to_toml().unwrap(); + let pasted = paste(&collection, &Layer::Scene("Scene".into()), &text, DIR); + assert_eq!(pasted.scenes[0].items[1], collection.scenes[0].items[0]); + + let Clip::Items(items) = Clip::from_toml(&text).unwrap() else { + panic!("not items") + }; + let mut draft = collection.draft("Paste"); + draft.paste(&Layer::Top, &items, Path::new(DIR)).unwrap(); + assert!(matches!(draft.finish(), Err(Error::Invalid(_)))); + } + /// Into another collection, a clip brings what its items show, renaming a /// source whose name is taken by another, and its clone follows the rename. #[test] diff --git a/rs/moq-studio/src/interact.rs b/rs/moq-studio/src/interact.rs index 5d287b3f..8eca809d 100644 --- a/rs/moq-studio/src/interact.rs +++ b/rs/moq-studio/src/interact.rs @@ -67,13 +67,14 @@ struct Place { impl Interact { /// Turn this frame's input over `image`, which shows `scene` of /// `collection` under its top layer, into what each page it touches should receive, by page - /// name. + /// name. `scene` is the scene's name and how long it has been in the + /// preview, which places its items where their keyframes have them. pub fn update( &mut self, ui: &egui::Ui, image: egui::Rect, collection: &Collection, - scene: Option<&str>, + scene: Option<(&str, Duration)>, ) -> Vec<(String, Input)> { let canvas = collection.canvas; let on_canvas = |pos: egui::Pos2| { @@ -395,14 +396,17 @@ impl Interact { } /// The topmost visible browser source, or clone of one, at canvas point (`x`, -/// `y`) in `scene` and the top layer over it, and the point on its page. -fn page_at(collection: &Collection, scene: Option<&str>, x: f32, y: f32) -> Option { - let scene = scene.and_then(|name| collection.scene(name)); +/// `y`) in `scene`, where its keyframes have each item that long after it went +/// on, and the top layer over it, and the point on its page. +fn page_at(collection: &Collection, scene: Option<(&str, Duration)>, x: f32, y: f32) -> Option { + let since = scene.map_or(Duration::ZERO, |(_, since)| since); + let scene = scene.and_then(|(name, _)| collection.scene(name)); let items = scene.map(|scene| scene.items.as_slice()).unwrap_or_default(); items .iter() .chain(&collection.top) .rev() + .map(|item| item.posed(since)) .filter(|item| item.visible) .find_map(|item| { let rect = item.rect; @@ -414,7 +418,7 @@ fn page_at(collection: &Collection, scene: Option<&str>, x: f32, y: f32) -> Opti let page = collection.capture(&item.source); let place = Place { rect, - crop: collection.crop(item), + crop: collection.crop(&item), }; let (x, y) = scale(collection, page, place, x, y)?; Some(Hit { @@ -590,8 +594,8 @@ mod tests { name: "Main".into(), items: vec![item("alerts", fill, true), item("chat", corner, true)], }]; - let at = - |collection: &Collection, x, y| page_at(collection, Some("Main"), x, y).map(|hit| (hit.page, hit.x, hit.y)); + let main = Some(("Main", Duration::ZERO)); + let at = |collection: &Collection, x, y| page_at(collection, main, x, y).map(|hit| (hit.page, hit.x, hit.y)); assert_eq!(at(&collection, 1720.0, 680.0), Some(("chat".into(), 200, 400))); assert_eq!(at(&collection, 960.0, 540.0), Some(("alerts".into(), 400, 300))); @@ -602,7 +606,7 @@ mod tests { Some(("alerts".into(), 716, 377)), "a hidden page takes no input" ); - assert!(page_at(&collection, Some("Other"), 1.0, 1.0).is_none()); + assert!(page_at(&collection, Some(("Other", Duration::ZERO)), 1.0, 1.0).is_none()); // The top layer draws over every scene, and a clone shows its source's // page, which runs under the source's name. @@ -644,6 +648,19 @@ mod tests { }; assert_eq!(at(&collection, 1520.0, 280.0), Some(("chat".into(), 200, 400))); assert_eq!(at(&collection, 1720.0, 680.0), Some(("chat".into(), 300, 600))); + + // A keyframed item takes input where its keyframes have it. + collection.scenes[0].items[1].keyframes = vec![moq_scene::Keyframe { + x: -400.0, + ..moq_scene::Keyframe::new(Duration::ZERO) + }]; + assert_eq!(at(&collection, 1320.0, 680.0), Some(("chat".into(), 300, 600))); + let later = page_at(&collection, Some(("Main", Duration::from_secs(9))), 1320.0, 680.0); + assert_eq!( + later.map(|hit| hit.page), + Some("chat".into()), + "the last keyframe holds" + ); } /// One frame of `events` over a preview showing the canvas at its own @@ -663,7 +680,7 @@ mod tests { }; let mut delivered = Vec::new(); let mut output = ctx.run_ui(input, |ui| { - delivered = interact.update(ui, image, collection, Some("Main")); + delivered = interact.update(ui, image, collection, Some(("Main", Duration::ZERO))); }); // Nothing renders, so the font atlas egui uploaded goes nowhere. output.textures_delta.clear(); diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index ff33e380..a607e75a 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -752,7 +752,8 @@ impl Panels { // Interacting, the pointer belongs to the pages rather than to arranging. if self.interacting { let image = picture(ui, self.preview.id, canvas, egui::Sense::hover()).rect; - let scene = state.compositor.preview_scene(); + let since = state.compositor.preview_since(kio::clock::Instant::now()); + let scene = state.compositor.preview_scene().map(|scene| (scene, since)); self.rebind(&state.sources); let delivered = self.interact.update(ui, image, state.collection(), scene); deliver(&state.sources, delivered); From 76b908ba07777367f4dad0e9ec80bb9e1bb41225 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 20:56:52 -0800 Subject: [PATCH 4/7] fix(studio): a page drag follows its item as keyframes move it Co-Authored-By: Claude Opus 5.5 --- rs/moq-studio/src/interact.rs | 154 +++++++++++++++++++++++++++++----- 1 file changed, 135 insertions(+), 19 deletions(-) diff --git a/rs/moq-studio/src/interact.rs b/rs/moq-studio/src/interact.rs index 8eca809d..361d74cb 100644 --- a/rs/moq-studio/src/interact.rs +++ b/rs/moq-studio/src/interact.rs @@ -1,12 +1,13 @@ //! Interaction with browser sources through the preview: the pointer goes to //! the page under it, and the keyboard to the page last clicked. +use std::borrow::Cow; use std::cell::RefCell; use std::time::Duration; use kio::clock::Instant; use moq_scene::browser::{Button, Input, Modifiers}; -use moq_scene::{Collection, Crop, Rect, Source}; +use moq_scene::{Collection, Crop, Item, Rect, Source}; /// How close in time and space a press must follow the last to count toward a /// double or triple click. @@ -39,8 +40,12 @@ struct Press { struct Capture { page: String, - /// Where the pressed item places the page, since the same page can be in - /// several. + /// The pressed item, by its index among the scene's items then the top + /// layer's, since the same page can be in several, and its keyframes move + /// it during the drag. + slot: usize, + /// Where the pressed item placed the page, for when the slot no longer + /// shows it. place: Place, /// The last point the page heard, where a release lands that comes from /// no pointer. @@ -49,9 +54,10 @@ struct Capture { held: Vec