From 38705b0361dc33c5e6c1da72ba63112a3e17d988 Mon Sep 17 00:00:00 2001 From: Gunwoo Kim Date: Sun, 8 Nov 2026 21:28:17 +0900 Subject: [PATCH 1/3] docs(h264): design plane-owned row completion Signed-off-by: Gunwoo Kim --- docs/design/h264-row-deblocking.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/design/h264-row-deblocking.md b/docs/design/h264-row-deblocking.md index ccfec8856..24cb6c321 100644 --- a/docs/design/h264-row-deblocking.md +++ b/docs/design/h264-row-deblocking.md @@ -1,15 +1,25 @@ # H.264 ordered row deblocking -The caller owns syntax output and the three reconstruction planes. Decision workers publish immutable, unfiltered macroblock candidates. Once a row is complete, the caller writes its syntax, commits its tiles, and filters that row while workers decide later rows. Prediction reads only published candidate tiles, never the filtered output planes. +The caller owns syntax output. Decision workers publish immutable, unfiltered macroblock candidates. Once a row is complete, the caller writes its syntax while the caller and decision workers can complete its reconstruction planes. Prediction reads only published candidate tiles, never the filtered output planes. Rows are consumed in raster order. Within each plane, each macroblock retains its vertical-then-horizontal edge order. Filtering a row can change the bottom border of the preceding row, but needs no samples from a later row. Cropping and reference publication happen only after every row is filtered and every worker has joined successfully. The scheduler publishes each row outcome before sending a completion notification through a channel bounded by the picture height. Each worker owns its sender, and the caller drops the original sender before receiving. A failed or panicking decision propagates failure to dependent rows before its outcome is published. Every claimed row's compute callback stays inside the unwind boundary. -The caller remembers a consumer error or panic, stops consuming, and joins every worker. The first raster decision error takes precedence over a consumer error, preserving the earlier decide-all-before-writing failure order. A one-worker picture completes decisions before consuming, without channels or threads. With one requested worker, filtering stays after committing the complete picture: no overlap is possible, and retaining whole-picture staging avoids interleaving costs. If no worker can start, the caller completes the decision work before consuming; partial starts retain the workers that did start. +The caller remembers a consumer error or panic, stops consuming, and joins every worker. Plane-completion errors and panics stop further completion attempts but not decisions. The first raster decision error takes precedence over an output error, preserving the earlier decide-all-before-writing failure order. A one-worker picture completes decisions before consuming, without channels or threads. With one requested worker, filtering stays after committing the complete picture: no overlap is possible, and retaining whole-picture staging avoids interleaving costs. If no worker can start, the caller completes the decision work before consuming; partial starts retain the workers that did start. The requested worker count still limits decision workers, with the existing geometry cap. The caller now does useful work concurrently but adds no operating-system thread. It can compete with decision workers on a limited CPU set, so the serial deblocking fraction is not a predicted speedup. Band barriers would simplify completion ownership but add repeated spawning and idle boundaries. Moving mutable strips between workers would complicate border ownership while immutable candidates remain necessary. This design keeps the existing bounded decision table and picture allocations instead. -Verification compares complete streams and reconstructions with the post-ownership baseline at one, two, three and eight workers in automatic and scalar builds. Scheduler tests cover ordered overlap, decision and consumer failures and panics, raster error precedence, and zero or partial thread-start refusal. Row filtering is compared with whole-picture filtering on mixed edges and narrow pictures. Alternating explicit-SHA native reference calls at one and four workers must show an improvement beyond variation before this change ships. +## Plane-owned completion + +For multiple requested workers, each reconstruction plane has one private owner containing its pixels and next uncommitted row. A mutex protects that owner. After publishing a completed decision row, a worker tries each owner without waiting. The caller does the same after writing a syntax row. A successful owner processes its consecutive ready rows, committing candidate tiles and applying the existing filter in raster order. A busy owner is skipped; a worker never waits for output while holding up another decision row. + +The last candidate in a row establishes readiness because the decision loop publishes its candidates in order. Completing a plane can modify its preceding row's border but reads no later row. Different planes have no filtering dependencies, so their owners can run concurrently without changing arithmetic or filter ordering within a plane. After every worker joins, the caller drains any remaining ready work before cropping and reference publication. That final drain also covers a last-row notification racing with an owner's release. + +The scheduler accepts a shared completion callback beside its caller-only syntax consumer. Completion callbacks run inside an unwind boundary; one retained completion failure prevents further attempts, while all decision rows and joins still finish. No callback owns candidate storage or adds a thread. With one requested worker, plain planes retain the existing syntax-and-commit traversal and whole-picture filter, avoiding mutexes and worker assistance. Both paths use one syntax traversal and the same commit and filtering operations. + +Dedicated plane threads would exceed the existing thread budget. Reserving decision workers permanently for planes would reduce search parallelism even when no output row is ready. Nonblocking assistance instead borrows existing worker capacity only at completed-row boundaries. Entropy fragments would require changes to the bit writer and its byte-joining contract, so they are outside this change. + +Verification compares complete streams and reconstructions with the post-ownership baseline at one, two, three and eight workers in automatic and scalar builds. Scheduler tests cover ordered overlap, decision and consumer failures and panics, raster error precedence, and zero or partial thread-start refusal. Row filtering is compared with whole-picture filtering on mixed edges and narrow pictures. Alternating explicit-SHA native reference calls quantify the one-worker cost and must show a four-worker improvement beyond variation before this change ships. From c47d9bb3ab90ef610fe97065475009cc730f9426 Mon Sep 17 00:00:00 2001 From: Gunwoo Kim Date: Sun, 8 Nov 2026 21:37:40 +0900 Subject: [PATCH 2/3] perf(h264): share plane completion work Signed-off-by: Gunwoo Kim --- crates/coreav-h264-encoder/src/codec.rs | 348 +++++++++++++++----- crates/coreav-h264-encoder/src/rows.rs | 3 + crates/coreav-h264-encoder/src/wavefront.rs | 148 ++++++++- docs/design/h264-row-deblocking.md | 2 +- 4 files changed, 422 insertions(+), 79 deletions(-) diff --git a/crates/coreav-h264-encoder/src/codec.rs b/crates/coreav-h264-encoder/src/codec.rs index e97c85ab4..bd4c80b30 100644 --- a/crates/coreav-h264-encoder/src/codec.rs +++ b/crates/coreav-h264-encoder/src/codec.rs @@ -10,6 +10,7 @@ use crate::{ vlc::{INTER_CBP, INTRA_CBP}, }; use coreav_kernels_h264::{BlockMut, IntraAvail, IntraMut, Kernels, Scalar}; +use std::sync::{Mutex, OnceLock, TryLockError}; #[path = "inter.rs"] mod inter; @@ -708,6 +709,62 @@ impl FramePlane { } Ok(()) } + + fn commit_candidate( + &mut self, + candidate: &Candidate, + component: usize, + mx: usize, + my: usize, + ) -> Result<()> { + let (tile, n) = if component == 0 { + (&candidate.luma.tile, 16) + } else { + ( + candidate + .chroma + .tiles + .get(component - 1) + .ok_or(Error::Internal)?, + 8, + ) + }; + self.commit(tile, mx * n, my * n, n) + } + + fn filter_rows( + &mut self, + qp: u8, + component: usize, + rows: std::ops::Range, + mb_width: usize, + candidates: &[OnceLock], + ) -> Result<()> { + deblock::rows( + &mut self.data, + self.width, + if component == 0 { qp } else { t::chroma_qp(qp) }, + component != 0, + rows, + |mx, my| { + let n = &candidates + .get(my * mb_width + mx) + .and_then(OnceLock::get) + .ok_or(Error::Internal)? + .state; + let mut nonzero = 0; + for (p, &count) in n.counts.first().ok_or(Error::Internal)?.iter().enumerate() { + if count != 0 { + nonzero |= 1 << p; + } + } + Ok(deblock::Record { + motion: n.motion, + nonzero, + }) + }, + ) + } } #[derive(Clone, PartialEq, Eq, PartialOrd, Ord)] @@ -1045,26 +1102,26 @@ pub(crate) struct EncodedPicture { pub(crate) previous_mv_count: u8, } -pub(crate) fn picture( - e: &Encoder, - input: Picture<'_>, - slice: syntax::Slice, -) -> Result { - let mut planes = [ - FramePlane::new(e.coded_width, e.coded_height)?, - FramePlane::new(e.coded_width / 2, e.coded_height / 2)?, - FramePlane::new(e.coded_width / 2, e.coded_height / 2)?, - ]; - let mb_width = e.coded_width / 16; - let mb_height = e.coded_height / 16; - let mut writer = Bits::new(e.limits.access_unit_bytes); - syntax::slice(&mut writer, e, slice)?; - let mut pending_skip = 0; - let mut previous_mv_count = e.previous_mv_count; - rows::decide(e, input, slice, |row, candidates| { +struct PictureSyntax { + writer: Bits, + pending_skip: u32, + previous_mv_count: u8, +} + +impl PictureSyntax { + fn row( + &mut self, + row: usize, + candidates: &[OnceLock], + mb_width: usize, + slice: syntax::Slice, + mut planes: Option<&mut [FramePlane; 3]>, + ) -> Result<()> { for address in row * mb_width..(row + 1) * mb_width { - let slot = candidates.get(address).ok_or(Error::Internal)?; - let candidate = slot.get().ok_or(Error::Internal)?; + let candidate = candidates + .get(address) + .and_then(OnceLock::get) + .ok_or(Error::Internal)?; let Candidate { luma, chroma, @@ -1076,70 +1133,137 @@ pub(crate) fn picture( let skip = inter.as_ref().is_some_and(|i| i.skip); if !slice.idr { if skip { - pending_skip += 1; + self.pending_skip += 1; } else { - writer.ue(pending_skip)?; - pending_skip = 0; + self.writer.ue(self.pending_skip)?; + self.pending_skip = 0; } } - macroblock(&mut writer, luma, chroma, inter.as_ref(), context)?; - previous_mv_count = inter.as_ref().map_or(0, |i| i.len as u8); - planes - .first_mut() - .ok_or(Error::Internal)? - .commit(&luma.tile, mx * 16, my * 16, 16)?; - for (c, tile) in chroma.tiles.iter().enumerate() { - planes - .get_mut(c + 1) - .ok_or(Error::Internal)? - .commit(tile, mx * 8, my * 8, 8)?; + macroblock(&mut self.writer, luma, chroma, inter.as_ref(), context)?; + self.previous_mv_count = inter.as_ref().map_or(0, |i| i.len as u8); + if let Some(planes) = planes.as_deref_mut() { + for (component, plane) in planes.iter_mut().enumerate() { + plane.commit_candidate(candidate, component, mx, my)?; + } } } - if e.settings.deblocking == Deblocking::Enabled && (e.threads != 1 || row + 1 == mb_height) - { - let rows = if e.threads == 1 { - 0..mb_height - } else { - row..row + 1 - }; - for (c, plane) in planes.iter_mut().enumerate() { - let qp = if c == 0 { - slice.qp - } else { - t::chroma_qp(slice.qp) - }; - deblock::rows( - &mut plane.data, - plane.width, - qp, - c != 0, - rows.clone(), - |mx, my| { - let n = &candidates - .get(my * mb_width + mx) - .and_then(std::sync::OnceLock::get) - .ok_or(Error::Internal)? - .state; - let mut nonzero = 0; - for (p, &count) in - n.counts.first().ok_or(Error::Internal)?.iter().enumerate() - { - if count != 0 { - nonzero |= 1 << p; - } - } - Ok(deblock::Record { - motion: n.motion, - nonzero, - }) - }, - )?; + Ok(()) + } +} + +struct PlaneRows { + plane: FramePlane, + next_row: usize, +} + +impl PlaneRows { + fn advance( + &mut self, + e: &Encoder, + slice: syntax::Slice, + candidates: &[OnceLock], + component: usize, + ) -> Result<()> { + let width = e.coded_width / 16; + while self.next_row < e.coded_height / 16 { + let row = self.next_row; + if candidates + .get((row + 1) * width - 1) + .and_then(OnceLock::get) + .is_none() + { + break; } + for mx in 0..width { + let candidate = candidates + .get(row * width + mx) + .and_then(OnceLock::get) + .ok_or(Error::Internal)?; + self.plane.commit_candidate(candidate, component, mx, row)?; + } + if e.settings.deblocking == Deblocking::Enabled { + self.plane + .filter_rows(slice.qp, component, row..row + 1, width, candidates)?; + } + self.next_row += 1; } Ok(()) - })?; - if pending_skip != 0 { - writer.ue(pending_skip)?; + } +} + +pub(crate) fn picture( + e: &Encoder, + input: Picture<'_>, + slice: syntax::Slice, +) -> Result { + let planes = [ + FramePlane::new(e.coded_width, e.coded_height)?, + FramePlane::new(e.coded_width / 2, e.coded_height / 2)?, + FramePlane::new(e.coded_width / 2, e.coded_height / 2)?, + ]; + let mb_width = e.coded_width / 16; + let mb_height = e.coded_height / 16; + let mut writer = Bits::new(e.limits.access_unit_bytes); + syntax::slice(&mut writer, e, slice)?; + let mut output = PictureSyntax { + writer, + pending_skip: 0, + previous_mv_count: e.previous_mv_count, + }; + let planes = if e.threads == 1 { + let mut planes = planes; + rows::decide( + e, + input, + slice, + |row, candidates| { + output.row(row, candidates, mb_width, slice, Some(&mut planes))?; + if row + 1 == mb_height && e.settings.deblocking == Deblocking::Enabled { + for (component, plane) in planes.iter_mut().enumerate() { + plane.filter_rows( + slice.qp, + component, + 0..mb_height, + mb_width, + candidates, + )?; + } + } + Ok(()) + }, + |_| Ok(()), + )?; + planes + } else { + let planes = planes.map(|plane| Mutex::new(PlaneRows { plane, next_row: 0 })); + let candidates = rows::decide( + e, + input, + slice, + |row, candidates| output.row(row, candidates, mb_width, slice, None), + |candidates| { + for (component, plane) in planes.iter().enumerate() { + match plane.try_lock() { + Ok(mut plane) => plane.advance(e, slice, candidates, component)?, + Err(TryLockError::WouldBlock) => {} + Err(TryLockError::Poisoned(_)) => return Err(Error::Internal), + } + } + Ok(()) + }, + )?; + let [y, cb, cr] = planes.map(|plane| plane.into_inner().map_err(|_| Error::Internal)); + let mut planes = [y?, cb?, cr?]; + for (component, plane) in planes.iter_mut().enumerate() { + plane.advance(e, slice, &candidates, component)?; + if plane.next_row != mb_height { + return Err(Error::Internal); + } + } + planes.map(|plane| plane.plane) + }; + if output.pending_skip != 0 { + output.writer.ue(output.pending_skip)?; } let visible = e.settings.width * e.settings.height * 3 / 2; let mut reconstruction = zeros(visible)?; @@ -1165,10 +1289,10 @@ pub(crate) fn picture( } } Ok(EncodedPicture { - nal: writer.nal(if slice.idr { 0x65 } else { 0x61 })?, + nal: output.writer.nal(if slice.idr { 0x65 } else { 0x61 })?, reconstruction, planes, - previous_mv_count, + previous_mv_count: output.previous_mv_count, }) } @@ -1181,6 +1305,76 @@ mod tests { pub(super) static INTRA_PAIRS: std::cell::Cell = const { std::cell::Cell::new(0) }; } + #[test] + fn plane_completion_waits_for_a_row_and_does_not_filter_it_twice() { + for deblocking in [Deblocking::Disabled, Deblocking::Enabled] { + let settings = crate::Settings { + width: 64, + height: 48, + frame_rate: Default::default(), + sample_aspect_ratio: Default::default(), + rate_mode: crate::RateMode::FixedQp(28), + deblocking, + gop: crate::Gop::AllIntra, + colour: crate::Colour { + full_range: false, + primaries: 1, + transfer: 1, + matrix: 1, + chroma_location: 0, + }, + }; + let encoder = Encoder::with_threads(settings, crate::Limits::default(), 1).unwrap(); + let y: Vec<_> = (0..64 * 48) + .map(|p| 100 + ((p % 64 / 4 * 2 + p / 64 / 4 * 3) % 16) as u8) + .collect(); + let chroma: Vec<_> = (0..32 * 24) + .map(|p| 110 + ((p % 32 / 4 * 3 + p / 32 / 4 * 2) % 16) as u8) + .collect(); + let input = Picture { + y: Plane::new(&y, 64, 64, 48).unwrap(), + cb: Plane::new(&chroma, 32, 32, 24).unwrap(), + cr: Plane::new(&chroma, 32, 32, 24).unwrap(), + }; + let slice = syntax::Slice { + idr: true, + id: 0, + frame_num: 0, + qp: 28, + }; + let expected = picture(&encoder, input, slice).unwrap(); + let mut candidates = + rows::decide(&encoder, input, slice, |_, _| Ok(()), |_| Ok(())).unwrap(); + let delayed = candidates[7].take().unwrap(); + for (component, reference) in expected.planes.iter().enumerate() { + let mut output = PlaneRows { + plane: FramePlane::new(reference.width, reference.height).unwrap(), + next_row: 0, + }; + output + .advance(&encoder, slice, &candidates, component) + .unwrap(); + assert_eq!(output.next_row, 1); + assert!( + output.plane.data[(reference.height / 3) * reference.width..] + .iter() + .all(|&sample| sample == 0) + ); + assert!(candidates[7].set(delayed.clone()).is_ok()); + output + .advance(&encoder, slice, &candidates, component) + .unwrap(); + assert_eq!(output.next_row, 3); + assert_eq!(output.plane.data, reference.data); + output + .advance(&encoder, slice, &candidates, component) + .unwrap(); + assert_eq!(output.plane.data, reference.data); + assert!(candidates[7].take().is_some()); + } + } + } + #[test] fn complete_cost_uses_quantizer_energy_and_keeps_idr_ordering() { let source = Tile::empty(); diff --git a/crates/coreav-h264-encoder/src/rows.rs b/crates/coreav-h264-encoder/src/rows.rs index b9f65e7a9..b6f5c075a 100644 --- a/crates/coreav-h264-encoder/src/rows.rs +++ b/crates/coreav-h264-encoder/src/rows.rs @@ -123,6 +123,7 @@ pub(super) fn decide( input: Picture<'_>, slice: syntax::Slice, consume: impl FnMut(usize, &[OnceLock]) -> Result<()>, + prepare: impl Fn(&[OnceLock]) -> Result<()> + Sync, ) -> Result>> { let width = e.coded_width / 16; crate::wavefront::fill( @@ -200,6 +201,7 @@ pub(super) fn decide( .map_err(|_| Error::Internal) }, consume, + prepare, ) } @@ -244,6 +246,7 @@ mod tests { qp: 28, }, |_, _| Ok(()), + |_| Ok(()), ) .unwrap(); let address = 5; diff --git a/crates/coreav-h264-encoder/src/wavefront.rs b/crates/coreav-h264-encoder/src/wavefront.rs index a11c5ae45..8bb8fff17 100644 --- a/crates/coreav-h264-encoder/src/wavefront.rs +++ b/crates/coreav-h264-encoder/src/wavefront.rs @@ -9,6 +9,7 @@ pub(crate) fn fill( threads: usize, compute: impl Fn(usize, &[OnceLock]) -> Result<()> + Sync, mut consume: impl FnMut(usize, &[OnceLock]) -> Result<()>, + prepare: impl Fn(&[OnceLock]) -> Result<()> + Sync, ) -> Result>> { if width == 0 || height == 0 || threads == 0 { return Err(Error::Internal); @@ -27,6 +28,7 @@ pub(crate) fn fill( } for row in 0..height { consume(row, &table)?; + prepare(&table)?; } Ok(table) })) @@ -47,6 +49,16 @@ pub(crate) fn fill( outcomes.resize_with(height, OnceLock::new); let next_row = std::sync::atomic::AtomicUsize::new(0); let (complete, completed) = mpsc::sync_channel(height); + let preparation_error = OnceLock::new(); + let prepare_ready = || { + if preparation_error.get().is_none() { + let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| prepare(&table))) + .unwrap_or(Err(Error::Internal)); + if let Err(error) = result { + let _ = preparation_error.set(error); + } + } + }; let work = |complete: mpsc::SyncSender<()>| { loop { let row = next_row.fetch_add(1, std::sync::atomic::Ordering::Relaxed); @@ -97,6 +109,9 @@ pub(crate) fn fill( let _ = outcome.set(result); } let _ = complete.send(()); + if result.is_ok() { + prepare_ready(); + } } }; std::thread::scope(|scope| -> Result<()> { @@ -135,6 +150,9 @@ pub(crate) fn fill( consumed = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| consume(row, &table))) .unwrap_or(Err(Error::Internal)); + if consumed.is_ok() { + prepare_ready(); + } } else if consumed.is_ok() { consumed = Err(Error::Internal); } @@ -149,7 +167,8 @@ pub(crate) fn fill( (*outcome.get().ok_or(Error::Internal)?)?; } joined?; - consumed + consumed?; + preparation_error.get().copied().map_or(Ok(()), Err) })?; Ok(table) } @@ -173,6 +192,7 @@ fn run( .map_err(|_| Error::Internal) }, |_, _| Ok(()), + |_| Ok(()), ) } @@ -186,6 +206,130 @@ mod tests { pub(super) static SPAWN_LIMIT: std::cell::Cell = const { std::cell::Cell::new(usize::MAX) }; } + #[test] + fn preparation_uses_existing_workers_and_the_ordered_caller() { + for threads in [2, 3, 8] { + let caller = std::thread::current().id(); + let compute_threads = Mutex::new(std::collections::HashSet::new()); + let prepare_threads = Mutex::new(std::collections::HashSet::new()); + let first = std::sync::atomic::AtomicBool::new(true); + let (entered, started) = mpsc::sync_channel(1); + let (release, released) = mpsc::sync_channel(1); + let released = Mutex::new(released); + let table = fill( + 12, + 8, + threads, + |address, table| { + compute_threads + .lock() + .unwrap() + .insert(std::thread::current().id()); + table[address].set(address).map_err(|_| Error::Internal) + }, + |row, _| { + if row == 0 { + started + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + release.send(()).unwrap(); + } + Ok(()) + }, + |table| { + let current = std::thread::current().id(); + prepare_threads.lock().unwrap().insert(current); + if current != caller && first.swap(false, Ordering::Relaxed) { + entered.send(()).unwrap(); + released + .lock() + .unwrap() + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + } + for row in 0..8 { + if table[(row + 1) * 12 - 1].get().is_some() { + for (address, slot) in + table.iter().enumerate().take((row + 1) * 12).skip(row * 12) + { + assert_eq!(slot.get(), Some(&address)); + } + } + } + Ok(()) + }, + ) + .unwrap(); + assert!(table.iter().all(|value| value.get().is_some())); + let compute_threads = compute_threads.into_inner().unwrap(); + let prepare_threads = prepare_threads.into_inner().unwrap(); + assert!(compute_threads.len() <= threads); + assert!(!compute_threads.contains(&caller)); + assert!(prepare_threads.contains(&caller)); + assert!(prepare_threads.iter().any(|thread| *thread != caller)); + assert!( + prepare_threads + .iter() + .all(|thread| *thread == caller || compute_threads.contains(thread)) + ); + } + } + + #[test] + fn preparation_failures_finish_decisions_and_keep_raster_error_precedence() { + struct Active<'a>(&'a AtomicUsize); + impl Drop for Active<'_> { + fn drop(&mut self) { + self.0.fetch_sub(1, Ordering::Relaxed); + } + } + for threads in [1, 2, 8] { + for limit in [0, 1, 2, usize::MAX] { + for unwind in [false, true] { + for decision_failure in [false, true] { + let previous = SPAWN_LIMIT.with(|value| value.replace(limit)); + let calls = AtomicUsize::new(0); + let active = AtomicUsize::new(0); + let result = fill( + 12, + 8, + threads, + |address, table| { + calls.fetch_add(1, Ordering::Relaxed); + if decision_failure && address == 95 { + return Err(Error::Coefficient); + } + table[address].set(address).map_err(|_| Error::Internal) + }, + |_, _| Ok(()), + |_| { + active.fetch_add(1, Ordering::Relaxed); + let _active = Active(&active); + if unwind { + panic!("injected preparation unwind"); + } + Err(Error::Syntax) + }, + ); + SPAWN_LIMIT.with(|value| value.set(previous)); + assert_eq!( + result.err(), + Some(if decision_failure { + Error::Coefficient + } else if unwind { + Error::Internal + } else { + Error::Syntax + }) + ); + assert_eq!(calls.load(Ordering::Relaxed), 96); + assert_eq!(active.load(Ordering::Relaxed), 0); + } + } + } + } + } + #[test] fn caller_consumes_ordered_rows_before_later_decisions_finish() { for threads in [2, 3, 8] { @@ -224,6 +368,7 @@ mod tests { } Ok(()) }, + |_| Ok(()), ) .unwrap(); assert_eq!(rows, (0..8).collect::>()); @@ -261,6 +406,7 @@ mod tests { } Err(Error::Syntax) }, + |_| Ok(()), ); SPAWN_LIMIT.with(|value| value.set(previous)); assert_eq!( diff --git a/docs/design/h264-row-deblocking.md b/docs/design/h264-row-deblocking.md index 24cb6c321..6e463c1c9 100644 --- a/docs/design/h264-row-deblocking.md +++ b/docs/design/h264-row-deblocking.md @@ -22,4 +22,4 @@ The scheduler accepts a shared completion callback beside its caller-only syntax Dedicated plane threads would exceed the existing thread budget. Reserving decision workers permanently for planes would reduce search parallelism even when no output row is ready. Nonblocking assistance instead borrows existing worker capacity only at completed-row boundaries. Entropy fragments would require changes to the bit writer and its byte-joining contract, so they are outside this change. -Verification compares complete streams and reconstructions with the post-ownership baseline at one, two, three and eight workers in automatic and scalar builds. Scheduler tests cover ordered overlap, decision and consumer failures and panics, raster error precedence, and zero or partial thread-start refusal. Row filtering is compared with whole-picture filtering on mixed edges and narrow pictures. Alternating explicit-SHA native reference calls quantify the one-worker cost and must show a four-worker improvement beyond variation before this change ships. +Verification compares complete streams and reconstructions with an explicit current-main baseline at one, two, three and eight workers in automatic and scalar builds. Scheduler tests cover ordered overlap, existing-thread assistance, decision and output failures and panics, raster error precedence, and zero or partial thread-start refusal. Plane-completion tests withhold a row's last candidate, drain the remaining work, and require repeated completion to leave pixels unchanged. Row filtering is compared with whole-picture filtering on mixed edges and narrow pictures. Alternating explicit-SHA native reference calls quantify the one-worker cost and must show a four-worker improvement beyond variation before this change ships. From 566842601f8f00608a53359c28d2b544f17aac40 Mon Sep 17 00:00:00 2001 From: Gunwoo Kim Date: Sun, 8 Nov 2026 21:54:22 +0900 Subject: [PATCH 3/3] perf(h264): specialize neighbour tile preparation Signed-off-by: Gunwoo Kim --- crates/coreav-h264-encoder/src/rows.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/crates/coreav-h264-encoder/src/rows.rs b/crates/coreav-h264-encoder/src/rows.rs index b6f5c075a..5bf53e83a 100644 --- a/crates/coreav-h264-encoder/src/rows.rs +++ b/crates/coreav-h264-encoder/src/rows.rs @@ -38,6 +38,7 @@ impl<'a> Neighbours<'a> { }) } + #[inline(always)] fn tiles( &self, source: Plane<'_>,