From 830090e9822b130c3c3691bf156ff755ad4eb606 Mon Sep 17 00:00:00 2001 From: gunwooterry Date: Thu, 5 Nov 2026 08:54:41 +0000 Subject: [PATCH 1/4] docs(ffprobe): plan bounded structured stdin Signed-off-by: gunwooterry --- docs/design/ffprobe-stdin-json.md | 32 +++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 docs/design/ffprobe-stdin-json.md diff --git a/docs/design/ffprobe-stdin-json.md b/docs/design/ffprobe-stdin-json.md new file mode 100644 index 000000000..1890e3fbc --- /dev/null +++ b/docs/design/ffprobe-stdin-json.md @@ -0,0 +1,32 @@ +# Structured ffprobe input from standard input + +## Goal and ordering + +Commit this plan before implementation. Qualify `coreav ffprobe -of json -show_format -show_streams -i pipe:0` without changing local-file probing. The reference is the pinned FFmpeg/ffprobe 9.0.2, used only as black-box executables. Native admission requires complete JSON byte equality and the existing structured command's exit and diagnostic contract; unsupported transport or container facts refuse before native output. Do not remove fields, change comparison predicates or add a tolerance to make a comparison pass. + +This work handles structured stdin only. No-section input summaries, ffmpeg stdin jobs, packet/frame listings, counts, read intervals, arbitrary pipe descriptors, remote protocols and new codecs remain outside this qualification. In particular, this does not widen H.264 summary authority or change oracle selection and qualification. + +## Ownership and alternatives + +Keep option and command-shape admission in `compat_cmd::run_ffprobe` before consuming stdin. Reuse the existing `EncodedInput` owner for exclusive mode-0600 temporary creation, bounded 64 KiB acquisition, the existing 1 GiB input limit, rewind, cleanup and replay lifetime. Continue using the existing native container readers; do not add another spool, stdin reader, parser or fallback-program selector. + +The alternatives are a forward-only source implemented separately for each container, or acquisition into the existing replay owner followed by the existing bounded random-access readers. Choose the replay owner: it keeps one acquisition and cleanup contract and can preserve all consumed bytes for the existing fallback. A seekable temporary backing file does not make the logical input seekable, and its physical size must not become a reported pipe size or an invented bitrate. + +Represent the logical input separately from the backing file. The report's input name remains the original stdin spelling. Derive transport-visible fields from proved pipe behavior and container declarations, not from the temporary path or physical spool length. Qualify each container and layout only after the generated pipe/file contrasts establish which values remain authoritative. Do not fix an unknown field by deleting a whole category of metadata or by borrowing a local-file value. + +## Execution and refusal boundary + +1. Parse and validate the complete supported command shape before stdin acquisition. Unsupported options keep the existing refusal/fallback with stdin untouched. +2. Acquire encoded input once through the existing owner. Exhausted acquisition limits, input read errors and temporary-file failures are terminal without native output, with cleanup; do not attempt fallback with an incomplete replay. +3. Probe the owned backing source under the existing container limits while retaining the logical pipe identity. Before emission, establish complete transport-correct format and stream facts and render the entire supported JSON result into bounded staging. An unanswered fact, unsupported layout or late probe failure cannot leak a JSON prefix. +4. Emit only a completely qualified result. After successful acquisition, a native disqualification uses the existing `refuse_with_input` replay route with the original argv and unchanged fallback selection. Strict-safe mode refuses atomically. Keep the replay owner alive until the child has consumed stdin and remove owned temporary files on every exit path. + +The FFmpeg [pipe protocol documentation](https://ffmpeg.org/ffmpeg-protocols.html#pipe) defines descriptor spellings and distinguishes pipe input from seekable protocols. Container declarations retain their existing specification owners. Binary observations qualify diagnostic representation; they do not override contradictory container bytes or specifications. + +## Verification and boundaries + +First build the independent main baseline and reproduce native refusal with generated media. Compare the same generated bytes passed by stdin and by a local file, using the verified pinned pair. Include ordinary and faststart ISO layouts, a payload larger than the oracle's initial buffering window, fragmented ISO, Matroska with declared and undeclared timing, transport streams, empty/truncated input and unsupported content. This matrix determines the admitted subset; it is not permission to guess support for every container. + +Accepted cases require exact complete JSON, the intended exit status and native strict-safe execution. Refused cases require empty stdout and no partial native output; after successful acquisition, real pinned fallback must receive the original complete bytes and original argv. Check all admitted stdin spellings, malformed commands without stdin reads, cleanup, the byte-cap boundary, read/temp-file failure and ordinary local-file regressions. Keep the tests in a new non-frozen target and reuse the existing platform pin verifier without editing it. + +Run the smallest affected Rust and fixture tests, Clippy, formatting, public-text and PR-shape checks, and the canonical similarity guard. No real-file scan is required to qualify this transport boundary. Heavy checks belong on Actions. Do not execute or amend frozen inputs, registered harnesses, source closures, judging registrations, known-divergence rules, bars or tolerances. Historical command labels and published coverage counts remain unchanged. Any later input-summary diagnostic must use the established custody certificate/key and encrypted-evidence procedure, but no input-summary run is part of this plan. From 84ac212d43cd3747abb397c1165381f2d7bf4fd2 Mon Sep 17 00:00:00 2001 From: gunwooterry Date: Thu, 5 Nov 2026 09:41:24 +0000 Subject: [PATCH 2/4] feat(ffprobe): qualify structured stdin probes Signed-off-by: gunwooterry --- crates/coreav-cli/src/compat_cmd.rs | 90 ++++++-- .../coreav-cli/src/compat_cmd/probe_stdin.rs | 192 ++++++++++++++++++ crates/coreav-cli/src/ffprobe/args.rs | 64 +++++- crates/coreav-cli/src/ffprobe/model.rs | 2 +- crates/coreav-cli/tests/ffprobe_stdin.rs | 39 ++++ docs/agents/ffmpeg-cli.md | 1 + docs/design/ffprobe-stdin-json.md | 8 + scripts/ffprobe_stdin_check.py | 181 +++++++++++++++++ scripts/tests/test_ffprobe_stdin_check.py | 35 ++++ 9 files changed, 589 insertions(+), 23 deletions(-) create mode 100644 crates/coreav-cli/src/compat_cmd/probe_stdin.rs create mode 100644 crates/coreav-cli/tests/ffprobe_stdin.rs create mode 100644 scripts/ffprobe_stdin_check.py create mode 100644 scripts/tests/test_ffprobe_stdin_check.py diff --git a/crates/coreav-cli/src/compat_cmd.rs b/crates/coreav-cli/src/compat_cmd.rs index d1bd31b05..b8a0652bb 100644 --- a/crates/coreav-cli/src/compat_cmd.rs +++ b/crates/coreav-cli/src/compat_cmd.rs @@ -22,6 +22,8 @@ use coreav_isobmff::{BufferedSource, FileSource, Source}; use super::{EXIT_IO, EXIT_REJECTED, EXIT_USAGE, fallback_cmd}; +mod probe_stdin; + /// Exit status of a command the compatibility layer does not implement: /// an option, a value or a property outside its vocabulary. pub const EXIT_UNSUPPORTED: u8 = 4; @@ -112,17 +114,9 @@ pub fn run_ffmpeg(args: &[OsString], strict_safe: StrictSafe) -> ExitCode { match result { Ok(_) => ExitCode::SUCCESS, Err(Stop::Refused(Refusal::NotNative(refusal))) => { - if let Some(mut input) = spool.take() { - let replay = input - .file - .as_mut() - .ok_or_else(|| std::io::Error::other("encoded stdin file is closed")) - .and_then(|file| file.rewind().and_then(|()| file.try_clone())); - #[cfg(unix)] - let replay = - replay.and_then(|file| std::fs::remove_file(&input.path).map(|()| file)); - match replay { - Ok(file) => fallback_cmd::refuse_with_input( + if let Some(input) = spool.take() { + match input.into_replay() { + Ok((file, input)) => fallback_cmd::refuse_with_input( Tool::Ffmpeg, &args, refusal, @@ -178,6 +172,10 @@ pub(super) struct EncodedInput { impl EncodedInput { fn read() -> std::io::Result { + Self::read_from(&mut std::io::stdin().lock(), 1_073_741_824) + } + + fn read_from(reader: &mut dyn Read, limit: u64) -> std::io::Result { let tag = std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) .map_err(std::io::Error::other)? @@ -214,11 +212,10 @@ impl EncodedInput { .file .as_mut() .ok_or_else(|| std::io::Error::other("encoded stdin file is closed"))?; - let mut stdin = std::io::stdin().lock(); let mut bytes = [0u8; 65_536]; let mut size = 0u64; loop { - let count = match stdin.read(&mut bytes) { + let count = match reader.read(&mut bytes) { Err(error) if error.kind() == std::io::ErrorKind::Interrupted => continue, result => result?, }; @@ -226,7 +223,7 @@ impl EncodedInput { break; } size += count as u64; - if size > 1_073_741_824 { + if size > limit { return Err(std::io::Error::other( "encoded stdin exceeds the 1 GiB limit", )); @@ -240,6 +237,18 @@ impl EncodedInput { file.rewind()?; Ok(input) } + + fn into_replay(mut self) -> std::io::Result<(std::fs::File, Self)> { + let file = self + .file + .as_mut() + .ok_or_else(|| std::io::Error::other("encoded stdin file is closed"))?; + file.rewind()?; + let replay = file.try_clone()?; + #[cfg(unix)] + std::fs::remove_file(&self.path)?; + Ok((replay, self)) + } } impl Drop for EncodedInput { @@ -314,6 +323,9 @@ pub fn run_ffprobe(args: &[OsString], strict_safe: StrictSafe) -> ExitCode { return ExitCode::from(EXIT_USAGE); } }; + if ffmpeg::args::stdin(&options.input) { + return probe_stdin::run(&args, &options, strict_safe); + } #[cfg(not(feature = "net"))] if options.input.starts_with("http://") || options.input.starts_with("https://") { message( @@ -603,3 +615,53 @@ fn run( Err(other) => Err(Outcome::Read(other)), } } + +#[cfg(test)] +mod encoded_input_tests { + #![allow(clippy::unwrap_used)] + + use super::*; + + fn owned_names() -> std::collections::BTreeSet { + let prefix = format!("coreav-encoded-stdin-{}-", std::process::id()); + std::fs::read_dir(std::env::temp_dir()) + .unwrap() + .filter_map(Result::ok) + .filter(|entry| entry.file_name().to_string_lossy().starts_with(&prefix)) + .map(|entry| entry.path()) + .collect() + } + + #[test] + fn acquisition_limit_read_failure_replay_and_cleanup_share_one_owner() { + let before = owned_names(); + let input = EncodedInput::read_from(&mut std::io::Cursor::new(b"12345678"), 8).unwrap(); + let path = input.path.clone(); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + assert_eq!( + std::fs::metadata(&path).unwrap().permissions().mode() & 0o777, + 0o600 + ); + } + let (mut replay, owner) = input.into_replay().unwrap(); + #[cfg(unix)] + assert!(!path.exists()); + let mut bytes = Vec::new(); + replay.read_to_end(&mut bytes).unwrap(); + assert_eq!(bytes, b"12345678"); + drop(replay); + drop(owner); + assert!(!path.exists()); + assert!(EncodedInput::read_from(&mut std::io::Cursor::new(b"123456789"), 8).is_err()); + struct Failed; + impl Read for Failed { + fn read(&mut self, _bytes: &mut [u8]) -> std::io::Result { + Err(std::io::Error::other("read failed")) + } + } + assert!(EncodedInput::read_from(&mut Failed, 8).is_err()); + assert_eq!(owned_names(), before); + } +} diff --git a/crates/coreav-cli/src/compat_cmd/probe_stdin.rs b/crates/coreav-cli/src/compat_cmd/probe_stdin.rs new file mode 100644 index 000000000..eff1923ef --- /dev/null +++ b/crates/coreav-cli/src/compat_cmd/probe_stdin.rs @@ -0,0 +1,192 @@ +use std::io::Write; +use std::process::ExitCode; + +use coreav_cli::error::Failure; +use coreav_cli::fallback::{NotNative, Tool}; +use coreav_cli::ffprobe::{self, args::Options}; +use coreav_cli::strict_safe::StrictSafe; +use coreav_isobmff::{BufferedSource, FileSource, Limits, Parsed, Source, parse_box_header}; + +use super::{EXIT_IO, EncodedInput, fallback_cmd, message}; + +pub(super) fn run(args: &[String], options: &Options, strict_safe: StrictSafe) -> ExitCode { + let input = match EncodedInput::read() { + Ok(input) => input, + Err(error) => { + message( + "ffprobe", + &format!("could not acquire encoded stdin: {error}"), + ); + return ExitCode::from(EXIT_IO); + } + }; + match render(&input, options) { + Ok(bytes) => { + let mut out = std::io::stdout().lock(); + match out.write_all(&bytes).and_then(|()| out.flush()) { + Ok(()) => ExitCode::SUCCESS, + Err(error) => super::super::write_error(&error), + } + } + Err(_) => { + let refusal = NotNative::property( + "standard input facts", + "structured stdin requires a qualified front-loaded ISO movie", + ); + match input.into_replay() { + Ok((file, input)) => fallback_cmd::refuse_with_input( + Tool::Ffprobe, + args, + refusal, + strict_safe, + file, + input, + ), + Err(error) => { + message( + "ffprobe", + &format!("could not replay encoded stdin: {error}"), + ); + ExitCode::from(EXIT_IO) + } + } + } + } +} + +fn render(input: &EncodedInput, options: &Options) -> Result, Failure> { + let mut source = FileSource::open(&input.path).map_err(Failure::library)?; + let size = source.len().map_err(Failure::library)?; + let limits = Limits::default(); + front_loaded(&mut source, size, &limits)?; + let mut source = BufferedSource::new(source); + let mut probed = + ffprobe::probe_limited(&mut source, size, &options.input, options.force, &limits)?; + probed + .report + .format + .retain(|(key, _)| !matches!(*key, "size" | "bit_rate")); + let mut output = Staged { + bytes: Vec::new(), + limit: usize::try_from(limits.max_retained_bytes).map_err(|_| unsupported())?, + }; + ffprobe::emit(options, &mut probed, &mut output)?; + Ok(output.bytes) +} + +fn unsupported() -> Failure { + Failure::Unsupported("unqualified stdin layout or exhausted staging limit".to_owned()) +} + +fn front_loaded(source: &mut S, size: u64, limits: &Limits) -> Result<(), Failure> { + let mut at = 0u64; + let mut boxes = 0u32; + let (mut movie, mut media) = (false, false); + while at < size { + boxes += 1; + if boxes > limits.max_top_level_boxes { + return Err(unsupported()); + } + let room = size.checked_sub(at).ok_or_else(unsupported)?; + let mut bytes = [0u8; coreav_isobmff::MAX_HEADER_LEN]; + let count = usize::try_from(room.min(bytes.len() as u64)).map_err(|_| unsupported())?; + let header = bytes.get_mut(..count).ok_or_else(unsupported)?; + source.read_exact_at(at, header).map_err(Failure::library)?; + let Parsed::Done(header) = parse_box_header(header, at, room).map_err(Failure::library)? + else { + return Err(unsupported()); + }; + match &header.kind.0 { + b"moov" if !movie && !media => movie = true, + b"mdat" if movie => media = true, + b"ftyp" | b"free" | b"skip" | b"wide" => {} + _ => return Err(unsupported()), + } + at = header.end(); + } + if movie && media { + Ok(()) + } else { + Err(unsupported()) + } +} + +struct Staged { + bytes: Vec, + limit: usize, +} + +impl Write for Staged { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + let end = self + .bytes + .len() + .checked_add(bytes.len()) + .filter(|end| *end <= self.limit) + .ok_or_else(|| std::io::Error::other("structured stdin staging limit"))?; + self.bytes + .try_reserve_exact(end - self.bytes.len()) + .map_err(std::io::Error::other)?; + self.bytes.extend_from_slice(bytes); + Ok(bytes.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + +#[cfg(test)] +mod tests { + #![allow(clippy::unwrap_used, clippy::indexing_slicing)] + + use super::*; + use coreav_isobmff::SliceSource; + + fn atom(kind: &[u8; 4]) -> Vec { + [8u32.to_be_bytes().as_slice(), kind].concat() + } + + #[test] + fn only_complete_front_loaded_nonfragmented_iso_layouts_are_qualified() { + let valid = [atom(b"ftyp"), atom(b"moov"), atom(b"mdat")].concat(); + let check = |bytes: &[u8], limits: Limits| { + front_loaded(&mut SliceSource::new(bytes), bytes.len() as u64, &limits) + }; + assert!(check(&valid, Limits::default()).is_ok()); + for bytes in [ + [atom(b"mdat"), atom(b"moov")].concat(), + [atom(b"moov"), atom(b"moof"), atom(b"mdat")].concat(), + [atom(b"moov"), atom(b"mdat"), atom(b"moov")].concat(), + [atom(b"moov"), atom(b"mdat"), atom(b"uuid")].concat(), + atom(b"moov"), + atom(b"mdat"), + Vec::new(), + valid[..valid.len() - 1].to_vec(), + ] { + assert!(check(&bytes, Limits::default()).is_err()); + } + assert!( + check( + &valid, + Limits { + max_top_level_boxes: 2, + ..Limits::default() + } + ) + .is_err() + ); + } + + #[test] + fn staging_limit_fails_without_committing_a_partial_chunk() { + let mut output = Staged { + bytes: Vec::new(), + limit: 8, + }; + output.write_all(b"12345678").unwrap(); + assert!(output.write_all(b"9").is_err()); + assert_eq!(output.bytes, b"12345678"); + assert!(output.flush().is_ok()); + } +} diff --git a/crates/coreav-cli/src/ffprobe/args.rs b/crates/coreav-cli/src/ffprobe/args.rs index 281e29b73..642f64930 100644 --- a/crates/coreav-cli/src/ffprobe/args.rs +++ b/crates/coreav-cli/src/ffprobe/args.rs @@ -368,7 +368,36 @@ pub fn parse(args: &[String]) -> Result { "You have to specify one input file.".to_owned(), )); }; - if !input.starts_with("file:") + selection.finish(); + let stdin = crate::ffmpeg::args::stdin(&input); + if stdin { + let mut full = Selection::default(); + full.select_all(Sid::Format); + full.select_all(Sid::Stream); + full.select_bare(Sid::ProgramStreamDisposition); + full.select_bare(Sid::ProgramStreamTags); + full.finish(); + if input == "-" + || !matches!(writer, Kind::Json { .. }) + || selection != full + || select_streams.is_some() + || count_packets + || count_frames + || show_error + || !read_intervals.is_empty() + || style.units + || style.sexagesimal + || force.is_some_and(|kind| kind != Container::Isobmff) + || summary_level(&diagnostic_level).is_err() + { + return Err(Refusal::Unsupported(NotNative::property( + "standard input command", + "pipe: and pipe:0 require JSON format and stream sections without inspection options", + ))); + } + } + if !stdin + && !input.starts_with("file:") && !input.starts_with("http://") && !input.starts_with("https://") && input.contains(':') @@ -384,13 +413,6 @@ pub fn parse(args: &[String]) -> Result { } } } - if input == "-" { - return Err(Refusal::Unsupported(NotNative::property( - "standard input", - "input '-' is not supported: coreav ffprobe reads files and HTTP(S) URLs", - ))); - } - selection.finish(); Ok(Options { input, writer, @@ -905,4 +927,30 @@ mod tests { Err(Refusal::Unsupported(_)) )); } + #[test] + fn stdin_admits_only_complete_json_probe_shapes_before_acquisition() { + for name in ["pipe:", "pipe:0"] { + assert!( + parse(&args(&format!( + "-of json -show_format -show_streams -i {name}" + ))) + .is_ok() + ); + } + for text in [ + "-i pipe:0", + "-of json -show_streams -i pipe:0", + "-of json -show_format -show_streams -show_packets -i pipe:0", + "-of json -show_format -show_streams -count_frames -i pipe:0", + "-of json -show_format -show_streams -select_streams v -i pipe:0", + "-of json -show_format -show_streams -f matroska -i pipe:0", + "-of json -show_format -show_streams -i -", + "-of json -show_format -show_streams -i pipe:1", + ] { + assert!( + matches!(parse(&args(text)), Err(Refusal::Unsupported(_))), + "{text}" + ); + } + } } diff --git a/crates/coreav-cli/src/ffprobe/model.rs b/crates/coreav-cli/src/ffprobe/model.rs index e6cb18a9f..24d2b5e5c 100644 --- a/crates/coreav-cli/src/ffprobe/model.rs +++ b/crates/coreav-cli/src/ffprobe/model.rs @@ -345,7 +345,7 @@ pub fn time(ts: i64, num: i64, den: i64) -> Value { } /// What `-show_entries` and the `-show_*` options select. -#[derive(Clone, Debug, Default)] +#[derive(Clone, Debug, Default, PartialEq, Eq)] pub struct Selection { /// Every entry and every section inside, per section. all: [bool; SECTIONS], diff --git a/crates/coreav-cli/tests/ffprobe_stdin.rs b/crates/coreav-cli/tests/ffprobe_stdin.rs new file mode 100644 index 000000000..8ce30044a --- /dev/null +++ b/crates/coreav-cli/tests/ffprobe_stdin.rs @@ -0,0 +1,39 @@ +//! Structured stdin probing and complete pinned replay on generated inputs. +#![cfg(all(target_os = "linux", target_arch = "x86_64"))] +#![allow(clippy::unwrap_used, clippy::panic)] + +use std::path::{Path, PathBuf}; +use std::process::Command; + +#[test] +fn structured_stdin_and_complete_pinned_replay() { + let ffmpeg = std::env::var_os("COREAV_FFMPEG") + .map(PathBuf::from) + .unwrap_or_else(|| PathBuf::from("/opt/coreav-oracle-bin/ffmpeg")); + assert_eq!(ffmpeg.file_name(), Some(std::ffi::OsStr::new("ffmpeg"))); + if let Some(ffprobe) = std::env::var_os("COREAV_FFPROBE") { + assert_eq!( + PathBuf::from(ffprobe).canonicalize().unwrap(), + ffmpeg + .parent() + .unwrap() + .join("ffprobe") + .canonicalize() + .unwrap() + ); + } + let result = Command::new("python3") + .arg("-I") + .arg(Path::new(env!("CARGO_MANIFEST_DIR")).join("../../scripts/ffprobe_stdin_check.py")) + .args(["--coreav", env!("CARGO_BIN_EXE_coreav"), "--oracle-dir"]) + .arg(ffmpeg.parent().unwrap()) + .arg("--directory") + .arg(env!("CARGO_TARGET_TMPDIR")) + .output() + .unwrap(); + assert!( + result.status.success(), + "{}", + String::from_utf8_lossy(&result.stderr) + ); +} diff --git a/docs/agents/ffmpeg-cli.md b/docs/agents/ffmpeg-cli.md index aff370964..67d420c1c 100644 --- a/docs/agents/ffmpeg-cli.md +++ b/docs/agents/ffmpeg-cli.md @@ -70,6 +70,7 @@ CLI and ffprobe behavior, the FFmpeg pin, and fallback names. | A conformance-only PCM decoder preserves its complete presented samples without claiming oracle byte identity, while disabled presentations still refuse and replay stdin | The checked-in-record regression admits only qualified MPEG-1 MP3 in MP4. `raw_audio_pipe_check.py` compares native file and pipe packing with `decode-audio`, checks the finite edit count, and tests AAC, Opus, Matroska MP3 and MPEG-2 refusals with original-byte fallback replay. Its Linux rate check uses identical native decoded floats as oracle input, so decoder differences cannot hide a conversion error | | Encoded stdin fallback receives the original bytes and arguments, while decoded audio stays bounded and poststart errors never fall back | `EncodedInput` owns the capped private spool and replay; raw stdin uses the bounded stream engine directly. `encoded_stdin_is_replayed_for_native_raw_output`, `consumed_encoded_stdin_reaches_fallback_with_original_arguments`, `raw_stdin_writes_before_eof_without_a_whole_input_spool` and `a_partial_raw_frame_after_output_never_invokes_fallback` exercise native output, replay, cleanup and the irreversible stdout boundary | | Replayed Unix fallback uses the existing process-replacement owner and removes its named spool before handoff | `fallback_cmd::run` owns Unix `exec` for inherited and replayed stdin. `encoded_stdin_fallback_replaces_the_process_without_leaving_its_spool` requires the same PID, complete input and no temporary file. A separate spawn-and-wait path unnecessarily links another launcher into the binary; types cannot prove process replacement or filesystem cleanup | +| Structured stdin keeps logical pipe facts separate from its replay file and never emits a partial native JSON result before refusal | Shape admission precedes acquisition in `args::parse`; `probe_stdin` reuses the bounded owner, requires front-loaded nonfragmented ISO and stages complete JSON before emission. `ffprobe_stdin` checks exact generated pipe output, unsupported live-pipe commands, complete pinned replay, read/temp-file failures and cleanup; layout, acquisition-cap and output-cap units cover bounded failures. A file handle does not encode the original pipe's seekability or size | | Raw audio stdout checks final flush and preserves quiet EPIPE exit 141 | The shared media writer and explicit terminal-error mapping are covered by `tiny_pcm_stdout_reports_a_closed_reader_during_final_flush`; a nonempty output is not proof that its buffered tail reached the reader | | Every headerless raw-audio rate conversion needs its platform registration, including pre-existing raw S16 file commands | `raw_audio::write_raw` owns file and pipe admission. `short_raw_inputs_keep_the_compatibility_startup_and_drain_lengths` retains exact Linux bytes and lengths and requires refusal without output elsewhere; `raw_audio_pipe_check.py` compares unregistered-platform file and stdin fallback with the pinned oracle. Historical native behavior does not establish correctness | | Input thread syntax outside the native grammar reaches fallback, and decoder capacity does not restrict copy jobs | `threads` owns shared syntax classification and `Command::decoder_threads` owns the capacity limit only where decoder options consume it. Parser regressions, `input_thread_expressions_reach_fallback_without_consuming_stdin` and `copy_jobs_ignore_decoder_capacity_but_audio_decode_refuses` cover original-byte fallback and native copy above the decoder cap. A parsed count alone cannot identify which job will use it | diff --git a/docs/design/ffprobe-stdin-json.md b/docs/design/ffprobe-stdin-json.md index 1890e3fbc..433c7f92d 100644 --- a/docs/design/ffprobe-stdin-json.md +++ b/docs/design/ffprobe-stdin-json.md @@ -30,3 +30,11 @@ First build the independent main baseline and reproduce native refusal with gene Accepted cases require exact complete JSON, the intended exit status and native strict-safe execution. Refused cases require empty stdout and no partial native output; after successful acquisition, real pinned fallback must receive the original complete bytes and original argv. Check all admitted stdin spellings, malformed commands without stdin reads, cleanup, the byte-cap boundary, read/temp-file failure and ordinary local-file regressions. Keep the tests in a new non-frozen target and reuse the existing platform pin verifier without editing it. Run the smallest affected Rust and fixture tests, Clippy, formatting, public-text and PR-shape checks, and the canonical similarity guard. No real-file scan is required to qualify this transport boundary. Heavy checks belong on Actions. Do not execute or amend frozen inputs, registered harnesses, source closures, judging registrations, known-divergence rules, bars or tolerances. Historical command labels and published coverage counts remain unchanged. Any later input-summary diagnostic must use the established custody certificate/key and encrypted-evidence procedure, but no input-summary run is part of this plan. + +## Implemented qualification + +Native admission is limited to `pipe:` and `pipe:0`, complete JSON format/stream selections, and nonfragmented ISO files whose sole movie box precedes every media box. The existing box-header parser establishes complete top-level extents under its visit cap. Tail movie metadata, fragments, other containers, detailed diagnostics and inspection options remain unsupported. This deliberately avoids assuming an oracle seek-back buffer size or borrowing file-derived transport timing. + +The bytes are acquired through the existing owner and limits. The logical name is passed to probing independently of the backing path. ISO format size and the total-file-derived format bitrate are unknown on the logical pipe and are omitted; declared stream rates and timings remain unchanged. The complete JSON is staged with fallible allocation under the existing 16 MiB retained-byte default before stdout is touched. Probe, layout or staging failure discards that result and replays the complete owned input to the unchanged fallback; acquisition failure is terminal and cleans its partial spool. + +The `-` alias stays unsupported before acquisition. The pinned executable canonicalizes it to `fd:`, whose seekability can depend on the underlying descriptor; treating it as `pipe:0` would misreport both its identity and potentially its transport facts. No-section summaries remain on their original path. Existing encoded-audio stdin uses the same factored rewind/unlink/replay owner, including Unix process replacement and Windows lifetime cleanup. diff --git a/scripts/ffprobe_stdin_check.py b/scripts/ffprobe_stdin_check.py new file mode 100644 index 000000000..b5b90a0b0 --- /dev/null +++ b/scripts/ffprobe_stdin_check.py @@ -0,0 +1,181 @@ +#!/usr/bin/env python3 +"""Check complete structured stdin output and replay on generated inputs.""" + +import argparse +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +from input_summary_check import Oracle, CheckFailure as OracleFailure + + +class CheckFailure(Exception): + pass + + +def run(argv, data=None, *, env=None): + try: + return subprocess.run(list(map(str, argv)), input=data, capture_output=True, + timeout=30, env=env) + except subprocess.TimeoutExpired: + raise CheckFailure('child-timeout') from None + except OSError: + raise CheckFailure('child-spawn') from None + + +def check(coreav, oracle, root): + env = {**os.environ, 'COREAV_STRICT_SAFE': '1', 'TMPDIR': str(root)} + fallback_env = {**env, 'COREAV_FALLBACK': str(oracle.directory)} + fallback_env.pop('COREAV_STRICT_SAFE', None) + args = ['-v', 'error', '-of', 'json', '-show_format', '-show_streams'] + video = ['-f', 'lavfi', '-i', 'testsrc2=s=128x96:r=10:d=0.4'] + sound = ['-f', 'lavfi', '-i', 'sine=frequency=709:sample_rate=48000:duration=0.4'] + codecs = ['-c:v', 'libx264', '-threads:v', '1', '-pix_fmt', 'yuv420p', '-c:a', 'aac'] + cases = [] + for name, suffix, options, native in ( + ('faststart', 'mp4', [*video, *sound, *codecs, '-movflags', '+faststart'], True), + ('faststart-large', 'mp4', ['-f', 'lavfi', '-i', 'testsrc2=s=320x240:r=30:d=1', + '-c:v', 'libx264', '-threads:v', '1', '-crf', '8', + '-movflags', '+faststart'], True), + ('tail-moov', 'mp4', [*video, *sound, *codecs], False), + ('fragmented', 'mp4', [*video, *sound, *codecs, + '-movflags', 'frag_keyframe+empty_moov'], False), + ('matroska', 'mkv', [*video, *sound, *codecs], False), + ('transport', 'ts', [*video, *sound, *codecs], False), + ): + path = root / (name + '.' + suffix) + made = run([oracle.ffmpeg, '-nostdin', '-v', 'error', '-y', *options, path]) + if made.returncode or not path.is_file(): + raise CheckFailure('fixture-generation') + if name == 'faststart-large' and path.stat().st_size <= 65536: + raise CheckFailure('fixture-buffer-contrast') + cases.append((name, path.read_bytes(), native)) + malformed = bytearray(cases[0][1]) + child = malformed.index(b'moov') + 4 + malformed[child:child + 4] = b'\xff\xff\xff\xff' + cases.extend([('empty', b'', False), ('truncated', cases[0][1][:64], False), + ('malformed-movie', bytes(malformed), False)]) + for name, data, native in cases: + for spelling in ('pipe:0', 'pipe:'): + command = [*args, '-i', spelling] + reference = run([oracle.ffprobe, '-hide_banner', *command], data) + result = run([coreav, 'ffprobe', *command], data, env=env) + if native: + if reference.returncode or reference.stderr or result.returncode or result.stderr \ + or not result.stdout or result.stdout != reference.stdout: + raise CheckFailure('native-' + name) + parsed = json.loads(result.stdout) + if parsed['format']['filename'] != spelling or \ + any(key in parsed['format'] for key in ('size', 'bit_rate')): + raise CheckFailure('logical-pipe-facts') + else: + if result.returncode != 4 or result.stdout or b'Input #' in result.stderr: + raise CheckFailure('refusal-' + name) + fallback = run([coreav, 'ffprobe', *command], data, env=fallback_env) + notice, separator, diagnostics = fallback.stderr.partition(b'\n') + if fallback.returncode != reference.returncode or fallback.stdout != reference.stdout \ + or not separator or not notice.startswith(b'coreav: ran ') \ + or diagnostics != reference.stderr: + raise CheckFailure('replay-' + name) + if list(root.glob('coreav-encoded-stdin-*')): + raise CheckFailure('input-cleanup') + if native: + compact = ['-v', 'error', '-of', 'json=compact=1', '-show_format', '-show_streams', + '-i', 'pipe:0'] + reference = run([oracle.ffprobe, '-hide_banner', *compact], data) + result = run([coreav, 'ffprobe', *compact], data, env=env) + if reference.returncode or reference.stderr or result.returncode or result.stderr \ + or result.stdout != reference.stdout: + raise CheckFailure('native-' + name) + default = [*args[2:], '-i', 'pipe:0'] + reference = run([oracle.ffprobe, '-hide_banner', *default], data) + result = run([coreav, 'ffprobe', *default], data, env=env) + if reference.returncode or result.returncode or result.stderr \ + or result.stdout != reference.stdout: + raise CheckFailure('native-' + name) + for options in ( + ['-i', 'pipe:0'], + [*args, '-show_packets', '-i', 'pipe:0'], + [*args, '-count_frames', '-i', 'pipe:0'], + [*args, '-read_intervals', '%+#1', '-i', 'pipe:0'], + ['-show_streams', '-i', 'pipe:0'], + [*args, '-i', 'pipe:1'], + [*args, '-i', '-'], + ['-v', 'debug', *args[2:], '-i', 'pipe:0'], + ): + child = subprocess.Popen([str(coreav), 'ffprobe', *options], stdin=subprocess.PIPE, + stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env) + try: + try: + child.wait(timeout=5) + except subprocess.TimeoutExpired: + raise CheckFailure('shape-before-stdin') from None + output, diagnostics = child.communicate() + finally: + if child.poll() is None: + child.kill() + child.communicate() + if child.returncode != 4 or output or b'Input #' in diagnostics: + raise CheckFailure('shape-before-stdin') + bad_temp = root / 'not-a-directory' + bad_temp.write_bytes(b'preserve') + failed = run([coreav, 'ffprobe', *args, '-i', 'pipe:0'], cases[0][1], + env={**env, 'TMPDIR': str(bad_temp)}) + if failed.returncode == 0 or failed.stdout or bad_temp.read_bytes() != b'preserve': + raise CheckFailure('temporary-file-failure') + with (root / 'write-only').open('wb') as source: + failed = subprocess.run([str(coreav), 'ffprobe', *args, '-i', 'pipe:0'], stdin=source, + capture_output=True, timeout=30, env=env) + if failed.returncode == 0 or failed.stdout or list(root.glob('coreav-encoded-stdin-*')): + raise CheckFailure('input-read-failure') + minus = [*args, '-i', '-'] + reference = run([oracle.ffprobe, '-hide_banner', *minus], cases[0][1]) + fallback = run([coreav, 'ffprobe', *minus], cases[0][1], env=fallback_env) + if fallback.returncode != reference.returncode or fallback.stdout != reference.stdout \ + or fallback.stderr.partition(b'\n')[2] != reference.stderr: + raise CheckFailure('fd-alias-fallback') + local = root / 'faststart.mp4' + reference = run([oracle.ffprobe, '-hide_banner', *args, '-i', local]) + result = run([coreav, 'ffprobe', *args, '-i', local], env=env) + if reference.returncode or reference.stderr or result.returncode or result.stderr \ + or result.stdout != reference.stdout: + raise CheckFailure('local-file-regression') + print('PASS generated structured stdin and complete replay') + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--coreav', type=Path, required=True) + parser.add_argument('--oracle-dir', type=Path, required=True) + parser.add_argument('--directory', type=Path, required=True) + args = parser.parse_args() + try: + oracle = Oracle(args.oracle_dir) + args.directory.mkdir(parents=True, exist_ok=True) + with tempfile.TemporaryDirectory(prefix='ffprobe-stdin-', dir=args.directory) as owned: + check(args.coreav.resolve(), oracle, Path(owned)) + except CheckFailure as error: + known = {'fixture-generation', 'fixture-buffer-contrast', 'logical-pipe-facts', + 'input-cleanup', 'shape-before-stdin', 'temporary-file-failure', 'local-file-regression', + 'input-read-failure', 'fd-alias-fallback', 'child-timeout', 'child-spawn'} + known.update(category + '-' + name for category in ('native', 'refusal', 'replay') + for name in ('faststart', 'faststart-large', 'tail-moov', 'fragmented', + 'matroska', 'transport', 'empty', 'truncated', 'malformed-movie')) + label = str(error) + print('FAIL ' + (label if label in known else 'structured-stdin-check'), file=sys.stderr) + return 1 + except OracleFailure: + print('FAIL oracle-qualification', file=sys.stderr) + return 1 + except (OSError, ValueError, subprocess.SubprocessError): + print('FAIL structured-stdin-check', file=sys.stderr) + return 1 + return 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/scripts/tests/test_ffprobe_stdin_check.py b/scripts/tests/test_ffprobe_stdin_check.py new file mode 100644 index 000000000..da5edd2c9 --- /dev/null +++ b/scripts/tests/test_ffprobe_stdin_check.py @@ -0,0 +1,35 @@ +import contextlib +import io +from pathlib import Path +import sys +import tempfile +import unittest +from unittest.mock import patch + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +import ffprobe_stdin_check as check + + +class FailureDiagnostics(unittest.TestCase): + def test_oracle_failure_is_closed_and_precedes_any_fixture_work(self): + with tempfile.TemporaryDirectory() as root, \ + patch.object(sys, 'argv', ['check', '--coreav', 'native', '--oracle-dir', 'oracle', + '--directory', root]), \ + patch.object(check, 'Oracle', side_effect=check.OracleFailure('private-secret')), \ + patch.object(check, 'check') as work, contextlib.redirect_stderr(io.StringIO()) as error: + self.assertEqual(check.main(), 1) + work.assert_not_called() + self.assertEqual(error.getvalue(), 'FAIL oracle-qualification\n') + + def test_unknown_failure_details_never_enter_output(self): + with tempfile.TemporaryDirectory() as root, \ + patch.object(sys, 'argv', ['check', '--coreav', 'native', '--oracle-dir', 'oracle', + '--directory', root]), patch.object(check, 'Oracle'), \ + patch.object(check, 'check', side_effect=check.CheckFailure('private-secret')), \ + contextlib.redirect_stderr(io.StringIO()) as error: + self.assertEqual(check.main(), 1) + self.assertEqual(error.getvalue(), 'FAIL structured-stdin-check\n') + + +if __name__ == '__main__': + unittest.main() From adeb0b7e6c642d41a90e2f41183f6c64e5c5267c Mon Sep 17 00:00:00 2001 From: gunwooterry Date: Thu, 5 Nov 2026 09:57:51 +0000 Subject: [PATCH 3/4] fix(ffprobe): relay live stdin after bounded inspection Signed-off-by: gunwooterry --- .../coreav-cli/src/compat_cmd/probe_stdin.rs | 177 +++++++++++++++--- crates/coreav-cli/src/fallback_cmd.rs | 102 ++++++++-- docs/agents/ffmpeg-cli.md | 5 +- docs/design/ffprobe-stdin-json.md | 6 +- scripts/ffprobe_stdin_check.py | 58 +++++- 5 files changed, 297 insertions(+), 51 deletions(-) diff --git a/crates/coreav-cli/src/compat_cmd/probe_stdin.rs b/crates/coreav-cli/src/compat_cmd/probe_stdin.rs index eff1923ef..f7d314cf3 100644 --- a/crates/coreav-cli/src/compat_cmd/probe_stdin.rs +++ b/crates/coreav-cli/src/compat_cmd/probe_stdin.rs @@ -1,16 +1,36 @@ -use std::io::Write; +use std::io::{Read, Write}; use std::process::ExitCode; use coreav_cli::error::Failure; use coreav_cli::fallback::{NotNative, Tool}; use coreav_cli::ffprobe::{self, args::Options}; use coreav_cli::strict_safe::StrictSafe; -use coreav_isobmff::{BufferedSource, FileSource, Limits, Parsed, Source, parse_box_header}; +use coreav_isobmff::{ + BufferedSource, FileSource, Limits, Parsed, SliceSource, Source, parse_box_header, +}; use super::{EXIT_IO, EncodedInput, fallback_cmd, message}; +const PREFIX_LIMIT: usize = 1_048_576; + pub(super) fn run(args: &[String], options: &Options, strict_safe: StrictSafe) -> ExitCode { - let input = match EncodedInput::read() { + let acquired = prefix(&mut std::io::stdin().lock()); + let input = match acquired { + Ok(Acquired::Complete(bytes)) => { + EncodedInput::read_from(&mut std::io::Cursor::new(bytes), 1_073_741_824) + } + Ok(Acquired::Live(bytes)) => { + return fallback_cmd::refuse_with_prefix( + Tool::Ffprobe, + args, + refusal(), + strict_safe, + bytes, + ); + } + Err(error) => Err(error), + }; + let input = match input { Ok(input) => input, Err(error) => { message( @@ -28,28 +48,70 @@ pub(super) fn run(args: &[String], options: &Options, strict_safe: StrictSafe) - Err(error) => super::super::write_error(&error), } } - Err(_) => { - let refusal = NotNative::property( - "standard input facts", - "structured stdin requires a qualified front-loaded ISO movie", - ); - match input.into_replay() { - Ok((file, input)) => fallback_cmd::refuse_with_input( - Tool::Ffprobe, - args, - refusal, - strict_safe, - file, - input, - ), - Err(error) => { - message( - "ffprobe", - &format!("could not replay encoded stdin: {error}"), - ); - ExitCode::from(EXIT_IO) - } + Err(_) => match input.into_replay() { + Ok((file, input)) => fallback_cmd::refuse_with_input( + Tool::Ffprobe, + args, + refusal(), + strict_safe, + file, + input, + ), + Err(error) => { + message( + "ffprobe", + &format!("could not replay encoded stdin: {error}"), + ); + ExitCode::from(EXIT_IO) } + }, + } +} + +fn refusal() -> NotNative { + NotNative::property( + "standard input facts", + "structured stdin requires a complete qualified ISO movie within the prefix limit", + ) +} + +enum Acquired { + Complete(Vec), + Live(Vec), +} + +fn prefix(reader: &mut dyn Read) -> std::io::Result { + let limits = Limits::default(); + let mut bytes = Vec::new(); + let mut buffer = [0u8; 65_536]; + loop { + let room = (PREFIX_LIMIT - bytes.len()).min(buffer.len()); + let chunk = buffer + .get_mut(..room) + .ok_or_else(|| std::io::Error::other("stdin prefix limit"))?; + let count = match reader.read(chunk) { + Err(error) if error.kind() == std::io::ErrorKind::Interrupted => continue, + result => result?, + }; + if count == 0 { + return Ok(Acquired::Complete(bytes)); + } + bytes.try_reserve(count).map_err(std::io::Error::other)?; + bytes.extend_from_slice( + buffer + .get(..count) + .ok_or_else(|| std::io::Error::other("stdin prefix read"))?, + ); + if bytes.len() == PREFIX_LIMIT + || front_loaded( + &mut SliceSource::new(&bytes), + bytes.len() as u64, + &limits, + false, + ) + .is_err() + { + return Ok(Acquired::Live(bytes)); } } } @@ -58,7 +120,7 @@ fn render(input: &EncodedInput, options: &Options) -> Result, Failure> { let mut source = FileSource::open(&input.path).map_err(Failure::library)?; let size = source.len().map_err(Failure::library)?; let limits = Limits::default(); - front_loaded(&mut source, size, &limits)?; + front_loaded(&mut source, size, &limits, true)?; let mut source = BufferedSource::new(source); let mut probed = ffprobe::probe_limited(&mut source, size, &options.input, options.force, &limits)?; @@ -78,7 +140,12 @@ fn unsupported() -> Failure { Failure::Unsupported("unqualified stdin layout or exhausted staging limit".to_owned()) } -fn front_loaded(source: &mut S, size: u64, limits: &Limits) -> Result<(), Failure> { +fn front_loaded( + source: &mut S, + size: u64, + limits: &Limits, + complete: bool, +) -> Result<(), Failure> { let mut at = 0u64; let mut boxes = 0u32; let (mut movie, mut media) = (false, false); @@ -92,9 +159,11 @@ fn front_loaded(source: &mut S, size: u64, limits: &Limits) -> Result let count = usize::try_from(room.min(bytes.len() as u64)).map_err(|_| unsupported())?; let header = bytes.get_mut(..count).ok_or_else(unsupported)?; source.read_exact_at(at, header).map_err(Failure::library)?; - let Parsed::Done(header) = parse_box_header(header, at, room).map_err(Failure::library)? + let extent = if complete { room } else { u64::MAX - at }; + let Parsed::Done(header) = + parse_box_header(header, at, extent).map_err(Failure::library)? else { - return Err(unsupported()); + return if complete { Err(unsupported()) } else { Ok(()) }; }; match &header.kind.0 { b"moov" if !movie && !media => movie = true, @@ -102,9 +171,15 @@ fn front_loaded(source: &mut S, size: u64, limits: &Limits) -> Result b"ftyp" | b"free" | b"skip" | b"wide" => {} _ => return Err(unsupported()), } + if !complete && header.end() >= PREFIX_LIMIT as u64 { + return Err(unsupported()); + } + if header.end() > size && !complete { + return Ok(()); + } at = header.end(); } - if movie && media { + if !complete || movie && media { Ok(()) } else { Err(unsupported()) @@ -125,7 +200,7 @@ impl Write for Staged { .filter(|end| *end <= self.limit) .ok_or_else(|| std::io::Error::other("structured stdin staging limit"))?; self.bytes - .try_reserve_exact(end - self.bytes.len()) + .try_reserve(end - self.bytes.len()) .map_err(std::io::Error::other)?; self.bytes.extend_from_slice(bytes); Ok(bytes.len()) @@ -151,7 +226,12 @@ mod tests { fn only_complete_front_loaded_nonfragmented_iso_layouts_are_qualified() { let valid = [atom(b"ftyp"), atom(b"moov"), atom(b"mdat")].concat(); let check = |bytes: &[u8], limits: Limits| { - front_loaded(&mut SliceSource::new(bytes), bytes.len() as u64, &limits) + front_loaded( + &mut SliceSource::new(bytes), + bytes.len() as u64, + &limits, + true, + ) }; assert!(check(&valid, Limits::default()).is_ok()); for bytes in [ @@ -178,6 +258,41 @@ mod tests { ); } + #[test] + fn prefix_refuses_early_and_bounds_an_unending_large_movie() { + struct Live(bool); + impl Read for Live { + fn read(&mut self, bytes: &mut [u8]) -> std::io::Result { + assert!(!self.0); + self.0 = true; + bytes.get_mut(..9).unwrap().copy_from_slice(b"not media"); + Ok(9) + } + } + let Acquired::Live(bytes) = prefix(&mut Live(false)).unwrap() else { + panic!("unadmitted live content must not wait for EOF"); + }; + assert_eq!(bytes, b"not media"); + let header = [ + atom(b"ftyp"), + atom(b"moov"), + [2_147_483_648u32.to_be_bytes().as_slice(), b"mdat"].concat(), + ] + .concat(); + let mut reader = std::io::Cursor::new(&header).chain(std::io::repeat(0)); + let Acquired::Live(bytes) = prefix(&mut reader).unwrap() else { + panic!("a large live movie must reach fallback before EOF"); + }; + assert_eq!(bytes, header); + let mut reader = std::io::repeat(0); + let prefix = [1_000_000u32.to_be_bytes().as_slice(), b"free"].concat(); + let mut reader = std::io::Cursor::new(prefix).chain(&mut reader); + let Acquired::Live(bytes) = super::prefix(&mut reader).unwrap() else { + panic!("an undecidable prefix must reach fallback at its cap"); + }; + assert_eq!(bytes.len(), PREFIX_LIMIT); + } + #[test] fn staging_limit_fails_without_committing_a_partial_chunk() { let mut output = Staged { diff --git a/crates/coreav-cli/src/fallback_cmd.rs b/crates/coreav-cli/src/fallback_cmd.rs index afbfafb3d..41190daa7 100644 --- a/crates/coreav-cli/src/fallback_cmd.rs +++ b/crates/coreav-cli/src/fallback_cmd.rs @@ -7,7 +7,7 @@ use std::ffi::OsString; use std::fs::OpenOptions; -use std::io::Write; +use std::io::{Read, Write}; use std::path::{Path, PathBuf}; use std::process::{Command, ExitCode}; use std::time::{SystemTime, UNIX_EPOCH}; @@ -25,6 +25,11 @@ const GUARD: &str = "COREAV_FALLBACK_ACTIVE"; /// counts from. const SECONDS_1904_TO_1970: u64 = 2_082_844_800; +enum Input { + Complete(std::fs::File, super::compat_cmd::EncodedInput), + Prefix(Vec), +} + /// A command that is not native: the program that runs in its place, whose /// exit status becomes this one's, or the refusal with exit status 4. pub fn refuse( @@ -44,7 +49,29 @@ pub fn refuse_with_input( input: std::fs::File, spool: super::compat_cmd::EncodedInput, ) -> ExitCode { - refuse_inner(tool, args, refusal, strict_safe, Some((input, spool))) + refuse_inner( + tool, + args, + refusal, + strict_safe, + Some(Input::Complete(input, spool)), + ) +} + +pub fn refuse_with_prefix( + tool: Tool, + args: &[String], + refusal: NotNative, + strict_safe: StrictSafe, + prefix: Vec, +) -> ExitCode { + refuse_inner( + tool, + args, + refusal, + strict_safe, + Some(Input::Prefix(prefix)), + ) } fn refuse_inner( @@ -52,7 +79,7 @@ fn refuse_inner( args: &[String], refusal: NotNative, strict_safe: StrictSafe, - input: Option<(std::fs::File, super::compat_cmd::EncodedInput)>, + input: Option, ) -> ExitCode { #[cfg(not(feature = "net"))] if fallback::has_remote_input(tool, args) { @@ -214,21 +241,66 @@ fn log(tool: Tool, args: &[String], reason: &fallback::Reason, ran: Result<&str, } } -/// Runs the program in this process's place: on Unix it replaces this -/// process (`exec`), so the process id, signals and exit status are the -/// program's; elsewhere it waits for it and exits with its exit code. +/// Runs inherited stdin and complete replay through Unix `exec`. A live +/// prefix needs a spawned child and relay; other platforms also wait for +/// the child and preserve its full exit code. /// Returns only when the program could not start. -fn run( - path: &Path, - args: &[String], - input: Option<(std::fs::File, super::compat_cmd::EncodedInput)>, -) -> std::io::Error { +fn run(path: &Path, args: &[String], input: Option) -> std::io::Error { let mut command = Command::new(path); command.arg("-hide_banner").args(args).env(GUARD, "1"); - let spool = input.map(|(input, spool)| { - command.stdin(input); - spool - }); + let (spool, prefix) = match input { + Some(Input::Complete(input, spool)) => { + command.stdin(input); + (Some(spool), None) + } + Some(Input::Prefix(prefix)) => (None, Some(prefix)), + None => (None, None), + }; + if let Some(prefix) = prefix { + let mut child = match command.stdin(std::process::Stdio::piped()).spawn() { + Ok(child) => child, + Err(error) => return error, + }; + let Some(mut writer) = child.stdin.take() else { + let _ = child.kill(); + let _ = child.wait(); + return std::io::Error::other("fallback stdin pipe is unavailable"); + }; + let (sent, received) = std::sync::mpsc::channel(); + let pump = std::thread::Builder::new().spawn(move || { + let mut input = std::io::Cursor::new(prefix).chain(std::io::stdin().lock()); + let result = std::io::copy(&mut input, &mut writer); + let _ = sent.send(result); + }); + if let Err(error) = pump { + let _ = child.kill(); + let _ = child.wait(); + return error; + } + let status = match child.wait() { + Ok(status) => status, + Err(error) => { + let _ = child.kill(); + let _ = child.wait(); + return error; + } + }; + if let Ok(Err(error)) = received.try_recv() { + if error.kind() != std::io::ErrorKind::BrokenPipe { + return error; + } + } + #[cfg(unix)] + let code = { + use std::os::unix::process::ExitStatusExt; + status + .code() + .unwrap_or_else(|| 128 + status.signal().unwrap_or(1)) + }; + #[cfg(not(unix))] + let code = status.code().unwrap_or(1); + std::process::exit(code); + } #[cfg(unix)] let error = { use std::os::unix::process::CommandExt; diff --git a/docs/agents/ffmpeg-cli.md b/docs/agents/ffmpeg-cli.md index 67d420c1c..1e7a71813 100644 --- a/docs/agents/ffmpeg-cli.md +++ b/docs/agents/ffmpeg-cli.md @@ -69,8 +69,9 @@ CLI and ffprobe behavior, the FFmpeg pin, and fallback names. | Headerless audio admits decoder representations and exact conversion stages independently; a conformance decoder record cannot authorize a missing conversion or presentation predicate | `raw_audio::Plan` owns the resolved stage chain, `record` fails closed on unknown or disabled entries, and the exact channel/rate tables remain separate. Registry regressions and `nonstandard_pipe_descriptors_and_unregistered_compositions_refuse` check those boundaries. Runtime codec and presentation provenance cannot be encoded by command syntax alone | | A conformance-only PCM decoder preserves its complete presented samples without claiming oracle byte identity, while disabled presentations still refuse and replay stdin | The checked-in-record regression admits only qualified MPEG-1 MP3 in MP4. `raw_audio_pipe_check.py` compares native file and pipe packing with `decode-audio`, checks the finite edit count, and tests AAC, Opus, Matroska MP3 and MPEG-2 refusals with original-byte fallback replay. Its Linux rate check uses identical native decoded floats as oracle input, so decoder differences cannot hide a conversion error | | Encoded stdin fallback receives the original bytes and arguments, while decoded audio stays bounded and poststart errors never fall back | `EncodedInput` owns the capped private spool and replay; raw stdin uses the bounded stream engine directly. `encoded_stdin_is_replayed_for_native_raw_output`, `consumed_encoded_stdin_reaches_fallback_with_original_arguments`, `raw_stdin_writes_before_eof_without_a_whole_input_spool` and `a_partial_raw_frame_after_output_never_invokes_fallback` exercise native output, replay, cleanup and the irreversible stdout boundary | -| Replayed Unix fallback uses the existing process-replacement owner and removes its named spool before handoff | `fallback_cmd::run` owns Unix `exec` for inherited and replayed stdin. `encoded_stdin_fallback_replaces_the_process_without_leaving_its_spool` requires the same PID, complete input and no temporary file. A separate spawn-and-wait path unnecessarily links another launcher into the binary; types cannot prove process replacement or filesystem cleanup | -| Structured stdin keeps logical pipe facts separate from its replay file and never emits a partial native JSON result before refusal | Shape admission precedes acquisition in `args::parse`; `probe_stdin` reuses the bounded owner, requires front-loaded nonfragmented ISO and stages complete JSON before emission. `ffprobe_stdin` checks exact generated pipe output, unsupported live-pipe commands, complete pinned replay, read/temp-file failures and cleanup; layout, acquisition-cap and output-cap units cover bounded failures. A file handle does not encode the original pipe's seekability or size | +| Complete-file Unix fallback uses the existing process-replacement owner and removes its named spool before handoff | `fallback_cmd::Input::Complete` and inherited stdin retain Unix `exec`; `encoded_stdin_fallback_replaces_the_process_without_leaving_its_spool` requires the same PID, complete input and no temporary file. Only `Input::Prefix` needs a spawned child to concatenate already-read bytes with live stdin; types distinguish its transport but cannot prove process replacement or filesystem cleanup | +| Structured stdin checks a bounded prefix before acquisition, keeps logical pipe facts separate and never emits partial native JSON before refusal | `args::parse` guards command shape; incremental `probe_stdin::prefix` sends unsupported or cap-exhausted content to live fallback, while complete small ISO candidates reuse the file owner and staged JSON. Generated exact output, live-pipe startup, declared-large and undecidable prefixes, complete replay and failure/cleanup checks cover the boundaries; types cannot prove external EOF or transport facts | +| A live-prefix fallback forwards bytes in order and stops with the child even when its producer remains open | `fallback_cmd::Input::Prefix` owns one copy thread and the parent's child wait. `ffprobe_stdin` requires startup before EOF for invalid and declared-large streams, complete prefix/remainder SHA-256 and original arguments, exit code 37, early exit 29 and no spool. A reader thread cannot be joined while its external input may remain open | | Raw audio stdout checks final flush and preserves quiet EPIPE exit 141 | The shared media writer and explicit terminal-error mapping are covered by `tiny_pcm_stdout_reports_a_closed_reader_during_final_flush`; a nonempty output is not proof that its buffered tail reached the reader | | Every headerless raw-audio rate conversion needs its platform registration, including pre-existing raw S16 file commands | `raw_audio::write_raw` owns file and pipe admission. `short_raw_inputs_keep_the_compatibility_startup_and_drain_lengths` retains exact Linux bytes and lengths and requires refusal without output elsewhere; `raw_audio_pipe_check.py` compares unregistered-platform file and stdin fallback with the pinned oracle. Historical native behavior does not establish correctness | | Input thread syntax outside the native grammar reaches fallback, and decoder capacity does not restrict copy jobs | `threads` owns shared syntax classification and `Command::decoder_threads` owns the capacity limit only where decoder options consume it. Parser regressions, `input_thread_expressions_reach_fallback_without_consuming_stdin` and `copy_jobs_ignore_decoder_capacity_but_audio_decode_refuses` cover original-byte fallback and native copy above the decoder cap. A parsed count alone cannot identify which job will use it | diff --git a/docs/design/ffprobe-stdin-json.md b/docs/design/ffprobe-stdin-json.md index 433c7f92d..2c6ece6a7 100644 --- a/docs/design/ffprobe-stdin-json.md +++ b/docs/design/ffprobe-stdin-json.md @@ -33,8 +33,10 @@ Run the smallest affected Rust and fixture tests, Clippy, formatting, public-tex ## Implemented qualification -Native admission is limited to `pipe:` and `pipe:0`, complete JSON format/stream selections, and nonfragmented ISO files whose sole movie box precedes every media box. The existing box-header parser establishes complete top-level extents under its visit cap. Tail movie metadata, fragments, other containers, detailed diagnostics and inspection options remain unsupported. This deliberately avoids assuming an oracle seek-back buffer size or borrowing file-derived transport timing. +Native admission is limited to `pipe:` and `pipe:0`, complete JSON format/stream selections, and complete nonfragmented ISO files smaller than the 1 MiB prefix cap whose sole movie box precedes every media box. The existing box-header parser establishes complete top-level extents under its visit cap. Tail movie metadata, fragments, other containers, detailed diagnostics and inspection options remain unsupported. This deliberately avoids assuming an oracle seek-back buffer size or borrowing file-derived transport timing. Generated transport qualification covers interleaved ISO; other sample-placement layouts are not newly qualified here. -The bytes are acquired through the existing owner and limits. The logical name is passed to probing independently of the backing path. ISO format size and the total-file-derived format bitrate are unknown on the logical pipe and are omitted; declared stream rates and timings remain unchanged. The complete JSON is staged with fallible allocation under the existing 16 MiB retained-byte default before stdout is touched. Probe, layout or staging failure discards that result and replays the complete owned input to the unchanged fallback; acquisition failure is terminal and cleans its partial spool. +Prefix inspection is incremental. An unsupported box, a declared extent that cannot fit, or the prefix cap immediately selects fallback without awaiting EOF. Only a complete candidate within that prefix reaches the existing file owner. The logical name is passed to probing independently of the backing path. ISO format size and the total-file-derived format bitrate are unknown on the logical pipe and are omitted; declared stream rates and timings remain unchanged. The complete JSON is staged with fallible allocation under the existing 16 MiB retained-byte default before stdout is touched. Probe, layout or staging failure discards that result and replays the complete owned input; acquisition failure is terminal and cleans its partial spool. + +An incomplete or unsupported prefix needs its unread stdin remainder, so this fallback is the explicit exception to Unix process replacement: a spawned child receives the prefix followed by live stdin through one bounded-copy thread. The parent waits for the child, not for producer EOF or the relay thread, preserving prompt early child exit. Normal child exit codes are preserved, including full Windows codes; Unix signal termination maps to `128 + signal`. Inherited stdin and complete-file replay retain the existing Unix `exec` path, arguments, fallback selection, notice and guard. The `-` alias stays unsupported before acquisition. The pinned executable canonicalizes it to `fd:`, whose seekability can depend on the underlying descriptor; treating it as `pipe:0` would misreport both its identity and potentially its transport facts. No-section summaries remain on their original path. Existing encoded-audio stdin uses the same factored rewind/unlink/replay owner, including Unix process replacement and Windows lifetime cleanup. diff --git a/scripts/ffprobe_stdin_check.py b/scripts/ffprobe_stdin_check.py index b5b90a0b0..475c9ae2e 100644 --- a/scripts/ffprobe_stdin_check.py +++ b/scripts/ffprobe_stdin_check.py @@ -2,9 +2,11 @@ """Check complete structured stdin output and replay on generated inputs.""" import argparse +import hashlib import json import os from pathlib import Path +import selectors import subprocess import sys import tempfile @@ -144,6 +146,59 @@ def check(coreav, oracle, root): if reference.returncode or reference.stderr or result.returncode or result.stderr \ or result.stdout != reference.stdout: raise CheckFailure('local-file-regression') + relay = root / 'live-fallback' + relay.mkdir() + executable = relay / 'ffprobe' + executable.write_text('#!/usr/bin/env python3\n' + 'import hashlib, json, sys\n' + 'print("READY", flush=True)\n' + 'data = sys.stdin.buffer.read()\n' + 'print(json.dumps([sys.argv[1:], len(data), hashlib.sha256(data).hexdigest()]), flush=True)\n' + 'sys.exit(37)\n') + executable.chmod(0o700) + for data in (b'not media', b'\x00\x00\x00\x08ftyp\x00\x00\x00\x08moov' + + b'\x80\x00\x00\x00mdat' + bytes(1_048_576 - 24)): + child = subprocess.Popen([str(coreav), 'ffprobe', *args, '-i', 'pipe:0'], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + env={**fallback_env, 'COREAV_FALLBACK': str(relay)}) + try: + child.stdin.write(data) + child.stdin.flush() + with selectors.DefaultSelector() as ready: + ready.register(child.stdout, selectors.EVENT_READ) + if not ready.select(5) or child.stdout.readline() != b'READY\n': + raise CheckFailure('live-fallback-start') + tail = b'live remainder\x00\xff' + output, diagnostics = child.communicate(tail, timeout=5) + if child.returncode != 37 or json.loads(output) != [ + ['-hide_banner', *args, '-i', 'pipe:0'], len(data + tail), + hashlib.sha256(data + tail).hexdigest()] \ + or not diagnostics.startswith(b'coreav: ran '): + raise CheckFailure('live-fallback-replay') + finally: + if child.poll() is None: + child.kill() + child.communicate() + executable.write_text('#!/usr/bin/env python3\n' + 'import sys\n' + 'sys.exit(29)\n') + child = subprocess.Popen([str(coreav), 'ffprobe', *args, '-i', 'pipe:0'], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + env={**fallback_env, 'COREAV_FALLBACK': str(relay)}) + try: + child.stdin.write(b'not media') + child.stdin.flush() + child.wait(timeout=5) + if child.returncode != 29 or child.stdout.read(): + raise CheckFailure('live-fallback-early-exit') + finally: + if child.poll() is None: + child.kill() + child.communicate() + if list(root.glob('coreav-encoded-stdin-*')): + raise CheckFailure('input-cleanup') print('PASS generated structured stdin and complete replay') @@ -161,7 +216,8 @@ def main(): except CheckFailure as error: known = {'fixture-generation', 'fixture-buffer-contrast', 'logical-pipe-facts', 'input-cleanup', 'shape-before-stdin', 'temporary-file-failure', 'local-file-regression', - 'input-read-failure', 'fd-alias-fallback', 'child-timeout', 'child-spawn'} + 'input-read-failure', 'fd-alias-fallback', 'child-timeout', 'child-spawn', + 'live-fallback-start', 'live-fallback-replay', 'live-fallback-early-exit'} known.update(category + '-' + name for category in ('native', 'refusal', 'replay') for name in ('faststart', 'faststart-large', 'tail-moov', 'fragmented', 'matroska', 'transport', 'empty', 'truncated', 'malformed-movie')) From a3e5ec95cbce24caf380f2ae756ada36fab881bb Mon Sep 17 00:00:00 2001 From: gunwooterry Date: Thu, 5 Nov 2026 10:39:12 +0000 Subject: [PATCH 4/4] test(ffprobe): inject a real stdin read failure Signed-off-by: gunwooterry --- docs/agents/ffmpeg-cli.md | 1 + scripts/ffprobe_stdin_check.py | 9 +++++++-- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/agents/ffmpeg-cli.md b/docs/agents/ffmpeg-cli.md index 1e7a71813..a36ffc72d 100644 --- a/docs/agents/ffmpeg-cli.md +++ b/docs/agents/ffmpeg-cli.md @@ -72,6 +72,7 @@ CLI and ffprobe behavior, the FFmpeg pin, and fallback names. | Complete-file Unix fallback uses the existing process-replacement owner and removes its named spool before handoff | `fallback_cmd::Input::Complete` and inherited stdin retain Unix `exec`; `encoded_stdin_fallback_replaces_the_process_without_leaving_its_spool` requires the same PID, complete input and no temporary file. Only `Input::Prefix` needs a spawned child to concatenate already-read bytes with live stdin; types distinguish its transport but cannot prove process replacement or filesystem cleanup | | Structured stdin checks a bounded prefix before acquisition, keeps logical pipe facts separate and never emits partial native JSON before refusal | `args::parse` guards command shape; incremental `probe_stdin::prefix` sends unsupported or cap-exhausted content to live fallback, while complete small ISO candidates reuse the file owner and staged JSON. Generated exact output, live-pipe startup, declared-large and undecidable prefixes, complete replay and failure/cleanup checks cover the boundaries; types cannot prove external EOF or transport facts | | A live-prefix fallback forwards bytes in order and stops with the child even when its producer remains open | `fallback_cmd::Input::Prefix` owns one copy thread and the parent's child wait. `ffprobe_stdin` requires startup before EOF for invalid and declared-large streams, complete prefix/remainder SHA-256 and original arguments, exit code 37, early exit 29 and no spool. A reader thread cannot be joined while its external input may remain open | +| A stdin read-error fixture proves the terminal I/O path rather than accepting any nonzero refusal | `ffprobe_stdin_check.py` uses a directory descriptor to produce a real read error and requires exit 3, the acquisition diagnostic, empty stdout and cleanup. A write-only descriptor is treated as EOF by Rust stdin on Linux; subprocess exit values cannot establish which kernel error was injected | | Raw audio stdout checks final flush and preserves quiet EPIPE exit 141 | The shared media writer and explicit terminal-error mapping are covered by `tiny_pcm_stdout_reports_a_closed_reader_during_final_flush`; a nonempty output is not proof that its buffered tail reached the reader | | Every headerless raw-audio rate conversion needs its platform registration, including pre-existing raw S16 file commands | `raw_audio::write_raw` owns file and pipe admission. `short_raw_inputs_keep_the_compatibility_startup_and_drain_lengths` retains exact Linux bytes and lengths and requires refusal without output elsewhere; `raw_audio_pipe_check.py` compares unregistered-platform file and stdin fallback with the pinned oracle. Historical native behavior does not establish correctness | | Input thread syntax outside the native grammar reaches fallback, and decoder capacity does not restrict copy jobs | `threads` owns shared syntax classification and `Command::decoder_threads` owns the capacity limit only where decoder options consume it. Parser regressions, `input_thread_expressions_reach_fallback_without_consuming_stdin` and `copy_jobs_ignore_decoder_capacity_but_audio_decode_refuses` cover original-byte fallback and native copy above the decoder cap. A parsed count alone cannot identify which job will use it | diff --git a/scripts/ffprobe_stdin_check.py b/scripts/ffprobe_stdin_check.py index 475c9ae2e..6b697f491 100644 --- a/scripts/ffprobe_stdin_check.py +++ b/scripts/ffprobe_stdin_check.py @@ -129,10 +129,15 @@ def check(coreav, oracle, root): env={**env, 'TMPDIR': str(bad_temp)}) if failed.returncode == 0 or failed.stdout or bad_temp.read_bytes() != b'preserve': raise CheckFailure('temporary-file-failure') - with (root / 'write-only').open('wb') as source: + source = os.open(root, os.O_RDONLY | os.O_DIRECTORY) + try: failed = subprocess.run([str(coreav), 'ffprobe', *args, '-i', 'pipe:0'], stdin=source, capture_output=True, timeout=30, env=env) - if failed.returncode == 0 or failed.stdout or list(root.glob('coreav-encoded-stdin-*')): + finally: + os.close(source) + if failed.returncode != 3 or failed.stdout \ + or not failed.stderr.startswith(b'coreav ffprobe: could not acquire encoded stdin:') \ + or list(root.glob('coreav-encoded-stdin-*')): raise CheckFailure('input-read-failure') minus = [*args, '-i', '-'] reference = run([oracle.ffprobe, '-hide_banner', *minus], cases[0][1])