diff --git a/openless-all/app/crates/micyou-dsp/UPSTREAM.md b/openless-all/app/crates/micyou-dsp/UPSTREAM.md index 4be8b707e..07cbd15e6 100644 --- a/openless-all/app/crates/micyou-dsp/UPSTREAM.md +++ b/openless-all/app/crates/micyou-dsp/UPSTREAM.md @@ -5,8 +5,9 @@ Source: [MicYou-Dev/MicYou](https://github.com/MicYou-Dev/MicYou), commit `src/lib.rs` comes from `tauri-app/crates/micyou-audio/src/dsp.rs`. `../micyou-infer/src/{lib,ops}.rs` and the two `src/assets/*.mcy` blobs -come from the corresponding upstream paths. The inference runtime and model -blobs are imported without algorithm changes. Original copyright notices are +come from the corresponding upstream paths. The model blobs are imported unchanged. The inference runtime retains the +upstream algorithms, with baseline x86-64 SSE and ARM64 NEON row projections using separate +multiply/add operations and the original accumulation order. Original copyright notices are retained in all source files. See `LICENSE` for MicYou's complete license and plugin exception; `LICENSE-PureVox.txt` and `LICENSE-AEC7.txt` are the upstream MIT model notices. The inference crate also carries MicYou's license. @@ -24,6 +25,9 @@ OpenLess changes: PureVox wet/dry energy ratio. This prevents AGC from undoing suppression when a conservative amount of dry signal is mixed back in. - Add serde defaults for DSP settings. +- Make the true FFT spectrum analyzer opt-in and throttle snapshots to 10 Hz. + Closing monitoring releases its FFT resources. +- Use reusable host buffers and cached rational-phase resampling kernels. - Define the small AEC failure enum locally instead of depending on MicYou's audio host. @@ -32,9 +36,17 @@ resampling, fixed 480-sample hops, model-delay compensation, end-of-stream flush and exact output-duration accounting in `openless-core/src/audio.rs` and `audio_dsp.rs`. The main microphone path does not install plugin hooks. +Android capture requests 48 kHz directly, and all enhancement algorithms keep +that processing rate. The original input filter is preserved even at 48 kHz, +using cached coefficients; ASR output is downsampled to 16 kHz. CPAL retains a device-native fallback if +48 kHz cannot be opened. Capture queues are not enlarged for playback or +monitoring. + AEC7 and its aligned-reference API are retained and tested here. OpenLess's microphone hosts do not yet provide synchronized speaker-reference capture, -so the application does not expose or claim active acoustic echo cancellation. +so the application does not expose AEC7 as an active microphone effect. +Android may instead use its session-bound platform AcousticEchoCanceler when +the device supports it; availability and activation are reported separately. The optional comb filter follows upstream's simple delay subtraction; it is not a learned room dereverberation model. @@ -81,3 +93,68 @@ do not automatically trigger separate PR workflows. Most upstream implementation/model fixes can follow this route without manual copying. Upstream API/dependency/format changes that break the adapter or tests still require review and a deliberate patch/dependency update. + +## Shared performance changes (2026-10-22) + +All hosts that use `AudioProcessor` (including desktop Tauri and Linux egui) +reuse mono, frame and resampled output scratch buffers. Rational resampling +phases are precomputed once per stream, capped at 1024 phases; unusual clocks +fall back to the original kernel calculation. The output duration and filter +formula are unchanged, including the original filter at native 48 kHz. + +Contiguous matrix and GRU projection rows use separate SIMD multiply and add: +SSE on baseline x86-64, NEON on ARM64 targets with NEON enabled. Other targets +keep the scalar fallback. No additional inference runtime or worker pool is +loaded. This does not change OS-specific capture APIs or require AVX support. + +The synthetic full-chain probe defaults to 48 kHz and checks exact output +length. Run on the target device before drawing battery or latency conclusions: + +```sh +cargo run --release --manifest-path openless-all/app/Cargo.toml -p openless-core --example audio_performance -- rnnoise 48000 +``` + +## Before/after equivalence validation (2026-10-24) + +The frozen baseline is commit `f0744e10`. The same `openless-core` example +`audio_equivalence` was built against that baseline and the optimized sources, +using their locked dependencies, in separate release build directories. +Each build exports 48 cases: 16/44.1/48 kHz, mono/stereo, four denoiser modes, +and default/combined gain-EQ-comb-VAD settings. Inputs contain silence, +harmonics, deterministic noise, high-frequency interference, an impulse, +irregular chunk boundaries, and a partial final frame. + +On Linux x86-64, all 48 output PCM files were byte-identical to the baseline: +maximum difference 0 LSB, with matching durations. The full hashes, errors and +diagnostic processing times are committed in +`../../scripts/fixtures/audio-equivalence-2026-10-24.json`. This establishes +equivalence for those inputs and host, not a listening study or ASR accuracy. + +Export both builds to separate directories with `audio_equivalence OUTPUT_DIR`, +then run `python3 scripts/compare-audio-outputs.py BEFORE AFTER --report REPORT`. +The comparator fails on any PCM difference or duration/case-set mismatch; +`python3 scripts/test-compare-audio-outputs.py` verifies a one-LSB mutation +and a truncated output cannot pass. The probe sources must be identical in +both builds. Original baseline sources/models must remain unchanged. + +Desktop Tauri and Linux egui use the shared processor directly. Harmony uses +the external PCM recorder, which reuses conversion scratch per session. +The iOS native recorder now optionally applies the same processor with +session-frozen settings and a reused input buffer; the stop/drain barrier +flushes its tail before finishing the archive. Enhancement remains off by +default. Its native Swift settings page persists the same software DSP +controls as egui and Tauri, with writes blocked while a capture is active. +Native iOS conversion uses a separate +Swift AVAudioConverter worker; its output scratch is now actor-owned and +reused with exact output capacities and a two-entry bound. +`ios/scripts/test-audio.sh` compares that worker byte-for-byte against +the frozen pre-optimization Swift worker using its complete, fixed 100 ms +input path, over up to 32 format/ownership/packetization combinations and +end-of-stream flush (older macOS runtimes only exercise legacy ownership). +The old variable-packet path itself lost 960 samples in the 44.1 kHz CI case. +The new worker honors the converter's requested input frame count and drains +available output before accepting another packet. Variable-packet output must +match the complete fixed-packet baseline, not its truncated old counterpart; +the duration assertion remains enabled. That Apple SDK test +was added but cannot be executed on this Linux host. No claim of on-device +equivalence, battery savings, or real-time capture latency is made here. diff --git a/openless-all/app/crates/micyou-dsp/src/lib.rs b/openless-all/app/crates/micyou-dsp/src/lib.rs index 7db9773db..d6742da6b 100644 --- a/openless-all/app/crates/micyou-dsp/src/lib.rs +++ b/openless-all/app/crates/micyou-dsp/src/lib.rs @@ -939,6 +939,9 @@ pub struct DspProcessor { agc_smoothed_gain: f32, // VAD fade state (0.0 = muted, 1.0 = full) vad_fade: f32, + #[cfg(feature = "noise-suppression")] + spectrum_analyzer: Option, + spectrum_tick: usize, // Spectrum snapshots raw_spectrum: Vec, processed_spectrum: Vec, @@ -981,6 +984,53 @@ pub fn parse_plugin_chain_node(node: &str) -> Option<&str> { const RNNOISE_FRAME_SIZE: usize = 480; +#[cfg(feature = "noise-suppression")] +struct SpectrumAnalyzer { + fft: Arc>, + buffer: Vec>, + scratch: Vec>, + window: Vec, +} + +#[cfg(feature = "noise-suppression")] +impl SpectrumAnalyzer { + fn new() -> Self { + let fft = rustfft::FftPlanner::new().plan_fft_forward(512); + let scratch = vec![Complex::default(); fft.get_inplace_scratch_len()]; + Self { + fft, + scratch, + buffer: vec![Complex::default(); 512], + window: (0..512) + .map(|i| 0.5 - 0.5 * (std::f32::consts::TAU * i as f32 / 511.0).cos()) + .collect(), + } + } + + fn analyze(&mut self, data: &[f32], target: &mut [f32]) { + self.buffer.fill(Complex::default()); + let count = data.len().min(512); + for (i, sample) in data[data.len() - count..].iter().enumerate() { + self.buffer[i].re = sample * self.window[i]; + } + self.fft + .process_with_scratch(&mut self.buffer, &mut self.scratch); + for (band, value) in target.iter_mut().enumerate() { + let low = 20.0 * 1200.0_f32.powf(band as f32 / 64.0); + let high = 20.0 * 1200.0_f32.powf((band + 1) as f32 / 64.0); + let start = ((low * 512.0 / 48_000.0) as usize).min(256); + let end = ((high * 512.0 / 48_000.0).ceil() as usize).clamp(start + 1, 257); + let magnitude = self.buffer[start..end] + .iter() + .map(|bin| bin.norm()) + .fold(0.0_f32, f32::max) + / 128.0; + let normalized = ((20.0 * magnitude.max(1e-8).log10() + 60.0) / 60.0).clamp(0.0, 1.0); + *value = normalized.max(*value * 0.8); + } + } +} + impl DspProcessor { pub fn new(settings: Arc>, _model_dir: Option) -> Self { Self { @@ -1020,6 +1070,9 @@ impl DspProcessor { agc_envelope: 0.0, agc_smoothed_gain: 1.0, vad_fade: 1.0, + #[cfg(feature = "noise-suppression")] + spectrum_analyzer: None, + spectrum_tick: 0, raw_spectrum: vec![0.0; 64], processed_spectrum: vec![0.0; 64], accum_buffer: Vec::new(), @@ -1045,6 +1098,7 @@ impl DspProcessor { } let raw_rms = compute_rms(data); + self.spectrum_tick = (self.spectrum_tick + 1) % 10; self.compute_spectrum(data, true); // Frame accumulation: align to 480*channels samples before processing, @@ -1670,59 +1724,34 @@ impl DspProcessor { // ── Spectrum ───────────────────────────────────────────────────────────── - fn compute_spectrum(&mut self, data: &[f32], is_raw: bool) { - let bands = 64; - let target = if is_raw { - &mut self.raw_spectrum - } else { - &mut self.processed_spectrum - }; - if target.len() != bands { - target.resize(bands, 0.0); + pub fn set_spectrum_enabled(&mut self, enabled: bool) { + #[cfg(feature = "noise-suppression")] + if enabled && self.spectrum_analyzer.is_none() { + self.spectrum_analyzer = Some(SpectrumAnalyzer::new()); + } else if !enabled { + self.spectrum_analyzer = None; + self.raw_spectrum.fill(0.0); + self.processed_spectrum.fill(0.0); } - if data.is_empty() { - for v in target.iter_mut() { - *v = 0.0; - } + #[cfg(not(feature = "noise-suppression"))] + let _ = enabled; + } + + fn compute_spectrum(&mut self, data: &[f32], is_raw: bool) { + if self.spectrum_tick != 1 { return; } - - const BANDS: usize = 64; - static BAND_LIMITS: std::sync::OnceLock<[f32; BANDS + 1]> = std::sync::OnceLock::new(); - let limits = BAND_LIMITS.get_or_init(|| { - let mut array = [0.0; BANDS + 1]; - for (i, limit) in array.iter_mut().enumerate() { - *limit = (i as f32 / BANDS as f32).powf(1.5); - } - array - }); - - for (band_idx, band_val) in target.iter_mut().enumerate() { - let start = limits[band_idx] * data.len() as f32; - let end = limits[band_idx + 1] * data.len() as f32; - let start = start as usize; - let end = (end as usize).min(data.len()); - if start >= end { - *band_val *= 0.85; - continue; - } - let mut sum = 0.0_f32; - for sample in &data[start..end] { - sum += sample * sample; - } - let rms = (sum / (end - start) as f32).sqrt(); - let db = if rms > 1e-10 { - 20.0 * rms.log10() + #[cfg(feature = "noise-suppression")] + if let Some(analyzer) = &mut self.spectrum_analyzer { + let target = if is_raw { + &mut self.raw_spectrum } else { - -100.0 + &mut self.processed_spectrum }; - let normalized = ((db + 60.0) / 60.0).clamp(0.0, 1.0); - if normalized > *band_val { - *band_val = normalized; - } else { - *band_val = *band_val * 0.85 + normalized * 0.15; - } + analyzer.analyze(data, target); } + #[cfg(not(feature = "noise-suppression"))] + let _ = (data, is_raw); } } @@ -2095,11 +2124,13 @@ mod native_tests { #[test] fn dsp_processor_chain_smoke() { // Full chain: AEC → PureVox NS through the public DspProcessor API. - let mut settings = AudioDspSettings::default(); - settings.ns_enabled = true; - settings.ns_type = "PureVox".to_string(); - settings.ns_intensity = 100.0; - settings.aec_enabled = true; + let settings = AudioDspSettings { + ns_enabled: true, + ns_type: "PureVox".to_string(), + ns_intensity: 100.0, + aec_enabled: true, + ..Default::default() + }; let settings = Arc::new(RwLock::new(settings)); let mut dsp = DspProcessor::new(settings, None); @@ -2126,3 +2157,42 @@ mod native_tests { ); } } + +#[cfg(all(test, feature = "noise-suppression"))] +mod spectrum_tests { + use super::*; + + #[test] + fn spectrum_is_opt_in_and_tracks_frequency_without_changing_audio() { + let settings = Arc::new(RwLock::new(AudioDspSettings { + gain: 6.0, + ..Default::default() + })); + let mut plain = DspProcessor::new(Arc::clone(&settings), None); + let mut monitored = DspProcessor::new(settings, None); + assert!(plain.spectrum_analyzer.is_none()); + monitored.set_spectrum_enabled(true); + let tone: Vec = (0..480) + .map(|i| (std::f32::consts::TAU * 1000.0 * i as f32 / 48_000.0).sin() * 0.05) + .collect(); + for _ in 0..12 { + let mut a = tone.clone(); + let mut b = tone.clone(); + plain.process(&mut a, 1, 0.0); + monitored.process(&mut b, 1, 0.0); + assert_eq!(a, b); + } + let (raw, processed) = monitored.get_spectrums(); + let peak = raw + .iter() + .enumerate() + .max_by(|a, b| a.1.total_cmp(b.1)) + .unwrap() + .0; + assert!((30..=40).contains(&peak), "1 kHz peak at {peak}"); + assert!(processed[peak] > raw[peak]); + monitored.set_spectrum_enabled(false); + assert!(monitored.spectrum_analyzer.is_none()); + assert!(monitored.get_spectrums().0.iter().all(|v| *v == 0.0)); + } +} diff --git a/openless-all/app/crates/micyou-infer/src/ops.rs b/openless-all/app/crates/micyou-infer/src/ops.rs index 23bdda258..ba94bbac8 100644 --- a/openless-all/app/crates/micyou-infer/src/ops.rs +++ b/openless-all/app/crates/micyou-infer/src/ops.rs @@ -396,9 +396,7 @@ pub(crate) unsafe fn matmul_impl( for kk in 0..k { let av = arow[kk]; let brow = &bbase[kk * n..(kk + 1) * n]; - for j in 0..n { - orow[j] += av * brow[j]; - } + axpy(orow, brow, av); } } if nb > 0 { @@ -501,9 +499,7 @@ pub(crate) unsafe fn conv_impl( for ci in 0..cpg { let wv = wg[mi * cpg + ci]; let xrow = &xg[ci * hw_in..(ci + 1) * hw_in]; - for p in 0..hw_out { - orow[p] += wv * xrow[p]; - } + axpy(orow, &xrow[..hw_out], wv); } } } @@ -753,9 +749,7 @@ pub(crate) unsafe fn gru_impl( for i in 0..batch { let xv = xb[i * in_dim + kk]; let prow = &mut pb[i * h3..(i + 1) * h3]; - for j in 0..h3 { - prow[j] += xv * wrow[j]; - } + axpy(prow, wrow, xv); } } } @@ -784,9 +778,7 @@ pub(crate) unsafe fn gru_impl( for i in 0..batch { let sv = state_s[i * h + kk]; let orow = &mut rproj_s[i * h3..(i + 1) * h3]; - for j in 0..h3 { - orow[j] += sv * wrow[j]; - } + axpy(orow, wrow, sv); } } // gates (elementwise passes — vectorize; rounding order matches @@ -1508,6 +1500,50 @@ fn resize_pass( // ─── AVX2+FMA runtime dispatch ───────────────────────────────────────────── // +// Use baseline SIMD on x86-64 and NEON-enabled ARM64 targets. Vectorize rows +// without changing reduction order or using fused multiply-add: recurrent +// models retain the scalar rounding contract. No extra model/runtime/thread +// pool is loaded alongside local ASR. +#[inline(always)] +fn axpy(output: &mut [f32], input: &[f32], scale: f32) { + assert_eq!(output.len(), input.len()); + #[cfg(all(target_arch = "aarch64", target_feature = "neon"))] + let offset = unsafe { + let mut offset = 0; + use std::arch::aarch64::{vaddq_f32, vld1q_f32, vmulq_n_f32, vst1q_f32}; + while offset + 4 <= output.len() { + let value = vmulq_n_f32(vld1q_f32(input.as_ptr().add(offset)), scale); + let sum = vaddq_f32(vld1q_f32(output.as_ptr().add(offset)), value); + vst1q_f32(output.as_mut_ptr().add(offset), sum); + offset += 4; + } + offset + }; + #[cfg(target_arch = "x86_64")] + let offset = unsafe { + // SSE2 is part of the x86-64 baseline. Unaligned loads/stores support + // every tensor row; no AVX CPU requirement or FMA rounding changes. + use std::arch::x86_64::{_mm_add_ps, _mm_loadu_ps, _mm_mul_ps, _mm_set1_ps, _mm_storeu_ps}; + let factor = _mm_set1_ps(scale); + let mut offset = 0; + while offset + 4 <= output.len() { + let value = _mm_mul_ps(_mm_loadu_ps(input.as_ptr().add(offset)), factor); + let sum = _mm_add_ps(_mm_loadu_ps(output.as_ptr().add(offset)), value); + _mm_storeu_ps(output.as_mut_ptr().add(offset), sum); + offset += 4; + } + offset + }; + #[cfg(not(any( + target_arch = "x86_64", + all(target_arch = "aarch64", target_feature = "neon") + )))] + let offset = 0; + for (dst, src) in output[offset..].iter_mut().zip(&input[offset..]) { + *dst += scale * src; + } +} + // The workspace builds for baseline x86-64 (SSE2) so the shipped binaries // run everywhere; ONNX Runtime's MLAS gets its speed from runtime CPUID // dispatch to AVX kernels. We mirror that: each hot kernel body is compiled @@ -1755,3 +1791,22 @@ unsafe fn layer_norm_avx2( ) { layer_norm_impl(xp, xsh, scale, bias, op_, axis, eps) } + +#[cfg(test)] +mod projection_tests { + #[test] + fn projections_keep_scalar_rounding_and_handle_vector_tails() { + for size in [0, 1, 3, 4, 5, 17, 192, 257] { + let input: Vec = (0..size).map(|i| (i as f32 * 0.7).sin()).collect(); + let mut actual = vec![0.17; size]; + let mut expected = actual.clone(); + for scale in [0.0, -0.37, 0.81, 1e-12] { + for (dst, src) in expected.iter_mut().zip(&input) { + *dst += scale * src; + } + super::axpy(&mut actual, &input, scale); + } + assert_eq!(actual, expected, "projection width {size}"); + } + } +} diff --git a/openless-all/app/crates/openless-core/examples/audio_equivalence.rs b/openless-all/app/crates/openless-core/examples/audio_equivalence.rs new file mode 100644 index 000000000..33dad637c --- /dev/null +++ b/openless-all/app/crates/openless-core/examples/audio_equivalence.rs @@ -0,0 +1,87 @@ +//! Export deterministic PCM for comparison across revisions and architectures. +//! Usage: audio_equivalence OUTPUT_DIRECTORY +use openless_core::{AudioDspConfig, AudioProcessingConfig, AudioProcessor, NoiseReduction}; +use std::{fs, path::PathBuf, time::Instant}; + +fn main() -> Result<(), Box> { + let root = PathBuf::from(std::env::args().nth(1).ok_or("output directory required")?); + fs::create_dir(&root)?; + let mut rows = Vec::new(); + for rate in [16_000, 44_100, 48_000] { + for channels in [1, 2] { + for (name, algorithm) in [ + ("off", NoiseReduction::Off), + ("rnnoise", NoiseReduction::Rnnoise), + ("purevox", NoiseReduction::Purevox), + ("spectral", NoiseReduction::Spectral), + ] { + for effects in [false, true] { + let config = AudioDspConfig { + noise_reduction: algorithm, + gain_db: if effects { 3 } else { 0 }, + equalizer_enabled: effects, + equalizer_bands_db: if effects { + [-2, -1, 0, 1, 2, 3, 4, 4, 3, 2] + } else { + [0; 10] + }, + dereverb_level: if effects { 12 } else { 0 }, + vad_enabled: effects, + vad_threshold_db: -55, + ..Default::default() + }; + let mut processor = AudioProcessor::new(AudioProcessingConfig { + enhancement_enabled: true, + dsp: config, + }); + let mut random = 42u32; + let mut input = Vec::new(); + // Silence, speech-band harmonics with deterministic noise, + // near-Nyquist interference, impulses, and an unaligned tail. + for i in 0..rate + 37 { + random = random.wrapping_mul(1664525).wrapping_add(1013904223); + let t = i as f32 / rate as f32; + let noise = ((random >> 8) as f32 / 16777216.0 - 0.5) * 0.025; + let signal = if i < rate / 5 { + 0.0 + } else { + 0.13 * (std::f32::consts::TAU * 230.0 * t).sin() + + 0.04 * (std::f32::consts::TAU * 1733.0 * t).sin() + + 0.02 * (std::f32::consts::TAU * rate as f32 * 0.43 * t).sin() + + noise + } + if i == rate / 2 { 0.4 } else { 0.0 }; + for channel in 0..channels { + input.push(signal * if channel == 0 { 1.0 } else { 0.7 }); + } + } + let start = Instant::now(); + let mut pcm = Vec::new(); + let mut offset = 0; + for size in [37, 960, 127, 4096].into_iter().cycle() { + if offset == input.len() { + break; + } + let end = (offset + size).min(input.len()); + if let Some(chunk) = processor.process(&input[offset..end], channels, rate) + { + pcm.extend(chunk.pcm_i16_le); + } + offset = end; + } + if let Some(chunk) = processor.finish() { + pcm.extend(chunk.pcm_i16_le); + } + assert_eq!( + pcm.len(), + ((rate as u64 + 37) * 16000).div_ceil(rate as u64) as usize * 2 + ); + let file = format!("{name}-{rate}-{channels}-{effects}.pcm"); + rows.push(serde_json::json!({"file":file,"seconds":start.elapsed().as_secs_f64(),"samples":pcm.len()/2})); + fs::write(root.join(file), pcm)?; + } + } + } + } + fs::write(root.join("results.json"), serde_json::to_vec_pretty(&rows)?)?; + Ok(()) +} diff --git a/openless-all/app/crates/openless-core/examples/audio_performance.rs b/openless-all/app/crates/openless-core/examples/audio_performance.rs new file mode 100644 index 000000000..a0555ae4e --- /dev/null +++ b/openless-all/app/crates/openless-core/examples/audio_performance.rs @@ -0,0 +1,55 @@ +//! Full capture-processing throughput probe; no ASR/network/device timing. +use openless_core::{AudioDspConfig, AudioProcessingConfig, AudioProcessor, NoiseReduction}; +use std::time::Instant; + +fn main() { + let args: Vec = std::env::args().collect(); + let name = args.get(1).map(String::as_str).unwrap_or("rnnoise"); + let rate: u32 = args.get(2).map(|s| s.parse().unwrap()).unwrap_or(48_000); + let mut processor = AudioProcessor::new(AudioProcessingConfig { + enhancement_enabled: name != "disabled", + dsp: AudioDspConfig { + noise_reduction: match name { + "purevox" => NoiseReduction::Purevox, + "spectral" => NoiseReduction::Spectral, + "off" | "disabled" => NoiseReduction::Off, + _ => NoiseReduction::Rnnoise, + }, + ..Default::default() + }, + }); + let samples: Vec = (0..rate / 50) + .map(|i| { + let t = i as f32 / rate as f32; + 0.1 * (std::f32::consts::TAU * 230.0 * t).sin() + + 0.015 * (std::f32::consts::TAU * 1733.0 * t).sin() + }) + .collect(); + let cold = Instant::now(); + let mut output_bytes = processor + .process(&samples, 1, rate) + .map_or(0, |c| c.pcm_i16_le.len()); + let first_ms = cold.elapsed().as_secs_f64() * 1000.0; + let mut times = Vec::with_capacity(1499); + let steady = Instant::now(); + for _ in 1..1500 { + let start = Instant::now(); + output_bytes += processor + .process(&samples, 1, rate) + .map_or(0, |c| c.pcm_i16_le.len()); + times.push(start.elapsed().as_secs_f64() * 1000.0); + } + let seconds = steady.elapsed().as_secs_f64(); + output_bytes += processor.finish().map_or(0, |c| c.pcm_i16_le.len()); + assert_eq!(output_bytes, 30 * 16_000 * 2); + times.sort_by(f64::total_cmp); + println!("algorithm={name} rate={rate} audio_seconds=30 first_callback_ms={first_ms:.3} steady_seconds={seconds:.3} rtf={:.4} p50_ms={:.3} p95_ms={:.3} max_ms={:.3} output_bytes={output_bytes}", seconds / 29.98, times[times.len()/2], times[times.len()*95/100], times[times.len()-1]); + if let Ok(status) = std::fs::read_to_string("/proc/self/status") { + for line in status + .lines() + .filter(|s| s.starts_with("VmRSS:") || s.starts_with("VmHWM:")) + { + println!("{line}"); + } + } +} diff --git a/openless-all/app/crates/openless-core/src/audio.rs b/openless-all/app/crates/openless-core/src/audio.rs index 46233a7ab..85e2437ba 100644 --- a/openless-all/app/crates/openless-core/src/audio.rs +++ b/openless-all/app/crates/openless-core/src/audio.rs @@ -16,6 +16,141 @@ pub struct AudioProcessingConfig { pub dsp: crate::audio_dsp::AudioDspConfig, } +/// 扬声器回环（far-end）参考音频的交接队列。 +/// +/// AEC 需要「扬声器此刻在放什么」。宿主在一条独立的采集流里拿到输出音频后， +/// 按自己的采样率塞进这里；DSP 线程在处理每一帧之前把它取走、重采样到 48 kHz +/// 单声道后交给 MicYou。队列有上限：AEC 只关心最近的参考，堆积只会拖慢。 +#[derive(Clone, Default)] +pub struct FarEndQueue { + inner: std::sync::Arc>>, +} + +#[derive(Debug)] +struct FarEndFrame { + rate: u32, + channels: usize, + samples: Vec, +} + +/// 参考队列最多保留的样本数（约 2 秒 @48 kHz）。超出时丢弃最旧的。 +const FAR_END_QUEUE_LIMIT: usize = 96_000; + +impl FarEndQueue { + /// 宿主采集线程调用。非有限值与空帧直接丢弃。 + pub fn push(&self, samples: &[f32], channels: usize, rate: u32) { + if samples.is_empty() || channels == 0 || rate == 0 { + return; + } + let clean: Vec = samples + .iter() + .map(|sample| if sample.is_finite() { sample.clamp(-1.0, 1.0) } else { 0.0 }) + .collect(); + let mut queue = self.inner.lock().unwrap_or_else(|poisoned| poisoned.into_inner()); + queue.push_back(FarEndFrame { + rate, + channels, + samples: clean, + }); + let mut total: usize = queue.iter().map(|frame| frame.samples.len()).sum(); + while total > FAR_END_QUEUE_LIMIT { + let Some(dropped) = queue.pop_front() else { break }; + total -= dropped.samples.len(); + } + } + + /// DSP 线程调用：取走全部待处理参考帧（保持到达顺序）。 + fn drain(&self) -> Vec { + let mut queue = self.inner.lock().unwrap_or_else(|poisoned| poisoned.into_inner()); + queue.drain(..).collect() + } + + fn is_empty(&self) -> bool { + self.inner + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .is_empty() + } +} + +const ECHO_FILTER_TAPS: usize = 1_024; +const ECHO_ADAPTATION_RATE: f32 = 0.12; +const ECHO_DOUBLE_TALK_RATIO: f32 = 2.5; +const ECHO_MIN_REFERENCE_POWER: f32 = 0.000_001; + +/// 轻量回声消除：有界归一化 LMS（沿用 #1228 的实现）。 +/// +/// 学的是「扬声器 → 麦克风」这条声学路径；近端能量明显强于估计回声（双讲）时 +/// 暂停自适应，避免把说话人自己的声音学掉。单核开销约 3%，弱机可用。 +#[derive(Debug, Clone)] +struct LightEchoCanceller { + coefficients: Vec, + history: Vec, + history_cursor: usize, +} + +impl Default for LightEchoCanceller { + fn default() -> Self { + Self { + coefficients: vec![0.0; ECHO_FILTER_TAPS], + history: vec![0.0; ECHO_FILTER_TAPS], + history_cursor: 0, + } + } +} + +impl LightEchoCanceller { + fn reset(&mut self) { + self.coefficients.fill(0.0); + self.history.fill(0.0); + self.history_cursor = 0; + } + + fn process(&mut self, microphone: &[f32], reference: &[f32]) -> Vec { + if microphone.is_empty() || reference.is_empty() { + return microphone.to_vec(); + } + let mut output = Vec::with_capacity(microphone.len()); + for (index, &sample) in microphone.iter().enumerate() { + // 块边界处重采样可能少一个样本：保留麦克风样本、等下一块重建对齐， + // 不要把上一个参考值铺满，那样会教坏滤波器。 + let Some(&reference_sample) = reference.get(index) else { + output.push(sample); + continue; + }; + self.history[self.history_cursor] = reference_sample; + let mut estimate = 0.0; + let mut reference_power = 0.0; + for tap in 0..ECHO_FILTER_TAPS { + let history_index = + (self.history_cursor + ECHO_FILTER_TAPS - tap) % ECHO_FILTER_TAPS; + let value = self.history[history_index]; + estimate += self.coefficients[tap] * value; + reference_power += value * value; + } + let error = sample - estimate; + let near_power = sample * sample; + let should_adapt = reference_power > ECHO_MIN_REFERENCE_POWER + && (estimate.abs() < 0.01 + || estimate * estimate * ECHO_DOUBLE_TALK_RATIO >= near_power + || near_power < ECHO_MIN_REFERENCE_POWER); + if should_adapt { + let normalization = reference_power + 0.000_01; + for tap in 0..ECHO_FILTER_TAPS { + let history_index = + (self.history_cursor + ECHO_FILTER_TAPS - tap) % ECHO_FILTER_TAPS; + self.coefficients[tap] += + ECHO_ADAPTATION_RATE * error * self.history[history_index] / normalization; + self.coefficients[tap] = self.coefficients[tap].clamp(-2.0, 2.0); + } + } + output.push(if error.is_finite() { error } else { 0.0 }); + self.history_cursor = (self.history_cursor + 1) % ECHO_FILTER_TAPS; + } + output + } +} + /// Shared processor for native and external microphone hosts. /// Enhancement starts with the session's settings and may be disabled under /// load. Hosts must deliver every returned tail before feeding more input or @@ -24,7 +159,15 @@ pub struct AudioProcessor { config: AudioProcessingConfig, normalizer: PcmNormalizer, enhanced: Option, + /// AEC 的扬声器参考输入（宿主在独立的回环采集流里喂）。 + far_end: Option, + /// 轻量 NLMS 抵消器（`AecMode::Light`）。 + light_aec: Option, + /// 轻量路径把参考重采样到输出采样率（16 kHz 单声道）。 + light_reference_resampler: Option, + light_reference_rate: Option, finished: bool, + monitoring: bool, } impl std::fmt::Debug for AudioProcessor { @@ -41,14 +184,88 @@ impl AudioProcessor { config, normalizer: Default::default(), enhanced: None, + far_end: None, + light_aec: matches!(config.dsp.aec_mode, crate::audio_dsp::AecMode::Light) + .then(LightEchoCanceller::default), + light_reference_resampler: None, + light_reference_rate: None, finished: false, + monitoring: false, } } + /// 把参考队列里的帧转成「与麦克风同一采样率」的单声道，供轻量抵消器使用。 + /// + /// NLMS 的抽头是按样本对齐的，采样率不一致就完全对不上 —— 早先版本把参考重 + /// 采样到 16 kHz 却拿 48 kHz 的麦克风样本做抵消，实测 ERLE ≈ 0（等于没做）。 + fn light_reference_at( + &mut self, + queue: &FarEndQueue, + rate: u32, + ) -> Option> { + if queue.is_empty() { + return None; + } + let frames = queue.drain(); + if frames.is_empty() { + return None; + } + if self.light_reference_rate != Some(rate) { + self.light_reference_rate = Some(rate); + self.light_reference_resampler = + Some(crate::audio_dsp::SpeechResampler::new(rate, rate)); + } + let mut reference = Vec::new(); + for frame in frames { + let mono = downmix_to_mono(&frame.samples, frame.channels); + if let Some(resampler) = self.light_reference_resampler.as_mut() { + resampler.push_into(&mono, false, &mut reference); + } + } + (!reference.is_empty()).then_some(reference) + } + + /// 会话结束时清掉抵消器状态，避免上一场的声学路径带到下一场。 + pub fn reset_echo_cancellers(&mut self) { + if let Some(canceller) = self.light_aec.as_mut() { + canceller.reset(); + } + self.light_reference_resampler = None; + self.light_reference_rate = None; + } + + /// 接上扬声器回环参考队列。只有 AEC 打开且宿主真的采到了输出音频时才有意义； + /// 没接或队列为空时 AEC 按「无参考」处理（MicYou 内部会退化）。 + pub fn attach_far_end(&mut self, queue: FarEndQueue) { + self.far_end = Some(queue); + } + pub fn reset(&mut self) { *self = Self::new(self.config); } + /// Visual analysis is opt-in and does not open a microphone or retain PCM. + pub fn set_monitoring_enabled(&mut self, enabled: bool) { + self.monitoring = enabled; + if let Some(stream) = &mut self.enhanced { + stream.dsp.set_spectrum_enabled(enabled); + } + } + + pub fn monitoring_sample_rate(&self) -> Option { + self.enhanced.as_ref().map(|stream| stream.work_rate) + } + + pub fn spectrums(&self) -> Option<(Vec, Vec)> { + self.monitoring + .then(|| { + self.enhanced + .as_ref() + .map(|stream| stream.dsp.get_spectrums()) + }) + .flatten() + } + /// Flush delayed enhanced samples before switching to basic normalization. /// The change lasts until the processor is replaced for a new capture. pub fn disable_enhancement(&mut self) -> Option { @@ -71,12 +288,30 @@ impl AudioProcessor { if self.finished || input.is_empty() || channels == 0 || rate == 0 { return None; } - if !self.config.enhancement_enabled { + // 轻量 AEC 在主链之前做：抵消后的信号再进降噪/增益，顺序与 MicYou 的 + // 链路一致（AEC 必须拿到尚未被其它处理改过的信号）。 + let mut cancelled: Option> = None; + if self.light_aec.is_some() { + let reference = self + .far_end + .clone() + .and_then(|queue| self.light_reference_at(&queue, rate)); + if let (Some(reference), Some(canceller)) = (reference, self.light_aec.as_mut()) { + cancelled = Some(canceller.process(input, &reference)); + } + } + let input = cancelled.as_deref().unwrap_or(input); + + // 模型 AEC（`AecMode::Model`）走 MicYou 链路；增强关掉时也要能单独跑。 + let aec_standalone = matches!(self.config.dsp.aec_mode, crate::audio_dsp::AecMode::Model) + && self.far_end.is_some(); + if !self.config.enhancement_enabled && !aec_standalone { return self.normalizer.process(input, channels, rate); } let stream = self .enhanced .get_or_insert_with(|| EnhancedStream::new(self.config.dsp, channels, rate)); + stream.dsp.set_spectrum_enabled(self.monitoring); if stream.channels != channels || stream.rate != rate { log::error!("audio format changed during an enhancement session; reset required"); return None; @@ -89,12 +324,26 @@ impl AudioProcessor { } })); let complete = stream.interleaved.len() / channels * channels; - let mono = downmix_to_mono(&stream.interleaved[..complete], channels); + stream.mono.clear(); + stream.mono.extend( + stream.interleaved[..complete] + .chunks_exact(channels) + .map(|frame| frame.iter().sum::() / channels as f32), + ); stream.interleaved.drain(..complete); - stream.input_frames += mono.len() as u64; - let working = stream.input_resampler.push(&mono, false); - let processed = stream.frames(&working, false); - pcm_chunk(&processed) + stream.input_frames += stream.mono.len() as u64; + // 参考音频必须早于本帧进入 AEC：MicYou 用最近的历史做自适应滤波。 + if let Some(far_end) = self.far_end.as_ref().filter(|queue| !queue.is_empty()) { + let frames = far_end.drain(); + stream.push_far_end(frames); + } + stream.working.clear(); + if let Some(resampler) = &mut stream.input_resampler { + resampler.push_into(&stream.mono, false, &mut stream.working); + } else { + stream.working.extend_from_slice(&stream.mono); + } + pcm_chunk(stream.frames(false)) } /// Idempotent end-of-stream barrier. Flushes fractional resampling state, @@ -105,70 +354,127 @@ impl AudioProcessor { } self.finished = true; let stream = self.enhanced.as_mut()?; - let working = stream.input_resampler.push(&[], true); - let processed = stream.frames(&working, true); - pcm_chunk(&processed) + stream.working.clear(); + if let Some(resampler) = &mut stream.input_resampler { + resampler.push_into(&[], true, &mut stream.working); + } + pcm_chunk(stream.frames(true)) } } struct EnhancedStream { channels: usize, rate: u32, - input_resampler: crate::audio_dsp::SpeechResampler, - output_resampler: crate::audio_dsp::SpeechResampler, + input_resampler: Option, + output_resampler: Option, + work_rate: u32, + hop: usize, dsp: micyou_dsp::DspProcessor, interleaved: Vec, + mono: Vec, + working: Vec, + frame: Vec, + output: Vec, pending: std::collections::VecDeque, skip: usize, model_delay: usize, input_frames: u64, emitted_frames: u64, + /// 参考音频的重采样器，按输入采样率惰性重建（回环流通常固定，重建仅一次）。 + far_end_rate: Option, + far_end_resampler: Option, + far_end_pending: Vec, } impl EnhancedStream { fn new(config: crate::audio_dsp::AudioDspConfig, channels: usize, rate: u32) -> Self { let config = config.normalized(); + let work_rate = 48_000; Self { channels, rate, - input_resampler: crate::audio_dsp::SpeechResampler::new(rate, 48_000), - output_resampler: crate::audio_dsp::SpeechResampler::new(48_000, DICTATION_SAMPLE_RATE), + // Preserve the original input filter even at 48 kHz: cached + // coefficients remove redundant computation without changing + // the signal presented to the denoiser. + input_resampler: Some(crate::audio_dsp::SpeechResampler::new(rate, work_rate)), + output_resampler: (work_rate != DICTATION_SAMPLE_RATE) + .then(|| crate::audio_dsp::SpeechResampler::new(work_rate, DICTATION_SAMPLE_RATE)), + work_rate, + hop: (work_rate / 100) as usize, dsp: micyou_dsp::DspProcessor::new( std::sync::Arc::new(std::sync::RwLock::new(config.settings())), None, ), interleaved: Vec::new(), + mono: Vec::new(), + working: Vec::new(), + frame: Vec::with_capacity(480), + output: Vec::new(), pending: Default::default(), skip: config.model_delay(), model_delay: config.model_delay(), input_frames: 0, emitted_frames: 0, + far_end_rate: None, + far_end_resampler: None, + far_end_pending: Vec::new(), + } + } + + /// 把宿主采到的回环参考帧转成 48 kHz 单声道后交给 AEC。 + /// 把宿主采到的回环参考帧转成 48 kHz 单声道后交给 AEC。 + fn push_far_end(&mut self, frames: Vec) { + for frame in frames { + let mono = downmix_to_mono(&frame.samples, frame.channels); + if self.far_end_rate != Some(frame.rate) { + self.far_end_rate = Some(frame.rate); + self.far_end_resampler = + Some(crate::audio_dsp::SpeechResampler::new(frame.rate, 48_000)); + self.far_end_pending.clear(); + } + if let Some(resampler) = self.far_end_resampler.as_mut() { + resampler.push_into(&mono, false, &mut self.far_end_pending); + } + } + if self.far_end_pending.is_empty() { + return; } + let reference = std::mem::take(&mut self.far_end_pending); + self.dsp.set_far_end_audio(&reference); } - fn frames(&mut self, input: &[f32], finish: bool) -> Vec { - self.pending.extend(input); + + fn frames(&mut self, finish: bool) -> &[f32] { + self.pending.extend(&self.working); if finish { - let tail = self.pending.len().div_ceil(480) * 480 + self.model_delay; + let tail = self.pending.len().div_ceil(self.hop) * self.hop + self.model_delay; self.pending.resize(tail, 0.0); } - let mut output = Vec::new(); - while self.pending.len() >= 480 { - let mut frame: Vec = self.pending.drain(..480).collect(); - self.dsp.process(&mut frame, 1, 0.0); - let skip = self.skip.min(frame.len()); + self.output.clear(); + while self.pending.len() >= self.hop { + self.frame.clear(); + self.frame.extend(self.pending.drain(..self.hop)); + self.dsp.process(&mut self.frame, 1, 0.0); + let skip = self.skip.min(self.frame.len()); self.skip -= skip; - output.extend(self.output_resampler.push(&frame[skip..], false)); + if let Some(resampler) = &mut self.output_resampler { + resampler.push_into(&self.frame[skip..], false, &mut self.output); + } else { + self.output.extend_from_slice(&self.frame[skip..]); + } } if finish { - output.extend(self.output_resampler.push(&[], true)); + if let Some(resampler) = &mut self.output_resampler { + resampler.push_into(&[], true, &mut self.output); + } } // The last padded model frame may produce additional samples. Bound // output by real capture duration, independently of callback sizes. let valid = (self.input_frames * DICTATION_SAMPLE_RATE as u64).div_ceil(self.rate as u64); - output.truncate(valid.saturating_sub(self.emitted_frames) as usize); - self.emitted_frames += output.len() as u64; - output + self.output + .truncate(valid.saturating_sub(self.emitted_frames) as usize); + self.emitted_frames += self.output.len() as u64; + &self.output } } @@ -354,6 +660,17 @@ fn downmix_to_mono(interleaved: &[f32], channels: usize) -> Vec { .collect() } +/// f32 交织采样 → 小端 i16 字节流，供宿主把「处理前」的原始 PCM 也写成 WAV +/// （调试时与处理后那份做 A/B 对照）。量化规则与 [`AudioProcessor`] 内部一致。 +pub fn pcm_f32_to_i16_le(samples: &[f32]) -> Vec { + let mut bytes = Vec::with_capacity(samples.len() * 2); + for sample in samples { + let quantized = (sample.clamp(-1.0, 1.0) * i16::MAX as f32) as i16; + bytes.extend_from_slice(&quantized.to_le_bytes()); + } + bytes +} + fn quantize_to_i16_le(samples: &[f32]) -> (Vec, f32) { let mut bytes = Vec::with_capacity(samples.len() * 2); let mut square_sum = 0.0_f64; diff --git a/openless-all/app/crates/openless-core/src/audio/dsp_tests.rs b/openless-all/app/crates/openless-core/src/audio/dsp_tests.rs index bd04ee7be..d1eae9371 100644 --- a/openless-all/app/crates/openless-core/src/audio/dsp_tests.rs +++ b/openless-all/app/crates/openless-core/src/audio/dsp_tests.rs @@ -154,7 +154,7 @@ fn partial_preferences_migrate_and_invalid_controls_are_bounded() { config.noise_strength = 255; config.processing_order.fill(crate::AudioDspStage::Agc); let normalized = config.normalized(); - assert_eq!(normalized.gain_db, 30); + assert_eq!(normalized.gain_db, 50); assert_eq!(normalized.noise_strength, 100); assert_eq!( normalized.processing_order, @@ -195,3 +195,111 @@ fn default_agc_does_not_amplify_denoised_room_noise_back_to_original_level() { ); } } + +#[test] +fn native_48k_capture_preserves_input_filter_for_every_denoiser() { + for algorithm in [ + NoiseReduction::Off, + NoiseReduction::Rnnoise, + NoiseReduction::Spectral, + NoiseReduction::Purevox, + ] { + let mut processor = AudioProcessor::new(config(algorithm)); + processor.process(&signal(960), 1, 48_000); + let stream = processor.enhanced.as_ref().expect("enhanced stream"); + assert_eq!(stream.work_rate, 48_000); + assert_eq!(stream.hop, 480); + assert!(stream.input_resampler.is_some()); + assert!(stream.output_resampler.is_some()); + } +} + +/// 老 preferences 里 `processingOrder` 只有 6 个阶段（没有 AEC）。数组长度从 6 +/// 变 7 之后必须仍能反序列化，缺的 AEC 补在默认位置，否则升级后用户的增强配置 +/// 会被整个丢掉。 +#[test] +fn legacy_six_stage_processing_order_still_deserializes() { + let legacy = r#"{ + "noiseReduction": "rnnoise", + "noiseStrength": 80, + "gainDb": 4, + "agcEnabled": true, + "agcTarget": 35, + "agcAttack": 50, + "agcDecay": 50, + "vadEnabled": false, + "vadThresholdDb": -50, + "dereverbLevel": 0, + "equalizerEnabled": false, + "equalizerPreampDb": 0, + "equalizerBandsDb": [0,0,0,0,0,0,0,0,0,0], + "processingOrder": ["noiseReduction","dereverb","equalizer","amplifier","agc","vad"] + }"#; + let config: crate::AudioDspConfig = serde_json::from_str(legacy).unwrap(); + assert_eq!(config.aec_mode, crate::AecMode::Off, "AEC 默认关"); + assert_eq!(config.output_buffer_ms, 300); + assert_eq!(config.gain_db, 4, "老配置里的值必须保留"); + assert_eq!( + config.processing_order, + [ + // AEC 补到最前，老配置的 6 级顺序原样保留。 + crate::AudioDspStage::AcousticEchoCancellation, + crate::AudioDspStage::NoiseReduction, + crate::AudioDspStage::Dereverb, + crate::AudioDspStage::Equalizer, + crate::AudioDspStage::Amplifier, + crate::AudioDspStage::Agc, + crate::AudioDspStage::Vad, + ], + "老配置的 6 级顺序必须保留，AEC 前置补齐" + ); +} + +#[test] +fn aec_and_buffer_reach_the_dsp_settings() { + let config = crate::AudioDspConfig { + aec_mode: crate::AecMode::Model, + output_buffer_ms: 900, + ..Default::default() + }; + let settings = config.settings(); + assert!(settings.aec_enabled, "只有 Model 模式才走 MicYou 的 aec7"); + assert_eq!(settings.output_buffer_ms, 900); + // 越界值必须夹回 MicYou 接受的范围。 + let clamped = crate::AudioDspConfig { + output_buffer_ms: 5000, + ..Default::default() + } + .normalized(); + assert_eq!(clamped.output_buffer_ms, 1200); +} + + +/// 参考队列限长：AEC 只关心最近的扬声器信号，堆积只会把 AEC 拖慢。 +#[test] +fn far_end_queue_drops_oldest_frames_beyond_the_limit() { + let queue = crate::FarEndQueue::default(); + // 每帧 48 kHz 单声道 1 秒 = 48000 样本；塞 3 秒应当留下最近 2 秒。 + for _ in 0..3 { + queue.push(&vec![0.1_f32; 48_000], 1, 48_000); + } + let frames = queue.drain(); + let total: usize = frames.iter().map(|frame| frame.samples.len()).sum(); + assert!(total <= 96_000, "队列长度应被限制在 2 秒内，实际 {total}"); + assert_eq!(frames.len(), 2, "最旧的一秒应当被丢弃"); +} + +/// 空帧/非法采样率不该进队列；非有限值被清洗成 0。 +#[test] +fn far_end_queue_rejects_empty_and_cleans_non_finite_samples() { + let queue = crate::FarEndQueue::default(); + queue.push(&[], 1, 48_000); + queue.push(&[0.0; 8], 0, 48_000); + queue.push(&[0.0; 8], 1, 0); + assert!(queue.is_empty(), "空帧与零通道/零采样率必须被拒绝"); + + queue.push(&[f32::NAN, f32::INFINITY, 0.5], 1, 48_000); + let frames = queue.drain(); + assert_eq!(frames.len(), 1); + assert!(frames[0].samples.iter().all(|sample| sample.is_finite())); +} diff --git a/openless-all/app/crates/openless-core/src/audio_dsp.rs b/openless-all/app/crates/openless-core/src/audio_dsp.rs index 3e639acaf..102101761 100644 --- a/openless-all/app/crates/openless-core/src/audio_dsp.rs +++ b/openless-all/app/crates/openless-core/src/audio_dsp.rs @@ -11,9 +11,26 @@ pub enum NoiseReduction { Spectral, } +/// 回声消除算法。 +/// +/// - `Off`：不做回声消除。 +/// - `Light`：Core 自带的归一化 LMS（单核约 3%），对稳态回声有效，双讲时暂停自适应。 +/// - `Model`：MicYou 的 aec7 神经网络（单核约 50%），非线性/强混响下更强，弱机慎用。 +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum AecMode { + #[default] + Off, + Light, + Model, +} + #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] pub enum AudioDspStage { + /// 声学回声消除：需要远端参考信号，只有拿到 far-end 时才真正生效（MicYou 的 + /// 链路里它是第一级）。 + AcousticEchoCancellation, NoiseReduction, Dereverb, Equalizer, @@ -25,6 +42,7 @@ pub enum AudioDspStage { impl AudioDspStage { fn name(self) -> &'static str { match self { + Self::AcousticEchoCancellation => "AEC", Self::NoiseReduction => "NoiseReduction", Self::Dereverb => "Dereverb", Self::Equalizer => "Equalizer", @@ -39,6 +57,10 @@ impl AudioDspStage { #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[serde(default, rename_all = "camelCase")] pub struct AudioDspConfig { + /// 声学回声消除算法。老版本 preferences 里存的是布尔 `aecEnabled`， + /// 反序列化时 `true` 映射为 `Light`（轻量、弱机可用），`false` 映射为 `Off`。 + #[serde(default, deserialize_with = "deserialize_aec_mode")] + pub aec_mode: AecMode, pub noise_reduction: NoiseReduction, pub noise_strength: u8, pub gain_db: i8, @@ -54,13 +76,76 @@ pub struct AudioDspConfig { pub equalizer_enabled: bool, pub equalizer_preamp_db: i8, pub equalizer_bands_db: [i8; 10], - pub processing_order: [AudioDspStage; 6], + /// 输出环形缓冲的余量（毫秒）。设备回调抖动大时调大，端到端延迟会跟着变大。 + /// MicYou 的范围是 100..=1200，默认 300。 + pub output_buffer_ms: u16, + /// 处理链路顺序，7 个阶段各一次。老版本 preferences 里存的是 6 个阶段（没有 + /// AEC），反序列化时把 AEC 前插补齐，不会因为数组长度变化而丢配置。 + #[serde( + default = "default_processing_order", + deserialize_with = "deserialize_processing_order" + )] + pub processing_order: [AudioDspStage; 7], + /// Android AudioRecord effects. Other hosts ignore these local choices. + pub android_aec_enabled: bool, + pub android_noise_suppression_enabled: bool, + pub android_agc_enabled: bool, +} + +fn default_processing_order() -> [AudioDspStage; 7] { + AudioDspConfig::default().processing_order +} + +fn deserialize_processing_order<'de, D>( + deserializer: D, +) -> Result<[AudioDspStage; 7], D::Error> +where + D: serde::Deserializer<'de>, +{ + let stored = Vec::::deserialize(deserializer)?; + let defaults = default_processing_order(); + // 老配置（6 级，没有 AEC）：把 AEC 补到最前面，其余顺序原样保留 —— 这既是 + // MicYou 的默认位置（AEC 必须拿到尚未被降噪改过的信号），也不会制造重复项 + // 而触发 `normalized()` 的「重复即重置」。 + if stored.len() == defaults.len() - 1 && !stored.contains(&AudioDspStage::AcousticEchoCancellation) + { + let mut order = [AudioDspStage::AcousticEchoCancellation; 7]; + for (slot, stage) in stored.into_iter().enumerate() { + order[slot + 1] = stage; + } + return Ok(order); + } + let mut order = defaults; + for (slot, stage) in order.iter_mut().enumerate() { + // 其它长度（损坏或未来版本）：按位置取，缺的用默认，多的丢弃。 + *stage = stored.get(slot).copied().unwrap_or(defaults[slot]); + } + Ok(order) +} + +/// 接受三种写法：新枚举名（`off`/`light`/`model`）、老布尔值（`true`→`light`）。 +fn deserialize_aec_mode<'de, D>(deserializer: D) -> Result +where + D: serde::Deserializer<'de>, +{ + #[derive(Deserialize)] + #[serde(untagged)] + enum Wire { + Mode(AecMode), + Enabled(bool), + } + Ok(match Wire::deserialize(deserializer)? { + Wire::Mode(mode) => mode, + Wire::Enabled(true) => AecMode::Light, + Wire::Enabled(false) => AecMode::Off, + }) } impl Default for AudioDspConfig { fn default() -> Self { use AudioDspStage::*; Self { + aec_mode: AecMode::Off, noise_reduction: crate::audio_dsp::NoiseReduction::Rnnoise, noise_strength: 80, gain_db: 0, @@ -74,7 +159,21 @@ impl Default for AudioDspConfig { equalizer_enabled: false, equalizer_preamp_db: 0, equalizer_bands_db: [0; 10], - processing_order: [NoiseReduction, Dereverb, Equalizer, Amplifier, Agc, Vad], + output_buffer_ms: 300, + // 与 MicYou 的默认链路一致：AEC 在最前，其后是降噪、去混响、均衡、 + // 增益、AGC、VAD。 + processing_order: [ + AcousticEchoCancellation, + NoiseReduction, + Dereverb, + Equalizer, + Amplifier, + Agc, + Vad, + ], + android_aec_enabled: false, + android_noise_suppression_enabled: false, + android_agc_enabled: false, } } } @@ -82,14 +181,15 @@ impl Default for AudioDspConfig { impl AudioDspConfig { pub fn normalized(mut self) -> Self { self.noise_strength = self.noise_strength.min(100); - self.gain_db = self.gain_db.clamp(-30, 30); - self.agc_target = self.agc_target.clamp(5, 90); + self.gain_db = self.gain_db.clamp(-50, 50); + self.agc_target = self.agc_target.min(100); self.agc_attack = self.agc_attack.clamp(1, 100); self.agc_decay = self.agc_decay.clamp(1, 100); - self.vad_threshold_db = self.vad_threshold_db.clamp(-80, -10); + self.vad_threshold_db = self.vad_threshold_db.clamp(-100, 0); self.dereverb_level = self.dereverb_level.min(100); self.equalizer_preamp_db = self.equalizer_preamp_db.clamp(-20, 20); - self.equalizer_bands_db = self.equalizer_bands_db.map(|gain| gain.clamp(-12, 12)); + self.equalizer_bands_db = self.equalizer_bands_db.map(|gain| gain.clamp(-15, 15)); + self.output_buffer_ms = self.output_buffer_ms.clamp(100, 1200); if self .processing_order .iter() @@ -117,6 +217,8 @@ impl AudioDspConfig { pub(crate) fn settings(self) -> micyou_dsp::AudioDspSettings { let c = self.normalized(); micyou_dsp::AudioDspSettings { + aec_enabled: matches!(c.aec_mode, AecMode::Model), + output_buffer_ms: u32::from(c.output_buffer_ms), gain: c.gain_db as f32, ns_enabled: c.noise_reduction != NoiseReduction::Off && c.noise_strength > 0, ns_type: match c.noise_reduction { @@ -140,7 +242,6 @@ impl AudioDspConfig { gains: c.equalizer_bands_db.map(f32::from).to_vec(), }, processing_chain: c.processing_order.map(|stage| stage.name().into()).to_vec(), - ..Default::default() } } } @@ -154,12 +255,33 @@ pub(crate) struct SpeechResampler { base: i64, total: u64, emitted: u64, + kernels: Vec, +} + +struct ResampleKernel { + weights: [f64; 48], + sum: f64, } impl SpeechResampler { const HALF: i64 = 24; pub fn new(source_rate: u32, target_rate: u32) -> Self { + let mut a = source_rate; + let mut b = target_rate; + while b != 0 { + (a, b) = (b, a % b); + } + let phases = target_rate / a; + // Common audio clocks need at most 640 phases. Bound uncommon, + // relatively-prime rates instead of allocating megabytes per stream. + let kernels = if phases <= 1024 { + (0..phases) + .map(|phase| Self::kernel(source_rate, target_rate, phase as f64 / phases as f64)) + .collect() + } else { + Vec::new() + }; Self { source_rate, target_rate, @@ -167,15 +289,32 @@ impl SpeechResampler { base: 0, total: 0, emitted: 0, + kernels, } } - pub fn push(&mut self, samples: &[f32], finish: bool) -> Vec { + fn kernel(source_rate: u32, target_rate: u32, fraction: f64) -> ResampleKernel { + let cutoff = (target_rate as f64 / source_rate as f64).min(1.0) * 0.94; + let weights = std::array::from_fn(|tap| { + let distance = tap as f64 - Self::HALF as f64 + 1.0 - fraction; + let x = std::f64::consts::PI * distance * cutoff; + let sinc = if x.abs() < 1e-12 { 1.0 } else { x.sin() / x }; + let window = 0.5 + 0.5 * (std::f64::consts::PI * distance / Self::HALF as f64).cos(); + sinc * window * cutoff + }); + ResampleKernel { + sum: weights.iter().sum(), + weights, + } + } + + /// Append into caller-owned scratch storage; steady-state processing + /// does not allocate a new vector for each 10 ms model frame. + pub fn push_into(&mut self, samples: &[f32], finish: bool, output: &mut Vec) { self.samples .extend(samples.iter().map(|s| if s.is_finite() { *s } else { 0.0 })); self.total += samples.len() as u64; let limit = (self.total * self.target_rate as u64).div_ceil(self.source_rate as u64); - let mut output = Vec::new(); while self.emitted < limit { let numerator = self.emitted * self.source_rate as u64; let center = (numerator / self.target_rate as u64) as i64; @@ -183,25 +322,26 @@ impl SpeechResampler { break; } let fraction = (numerator % self.target_rate as u64) as f64 / self.target_rate as f64; - let cutoff = (self.target_rate as f64 / self.source_rate as f64).min(1.0) * 0.94; + let uncached; + let kernel = if self.kernels.is_empty() { + uncached = Self::kernel(self.source_rate, self.target_rate, fraction); + &uncached + } else { + let phase = ((numerator % self.target_rate as u64) * self.kernels.len() as u64 + / self.target_rate as u64) as usize; + &self.kernels[phase] + }; let mut value = 0.0; - let mut weight_sum = 0.0; - for index in center - Self::HALF + 1..=center + Self::HALF { - let distance = (index - center) as f64 - fraction; - let x = std::f64::consts::PI * distance * cutoff; - let sinc = if x.abs() < 1e-12 { 1.0 } else { x.sin() / x }; - let window = - 0.5 + 0.5 * (std::f64::consts::PI * distance / Self::HALF as f64).cos(); - let weight = sinc * window * cutoff; + for (tap, weight) in kernel.weights.iter().enumerate() { + let index = center - Self::HALF + 1 + tap as i64; let sample = if index >= self.base && index < self.total as i64 { self.samples[(index - self.base) as usize] as f64 } else { 0.0 }; value += sample * weight; - weight_sum += weight; } - output.push((value / weight_sum) as f32); + output.push((value / kernel.sum) as f32); self.emitted += 1; } let next_center = (self.emitted * self.source_rate as u64 / self.target_rate as u64) as i64; @@ -210,6 +350,58 @@ impl SpeechResampler { let discard = discard.min(self.samples.len()); self.samples.drain(..discard); self.base += discard as i64; - output + } +} + +#[cfg(test)] +mod resampler_tests { + use super::SpeechResampler; + + #[test] + fn cached_phases_match_the_original_sinc_and_bound_unusual_clocks() { + let input: Vec = (0..997).map(|i| (i as f32 * 0.037).sin() * 0.3).collect(); + for (source, target) in [ + (16_000, 48_000), + (48_000, 16_000), + (44_100, 48_000), + (48_000, 48_000), + (48_001, 48_000), + ] { + let mut resampler = SpeechResampler::new(source, target); + assert!(resampler.kernels.len() <= 1024); + if source == 48_001 { + assert!(resampler.kernels.is_empty()); + } + let mut actual = Vec::new(); + for block in input.chunks(71) { + resampler.push_into(block, false, &mut actual); + } + resampler.push_into(&[], true, &mut actual); + assert_eq!( + actual.len(), + (input.len() as u64 * target as u64).div_ceil(source as u64) as usize + ); + for (i, sample) in actual.iter().enumerate() { + let numerator = i as u64 * source as u64; + let center = (numerator / target as u64) as i64; + let fraction = (numerator % target as u64) as f64 / target as f64; + let cutoff = (target as f64 / source as f64).min(1.0) * 0.94; + let (mut value, mut sum) = (0.0, 0.0); + for tap in center - 23..=center + 24 { + let distance = (tap - center) as f64 - fraction; + let x = std::f64::consts::PI * distance * cutoff; + let sinc = if x.abs() < 1e-12 { 1.0 } else { x.sin() / x }; + let weight = sinc + * (0.5 + 0.5 * (std::f64::consts::PI * distance / 24.0).cos()) + * cutoff; + value += input.get(tap as usize).copied().unwrap_or(0.0) as f64 * weight; + sum += weight; + } + assert!( + (*sample as f64 - value / sum).abs() < 1e-7, + "clock {source}/{target}, sample {i}" + ); + } + } } } diff --git a/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/registry.rs b/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/registry.rs index 44e900442..705e62d13 100644 --- a/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/registry.rs +++ b/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/registry.rs @@ -851,6 +851,27 @@ pub const PREFERENCE_FIELDS: &[PreferenceField] = &[ shape: PreferenceShape::Boolean, reason: "portable_setting", }, + PreferenceField { + rust_name: "debug_archive_processed", + key: "debugArchiveProcessed", + class: PreferenceClass::Portable, + shape: PreferenceShape::Boolean, + reason: "portable_setting", + }, + PreferenceField { + rust_name: "aec_reference_device_name", + key: "aecReferenceDeviceName", + class: PreferenceClass::Excluded, + shape: PreferenceShape::Text, + reason: "local_hardware", + }, + PreferenceField { + rust_name: "debug_archive_raw", + key: "debugArchiveRaw", + class: PreferenceClass::Portable, + shape: PreferenceShape::Boolean, + reason: "portable_setting", + }, PreferenceField { rust_name: "audio_recording_max_entries", key: "audioRecordingMaxEntries", diff --git a/openless-all/app/crates/openless-core/src/dictation_context.rs b/openless-all/app/crates/openless-core/src/dictation_context.rs index b82b76a0e..b986b5253 100644 --- a/openless-all/app/crates/openless-core/src/dictation_context.rs +++ b/openless-all/app/crates/openless-core/src/dictation_context.rs @@ -171,6 +171,8 @@ pub struct DictationInsertionContext { #[derive(Debug, Clone, PartialEq, Eq)] pub struct RecordingPlan { pub microphone_device_name: Option, + /// AEC 的扬声器参考源：空字符串 = 自动选默认监听源，`"off"` = 不采集参考。 + pub aec_reference_device_name: String, /// The physical hotkey generation that owns a provisional capture, when /// this context is promoted from the native staging path. pub capture_generation: Option, @@ -181,6 +183,11 @@ pub struct RecordingPlan { /// keep PCM in memory; successful-recording retention is a separate policy. pub archive_enabled: bool, pub archive_successful_recording: bool, + /// 主档案（`.wav`）写「处理后」的 PCM。调试录音两个开关都关时为 + /// false；quick note / undecided 这类功能必需档案永远为 true。 + pub archive_processed: bool, + /// 额外再写一份「处理前」的 `.raw.wav` 供 A/B 对照。 + pub archive_raw: bool, /// A quick-note or undecided Android capture must not silently continue /// without a disk archive when its path cannot be created. pub archive_required: bool, @@ -305,12 +312,21 @@ impl DictationContext { &asr, crate::shared_types::PlatformCapabilities::current().supports_local_asr, ); + // 老版本只有总开关 `recordAudioForDebug`，两个子开关是后加的。升级后这两个 + // 字段都是 false，如果直接按「两个都没开就不落盘」处理，已经打开调试录音的 + // 用户会突然一段都录不到。所以两个子开关都没开时按「保存处理后」算。 + let debug_both_off = + !preferences.debug_archive_processed && !preferences.debug_archive_raw; + let debug_keep_processed = + preferences.record_audio_for_debug && (preferences.debug_archive_processed || debug_both_off); + let debug_keep_raw = preferences.record_audio_for_debug && preferences.debug_archive_raw; Self { audio_source: options.audio_source, output_target: options.output_target, normal_archive_successful_recording: preferences.record_audio_for_debug, recording: RecordingPlan { microphone_device_name: non_blank(&preferences.microphone_device_name), + aec_reference_device_name: preferences.aec_reference_device_name.clone(), capture_generation: None, mute_during_recording: preferences.mute_during_recording, // TextStream is explicitly a live ASR module; do not let an old @@ -321,7 +337,15 @@ impl DictationContext { archive_successful_recording: matches!( options.output_target, DictationOutputTarget::QuickNote | DictationOutputTarget::Undecided - ) || preferences.record_audio_for_debug, + ) || debug_keep_processed + || debug_keep_raw, + // 功能必需档案（quick note / undecided）始终存处理后的 PCM； + // 调试录音则由两个开关决定，见上面 `debug_keep_*` 的迁移说明。 + archive_processed: matches!( + options.output_target, + DictationOutputTarget::QuickNote | DictationOutputTarget::Undecided + ) || debug_keep_processed, + archive_raw: debug_keep_raw, archive_required: matches!( options.output_target, DictationOutputTarget::QuickNote | DictationOutputTarget::Undecided @@ -928,6 +952,76 @@ mod tests { assert!(prompt.chars().count() <= ASR_PROMPT_CHAR_BUDGET); } + // 调试录音的两个开关独立生效： + // - 总开关关 → 不落盘 + // - 总开关开、子开关都关（老版本只有总开关的配置）→ 主档案存处理后 + // - 只开处理后 → 主档案存处理后 + // - 只开处理前 → 主档案存原始（功能不减），不额外多一份 + // - 都开 → 主档案 + `.raw.wav` + // quick note 这类功能必需档案不受开关影响。 + #[test] + fn debug_archive_switches_decide_what_the_host_writes() { + fn plan(processed: bool, raw: bool) -> (bool, bool, bool) { + let preferences = UserPreferences { + record_audio_for_debug: true, + debug_archive_processed: processed, + debug_archive_raw: raw, + ..UserPreferences::default() + }; + let pack = builtin_style_pack_for_mode(preferences.default_mode); + let context = DictationContext::capture( + &preferences, + &pack, + DictationProviderInvocations::new( + ProviderInvocation::for_provider(preferences.active_asr_provider.clone()), + ProviderInvocation::for_provider(preferences.active_llm_provider.clone()), + ProviderInvocation::for_provider("omni"), + ), + Vec::new(), + Vec::new(), + &DictationStartOptions::default(), + ); + ( + context.recording.archive_successful_recording, + context.recording.archive_processed, + context.recording.archive_raw, + ) + } + + // 老配置迁移：只有总开关，两个子开关都没开 → 仍然保存处理后。 + assert_eq!(plan(false, false), (true, true, false)); + assert_eq!(plan(true, false), (true, true, false)); + assert_eq!(plan(false, true), (true, false, true)); + assert_eq!(plan(true, true), (true, true, true)); + + // quick note 必须始终有处理后档案，否则播放与本地回退会失去 PCM。 + let preferences = UserPreferences { + record_audio_for_debug: false, + debug_archive_processed: false, + debug_archive_raw: false, + ..UserPreferences::default() + }; + let pack = builtin_style_pack_for_mode(preferences.default_mode); + let context = DictationContext::capture( + &preferences, + &pack, + DictationProviderInvocations::new( + ProviderInvocation::for_provider(preferences.active_asr_provider.clone()), + ProviderInvocation::for_provider(preferences.active_llm_provider.clone()), + ProviderInvocation::for_provider("omni"), + ), + Vec::new(), + Vec::new(), + &DictationStartOptions { + insert_text: false, + output_target: DictationOutputTarget::QuickNote, + ..DictationStartOptions::default() + }, + ); + assert!(context.recording.archive_processed); + assert!(!context.recording.archive_raw); + } + #[test] fn quick_note_target_forces_permanent_archive_and_skips_insertion() { let preferences = UserPreferences::default(); diff --git a/openless-all/app/crates/openless-core/src/external_audio.rs b/openless-all/app/crates/openless-core/src/external_audio.rs index 7e42b1ab7..2449362c6 100644 --- a/openless-all/app/crates/openless-core/src/external_audio.rs +++ b/openless-all/app/crates/openless-core/src/external_audio.rs @@ -38,6 +38,7 @@ struct ExternalRecordingState { consumer: Arc, progress: Arc, processor: Option, + samples: Vec, } struct ExternalActiveRecording { @@ -139,6 +140,7 @@ impl AudioRecorder for ExternalAudioRecorder { }); let session = Arc::new(ExternalRecordingSession { state: Mutex::new(ExternalRecordingState { + samples: Vec::new(), active: true, bytes_received: 0, consumer, @@ -192,14 +194,20 @@ impl AudioRecorder for ExternalAudioRecorder { } state.bytes_received = state.bytes_received.saturating_add(pcm.len() as u64); let elapsed_ms = state.bytes_received.saturating_mul(1_000) / 32_000; - if let Some(processor) = &mut state.processor { - let samples: Vec = pcm - .as_chunks::<2>() - .0 - .iter() - .map(|pair| i16::from_le_bytes([pair[0], pair[1]]) as f32 / 32768.0) - .collect(); - if let Some(chunk) = processor.process(&samples, 1, crate::audio::DICTATION_SAMPLE_RATE) + if let ExternalRecordingState { + processor: Some(processor), + samples, + .. + } = &mut *state + { + samples.clear(); + samples.extend( + pcm.as_chunks::<2>() + .0 + .iter() + .map(|pair| i16::from_le_bytes([pair[0], pair[1]]) as f32 / 32768.0), + ); + if let Some(chunk) = processor.process(samples, 1, crate::audio::DICTATION_SAMPLE_RATE) { state.consumer.consume_pcm_chunk(&chunk.pcm_i16_le); if let Some(archive) = &session.archive { diff --git a/openless-all/app/crates/openless-core/src/lib.rs b/openless-all/app/crates/openless-core/src/lib.rs index 1f363893f..c54880299 100644 --- a/openless-all/app/crates/openless-core/src/lib.rs +++ b/openless-all/app/crates/openless-core/src/lib.rs @@ -10,7 +10,7 @@ pub mod api; pub mod asr; pub mod audio; pub mod audio_dsp; -pub use audio_dsp::{AudioDspConfig, AudioDspStage, NoiseReduction}; +pub use audio_dsp::{AecMode, AudioDspConfig, AudioDspStage, NoiseReduction}; pub mod auxiliary; pub mod cli; mod cloud_providers; @@ -227,8 +227,8 @@ pub use api::{ QaVoiceCaptureSession, StartupSnapshot, VoiceTranscriptionSession, }; pub use audio::{ - encode_dictation_wav, AudioProcessingConfig, AudioProcessor, NormalizedPcmChunk, - PcmNormalizer, DICTATION_SAMPLE_RATE, + encode_dictation_wav, pcm_f32_to_i16_le, AudioProcessingConfig, AudioProcessor, FarEndQueue, + NormalizedPcmChunk, PcmNormalizer, DICTATION_SAMPLE_RATE, }; pub use auxiliary::{ AsrCallLabel, AuxiliaryApi, RepolishRequest, RetranscriptionFailure, RetranscriptionResult, diff --git a/openless-all/app/crates/openless-core/src/shared_types.rs b/openless-all/app/crates/openless-core/src/shared_types.rs index ea06f2ee9..4aa888205 100644 --- a/openless-all/app/crates/openless-core/src/shared_types.rs +++ b/openless-all/app/crates/openless-core/src/shared_types.rs @@ -870,6 +870,20 @@ pub struct UserPreferences { /// policy as `history_retention_days`. #[serde(default)] pub record_audio_for_debug: bool, + /// 调试录音里保留「处理后」的音频（降噪/增益/AGC 之后）。占用现有档案名 + /// `.wav`，quick note 播放与本地 ASR 回退都用它。默认 false： + /// 只有显式选择才会落盘。 + #[serde(default)] + pub debug_archive_processed: bool, + /// 调试录音里额外保留「处理前」的原始音频，写成同目录的 + /// `.raw.wav`，用于和 `debug_archive_processed` 的那份做 A/B 对照。 + /// 只选它、不选处理后时，主档案直接存原始 PCM（功能不受影响）。 + #[serde(default)] + pub debug_archive_raw: bool, + /// AEC 的扬声器参考源（回环/监听设备名）。空字符串 = 自动选默认扬声器的 + /// 监听源；`"off"` = 不采集参考（AEC 退化为无参考）。 + #[serde(default)] + pub aec_reference_device_name: String, /// How many recent wav files `recordings/` keeps (newest first by mtime). /// `None` = follow `HISTORY_CAP` (200); `Some(n)` clamps to 1..=200. /// Trimmed before each new session. Lets users tune combinations like @@ -1225,6 +1239,12 @@ struct UserPreferencesWire { #[serde(default)] record_audio_for_debug: bool, #[serde(default)] + debug_archive_processed: bool, + #[serde(default)] + debug_archive_raw: bool, + #[serde(default)] + aec_reference_device_name: String, + #[serde(default)] audio_recording_max_entries: Option, #[serde(default)] quick_note_export_directory: String, @@ -1417,6 +1437,9 @@ impl Default for UserPreferencesWire { auto_update_check: prefs.auto_update_check, history_max_entries: prefs.history_max_entries, record_audio_for_debug: prefs.record_audio_for_debug, + debug_archive_processed: prefs.debug_archive_processed, + debug_archive_raw: prefs.debug_archive_raw, + aec_reference_device_name: prefs.aec_reference_device_name.clone(), audio_recording_max_entries: prefs.audio_recording_max_entries, quick_note_export_directory: prefs.quick_note_export_directory.clone(), marketplace_base_url: prefs.marketplace_base_url, @@ -1654,6 +1677,9 @@ impl<'de> Deserialize<'de> for UserPreferences { auto_update_check: wire.auto_update_check, history_max_entries: wire.history_max_entries, record_audio_for_debug: wire.record_audio_for_debug, + debug_archive_processed: wire.debug_archive_processed, + debug_archive_raw: wire.debug_archive_raw, + aec_reference_device_name: wire.aec_reference_device_name, audio_recording_max_entries: wire.audio_recording_max_entries, quick_note_export_directory: wire.quick_note_export_directory, marketplace_base_url: wire.marketplace_base_url, @@ -2034,6 +2060,9 @@ impl Default for UserPreferences { auto_update_check: true, history_max_entries: None, record_audio_for_debug: false, + debug_archive_processed: false, + debug_archive_raw: false, + aec_reference_device_name: String::new(), audio_recording_max_entries: None, quick_note_export_directory: String::new(), marketplace_base_url: String::new(), diff --git a/openless-all/app/harmony/ipc-audit.json b/openless-all/app/harmony/ipc-audit.json index d051d9e62..a07c01ce3 100644 --- a/openless-all/app/harmony/ipc-audit.json +++ b/openless-all/app/harmony/ipc-audit.json @@ -1,8 +1,8 @@ { "scope": "Static shared IPC command coverage; excludes event subscriptions, dynamic commands and desktop-only calls outside src/lib/ipc.", - "total": 198, + "total": 200, "wired": 103, - "missing": 95, + "missing": 97, "commands": [ { "name": "accept_pending_correction", @@ -452,6 +452,13 @@ ], "implementation": "rust" }, + { + "name": "get_android_audio_effects_status", + "calledFrom": [ + "src/lib/ipc/android-audio.ts" + ], + "implementation": "unimplemented" + }, { "name": "get_capsule_snapshot", "calledFrom": [ @@ -1089,6 +1096,13 @@ ], "implementation": "rust" }, + { + "name": "set_android_audio_monitoring", + "calledFrom": [ + "src/lib/ipc/android-audio.ts" + ], + "implementation": "unimplemented" + }, { "name": "set_capsule_language_menu", "calledFrom": [ diff --git a/openless-all/app/linux-egui/package-revision b/openless-all/app/linux-egui/package-revision index b4f334f26..15c44e939 100644 --- a/openless-all/app/linux-egui/package-revision +++ b/openless-all/app/linux-egui/package-revision @@ -1 +1 @@ -141 +149 diff --git a/openless-all/app/linux-egui/src/audio.rs b/openless-all/app/linux-egui/src/audio.rs index adb75bc7a..e02325fbf 100644 --- a/openless-all/app/linux-egui/src/audio.rs +++ b/openless-all/app/linux-egui/src/audio.rs @@ -50,6 +50,12 @@ struct LinuxRecordingOptions { require_archive: bool, mute_during_recording: bool, audio_processing: openless_core::AudioProcessingConfig, + /// 主档案（`.wav`）写「处理后」的 PCM。 + archive_processed: bool, + /// 额外再写一份「处理前」的 `.raw.wav` 做 A/B 对照。 + archive_raw: bool, + /// AEC 扬声器参考源：空 = 自动，`"off"` = 不采集。 + aec_reference_device_name: String, } #[cfg(target_os = "linux")] @@ -58,6 +64,12 @@ struct LinuxAudioThreadArgs { consumer: Arc, progress: Arc, writer: Option>>, + /// 主档案是否写「处理后」的 PCM；false 时主档案存原始 PCM。 + write_processed: bool, + /// 「处理前」的额外档案（只在同时要两份时创建）。 + raw_writer: Option>>, + /// AEC 的扬声器参考队列（None = 本次录音不接参考）。 + far_end: Option, audio_processing: openless_core::AudioProcessingConfig, stop: Arc, runtime_error: Arc>>, @@ -76,6 +88,9 @@ impl AudioProcessingWorker { consumer: Arc, progress: Arc, writer: Option>>, + write_processed: bool, + raw_writer: Option>>, + far_end: Option, ) -> Result<(Self, SyncSender), BackendError> { let (sender, receiver) = sync_channel::(AUDIO_PROCESSING_QUEUE_CAPACITY); let started = std::time::Instant::now(); @@ -83,18 +98,47 @@ impl AudioProcessingWorker { .name("openless-linux-audio-processor".to_string()) .spawn(move || { let mut processor = openless_core::AudioProcessor::new(processing); + if let Some(queue) = far_end { + processor.attach_far_end(queue); + } + // 处理前的原始 PCM：只在需要时量化，避免常态多一次分配。 + let write_raw = + |frame: &RawAudioFrame, sink: &Option>>| { + let Some(writer) = sink.as_ref() else { + return; + }; + let pcm = openless_core::pcm_f32_to_i16_le(&frame.samples); + if let Err(error) = writer + .lock() + .expect("Linux WAV writer lock poisoned") + .append(&pcm) + { + log::warn!("Linux raw recording write failed: {error}"); + } + }; while let Ok(frame) = receiver.recv() { + // 「处理前」的额外档案永远拿原始帧。 + if raw_writer.is_some() { + write_raw(&frame, &raw_writer); + } + // 只要了处理前、没要处理后：主档案也存原始帧，功能（回退/quick + // note 播放）继续可用，且不会多出一份重复文件。 + if !write_processed { + write_raw(&frame, &writer); + } if let Some(chunk) = processor.process(&frame.samples, frame.channels, frame.input_sample_rate) { consumer.consume_pcm_chunk(&chunk.pcm_i16_le); - if let Some(writer) = &writer { - if let Err(error) = writer - .lock() - .expect("Linux WAV writer lock poisoned") - .append(&chunk.pcm_i16_le) - { - log::warn!("Linux recording archive write failed: {error}"); + if write_processed { + if let Some(writer) = &writer { + if let Err(error) = writer + .lock() + .expect("Linux WAV writer lock poisoned") + .append(&chunk.pcm_i16_le) + { + log::warn!("Linux recording archive write failed: {error}"); + } } } let _ = progress @@ -103,7 +147,7 @@ impl AudioProcessingWorker { } if let Some(chunk) = processor.finish() { consumer.consume_pcm_chunk(&chunk.pcm_i16_le); - if let Some(writer) = &writer { + if let Some(writer) = writer.as_ref().filter(|_| write_processed) { if let Err(error) = writer .lock() .expect("Linux WAV writer lock poisoned") @@ -163,6 +207,9 @@ impl AudioRecorder for LinuxCpalRecorder { let recordings_dir = self.recordings_dir.clone(); let require_archive = context.output_target == openless_core::DictationOutputTarget::QuickNote; + let archive_processed = context.recording.archive_processed; + let archive_raw = context.recording.archive_raw; + let aec_reference_device_name = context.recording.aec_reference_device_name.clone(); Box::pin(async move { #[cfg(target_os = "linux")] { @@ -175,6 +222,9 @@ impl AudioRecorder for LinuxCpalRecorder { require_archive, mute_during_recording, audio_processing, + archive_processed, + archive_raw, + aec_reference_device_name, }, consumer, progress, @@ -209,10 +259,16 @@ struct LinuxActiveRecording { /// Holds the output-mute guard while capture is live. Its `Drop` restores /// the sink on every terminal path (stop/cancel/error/drop/shutdown). mute: Option, + /// AEC 的扬声器参考采集；随录音一起停（字段顺序即 Drop 顺序，先停参考流再 + /// 拆主线程，避免参考线程还在往已释放的队列里写）。 + reference: Option, } impl Drop for LinuxActiveRecording { fn drop(&mut self) { + // 先显式停掉 AEC 参考流：它只在录音期间有意义，且必须在主采集线程之前 + // 结束，避免参考线程还在往已释放的队列写。 + drop(self.reference.take()); // Taking the guard here forces the field to be consumed (and therefore // restored) even if `stop()` is never reached — e.g. the handle is // dropped directly on an early error or during Core shutdown before it @@ -322,8 +378,34 @@ fn start_linux_recording( require_archive, mute_during_recording, audio_processing, + archive_processed, + archive_raw, + aec_reference_device_name, } = options; + // AEC 打开且参考没被显式关掉时，单独起一条回环采集流喂参考队列。 + let far_end = if audio_processing.enhancement_enabled + && audio_processing.dsp.aec_mode != openless_core::AecMode::Off + && aec_reference_device_name != "off" + { + Some(openless_core::FarEndQueue::default()) + } else { + None + }; + let mut reference_capture = None; + if let Some(queue) = far_end.clone() { + let preferred = (!aec_reference_device_name.is_empty()) + .then(|| aec_reference_device_name.clone()); + match start_reference_capture(preferred, queue) { + Ok(capture) => reference_capture = Some(capture), + // 没有回环源（典型是纯 ALSA）不该让整次录音失败：AEC 退化为无参考。 + Err(error) => log::warn!( + "[aec] speaker reference unavailable; continuing without it: {}", + error.message + ), + } + } + // Mute is best-effort and independent of capture availability (mirrors the // Tauri reference): if it fails we log and continue recording. If capture // later fails on this path, the guard drops here and restores the sink. @@ -343,11 +425,33 @@ fn start_linux_recording( let (startup_tx, startup_rx) = std::sync::mpsc::sync_channel(1); let stop_for_thread = Arc::clone(&stop); let runtime_error_for_thread = Arc::clone(&runtime_error); - let (writer, archive) = match recordings_dir { + // 调试录音两个开关都关时不落盘；quick note / undecided 这类功能必需档案由 + // Core 把 archive_processed 置为 true，所以这里统一按开关走。 + let wants_archive = archive_processed || archive_raw; + let (writer, archive, raw_writer) = match recordings_dir.filter(|_| wants_archive) { Some(directory) => { match LinuxWavWriter::create(directory.join(format!("{session_id}.wav"))) { Ok((writer, archive)) => { - (Some(Arc::new(std::sync::Mutex::new(writer))), Some(archive)) + // 两份都要时，另开一个 .raw.wav 存处理前的 PCM。 + let raw = if archive_processed && archive_raw { + match LinuxWavWriter::create(directory.join(format!("{session_id}.raw.wav"))) + { + Ok((raw_writer, _raw_archive)) => { + Some(Arc::new(std::sync::Mutex::new(raw_writer))) + } + Err(error) => { + log::warn!("failed to create raw recording archive: {error}"); + None + } + } + } else { + None + }; + ( + Some(Arc::new(std::sync::Mutex::new(writer))), + Some(archive), + raw, + ) } Err(error) => { if require_archive { @@ -357,7 +461,7 @@ fn start_linux_recording( )); } log::warn!("failed to create Linux recording archive: {error}"); - (None, None) + (None, None, None) } } } @@ -367,7 +471,7 @@ fn start_linux_recording( "quick note recording archive directory is missing", )) } - None => (None, None), + None => (None, None, None), }; let thread = std::thread::Builder::new() .name("openless-linux-recorder".to_string()) @@ -377,6 +481,9 @@ fn start_linux_recording( consumer, progress, writer, + write_processed: archive_processed, + raw_writer, + far_end, audio_processing, stop: stop_for_thread, runtime_error: runtime_error_for_thread, @@ -397,6 +504,7 @@ fn start_linux_recording( runtime_error, archive, mute, + reference: reference_capture, })), Ok(Err(error)) => { let _ = thread.join(); @@ -421,13 +529,24 @@ fn run_audio_thread(args: LinuxAudioThreadArgs) { consumer, progress, writer, + write_processed, + raw_writer, + far_end, audio_processing, stop, runtime_error, startup, } = args; let (processor, frame_tx) = - match AudioProcessingWorker::start(audio_processing, consumer, progress, writer) { + match AudioProcessingWorker::start( + audio_processing, + consumer, + progress, + writer, + write_processed, + raw_writer, + far_end, + ) { Ok(worker) => worker, Err(error) => { let _ = startup.send(Err(error)); @@ -504,6 +623,248 @@ fn run_audio_thread(args: LinuxAudioThreadArgs) { let _ = processor.finish(); } +/// 判断一个输入设备名是否是「扬声器回环 / 监听」源。 +/// +/// - PulseAudio / PipeWire（走 pulse 兼容层）：`.monitor`，例如 +/// `alsa_output.pci-0000_00_1f.3.analog-stereo.monitor` +/// - 部分 PipeWire 版本直接暴露为 `...output...` 的 monitor 节点 +/// - ALSA 没有回环源（除非加载 `snd-aloop`，那会表现为 `Loopback, ...`） +fn is_loopback_device_name(name: &str) -> bool { + let lower = name.to_ascii_lowercase(); + lower.contains(".monitor") + || lower.contains("monitor of") + || lower.starts_with("loopback") + || lower.contains(", loopback") +} + +/// 列出所有可用的扬声器回环源，按后端顺序（pipewire → pulseaudio → alsa）去重。 +/// +/// 用户在下拉里选的是设备名原文；空字符串表示「自动」。返回空列表说明当前 +/// 环境没有回环源（典型是纯 ALSA），此时 AEC 只能在没有参考信号的情况下工作。 +pub fn list_loopback_devices() -> Vec { + use cpal::traits::HostTrait; + + let hosts = cpal::available_hosts(); + let mut names: Vec = Vec::new(); + for backend in audio_backend_order() { + let Some(host_id) = hosts + .iter() + .copied() + .find(|id| id.name().eq_ignore_ascii_case(backend)) + else { + continue; + }; + let Ok(host) = cpal::host_from_id(host_id) else { + continue; + }; + let Ok(devices) = host.input_devices() else { + continue; + }; + for device in devices { + let name = device.to_string(); + if is_loopback_device_name(&name) && !names.contains(&name) { + names.push(name); + } + } + } + names +} + +/// 一次录音期间的扬声器参考采集。Drop 时停流并回收线程。 +#[cfg(target_os = "linux")] +pub struct ReferenceCapture { + stop: Arc, + thread: Option>, +} + +#[cfg(target_os = "linux")] +impl Drop for ReferenceCapture { + fn drop(&mut self) { + self.stop + .store(true, std::sync::atomic::Ordering::Release); + if let Some(thread) = self.thread.take() { + let _ = thread.join(); + } + } +} + +/// 启动扬声器回环采集，把参考音频喂进 AEC 队列。 +/// +/// 与麦克风采集一样按 `pipewire → pulseaudio → alsa` 顺序尝试：只有前两者提供 +/// `.monitor`；ALSA 在没有 `snd-aloop` 时必然失败，此时返回错误，调用方 +/// 只记一条日志（AEC 退化为无参考，录音本身照常进行）。 +#[cfg(target_os = "linux")] +pub fn start_reference_capture( + preferred_device_name: Option, + queue: openless_core::FarEndQueue, +) -> Result { + use cpal::traits::{DeviceTrait, HostTrait, StreamTrait}; + + let stop = Arc::new(std::sync::atomic::AtomicBool::new(false)); + let stop_for_thread = Arc::clone(&stop); + let (startup_tx, startup_rx) = std::sync::mpsc::sync_channel(1); + let thread = std::thread::Builder::new() + .name("openless-linux-aec-reference".to_string()) + .spawn(move || { + let hosts = cpal::available_hosts(); + let mut stream_slot: Option = None; + let mut last_error = None; + for backend in audio_backend_order() { + let Some(host_id) = hosts + .iter() + .copied() + .find(|id| id.name().eq_ignore_ascii_case(backend)) + else { + continue; + }; + let Ok(host) = cpal::host_from_id(host_id) else { + continue; + }; + let Ok(devices) = host.input_devices() else { + continue; + }; + let wanted = preferred_device_name.as_deref().filter(|name| !name.is_empty()); + let device = devices.into_iter().find(|device| { + let name = device.to_string(); + match wanted { + Some(wanted) => name == wanted, + None => is_loopback_device_name(&name), + } + }); + let Some(device) = device else { + last_error = Some(BackendError::new( + BackendErrorCode::Platform, + format!("{backend}: no speaker loopback source"), + )); + continue; + }; + let Ok(config) = device.default_input_config() else { + last_error = Some(BackendError::new( + BackendErrorCode::Platform, + format!("{backend}: loopback device has no input config"), + )); + continue; + }; + let channels = usize::from(config.channels()); + let rate = config.sample_rate(); + let queue_for_stream = queue.clone(); + let built = match config.sample_format() { + cpal::SampleFormat::F32 => device.build_input_stream( + config.config(), + move |data: &[f32], _| queue_for_stream.push(data, channels, rate), + |error| log::warn!("[aec] reference stream error: {error}"), + None, + ), + cpal::SampleFormat::I16 => { + let queue_for_stream = queue.clone(); + device.build_input_stream( + config.config(), + move |data: &[i16], _| { + let converted: Vec = + data.iter().map(|s| *s as f32 / i16::MAX as f32).collect(); + queue_for_stream.push(&converted, channels, rate) + }, + |error| log::warn!("[aec] reference stream error: {error}"), + None, + ) + } + other => { + last_error = Some(BackendError::new( + BackendErrorCode::Unsupported, + format!("{backend}: unsupported loopback sample format {other:?}"), + )); + continue; + } + }; + match built { + Ok(stream) => { + if let Err(error) = stream.play() { + last_error = Some(classify_audio_error( + "start loopback stream", + error.to_string(), + )); + continue; + } + log::info!( + "[aec] speaker reference capturing via {backend}: {} ({rate} Hz, {channels} ch)", + device + ); + stream_slot = Some(stream); + break; + } + Err(error) => { + last_error = Some(classify_audio_error( + "build loopback stream", + error.to_string(), + )); + } + } + } + match stream_slot { + Some(stream) => { + let _ = startup_tx.send(Ok(())); + while !stop_for_thread.load(std::sync::atomic::Ordering::Acquire) { + std::thread::park_timeout(std::time::Duration::from_millis(25)); + } + drop(stream); + } + None => { + let error = last_error.unwrap_or_else(|| { + BackendError::new( + BackendErrorCode::Platform, + "no speaker loopback source is available", + ) + }); + let _ = startup_tx.send(Err(error)); + } + } + }) + .map_err(|error| { + BackendError::new( + BackendErrorCode::Platform, + format!("spawn AEC reference thread: {error}"), + ) + })?; + + match startup_rx.recv() { + Ok(Ok(())) => Ok(ReferenceCapture { + stop, + thread: Some(thread), + }), + Ok(Err(error)) => { + let _ = thread.join(); + Err(error) + } + Err(error) => { + let _ = thread.join(); + Err(BackendError::new( + BackendErrorCode::Platform, + format!("AEC reference thread exited during startup: {error}"), + )) + } + } +} + +#[cfg(all(test, target_os = "linux"))] +mod loopback_tests { + use super::is_loopback_device_name; + + #[test] + fn recognises_pulse_and_pipewire_monitor_sources() { + // PulseAudio / PipeWire 的经典形态 + assert!(is_loopback_device_name( + "alsa_output.pci-0000_00_1f.3.analog-stereo.monitor" + )); + // 部分版本带 "Monitor of" 前缀 + assert!(is_loopback_device_name("Monitor of Built-in Audio Analog Stereo")); + // ALSA 的 snd-aloop 模块 + assert!(is_loopback_device_name("Loopback, PCM Capture")); + // 普通麦克风不能被误判成回环源，否则 AEC 会拿麦克风当参考 + assert!(!is_loopback_device_name("alsa_input.pci-0000_00_1f.3.analog-stereo")); + assert!(!is_loopback_device_name("Yeti Stereo Microphone")); + } +} + #[cfg(target_os = "linux")] fn audio_backend_order() -> [&'static str; 3] { // Native desktop servers are preferred because they handle device policy, diff --git a/openless-all/app/linux-egui/src/i18n.rs b/openless-all/app/linux-egui/src/i18n.rs index 13ca6fe9e..4c29d092f 100644 --- a/openless-all/app/linux-egui/src/i18n.rs +++ b/openless-all/app/linux-egui/src/i18n.rs @@ -149,11 +149,99 @@ const fn row( // that the zh-CN column is fully populated and that every key actually // referenced resolves. pub const CATALOG: &[Msg] = &[ + Msg { key: "settings.audioDsp.eqPresets", text: row("均衡器预设", "均衡器預設", "Equalizer presets", "イコライザープリセット", "이퀄라이저 프리셋") }, + Msg { key: "settings.audioDsp.eqNormal", text: row("平直", "平直", "Flat", "フラット", "플랫") }, + Msg { key: "settings.audioDsp.eqBright", text: row("明亮人声", "明亮人聲", "Bright vocal", "明るい声", "밝은 음성") }, + Msg { key: "settings.audioDsp.eqDeep", text: row("低沉人声", "低沉人聲", "Deep voice", "低い声", "낮은 음성") }, + Msg { key: "settings.audioDsp.eqPodcast", text: row("播客", "播客", "Podcast", "ポッドキャスト", "팟캐스트") }, + Msg { key: "settings.audioDsp.eqClarity", text: row("清晰人声", "清晰人聲", "Vocal clarity", "明瞭な声", "선명한 음성") }, + Msg { key: "settings.audioDsp.eqWarm", text: row("温暖人声", "溫暖人聲", "Warm vocal", "温かい声", "따뜻한 음성") }, Msg { key: "settings.audioDsp.disabledHint", text: row("开启上方语音增强开关后可调整，当前配置已保留。", "開啟上方語音增強開關後可調整，目前設定已保留。", "Enable speech enhancement above to adjust these settings. Your configuration is retained.", "上の音声強調を有効にすると調整できます。設定は保持されています。", "위에서 음성 향상을 켜면 조정할 수 있습니다. 설정은 유지됩니다.") }, Msg { key: "settings.audioDsp.moveUp", text: row("上移处理阶段", "上移處理階段", "Move stage up", "処理を上へ移動", "처리 단계 위로 이동") }, Msg { key: "settings.audioDsp.moveDown", text: row("下移处理阶段", "下移處理階段", "Move stage down", "処理を下へ移動", "처리 단계 아래로 이동") }, + Msg { + key: "settings.audioDsp.aecStage", + text: row("回声消除", "回聲消除", "Echo cancellation", "エコーキャンセル", "에코 제거"), + }, + Msg { + key: "settings.audioDsp.echoStage", + text: row("回声消除与缓冲", "回聲消除與緩衝", "Echo cancellation and buffering", "エコーキャンセルとバッファ", "에코 제거 및 버퍼"), + }, + Msg { + key: "settings.audioDsp.aecOff", + text: row("关闭", "關閉", "Off", "オフ", "끄기"), + }, + Msg { + key: "settings.audioDsp.aecLight", + text: row("轻量（省 CPU）", "輕量（省 CPU）", "Light (low CPU)", "軽量（低 CPU）", "경량(저 CPU)"), + }, + Msg { + key: "settings.audioDsp.aecModel", + text: row("模型（更强）", "模型（更強）", "Model (stronger)", "モデル（高精度）", "모델(강화)"), + }, + Msg { + key: "settings.audioDsp.aecModelCost", + text: row( + "模型版抑制更强，但约是轻量版的 3 倍 CPU；老机器建议用轻量。", + "模型版抑制更強，但約是輕量版的 3 倍 CPU；舊機器建議用輕量。", + "The model cancels more echo but costs about 3× the CPU; prefer Light on older machines.", + "モデル版はより強く抑圧しますが CPU は約 3 倍です。古い端末では軽量版を推奨します。", + "모델 버전은 더 강하게 제거하지만 CPU를 약 3배 사용합니다. 오래된 기기에서는 경량을 권장합니다.", + ), + }, + Msg { + key: "settings.audioDsp.aecReferenceLabel", + text: row("扬声器参考源", "揚聲器參考源", "Speaker reference", "スピーカー参照", "스피커 참조"), + }, + Msg { + key: "settings.audioDsp.aecReferenceDesc", + text: row( + "采集哪台扬声器的回环信号供回声消除使用；没有回环源时 AEC 只能在没有参考的情况下工作。", + "擷取哪台揚聲器的回環訊號供回聲消除使用；沒有回環源時 AEC 只能在沒有參考的情況下運作。", + "Which speaker loopback feeds echo cancellation; without a loopback source AEC runs without a reference.", + "どのスピーカーのループバックをエコーキャンセルに使うか。ループバック元がない場合、AEC は参照なしで動作します。", + "어떤 스피커의 루프백을 에코 제거에 사용할지 선택합니다. 루프백 소스가 없으면 AEC는 참조 없이 동작합니다.", + ), + }, + Msg { + key: "settings.audioDsp.aecReferenceAuto", + text: row("自动（默认扬声器）", "自動（預設揚聲器）", "Automatic (default speaker)", "自動（既定のスピーカー）", "자동(기본 스피커)"), + }, + Msg { + key: "settings.audioDsp.aecReferenceOff", + text: row("不采集参考", "不擷取參考", "Do not capture a reference", "参照を取得しない", "참조를 수집하지 않음"), + }, + Msg { + key: "settings.audioDsp.acousticEchoCancellation", + text: row("声学回声消除（AEC）", "聲學回聲消除（AEC）", "Acoustic echo cancellation (AEC)", "音響エコーキャンセル（AEC）", "음향 에코 제거(AEC)"), + }, + Msg { + key: "settings.audioDsp.AEC_needs_a_loudspeaker_reference", + text: row( + "外放时压掉扬声器回授；没有参考信号时不会起作用，戴耳机可以关掉。", + "外放時壓掉揚聲器回授；沒有參考訊號時不會起作用，戴耳機可以關掉。", + "Suppresses loudspeaker feedback while on speaker; it does nothing without a reference signal, so leave it off with headphones.", + "スピーカー使用時の回り込みを抑えます。参照信号がないと効果はないため、ヘッドホンならオフのままで構いません。", + "스피커 사용 시 되먹임을 줄입니다. 참조 신호가 없으면 효과가 없으므로 헤드폰이면 꺼 두세요.", + ), + }, + Msg { + key: "settings.audioDsp.Output_buffer_ms", + text: row("输出缓冲区", "輸出緩衝區", "Output buffer", "出力バッファ", "출력 버퍼"), + }, + Msg { + key: "settings.audioDsp.A_larger_buffer_survives_jitter_but_adds_latency", + text: row( + "调大可吸收声卡回调抖动，代价是端到端延迟增加。默认 300 ms。", + "調大可吸收音效卡回呼抖動，代價是端到端延遲增加。預設 300 ms。", + "A larger buffer absorbs sound-card callback jitter at the cost of end-to-end latency. Default 300 ms.", + "大きくするとサウンドカードのコールバック揺らぎに強くなりますが、遅延が増えます。既定は 300 ms です。", + "크게 하면 사운드 카드 콜백 지터를 흡수하지만 지연이 늘어납니다. 기본값은 300 ms입니다.", + ), + }, Msg { key: "settings.audioDsp.noiseStage", text: row("降噪", "降噪", "Noise reduction", "ノイズ低減", "잡음 감소") }, - Msg { key: "settings.audioDsp.combStage", text: row("梳状滤波", "梳狀濾波", "Comb filter", "コムフィルター", "콤 필터") }, + Msg { key: "settings.audioDsp.roomStage", text: row("音色与房间", "音色與房間", "Tone and room", "音色と残響", "음색과 잔향") }, + Msg { key: "settings.audioDsp.combStage", text: row("梳状滤波（简易去混响）", "梳狀濾波（簡易去混響）", "Comb filter (simple dereverb)", "コムフィルター（簡易デリバーブ）", "콤 필터(간이 디버브)") }, Msg { key: "settings.audioDsp.Speech_enhancement_settings", text: row("语音增强设置", "語音增強設置", "Speech enhancement settings", "音声強調設定", "음성 향상 설정") }, Msg { key: "settings.audioDsp.RNNoise_is_lightweight_PureVox_uses_more_CPU_Changes_apply_to_the_next_recording", text: row("RNNoise 更轻量，PureVox 模型降噪耗时较高。设置在下一次录音生效。", "RNNoise 更輕量，PureVox 模型降噪耗時較高。設置在下一次錄音生效。", "RNNoise is lightweight; PureVox uses more CPU. Changes apply to the next recording.", "RNNoise は軽量、PureVox は CPU 使用量が多めです。設定は次の録音に適用されます。", "RNNoise는 가볍고 PureVox는 CPU를 더 사용합니다. 설정은 다음 녹음에 적용됩니다.") }, Msg { key: "settings.audioDsp.Noise_reduction_off", text: row("关闭降噪", "關闭降噪", "Noise reduction off", "ノイズ低減オフ", "잡음 감소 끄기") }, @@ -165,18 +253,18 @@ pub const CATALOG: &[Msg] = &[ Msg { key: "settings.audioDsp.Target_level", text: row("目标电平 %", "目標電平 %", "Target level %", "目標レベル %", "목표 레벨 %") }, Msg { key: "settings.audioDsp.Attack_speed", text: row("启动速度", "啟動速度", "Attack speed", "アタック速度", "어택 속도") }, Msg { key: "settings.audioDsp.Release_speed", text: row("释放速度", "釋放速度", "Release speed", "リリース速度", "릴리스 속도") }, - Msg { key: "settings.audioDsp.Speech_gate_does_not_stop_recording", text: row("语音门限（不自动停止录音）", "語音門限（不自動停止錄音）", "Speech gate (does not stop recording)", "音声ゲート（録音は停止しません）", "음성 게이트 (녹음을 중지하지 않음)") }, + Msg { key: "settings.audioDsp.Speech_gate_does_not_stop_recording", text: row("低于门限的片段被衰减，录音不会因此停止。", "低於門檻的片段被衰減，錄音不會因此停止。", "Passages below the threshold are attenuated; recording never stops for them.", "しきい値を下回る区間は減衰されますが、録音は停止しません。", "임계값 미만 구간은 감쇠되지만 녹음은 중단되지 않습니다.") }, Msg { key: "settings.audioDsp.Threshold_dB", text: row("门限 dB", "門限 dB", "Threshold dB", "しきい値 dB", "임계값 dB") }, Msg { key: "settings.audioDsp.A_high_threshold_can_remove_quiet_speech", text: row("门限过高可能丢失轻声。", "門限過高可能丟失輕聲。", "A high threshold can remove quiet speech.", "しきい値が高いと小さな声が失われる場合があります。", "임계값이 높으면 작은 목소리가 사라질 수 있습니다.") }, Msg { key: "settings.audioDsp.Tone_and_room_adjustments", text: row("音色与房间调整", "音色與房間調整", "Tone and room adjustments", "音色と室内音響の調整", "음색 및 실내 음향 조정") }, - Msg { key: "settings.audioDsp.The_MicYou_comb_filter_can_color_speech_Start_at_0", text: row("梳状滤波沿用 MicYou 的延迟相减方式，可能改变音色，建议从 0% 开始。", "梳狀濾波沿用 MicYou 的延遲相減方式，可能改變音色，建議從 0% 开始。", "The MicYou comb filter can color speech. Start at 0%.", "MicYou のコムフィルターは音色を変える場合があります。0% から始めてください。", "MicYou 콤 필터는 음색을 바꿀 수 있습니다. 0%부터 시작하세요.") }, + Msg { key: "settings.audioDsp.The_MicYou_comb_filter_can_color_speech_Start_at_0", text: row("把 10 ms 前的信号按比例减掉，能压掉一点早期反射，但会给音色染上金属感；从 0 开始试。真正的去混响需要频谱或模型方法，这里没做。", "把 10 ms 前的訊號按比例減掉，能壓掉一點早期反射，但會讓音色帶金屬感；從 0 開始試。真正的去混響需要頻譜或模型方法，這裡沒做。", "Subtracts the signal from 10 ms ago, trimming some early reflections at the cost of a metallic color; start at 0. Real dereverberation needs spectral or model-based methods, which this is not.", "10 ms 前の信号を差し引いて初期反射を少し抑えますが、金属的な音色になります。0 から試してください。本格的なデリバーブは周波数領域かモデル方式が必要で、ここでは未実装です。", "10 ms 전 신호를 비율로 빼서 초기 반사를 조금 줄이지만 금속성 음색이 생깁니다. 0부터 시작하세요. 실제 디버브는 주파수 영역이나 모델 방식이 필요하며 여기서는 구현되지 않았습니다.") }, Msg { key: "settings.audioDsp.Comb_filter", text: row("梳状滤波 %", "梳狀濾波 %", "Comb filter %", "コムフィルター %", "콤 필터 %") }, Msg { key: "settings.audioDsp.10_band_equalizer", text: row("十段均衡器", "十段均衡器", "10-band equalizer", "10 バンドイコライザー", "10밴드 이퀄라이저") }, Msg { key: "settings.audioDsp.Preamp_dB", text: row("前置增益 dB", "前置增益 dB", "Preamp dB", "プリアンプ dB", "프리앰프 dB") }, Msg { key: "settings.audioDsp.Processing_order", text: row("处理顺序", "處理順序", "Processing order", "処理順序", "처리 순서") }, Msg { key: "settings.audioDsp.Equalizer", text: row("均衡器", "均衡器", "Equalizer", "イコライザー", "이퀄라이저") }, Msg { key: "settings.audioDsp.Input_gain", text: row("输入增益", "輸入增益", "Input gain", "入力ゲイン", "입력 게인") }, - Msg { key: "settings.audioDsp.Speech_gate", text: row("语音门限", "語音門限", "Speech gate", "音声ゲート", "음성 게이트") }, + Msg { key: "settings.audioDsp.Speech_gate", text: row("噪声门限（不是人声检测）", "雜訊門檻（不是人聲偵測）", "Noise gate (not speech detection)", "ノイズゲート（音声検出ではありません）", "노이즈 게이트(음성 검출 아님)") }, Msg { key: "settings.audioDsp.Reset_enhancement_settings", text: row("恢复增强默认设置", "恢復增強默認設置", "Reset enhancement settings", "強調設定を初期化", "향상 설정 초기화") }, Msg { key: "settings.providers.presets.chatgptSiwc", text: row("ChatGPT Plan", "ChatGPT Plan", "ChatGPT Plan", "ChatGPT Plan", "ChatGPT Plan") }, Msg { key: "siwc.description", text: row("符合条件的文本润色和翻译使用你的 ChatGPT 计划或积分余额。ASR 仍使用原渠道；额度不足时不会自动切换到 API Key。", "符合條件的文字潤飾和翻譯使用你的 ChatGPT 方案或點數。ASR 仍使用原管道；額度不足時不會自動切換至 API Key。", "Eligible text polishing and translation use your ChatGPT plan or credits. ASR keeps its existing provider. Usage limits never switch you to an API key automatically.", "対象の文章校正と翻訳には ChatGPT プランまたはクレジットを使用します。音声認識は従来のプロバイダーを使い、上限到達時に API キーへ自動切り替えしません。", "지원되는 텍스트 다듬기와 번역에 ChatGPT 요금제 또는 크레딧을 사용합니다. 음성 인식은 기존 서비스를 사용하며 한도에 도달해도 API 키로 자동 전환하지 않습니다.") }, @@ -3809,6 +3897,34 @@ pub const CATALOG: &[Msg] = &[ "debug는 하트비트와 IPC 세부 정보를 기록합니다(문제 해결용). OpenLess를 다시 시작해야 적용됩니다.", ), }, + Msg { + key: "settings.recording.archive_processed_label", + text: row("保存处理后的音频", "儲存處理後的音訊", "Keep processed audio", "処理後の音声を保存", "처리된 오디오 저장"), + }, + Msg { + key: "settings.recording.archive_processed_desc", + text: row( + "降噪与增益之后的版本，仍保存在原来的录制文件里（quick note 播放、本地模型回退都用它）。", + "降噪與增益之後的版本，仍存於原本的錄製檔案（quick note 播放、本機模型回退都用它）。", + "The version after noise reduction and gain, still stored in the existing recording file (used by quick-note playback and the local model fallback).", + "ノイズ低減とゲイン適用後の音声で、従来の録音ファイルに保存されます（クイックノート再生とローカルモデルのフォールバックが使います）。", + "노이즈 감소와 게인 적용 후의 오디오로, 기존 녹음 파일에 저장됩니다(퀵노트 재생과 로컬 모델 폴백이 사용).", + ), + }, + Msg { + key: "settings.recording.archive_raw_label", + text: row("另存一份处理前的音频", "另存一份處理前的音訊", "Also keep pre-processing audio", "処理前の音声も保存", "처리 전 오디오도 저장"), + }, + Msg { + key: "settings.recording.archive_raw_desc", + text: row( + "同一目录下另存 <会话>.raw.wav，可与上面那份逐段对照；只开这一项时，原始音频直接存进主档案。", + "同一目錄下另存 .raw.wav，可與上面那份逐段對照；只開這一項時，原始音訊直接存進主檔案。", + "Writes .raw.wav next to it so you can compare segment by segment; with only this enabled the raw audio goes into the main archive.", + "同じフォルダに .raw.wav を追加保存し、上記と区間ごとに比較できます。これだけを有効にすると元音声がメインファイルに保存されます。", + "같은 폴더에 .raw.wav를 함께 저장해 구간별로 비교할 수 있습니다. 이것만 켜면 원본 오디오가 기본 파일에 저장됩니다.", + ), + }, Msg { key: "settings.recording.record_audio_for_debug_label", text: row( diff --git a/openless-all/app/linux-egui/src/lib.rs b/openless-all/app/linux-egui/src/lib.rs index 765508486..69e330f53 100644 --- a/openless-all/app/linux-egui/src/lib.rs +++ b/openless-all/app/linux-egui/src/lib.rs @@ -50,6 +50,8 @@ pub mod wgpu_device; pub mod siri_wgpu; pub use audio::LinuxCpalRecorder; +#[cfg(target_os = "linux")] +pub use audio::{list_loopback_devices, start_reference_capture, ReferenceCapture}; pub use audio_cue::{play_cue_start, play_cue_stop, CueTone}; pub use audio_mute::AudioMuteGuard; pub use audio_player::ClipPlayer; diff --git a/openless-all/app/linux-egui/src/linux_app/settings_save.rs b/openless-all/app/linux-egui/src/linux_app/settings_save.rs index 6bb556f03..3c1002117 100644 --- a/openless-all/app/linux-egui/src/linux_app/settings_save.rs +++ b/openless-all/app/linux-egui/src/linux_app/settings_save.rs @@ -235,6 +235,20 @@ impl OpenLessEguiApp { } frontend::view_model::SettingsField::RecordAudioForDebug => { preferences.record_audio_for_debug = !preferences.record_audio_for_debug; + // 打开主开关时默认选「处理后」，否则主开关打开了却什么都不落盘、 + // 用户以为坏了；「处理前」保持用户上一次的选择。 + if preferences.record_audio_for_debug + && !preferences.debug_archive_processed + && !preferences.debug_archive_raw + { + preferences.debug_archive_processed = true; + } + } + frontend::view_model::SettingsField::DebugArchiveProcessed => { + preferences.debug_archive_processed = !preferences.debug_archive_processed; + } + frontend::view_model::SettingsField::DebugArchiveRaw => { + preferences.debug_archive_raw = !preferences.debug_archive_raw; } frontend::view_model::SettingsField::StreamingSaveClipboard => { preferences.streaming_insert_save_clipboard = @@ -253,6 +267,28 @@ impl OpenLessEguiApp { field: frontend::view_model::SettingsComboField, index: usize, ) { + // AEC 参考源是 Core 偏好里的字符串（空 = 自动，"off" = 不采集）， + // 界面用下标表达，这里换算回去。 + if matches!(field, frontend::view_model::SettingsComboField::AecReference) { + let value = match index { + 0 => String::new(), + 1 => "off".to_string(), + other => self + .frontend_vm + .settings + .aec_reference_devices + .get(other - 2) + .cloned() + .unwrap_or_default(), + }; + self.frontend_vm.settings.aec_reference_index = index; + if let Some(preferences) = self.preferences.as_mut() { + preferences.aec_reference_device_name = value; + self.save_settings_if_dirty(); + } + return; + } + // 日志级别是宿主本地设置（`/logs/level`），不经过 Core 偏好， // 因此要放在下面 `self.preferences` 的守卫之前 —— 否则偏好还没加载时 // 连日志级别都改不了。 @@ -287,6 +323,8 @@ impl OpenLessEguiApp { match field { // 已在函数开头处理（不走 Core 偏好）。 frontend::view_model::SettingsComboField::LogLevel => {} + // 已在函数开头处理（换算成偏好里的设备名字符串）。 + frontend::view_model::SettingsComboField::AecReference => {} frontend::view_model::SettingsComboField::Theme => { preferences.theme_mode = match index { 0 => openless_core::shared_types::ThemeMode::System, diff --git a/openless-all/app/linux-egui/src/main.rs b/openless-all/app/linux-egui/src/main.rs index 63fe56c4c..223d16ce9 100644 --- a/openless-all/app/linux-egui/src/main.rs +++ b/openless-all/app/linux-egui/src/main.rs @@ -589,6 +589,8 @@ mod linux_app { hydrate_text_fields: bool, overview: OverviewState, microphones: Vec, + /// AEC 可选的扬声器回环源（缓存一次 cpal 枚举结果）。 + aec_reference_devices: Vec, /// 麦克风枚举失败的原因（设置页据此显示 `microphoneLoadError`）。 microphone_error: Option, transcript: String, @@ -853,6 +855,7 @@ mod linux_app { hydrate_text_fields: true, overview: OverviewState::Loading, microphones: Vec::new(), + aec_reference_devices: Vec::new(), microphone_error: None, transcript: String::new(), transcript_state: TranscriptAccumulator::default(), @@ -974,6 +977,7 @@ mod linux_app { service_configured: [false; 2], overview: OverviewState::Loading, microphones: Vec::new(), + aec_reference_devices: Vec::new(), microphone_error: None, transcript: String::new(), transcript_state: TranscriptAccumulator::default(), @@ -5690,6 +5694,27 @@ mod linux_app { // 文本型设置行只在偏好载入 / 外部变更时回灌一次，避免把输入中的 // 内容每帧弹回旧值。 let hydrate_text = std::mem::replace(&mut self.hydrate_text_fields, false); + // AEC 参考源：设备列表要枚举一次 cpal 主机，缓存到应用里；下标由 + // 偏好里的设备名换算（0 = 自动，1 = 不采集）。 + if self.aec_reference_devices.is_empty() && hydrate_text { + self.aec_reference_devices = openless_linux_egui::list_loopback_devices(); + } + let aec_reference_devices = self.aec_reference_devices.clone(); + let aec_reference_index = match self + .preferences + .as_ref() + .map(|prefs| prefs.aec_reference_device_name.as_str()) + .unwrap_or_default() + { + "" => 0, + "off" => 1, + current => self + .aec_reference_devices + .iter() + .position(|name| name == current) + .map_or(0, |index| index + 2), + }; + // 日志级别是宿主本地文件（不在 Core 偏好里），要在借用 vm 之前读。 let log_level_index = self .backend() @@ -5879,8 +5904,12 @@ mod linux_app { s.launch_at_login = prefs.launch_at_login; s.streaming_save_clipboard = prefs.streaming_insert_save_clipboard; s.record_audio_for_debug = prefs.record_audio_for_debug; + s.debug_archive_processed = prefs.debug_archive_processed; + s.debug_archive_raw = prefs.debug_archive_raw; // 日志级别是宿主本地文件，不在 Core 偏好里；读不到就按 info 显示。 s.log_level = log_level_index; + s.aec_reference_devices = aec_reference_devices.clone(); + s.aec_reference_index = aec_reference_index; if hydrate_text { s.history_max_entries = prefs .history_max_entries diff --git a/openless-all/app/linux-egui/src/ui/frontend/audio_dsp.rs b/openless-all/app/linux-egui/src/ui/frontend/audio_dsp.rs index 0430a5424..efd02e97e 100644 --- a/openless-all/app/linux-egui/src/ui/frontend/audio_dsp.rs +++ b/openless-all/app/linux-egui/src/ui/frontend/audio_dsp.rs @@ -1,5 +1,5 @@ use eframe::egui; -use openless_core::{AudioDspConfig, AudioDspStage, NoiseReduction}; +use openless_core::{AecMode, AudioDspConfig, AudioDspStage, NoiseReduction}; use openless_linux_egui::{tr_l10n, Lang}; pub fn settings(ui: &mut egui::Ui, config: &mut AudioDspConfig, lang: Lang, enabled: bool) -> bool { @@ -11,6 +11,27 @@ pub fn settings(ui: &mut egui::Ui, config: &mut AudioDspConfig, lang: Lang, enab ui.add_enabled_ui(enabled, |ui| { ui.small(label("settings.audioDsp.RNNoise_is_lightweight_PureVox_uses_more_CPU_Changes_apply_to_the_next_recording")); ui.add_space(8.0); + // 回声消除与输出缓冲：MicYou 的链路里 AEC 在最前，缓冲区是整条链的全局 + // 参数（设备回调抖动大时调大，代价是延迟）。 + ui.strong(label("settings.audioDsp.echoStage")); + // 三选一：轻量 NLMS 省 CPU（弱机可用），aec7 模型抑制更强但约 3 倍开销。 + let aec_labels = [ + (AecMode::Off, label("settings.audioDsp.aecOff")), + (AecMode::Light, label("settings.audioDsp.aecLight")), + (AecMode::Model, label("settings.audioDsp.aecModel")), + ]; + ui.horizontal_wrapped(|ui| { + for (mode, name) in aec_labels { + ui.selectable_value(&mut config.aec_mode, mode, name); + } + }); + ui.small(label("settings.audioDsp.AEC_needs_a_loudspeaker_reference")); + if config.aec_mode == AecMode::Model { + ui.small(label("settings.audioDsp.aecModelCost")); + } + slider(ui, label("settings.audioDsp.Output_buffer_ms"), &mut config.output_buffer_ms, 100..=1200, " ms"); + ui.small(label("settings.audioDsp.A_larger_buffer_survives_jitter_but_adds_latency")); + ui.separator(); ui.strong(label("settings.audioDsp.noiseStage")); egui::ComboBox::from_id_salt("audio-dsp-algorithm") .width(ui.available_width().min(280.0)) @@ -30,10 +51,10 @@ pub fn settings(ui: &mut egui::Ui, config: &mut AudioDspConfig, lang: Lang, enab } ui.separator(); ui.strong(label("settings.audioDsp.Input_gain")); - slider(ui, label("settings.audioDsp.Input_gain_dB"), &mut config.gain_db, -30..=30, " dB"); + slider(ui, label("settings.audioDsp.Input_gain_dB"), &mut config.gain_db, -50..=50, " dB"); ui.checkbox(&mut config.agc_enabled, label("settings.audioDsp.Automatic_gain")); if config.agc_enabled { - slider(ui, label("settings.audioDsp.Target_level"), &mut config.agc_target, 5..=90, "%"); + slider(ui, label("settings.audioDsp.Target_level"), &mut config.agc_target, 0..=100, "%"); slider(ui, label("settings.audioDsp.Attack_speed"), &mut config.agc_attack, 1..=100, ""); slider(ui, label("settings.audioDsp.Release_speed"), &mut config.agc_decay, 1..=100, ""); } @@ -41,35 +62,21 @@ pub fn settings(ui: &mut egui::Ui, config: &mut AudioDspConfig, lang: Lang, enab ui.strong(label("settings.audioDsp.Speech_gate")); ui.checkbox(&mut config.vad_enabled, label("settings.audioDsp.Speech_gate_does_not_stop_recording")); if config.vad_enabled { - slider(ui, label("settings.audioDsp.Threshold_dB"), &mut config.vad_threshold_db, -80..=-10, " dB"); + slider(ui, label("settings.audioDsp.Threshold_dB"), &mut config.vad_threshold_db, -100..=0, " dB"); ui.small(label("settings.audioDsp.A_high_threshold_can_remove_quiet_speech")); } ui.add_space(8.0); - ui.collapsing(label("settings.audioDsp.Tone_and_room_adjustments"), |ui| { - ui.small(label("settings.audioDsp.The_MicYou_comb_filter_can_color_speech_Start_at_0")); - slider(ui, label("settings.audioDsp.Comb_filter"), &mut config.dereverb_level, 0..=100, "%"); - ui.checkbox(&mut config.equalizer_enabled, label("settings.audioDsp.10_band_equalizer")); - if config.equalizer_enabled { - slider(ui, label("settings.audioDsp.Preamp_dB"), &mut config.equalizer_preamp_db, -20..=20, " dB"); - let columns = if ui.available_width() >= 540.0 { 2 } else { 1 }; - egui::Grid::new("audio-dsp-eq").num_columns(columns).spacing([16.0, 8.0]).show(ui, |ui| { - let width = ((ui.available_width() - 16.0 * (columns - 1) as f32) / columns as f32).max(100.0); - for (index, (band, name)) in config.equalizer_bands_db.iter_mut().zip(["31 Hz", "63 Hz", "125 Hz", "250 Hz", "500 Hz", "1 kHz", "2 kHz", "4 kHz", "8 kHz", "16 kHz"]).enumerate() { - ui.push_id(index, |ui| ui.vertical(|ui| { - ui.set_width(width); - ui.label(name); - ui.spacing_mut().slider_width = (width - 80.0).clamp(60.0, 220.0); - ui.add(egui::Slider::new(band, -12..=12).suffix(" dB")); - })); - if (index + 1) % columns == 0 { ui.end_row(); } - } - }); - } - }); + // 房间/音色只剩「梳状滤波」一项：十段均衡器按需求移除（配置字段保留， + // 旧 preferences 仍能反序列化），界面上不再暴露。 + ui.add_space(8.0); + ui.strong(label("settings.audioDsp.roomStage")); + ui.small(label("settings.audioDsp.The_MicYou_comb_filter_can_color_speech_Start_at_0")); + slider(ui, label("settings.audioDsp.Comb_filter"), &mut config.dereverb_level, 0..=100, "%"); ui.collapsing(label("settings.audioDsp.Processing_order"), |ui| { let mut swap = None; for (i, stage) in config.processing_order.iter().enumerate() { let name = match stage { + AudioDspStage::AcousticEchoCancellation => label("settings.audioDsp.aecStage"), AudioDspStage::NoiseReduction => label("settings.audioDsp.noiseStage"), AudioDspStage::Dereverb => label("settings.audioDsp.combStage"), AudioDspStage::Equalizer => label("settings.audioDsp.Equalizer"), @@ -80,6 +87,7 @@ pub fn settings(ui: &mut egui::Ui, config: &mut AudioDspConfig, lang: Lang, enab ui.push_id(i, |ui| ui.horizontal(|ui| { ui.monospace(format!("{}", i + 1)); let active = match stage { + AudioDspStage::AcousticEchoCancellation => config.aec_mode != AecMode::Off, AudioDspStage::NoiseReduction => config.noise_reduction != NoiseReduction::Off && config.noise_strength > 0, AudioDspStage::Dereverb => config.dereverb_level > 0, AudioDspStage::Equalizer => config.equalizer_enabled, diff --git a/openless-all/app/linux-egui/src/ui/frontend/settings.rs b/openless-all/app/linux-egui/src/ui/frontend/settings.rs index 493a62327..b11dda971 100644 --- a/openless-all/app/linux-egui/src/ui/frontend/settings.rs +++ b/openless-all/app/linux-egui/src/ui/frontend/settings.rs @@ -639,6 +639,28 @@ fn general(ui: &mut egui::Ui, vm: &mut FrontendViewModel, actions: &mut Vec = vec![ + tr_l10n(lang, "settings.audioDsp.aecReferenceAuto"), + tr_l10n(lang, "settings.audioDsp.aecReferenceOff"), + ]; + options.extend(vm.settings.aec_reference_devices.iter().map(String::as_str)); + combo_index_row( + ui, + tr_l10n(lang, "settings.audioDsp.aecReferenceLabel"), + tr_l10n(lang, "settings.audioDsp.aecReferenceDesc"), + vm.settings.aec_reference_index.min(options.len().saturating_sub(1)), + &options, + |index| { + actions.push(FrontendAction::SettingsCombo( + SettingsComboField::AecReference, + index, + )); + }, + ); + } // #4：录音胶囊开关与样式（Tauri `capsuleLabel` / `capsuleStyleLabel`）。 toggle_row( ui, @@ -3197,6 +3219,34 @@ fn advanced(ui: &mut egui::Ui, vm: &mut FrontendViewModel, actions: &mut Vec/logs/level`， /// 重启后生效；不是 Core 偏好，所以不上云、不影响 Tauri。 LogLevel, @@ -1204,6 +1210,14 @@ pub struct SettingsFields { #[serde(default)] pub llm_thinking_enabled: bool, pub record_audio_for_debug: bool, + /// 调试录音：保留「处理后」的音频（主档案 `.wav`）。 + pub debug_archive_processed: bool, + /// 调试录音：额外保留「处理前」的 `.raw.wav`，用于 A/B 对照。 + pub debug_archive_raw: bool, + /// AEC 的扬声器参考源选项：0 = 自动，1 = 不采集，其余为回环设备名。 + pub aec_reference_devices: Vec, + /// 当前选择的下标（见 `aec_reference_devices`）。 + pub aec_reference_index: usize, /// 日志级别：0 = info，1 = debug，2 = trace。宿主本地设置，不进 Core 偏好。 pub log_level: usize, pub history_max_entries: String, @@ -1273,6 +1287,10 @@ impl Default for SettingsFields { capsule_transcript_font_size: 14, llm_thinking_enabled: false, record_audio_for_debug: false, + debug_archive_processed: false, + debug_archive_raw: false, + aec_reference_devices: Vec::new(), + aec_reference_index: 0, log_level: 0, history_max_entries: String::new(), style_pack_hotkeys: Vec::new(), diff --git a/openless-all/app/scripts/compare-audio-outputs.py b/openless-all/app/scripts/compare-audio-outputs.py new file mode 100644 index 000000000..6eea95b50 --- /dev/null +++ b/openless-all/app/scripts/compare-audio-outputs.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +"""Compare audio_equivalence exports from two builds; exact PCM is the default.""" +import argparse +import array +import hashlib +import json +import math +from pathlib import Path +import sys + +parser = argparse.ArgumentParser(description=__doc__) +parser.add_argument("before", type=Path) +parser.add_argument("after", type=Path) +parser.add_argument("--report", type=Path, required=True) +args = parser.parse_args() +before = {row["file"]: row for row in json.loads((args.before / "results.json").read_text())} +after = {row["file"]: row for row in json.loads((args.after / "results.json").read_text())} +if not before or before.keys() != after.keys(): + raise SystemExit("nonempty matching case sets required") +rows = [] +for name in sorted(before): + raw_a, raw_b = (args.before / name).read_bytes(), (args.after / name).read_bytes() + if not raw_a or len(raw_a) != len(raw_b) or len(raw_a) % 2: + raise SystemExit(f"invalid or different PCM duration: {name}") + a, b = array.array("h"), array.array("h") + a.frombytes(raw_a) + b.frombytes(raw_b) + if sys.byteorder != "little": + a.byteswap() + b.byteswap() + differences = [int(x) - int(y) for x, y in zip(a, b)] + error = sum(d * d for d in differences) + power = sum(int(x) * int(x) for x in a) + rows.append({"file": name, "samples": len(a), "identical": raw_a == raw_b, + "different_samples": sum(d != 0 for d in differences), + "max_error_lsb": max(map(abs, differences)), + "rms_error_lsb": math.sqrt(error / len(a)), + "snr_db": 10 * math.log10(power / error) if power and error else None, + "before_sha256": hashlib.sha256(raw_a).hexdigest(), + "after_sha256": hashlib.sha256(raw_b).hexdigest(), + "before_seconds": before[name]["seconds"], + "after_seconds": after[name]["seconds"]}) +report = {"cases": len(rows), "identical_cases": sum(r["identical"] for r in rows), + "max_error_lsb": max(r["max_error_lsb"] for r in rows), "results": rows} +args.report.write_text(json.dumps(report, indent=2) + "\n") +print(json.dumps({k: v for k, v in report.items() if k != "results"})) +raise SystemExit(0 if report["identical_cases"] == report["cases"] else 1) diff --git a/openless-all/app/scripts/fixtures/audio-equivalence-2026-10-24.json b/openless-all/app/scripts/fixtures/audio-equivalence-2026-10-24.json new file mode 100644 index 000000000..18743747e --- /dev/null +++ b/openless-all/app/scripts/fixtures/audio-equivalence-2026-10-24.json @@ -0,0 +1,635 @@ +{ + "cases": 48, + "identical_cases": 48, + "max_error_lsb": 0, + "results": [ + { + "file": "off-16000-1-false.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "e11f9e2e59fb5a868ab9a22fbf5d74adfc47f22aade92cdad6c89d19a3d19919", + "after_sha256": "e11f9e2e59fb5a868ab9a22fbf5d74adfc47f22aade92cdad6c89d19a3d19919", + "before_seconds": 0.116754236, + "after_seconds": 0.005358786 + }, + { + "file": "off-16000-1-true.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "933e8f5a23b37571a1cc3facb349aad0e307493f57b1026e19f6b5f6dc48e48e", + "after_sha256": "933e8f5a23b37571a1cc3facb349aad0e307493f57b1026e19f6b5f6dc48e48e", + "before_seconds": 0.086573904, + "after_seconds": 0.007005361 + }, + { + "file": "off-16000-2-false.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "ee16525ad1f9a00fb458058a140c9008563fb2f24d96bdc64e0d4cda82617e9c", + "after_sha256": "ee16525ad1f9a00fb458058a140c9008563fb2f24d96bdc64e0d4cda82617e9c", + "before_seconds": 0.07387148, + "after_seconds": 0.005151878 + }, + { + "file": "off-16000-2-true.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "b7e12695689c18329aaabee7e7775f183729ea107ccb2fabbde685d22e6549b6", + "after_sha256": "b7e12695689c18329aaabee7e7775f183729ea107ccb2fabbde685d22e6549b6", + "before_seconds": 0.075765705, + "after_seconds": 0.006915069 + }, + { + "file": "off-44100-1-false.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "e6efeaadc65c0a5ce9273ca3bbbd1de2a8d8a8012ef2209a9a729eb5ef4c6aa7", + "after_sha256": "e6efeaadc65c0a5ce9273ca3bbbd1de2a8d8a8012ef2209a9a729eb5ef4c6aa7", + "before_seconds": 0.077063813, + "after_seconds": 0.005544203 + }, + { + "file": "off-44100-1-true.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "8b2bc6fe8230137b73b18ac65d021b521c8bda8560260dcd8a62437eb26841a0", + "after_sha256": "8b2bc6fe8230137b73b18ac65d021b521c8bda8560260dcd8a62437eb26841a0", + "before_seconds": 0.078247096, + "after_seconds": 0.007225978 + }, + { + "file": "off-44100-2-false.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "5daffa30fba17c9c10ecc9c84114fe228972960828c3e9d624b7ff329bf0f812", + "after_sha256": "5daffa30fba17c9c10ecc9c84114fe228972960828c3e9d624b7ff329bf0f812", + "before_seconds": 0.075811189, + "after_seconds": 0.005497613 + }, + { + "file": "off-44100-2-true.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "271671a84913ac8e3a6037a3b87ed102dbd6ca156c7c237e7ceade030c459636", + "after_sha256": "271671a84913ac8e3a6037a3b87ed102dbd6ca156c7c237e7ceade030c459636", + "before_seconds": 0.077602554, + "after_seconds": 0.007278443 + }, + { + "file": "off-48000-1-false.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "0a844c08b6ee7276ec91da9ff9cdc3f3c84be8023179611ee49bd08c3fdd7167", + "after_sha256": "0a844c08b6ee7276ec91da9ff9cdc3f3c84be8023179611ee49bd08c3fdd7167", + "before_seconds": 0.073624321, + "after_seconds": 0.00515933 + }, + { + "file": "off-48000-1-true.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "23c03deecbfa201441d73bf4f3fab6bf1734aa4e35ea56b75794b0bf91d2faed", + "after_sha256": "23c03deecbfa201441d73bf4f3fab6bf1734aa4e35ea56b75794b0bf91d2faed", + "before_seconds": 0.074634896, + "after_seconds": 0.006924537 + }, + { + "file": "off-48000-2-false.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "497b5e23fd6b5dec50b1917f69c8d904e8ca89308e716d2f4c74392162146f0d", + "after_sha256": "497b5e23fd6b5dec50b1917f69c8d904e8ca89308e716d2f4c74392162146f0d", + "before_seconds": 0.072974345, + "after_seconds": 0.005360569 + }, + { + "file": "off-48000-2-true.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "4638ce855ba7f2ed394be976efdf2b89e430c6b0da497baa07dc91e140e57e7d", + "after_sha256": "4638ce855ba7f2ed394be976efdf2b89e430c6b0da497baa07dc91e140e57e7d", + "before_seconds": 0.074942485, + "after_seconds": 0.006904952 + }, + { + "file": "purevox-16000-1-false.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "9c0341d43644e43cfd6080afbcbb46fc2eb49f84ef070c544e46c5de39505044", + "after_sha256": "9c0341d43644e43cfd6080afbcbb46fc2eb49f84ef070c544e46c5de39505044", + "before_seconds": 0.185454531, + "after_seconds": 0.159913137 + }, + { + "file": "purevox-16000-1-true.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "6525c13175d121b01200a1b6a1c098dad7190cb6ae2c3123ca63a0da674f301f", + "after_sha256": "6525c13175d121b01200a1b6a1c098dad7190cb6ae2c3123ca63a0da674f301f", + "before_seconds": 0.181579766, + "after_seconds": 0.159810681 + }, + { + "file": "purevox-16000-2-false.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "812bcdd8544a26c8531599f37b6103ad114819e9bdf4bfa3305fe2784c6efa7c", + "after_sha256": "812bcdd8544a26c8531599f37b6103ad114819e9bdf4bfa3305fe2784c6efa7c", + "before_seconds": 0.196011856, + "after_seconds": 0.159011761 + }, + { + "file": "purevox-16000-2-true.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "3ffc5a0ca033af72680a72eb99a8c3fc97ca97ae99a8be6e7b22b7057e632960", + "after_sha256": "3ffc5a0ca033af72680a72eb99a8c3fc97ca97ae99a8be6e7b22b7057e632960", + "before_seconds": 0.186484676, + "after_seconds": 0.161438696 + }, + { + "file": "purevox-44100-1-false.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "06c38614f964c17a034c20084951107954dd13f694bfb1b907321c9295ece6ee", + "after_sha256": "06c38614f964c17a034c20084951107954dd13f694bfb1b907321c9295ece6ee", + "before_seconds": 0.184356045, + "after_seconds": 0.158482353 + }, + { + "file": "purevox-44100-1-true.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "1b06992420881943e8ae5bf912a98ff5cde2a6739a78b9f4cd324ae2d757f311", + "after_sha256": "1b06992420881943e8ae5bf912a98ff5cde2a6739a78b9f4cd324ae2d757f311", + "before_seconds": 0.183797703, + "after_seconds": 0.161311846 + }, + { + "file": "purevox-44100-2-false.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "889a0cfd345f69bf3128aa3a0cee81991f671e951a04e2f5ebc19318d7c255e3", + "after_sha256": "889a0cfd345f69bf3128aa3a0cee81991f671e951a04e2f5ebc19318d7c255e3", + "before_seconds": 0.184001547, + "after_seconds": 0.159463295 + }, + { + "file": "purevox-44100-2-true.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "a2a1613f24bde1dd9d0e0a8114f1954a31e59db429cf1a6e5f48f593d812818d", + "after_sha256": "a2a1613f24bde1dd9d0e0a8114f1954a31e59db429cf1a6e5f48f593d812818d", + "before_seconds": 0.187264544, + "after_seconds": 0.158695499 + }, + { + "file": "purevox-48000-1-false.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "6f824bdcd15d83886f8ee7c174030608e9ad3cb408676f4af140e263374d1a0c", + "after_sha256": "6f824bdcd15d83886f8ee7c174030608e9ad3cb408676f4af140e263374d1a0c", + "before_seconds": 0.179845299, + "after_seconds": 0.15897143 + }, + { + "file": "purevox-48000-1-true.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "978083461995c4b03f050f1660c2f85af5be94d40eeeee81a309a70da35367ac", + "after_sha256": "978083461995c4b03f050f1660c2f85af5be94d40eeeee81a309a70da35367ac", + "before_seconds": 0.186480047, + "after_seconds": 0.156610284 + }, + { + "file": "purevox-48000-2-false.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "438512f7739fcb258ff0b0bbd179b811ce4996106bb6651834821cea9cb0fe1b", + "after_sha256": "438512f7739fcb258ff0b0bbd179b811ce4996106bb6651834821cea9cb0fe1b", + "before_seconds": 0.185758329, + "after_seconds": 0.155331091 + }, + { + "file": "purevox-48000-2-true.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "8bdbc81e9dc86b0b493996be66239d86d299fbc7347d253f7080977c52498ebd", + "after_sha256": "8bdbc81e9dc86b0b493996be66239d86d299fbc7347d253f7080977c52498ebd", + "before_seconds": 0.184863034, + "after_seconds": 0.158239955 + }, + { + "file": "rnnoise-16000-1-false.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "b8ef881c4e9ec584b79531e3df91d28e8cd276573e8053e0afc5044868990df6", + "after_sha256": "b8ef881c4e9ec584b79531e3df91d28e8cd276573e8053e0afc5044868990df6", + "before_seconds": 0.08139987, + "after_seconds": 0.011418394 + }, + { + "file": "rnnoise-16000-1-true.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "02dcab55e0551531fb50d77b451d4593faa8bf83025a3cd1cdfc1c24afa3d245", + "after_sha256": "02dcab55e0551531fb50d77b451d4593faa8bf83025a3cd1cdfc1c24afa3d245", + "before_seconds": 0.081542452, + "after_seconds": 0.013149335 + }, + { + "file": "rnnoise-16000-2-false.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "81dc85cc66c69d9905db67626449ff5a803e9669734da610e9ab055a45f94e31", + "after_sha256": "81dc85cc66c69d9905db67626449ff5a803e9669734da610e9ab055a45f94e31", + "before_seconds": 0.080572818, + "after_seconds": 0.011260605 + }, + { + "file": "rnnoise-16000-2-true.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "301b0fa3828238acf28e3f99a330f4eff5784e15dc2af4741100ad2be1efa1e6", + "after_sha256": "301b0fa3828238acf28e3f99a330f4eff5784e15dc2af4741100ad2be1efa1e6", + "before_seconds": 0.083529118, + "after_seconds": 0.013055125 + }, + { + "file": "rnnoise-44100-1-false.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "007bb3e0089d91fa6a8c8c72c2352e290648aa3f99f850c296755906cf25c94b", + "after_sha256": "007bb3e0089d91fa6a8c8c72c2352e290648aa3f99f850c296755906cf25c94b", + "before_seconds": 0.082900206, + "after_seconds": 0.011355191 + }, + { + "file": "rnnoise-44100-1-true.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "2330fe8baf4efd698802249dfaab992baa8ceae6d116c9e03c7d156be201aac0", + "after_sha256": "2330fe8baf4efd698802249dfaab992baa8ceae6d116c9e03c7d156be201aac0", + "before_seconds": 0.08490873, + "after_seconds": 0.013156918 + }, + { + "file": "rnnoise-44100-2-false.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "c8f936ab76cffcdf588509aa3ec02fd74361ca093701a36a4cc7efb8f93069e8", + "after_sha256": "c8f936ab76cffcdf588509aa3ec02fd74361ca093701a36a4cc7efb8f93069e8", + "before_seconds": 0.082212834, + "after_seconds": 0.011510045 + }, + { + "file": "rnnoise-44100-2-true.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "5044542f4def4e3c50488fcf5b8581bd9e11d99fa8d7a16a6ae73095a665d538", + "after_sha256": "5044542f4def4e3c50488fcf5b8581bd9e11d99fa8d7a16a6ae73095a665d538", + "before_seconds": 0.083604619, + "after_seconds": 0.013184362 + }, + { + "file": "rnnoise-48000-1-false.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "2de2be3daf47ce414848687a74dcf0828ff5d37a387f4c713907ae1c4a5986d6", + "after_sha256": "2de2be3daf47ce414848687a74dcf0828ff5d37a387f4c713907ae1c4a5986d6", + "before_seconds": 0.079465597, + "after_seconds": 0.011077441 + }, + { + "file": "rnnoise-48000-1-true.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "87bb872eee2fb5e79808131ef64799931ff15476f8661557d72e80438c946660", + "after_sha256": "87bb872eee2fb5e79808131ef64799931ff15476f8661557d72e80438c946660", + "before_seconds": 0.080882969, + "after_seconds": 0.012818846 + }, + { + "file": "rnnoise-48000-2-false.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "ef43447a58302e67ec75ab8b51b659d4330a7110509769ac851481c8e37f502b", + "after_sha256": "ef43447a58302e67ec75ab8b51b659d4330a7110509769ac851481c8e37f502b", + "before_seconds": 0.08029677, + "after_seconds": 0.01115677 + }, + { + "file": "rnnoise-48000-2-true.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "dad5f57a9fb1658502b154707aa9e4ac7e090a5f9de39f3c3d44ccfd888e3774", + "after_sha256": "dad5f57a9fb1658502b154707aa9e4ac7e090a5f9de39f3c3d44ccfd888e3774", + "before_seconds": 0.082389417, + "after_seconds": 0.012953246 + }, + { + "file": "spectral-16000-1-false.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "6a59e58a8dd0620b06265ae20d36fd7d602c3fc28791b3b4ab18d5d19ea64bdb", + "after_sha256": "6a59e58a8dd0620b06265ae20d36fd7d602c3fc28791b3b4ab18d5d19ea64bdb", + "before_seconds": 0.073881205, + "after_seconds": 0.006052521 + }, + { + "file": "spectral-16000-1-true.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "a40ab5a854c09ec5f2cc8625cf674a6033f40aa3cfcfbb63a948d67d333ff1b1", + "after_sha256": "a40ab5a854c09ec5f2cc8625cf674a6033f40aa3cfcfbb63a948d67d333ff1b1", + "before_seconds": 0.076846195, + "after_seconds": 0.007796042 + }, + { + "file": "spectral-16000-2-false.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "d5abc138be586b562c9ce6717f29ee5388399d68f7242e2eea927f74e7118dba", + "after_sha256": "d5abc138be586b562c9ce6717f29ee5388399d68f7242e2eea927f74e7118dba", + "before_seconds": 0.075972349, + "after_seconds": 0.006051525 + }, + { + "file": "spectral-16000-2-true.pcm", + "samples": 16037, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "82ff018705bbb851226213ef58d2da8767d6a68b748f440e6a4f50428aed93a3", + "after_sha256": "82ff018705bbb851226213ef58d2da8767d6a68b748f440e6a4f50428aed93a3", + "before_seconds": 0.076897096, + "after_seconds": 0.007781968 + }, + { + "file": "spectral-44100-1-false.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "a01b6eaa9641f3e450ec3725127ace6106c8a3e3af447b7fdb5019a8e8131cce", + "after_sha256": "a01b6eaa9641f3e450ec3725127ace6106c8a3e3af447b7fdb5019a8e8131cce", + "before_seconds": 0.076590793, + "after_seconds": 0.006390747 + }, + { + "file": "spectral-44100-1-true.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "e09f3c92c03fd4813d5439621b5ccbdb9dd9a5eefae620b3760b2e31e75a20e6", + "after_sha256": "e09f3c92c03fd4813d5439621b5ccbdb9dd9a5eefae620b3760b2e31e75a20e6", + "before_seconds": 0.078474036, + "after_seconds": 0.008173782 + }, + { + "file": "spectral-44100-2-false.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "a5592134a4b01c4538d8c41ec4cb625cfcaab70e36a0f7a4861bc57304d5b19f", + "after_sha256": "a5592134a4b01c4538d8c41ec4cb625cfcaab70e36a0f7a4861bc57304d5b19f", + "before_seconds": 0.076863817, + "after_seconds": 0.006393915 + }, + { + "file": "spectral-44100-2-true.pcm", + "samples": 16014, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "77c84798a964dc8d2884530a0bf69b4036c3ebcf0cca9f61d151809ded9f6338", + "after_sha256": "77c84798a964dc8d2884530a0bf69b4036c3ebcf0cca9f61d151809ded9f6338", + "before_seconds": 0.078872631, + "after_seconds": 0.008147194 + }, + { + "file": "spectral-48000-1-false.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "3e40836ca79ea2ef9ee7e12016071fcdc4d0b3e995979062ba3d99f39e9a1a59", + "after_sha256": "3e40836ca79ea2ef9ee7e12016071fcdc4d0b3e995979062ba3d99f39e9a1a59", + "before_seconds": 0.073998635, + "after_seconds": 0.006071965 + }, + { + "file": "spectral-48000-1-true.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "1f0b6258dfd5bdfb99cec2dc702378cc1cc71add5800e06d31058f18a925fcd8", + "after_sha256": "1f0b6258dfd5bdfb99cec2dc702378cc1cc71add5800e06d31058f18a925fcd8", + "before_seconds": 0.075704484, + "after_seconds": 0.007758046 + }, + { + "file": "spectral-48000-2-false.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "cdbe91ced337c6e5264a3b2129dfcf642314ad3419a3e2703d317056a9a5e617", + "after_sha256": "cdbe91ced337c6e5264a3b2129dfcf642314ad3419a3e2703d317056a9a5e617", + "before_seconds": 0.073898392, + "after_seconds": 0.005995752 + }, + { + "file": "spectral-48000-2-true.pcm", + "samples": 16013, + "identical": true, + "different_samples": 0, + "max_error_lsb": 0, + "rms_error_lsb": 0.0, + "snr_db": null, + "before_sha256": "c99581bdbd1bd17c340fb0a536d5cc1fea8fd160529a5e9dcf216d9a38ed076d", + "after_sha256": "c99581bdbd1bd17c340fb0a536d5cc1fea8fd160529a5e9dcf216d9a38ed076d", + "before_seconds": 0.075725974, + "after_seconds": 0.008444904 + } + ], + "baseline_revision": "f0744e10", + "optimized_revision": "e690e5b3 plus external PCM scratch reuse", + "platform": "Linux x86_64, release, synthetic fixtures", + "limits": "No listening study or on-device validation. Timings are host-load dependent. Apple PCM comparison pending Apple SDK execution." +} diff --git a/openless-all/app/scripts/micyou-dsp.patch b/openless-all/app/scripts/micyou-dsp.patch index 12d2d8819..56eb88729 100644 --- a/openless-all/app/scripts/micyou-dsp.patch +++ b/openless-all/app/scripts/micyou-dsp.patch @@ -136,7 +136,68 @@ // Dereverb state dereverb_buffer_left: Vec, dereverb_buffer_right: Vec, -@@ -1042,6 +985,8 @@ +@@ -996,6 +939,9 @@ + agc_smoothed_gain: f32, + // VAD fade state (0.0 = muted, 1.0 = full) + vad_fade: f32, ++ #[cfg(feature = "noise-suppression")] ++ spectrum_analyzer: Option, ++ spectrum_tick: usize, + // Spectrum snapshots + raw_spectrum: Vec, + processed_spectrum: Vec, +@@ -1038,10 +984,59 @@ + + const RNNOISE_FRAME_SIZE: usize = 480; + ++#[cfg(feature = "noise-suppression")] ++struct SpectrumAnalyzer { ++ fft: Arc>, ++ buffer: Vec>, ++ scratch: Vec>, ++ window: Vec, ++} ++ ++#[cfg(feature = "noise-suppression")] ++impl SpectrumAnalyzer { ++ fn new() -> Self { ++ let fft = rustfft::FftPlanner::new().plan_fft_forward(512); ++ let scratch = vec![Complex::default(); fft.get_inplace_scratch_len()]; ++ Self { ++ fft, ++ scratch, ++ buffer: vec![Complex::default(); 512], ++ window: (0..512) ++ .map(|i| 0.5 - 0.5 * (std::f32::consts::TAU * i as f32 / 511.0).cos()) ++ .collect(), ++ } ++ } ++ ++ fn analyze(&mut self, data: &[f32], target: &mut [f32]) { ++ self.buffer.fill(Complex::default()); ++ let count = data.len().min(512); ++ for (i, sample) in data[data.len() - count..].iter().enumerate() { ++ self.buffer[i].re = sample * self.window[i]; ++ } ++ self.fft ++ .process_with_scratch(&mut self.buffer, &mut self.scratch); ++ for (band, value) in target.iter_mut().enumerate() { ++ let low = 20.0 * 1200.0_f32.powf(band as f32 / 64.0); ++ let high = 20.0 * 1200.0_f32.powf((band + 1) as f32 / 64.0); ++ let start = ((low * 512.0 / 48_000.0) as usize).min(256); ++ let end = ((high * 512.0 / 48_000.0).ceil() as usize).clamp(start + 1, 257); ++ let magnitude = self.buffer[start..end] ++ .iter() ++ .map(|bin| bin.norm()) ++ .fold(0.0_f32, f32::max) ++ / 128.0; ++ let normalized = ((20.0 * magnitude.max(1e-8).log10() + 60.0) / 60.0).clamp(0.0, 1.0); ++ *value = normalized.max(*value * 0.8); ++ } ++ } ++} ++ + impl DspProcessor { pub fn new(settings: Arc>, _model_dir: Option) -> Self { Self { settings: settings.clone(), @@ -145,7 +206,7 @@ #[cfg(feature = "dsp")] denoiser_left: DenoiseState::new(), #[cfg(feature = "dsp")] -@@ -1069,7 +1014,6 @@ +@@ -1069,13 +1064,15 @@ #[cfg(feature = "dsp")] speex_ns: SpeexStyleNS::new(), equalizer: EqualizerEffect::new(), @@ -153,7 +214,24 @@ dereverb_buffer_left: vec![0.0; 480], dereverb_buffer_right: vec![0.0; 480], dereverb_index: 0, -@@ -1176,6 +1120,14 @@ + agc_envelope: 0.0, + agc_smoothed_gain: 1.0, + vad_fade: 1.0, ++ #[cfg(feature = "noise-suppression")] ++ spectrum_analyzer: None, ++ spectrum_tick: 0, + raw_spectrum: vec![0.0; 64], + processed_spectrum: vec![0.0; 64], + accum_buffer: Vec::new(), +@@ -1101,6 +1098,7 @@ + } + + let raw_rms = compute_rms(data); ++ self.spectrum_tick = (self.spectrum_tick + 1) % 10; + self.compute_spectrum(data, true); + + // Frame accumulation: align to 480*channels samples before processing, +@@ -1176,6 +1174,14 @@ settings.agc_target, attack_rate, decay_rate, @@ -168,7 +246,7 @@ ); } } -@@ -1186,7 +1138,12 @@ +@@ -1186,7 +1192,12 @@ // Legacy synthetic plugin stage (may be absent): runs // every registered plugin at this chain position. if let Some(hook) = &mut self.external_hook { @@ -182,7 +260,7 @@ } } node if node.starts_with(PLUGIN_NODE_PREFIX) => { -@@ -1200,8 +1157,8 @@ +@@ -1200,8 +1211,8 @@ } } @@ -193,7 +271,7 @@ // Soft clip — avoids harsh hard-clipping artifacts (crackling/pops) for sample in self.output_buffer.iter_mut() { -@@ -1368,14 +1325,45 @@ +@@ -1368,14 +1379,45 @@ channels: usize, settings: &AudioDspSettings, ) { @@ -243,7 +321,7 @@ } } -@@ -1397,18 +1385,19 @@ +@@ -1397,18 +1439,19 @@ } // Process each channel with its own denoiser (no RNN state cross-contamination) @@ -265,7 +343,7 @@ data.clear(); for i in 0..frames { -@@ -1416,7 +1405,7 @@ +@@ -1416,7 +1459,7 @@ data.push(right[i]); } } else { @@ -274,7 +352,7 @@ data, &mut self.ns_buffer_left, &mut self.denoiser_left, -@@ -1431,12 +1420,13 @@ +@@ -1431,12 +1474,13 @@ ns_buffer: &mut Vec, denoiser: &mut DenoiseState<'static>, intensity: f32, @@ -289,7 +367,7 @@ while ns_buffer.len() >= RNNOISE_FRAME_SIZE { let frame: Vec = ns_buffer.drain(..RNNOISE_FRAME_SIZE).collect(); -@@ -1444,7 +1434,8 @@ +@@ -1444,7 +1488,8 @@ let input_frame: Vec = frame.iter().map(|s| s * 32767.0).collect(); let mut output_frame = vec![0.0f32; RNNOISE_FRAME_SIZE]; @@ -299,7 +377,7 @@ for i in 0..RNNOISE_FRAME_SIZE { let clean = output_frame[i] / 32767.0; -@@ -1463,6 +1454,7 @@ +@@ -1463,6 +1508,7 @@ } data.copy_from_slice(&output); @@ -307,7 +385,7 @@ } #[cfg(feature = "noise-suppression")] -@@ -1618,7 +1610,14 @@ +@@ -1618,7 +1664,14 @@ // ── AGC ───────────────────────────────────────────────────────────────── @@ -323,7 +401,7 @@ let target_linear = target / 32767.0; // Noise gate threshold: below this level, don't apply AGC gain // (prevents amplifying hiss/noise floor during speech pauses) -@@ -1633,7 +1632,7 @@ +@@ -1633,7 +1686,7 @@ } if self.agc_envelope > gate_threshold { let desired_gain = target_linear / self.agc_envelope; @@ -332,6 +410,151 @@ // Smooth gain transition to avoid pops (exponential moving average) let smooth_factor = 0.005_f32; self.agc_smoothed_gain += smooth_factor * (clamped_gain - self.agc_smoothed_gain); +@@ -1671,59 +1724,34 @@ + + // ── Spectrum ───────────────────────────────────────────────────────────── + ++ pub fn set_spectrum_enabled(&mut self, enabled: bool) { ++ #[cfg(feature = "noise-suppression")] ++ if enabled && self.spectrum_analyzer.is_none() { ++ self.spectrum_analyzer = Some(SpectrumAnalyzer::new()); ++ } else if !enabled { ++ self.spectrum_analyzer = None; ++ self.raw_spectrum.fill(0.0); ++ self.processed_spectrum.fill(0.0); ++ } ++ #[cfg(not(feature = "noise-suppression"))] ++ let _ = enabled; ++ } ++ + fn compute_spectrum(&mut self, data: &[f32], is_raw: bool) { +- let bands = 64; +- let target = if is_raw { +- &mut self.raw_spectrum +- } else { +- &mut self.processed_spectrum +- }; +- if target.len() != bands { +- target.resize(bands, 0.0); +- } +- if data.is_empty() { +- for v in target.iter_mut() { +- *v = 0.0; +- } ++ if self.spectrum_tick != 1 { + return; + } +- +- const BANDS: usize = 64; +- static BAND_LIMITS: std::sync::OnceLock<[f32; BANDS + 1]> = std::sync::OnceLock::new(); +- let limits = BAND_LIMITS.get_or_init(|| { +- let mut array = [0.0; BANDS + 1]; +- for (i, limit) in array.iter_mut().enumerate() { +- *limit = (i as f32 / BANDS as f32).powf(1.5); +- } +- array +- }); +- +- for (band_idx, band_val) in target.iter_mut().enumerate() { +- let start = limits[band_idx] * data.len() as f32; +- let end = limits[band_idx + 1] * data.len() as f32; +- let start = start as usize; +- let end = (end as usize).min(data.len()); +- if start >= end { +- *band_val *= 0.85; +- continue; +- } +- let mut sum = 0.0_f32; +- for sample in &data[start..end] { +- sum += sample * sample; +- } +- let rms = (sum / (end - start) as f32).sqrt(); +- let db = if rms > 1e-10 { +- 20.0 * rms.log10() ++ #[cfg(feature = "noise-suppression")] ++ if let Some(analyzer) = &mut self.spectrum_analyzer { ++ let target = if is_raw { ++ &mut self.raw_spectrum + } else { +- -100.0 ++ &mut self.processed_spectrum + }; +- let normalized = ((db + 60.0) / 60.0).clamp(0.0, 1.0); +- if normalized > *band_val { +- *band_val = normalized; +- } else { +- *band_val = *band_val * 0.85 + normalized * 0.15; +- } +- } ++ analyzer.analyze(data, target); ++ } ++ #[cfg(not(feature = "noise-suppression"))] ++ let _ = (data, is_raw); + } + } + +@@ -2096,11 +2124,13 @@ + #[test] + fn dsp_processor_chain_smoke() { + // Full chain: AEC → PureVox NS through the public DspProcessor API. +- let mut settings = AudioDspSettings::default(); +- settings.ns_enabled = true; +- settings.ns_type = "PureVox".to_string(); +- settings.ns_intensity = 100.0; +- settings.aec_enabled = true; ++ let settings = AudioDspSettings { ++ ns_enabled: true, ++ ns_type: "PureVox".to_string(), ++ ns_intensity: 100.0, ++ aec_enabled: true, ++ ..Default::default() ++ }; + let settings = Arc::new(RwLock::new(settings)); + let mut dsp = DspProcessor::new(settings, None); + +@@ -2127,3 +2157,42 @@ + ); + } + } ++ ++#[cfg(all(test, feature = "noise-suppression"))] ++mod spectrum_tests { ++ use super::*; ++ ++ #[test] ++ fn spectrum_is_opt_in_and_tracks_frequency_without_changing_audio() { ++ let settings = Arc::new(RwLock::new(AudioDspSettings { ++ gain: 6.0, ++ ..Default::default() ++ })); ++ let mut plain = DspProcessor::new(Arc::clone(&settings), None); ++ let mut monitored = DspProcessor::new(settings, None); ++ assert!(plain.spectrum_analyzer.is_none()); ++ monitored.set_spectrum_enabled(true); ++ let tone: Vec = (0..480) ++ .map(|i| (std::f32::consts::TAU * 1000.0 * i as f32 / 48_000.0).sin() * 0.05) ++ .collect(); ++ for _ in 0..12 { ++ let mut a = tone.clone(); ++ let mut b = tone.clone(); ++ plain.process(&mut a, 1, 0.0); ++ monitored.process(&mut b, 1, 0.0); ++ assert_eq!(a, b); ++ } ++ let (raw, processed) = monitored.get_spectrums(); ++ let peak = raw ++ .iter() ++ .enumerate() ++ .max_by(|a, b| a.1.total_cmp(b.1)) ++ .unwrap() ++ .0; ++ assert!((30..=40).contains(&peak), "1 kHz peak at {peak}"); ++ assert!(processed[peak] > raw[peak]); ++ monitored.set_spectrum_enabled(false); ++ assert!(monitored.spectrum_analyzer.is_none()); ++ assert!(monitored.get_spectrums().0.iter().all(|v| *v == 0.0)); ++ } ++} --- a/crates/micyou-infer/src/lib.rs +++ b/crates/micyou-infer/src/lib.rs @@ -48,6 +48,19 @@ @@ -354,3 +577,123 @@ use std::fmt; mod ops; +--- a/crates/micyou-infer/src/ops.rs ++++ b/crates/micyou-infer/src/ops.rs +@@ -396,9 +396,7 @@ + for kk in 0..k { + let av = arow[kk]; + let brow = &bbase[kk * n..(kk + 1) * n]; +- for j in 0..n { +- orow[j] += av * brow[j]; +- } ++ axpy(orow, brow, av); + } + } + if nb > 0 { +@@ -501,9 +499,7 @@ + for ci in 0..cpg { + let wv = wg[mi * cpg + ci]; + let xrow = &xg[ci * hw_in..(ci + 1) * hw_in]; +- for p in 0..hw_out { +- orow[p] += wv * xrow[p]; +- } ++ axpy(orow, &xrow[..hw_out], wv); + } + } + } +@@ -753,9 +749,7 @@ + for i in 0..batch { + let xv = xb[i * in_dim + kk]; + let prow = &mut pb[i * h3..(i + 1) * h3]; +- for j in 0..h3 { +- prow[j] += xv * wrow[j]; +- } ++ axpy(prow, wrow, xv); + } + } + } +@@ -784,9 +778,7 @@ + for i in 0..batch { + let sv = state_s[i * h + kk]; + let orow = &mut rproj_s[i * h3..(i + 1) * h3]; +- for j in 0..h3 { +- orow[j] += sv * wrow[j]; +- } ++ axpy(orow, wrow, sv); + } + } + // gates (elementwise passes — vectorize; rounding order matches +@@ -1508,6 +1500,50 @@ + + // ─── AVX2+FMA runtime dispatch ───────────────────────────────────────────── + // ++// Use baseline SIMD on x86-64 and NEON-enabled ARM64 targets. Vectorize rows ++// without changing reduction order or using fused multiply-add: recurrent ++// models retain the scalar rounding contract. No extra model/runtime/thread ++// pool is loaded alongside local ASR. ++#[inline(always)] ++fn axpy(output: &mut [f32], input: &[f32], scale: f32) { ++ assert_eq!(output.len(), input.len()); ++ #[cfg(all(target_arch = "aarch64", target_feature = "neon"))] ++ let offset = unsafe { ++ let mut offset = 0; ++ use std::arch::aarch64::{vaddq_f32, vld1q_f32, vmulq_n_f32, vst1q_f32}; ++ while offset + 4 <= output.len() { ++ let value = vmulq_n_f32(vld1q_f32(input.as_ptr().add(offset)), scale); ++ let sum = vaddq_f32(vld1q_f32(output.as_ptr().add(offset)), value); ++ vst1q_f32(output.as_mut_ptr().add(offset), sum); ++ offset += 4; ++ } ++ offset ++ }; ++ #[cfg(target_arch = "x86_64")] ++ let offset = unsafe { ++ // SSE2 is part of the x86-64 baseline. Unaligned loads/stores support ++ // every tensor row; no AVX CPU requirement or FMA rounding changes. ++ use std::arch::x86_64::{_mm_add_ps, _mm_loadu_ps, _mm_mul_ps, _mm_set1_ps, _mm_storeu_ps}; ++ let factor = _mm_set1_ps(scale); ++ let mut offset = 0; ++ while offset + 4 <= output.len() { ++ let value = _mm_mul_ps(_mm_loadu_ps(input.as_ptr().add(offset)), factor); ++ let sum = _mm_add_ps(_mm_loadu_ps(output.as_ptr().add(offset)), value); ++ _mm_storeu_ps(output.as_mut_ptr().add(offset), sum); ++ offset += 4; ++ } ++ offset ++ }; ++ #[cfg(not(any( ++ target_arch = "x86_64", ++ all(target_arch = "aarch64", target_feature = "neon") ++ )))] ++ let offset = 0; ++ for (dst, src) in output[offset..].iter_mut().zip(&input[offset..]) { ++ *dst += scale * src; ++ } ++} ++ + // The workspace builds for baseline x86-64 (SSE2) so the shipped binaries + // run everywhere; ONNX Runtime's MLAS gets its speed from runtime CPUID + // dispatch to AVX kernels. We mirror that: each hot kernel body is compiled +@@ -1755,3 +1791,22 @@ + ) { + layer_norm_impl(xp, xsh, scale, bias, op_, axis, eps) + } ++ ++#[cfg(test)] ++mod projection_tests { ++ #[test] ++ fn projections_keep_scalar_rounding_and_handle_vector_tails() { ++ for size in [0, 1, 3, 4, 5, 17, 192, 257] { ++ let input: Vec = (0..size).map(|i| (i as f32 * 0.7).sin()).collect(); ++ let mut actual = vec![0.17; size]; ++ let mut expected = actual.clone(); ++ for scale in [0.0, -0.37, 0.81, 1e-12] { ++ for (dst, src) in expected.iter_mut().zip(&input) { ++ *dst += scale * src; ++ } ++ super::axpy(&mut actual, &input, scale); ++ } ++ assert_eq!(actual, expected, "projection width {size}"); ++ } ++ } ++} diff --git a/openless-all/app/scripts/micyou-upstream.json b/openless-all/app/scripts/micyou-upstream.json index 611eefcf8..eea35e222 100644 --- a/openless-all/app/scripts/micyou-upstream.json +++ b/openless-all/app/scripts/micyou-upstream.json @@ -1,11 +1,11 @@ { "repository": "MicYou-Dev/MicYou", "revision": "d821da032e2b72330ced09678dd127acd2714bef", - "patchSha256": "7f204189f5ce7bb9f0250093bf6f5823e81e9c97777936d5f9bed54c210dd5ee", + "patchSha256": "aa74b2436d503cec561f76d31b3a69ee036005d1cc62c38cf757faf46091b67f", "files": { "crates/micyou-dsp/src/lib.rs": { "upstreamSha256": "9e3cb90b48c843554d75704188d4490d197f0134f57cf2fbc2f4a2aad1753d24", - "localSha256": "33174dfb93c42d4fd45dd36b721def47393bdce780324e8c58694f66f9585fa4" + "localSha256": "2efefcaf2563c167050623169e39f86da3366ecb751e27e3c63a8e126a4d9a78" }, "crates/micyou-infer/src/lib.rs": { "upstreamSha256": "a2d412f007ee15148e9a3684db8f59559e1057f62109f6ed5a1ca5fe951f5883", @@ -13,7 +13,7 @@ }, "crates/micyou-infer/src/ops.rs": { "upstreamSha256": "718c52a97d87a18043822f55f876ee03231429d5989361938f7e5b8d6dd459e9", - "localSha256": "718c52a97d87a18043822f55f876ee03231429d5989361938f7e5b8d6dd459e9" + "localSha256": "3963fbf11d5bce52ca042d56e64402d7ff0dfe2631e88eae7dce13fb8523263c" }, "crates/micyou-dsp/src/assets/purevox6.mcy": { "upstreamSha256": "88fe0a1f44b93942770c97c576eab84ff193370b96cb311671900c07ef213147", diff --git a/openless-all/app/scripts/test-compare-audio-outputs.py b/openless-all/app/scripts/test-compare-audio-outputs.py new file mode 100644 index 000000000..3b74c7436 --- /dev/null +++ b/openless-all/app/scripts/test-compare-audio-outputs.py @@ -0,0 +1,35 @@ +"""Negative controls for the before/after PCM comparison gate.""" +import json +from pathlib import Path +import subprocess +import sys +import tempfile +import unittest + + +class ComparisonTests(unittest.TestCase): + def test_exact_difference_and_truncation(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + before, after = root / "before", root / "after" + for folder in (before, after): + folder.mkdir() + (folder / "results.json").write_text(json.dumps([ + {"file": "signal.pcm", "seconds": 1, "samples": 3} + ])) + (folder / "signal.pcm").write_bytes(b"\x00\x00\x01\x00\xff\xff") + report = root / "report.json" + command = [sys.executable, str(Path(__file__).with_name("compare-audio-outputs.py")), + str(before), str(after), "--report", str(report)] + def run(): + return subprocess.run(command, capture_output=True, text=True).returncode + self.assertEqual(run(), 0) + (after / "signal.pcm").write_bytes(b"\x01\x00\x01\x00\xff\xff") + self.assertEqual(run(), 1) + self.assertEqual(json.loads(report.read_text())["max_error_lsb"], 1) + (after / "signal.pcm").write_bytes(b"\x00\x00") + self.assertNotEqual(run(), 0) + + +if __name__ == "__main__": + unittest.main() diff --git a/openless-all/app/scripts/test-recorder-module.py b/openless-all/app/scripts/test-recorder-module.py index 0e061e778..799fb35b7 100644 --- a/openless-all/app/scripts/test-recorder-module.py +++ b/openless-all/app/scripts/test-recorder-module.py @@ -37,9 +37,9 @@ "#[derive(Debug, Clone)] pub struct BackendError { pub code: BackendErrorCode, pub message: String }\n" "impl BackendError { pub fn new(code: BackendErrorCode, message: impl Into) -> Self { Self { code, message: message.into() } } }\n" + "#[path = " + json.dumps(str(recorder.parents[2] / "crates/openless-core/src/audio_dsp.rs"), ensure_ascii=False) + "]\npub mod audio_dsp;\n" - "pub use audio_dsp::{AudioDspConfig, AudioDspStage, NoiseReduction};\n" + "pub use audio_dsp::{AecMode, AudioDspConfig, AudioDspStage, NoiseReduction};\n" + "#[path = " + json.dumps(str(recorder.parents[2] / "crates/openless-core/src/audio.rs"), ensure_ascii=False) + "]\npub mod audio;\n" - "pub use audio::{AudioProcessingConfig, AudioProcessor};\n" + "pub use audio::{pcm_f32_to_i16_le, AudioProcessingConfig, AudioProcessor, FarEndQueue};\n" "pub trait AudioConsumer: Send + Sync { fn consume_pcm_chunk(&self, pcm: &[u8]); }\n" + "#[path = " + json.dumps(str(recorder), ensure_ascii=False) + "]\npub mod recorder;\n" ) diff --git a/openless-all/app/src-tauri/src/android/native_bridge.rs b/openless-all/app/src-tauri/src/android/native_bridge.rs index 268185d36..3754ca609 100644 --- a/openless-all/app/src-tauri/src/android/native_bridge.rs +++ b/openless-all/app/src-tauri/src/android/native_bridge.rs @@ -22,6 +22,14 @@ static OVERLAY_VISIBLE: std::sync::atomic::AtomicBool = std::sync::atomic::Atomi #[cfg(target_os = "android")] static APP_HANDLE: OnceLock = OnceLock::new(); +#[cfg(target_os = "android")] +pub(crate) fn emit_audio_spectrum(raw: Vec, processed: Vec) { + use tauri::Emitter; + if let Some(app) = APP_HANDLE.get() { + let _ = app.emit("android-audio-spectrum", serde_json::json!({"raw": raw, "processed": processed})); + } +} + // A panic from a synchronously polled future must not cross an extern JNI // boundary. Return an ordinary error so each caller keeps its wire envelope. fn catch_jni_panic( diff --git a/openless-all/app/src-tauri/src/commands/settings.rs b/openless-all/app/src-tauri/src/commands/settings.rs index 2b869abce..cef34f0d7 100644 --- a/openless-all/app/src-tauri/src/commands/settings.rs +++ b/openless-all/app/src-tauri/src/commands/settings.rs @@ -1,5 +1,25 @@ use super::*; +#[tauri::command] +pub fn get_android_audio_effects_status() -> serde_json::Value { + #[cfg(target_os = "android")] + { + crate::recorder::android_capture::status() + } + #[cfg(not(target_os = "android"))] + { + serde_json::json!({"aecAvailable": false, "noiseSuppressionAvailable": false, "agcAvailable": false, "runtime": null}) + } +} + +#[tauri::command] +pub fn set_android_audio_monitoring(enabled: bool) { + #[cfg(target_os = "android")] + crate::recorder::android_capture::set_monitoring(enabled); + #[cfg(not(target_os = "android"))] + let _ = enabled; +} + #[tauri::command] pub fn get_settings(core: CoreState<'_>) -> UserPreferences { let mut prefs = core.get_preferences(); @@ -239,7 +259,10 @@ pub(crate) fn sync_low_latency_capture_preferences(prefs: &UserPreferences) { #[cfg(not(target_os = "windows"))] pub(crate) fn sync_low_latency_capture_preferences(_prefs: &UserPreferences) {} -fn voice_activation_preferences_changed(previous: &UserPreferences, next: &UserPreferences) -> bool { +fn voice_activation_preferences_changed( + previous: &UserPreferences, + next: &UserPreferences, +) -> bool { previous.voice_activation_enabled != next.voice_activation_enabled || previous.voice_activation_wake_enabled != next.voice_activation_wake_enabled || previous.voice_activation_finish_enabled != next.voice_activation_finish_enabled @@ -250,9 +273,7 @@ fn voice_activation_preferences_changed(previous: &UserPreferences, next: &UserP || previous.microphone_device_name != next.microphone_device_name } -fn validate_voice_activation_preferences( - prefs: &UserPreferences, -) -> Result<(), String> { +fn validate_voice_activation_preferences(prefs: &UserPreferences) -> Result<(), String> { openless_core::VoiceActivationConfig { enabled: prefs.voice_activation_enabled, wake_enabled: prefs.voice_activation_wake_enabled, diff --git a/openless-all/app/src-tauri/src/core_adapters.rs b/openless-all/app/src-tauri/src/core_adapters.rs index d3b383201..6d1b63783 100644 --- a/openless-all/app/src-tauri/src/core_adapters.rs +++ b/openless-all/app/src-tauri/src/core_adapters.rs @@ -2964,13 +2964,13 @@ impl TauriStagedCapture { self, consumer: Arc, level_handler: Arc, - archive_path: Option, + archives: crate::recorder::ArchiveTargets, processing: openless_core::AudioProcessingConfig, ) -> Result { match self { Self::Cold(staged) => { let (recorder, runtime_errors, archive_active) = - staged.commit(consumer, level_handler, archive_path)?; + staged.commit(consumer, level_handler, archives)?; Ok(TauriCommittedCapture::Cold { recorder, runtime_errors, @@ -2983,8 +2983,8 @@ impl TauriStagedCapture { let fallback_hub = staged.capture_hub(); let fallback_consumer = Arc::clone(&consumer); let fallback_level_handler = Arc::clone(&level_handler); - let fallback_archive_path = archive_path.clone(); - match staged.commit(consumer, level_handler, archive_path) { + let fallback_archives = archives.clone(); + match staged.commit(consumer, level_handler, archives) { Ok((capture, runtime_errors, archive_active)) => { Ok(TauriCommittedCapture::Warm { capture, @@ -3022,7 +3022,7 @@ impl TauriStagedCapture { }) .unwrap_or(fallback_consumer), fallback_level_handler, - fallback_archive_path, + fallback_archives, ) { Ok(committed) => committed, Err(error) => { @@ -3128,6 +3128,15 @@ impl PendingAudioCapture for TauriPendingAudioCapture { }) .ok() .flatten(); + // 调试录音：主档案写处理后（或只勾处理前时写原始），需要对照时 + // 同目录再存一份 `.raw.wav`。 + let archives = crate::recorder::ArchiveTargets { + processed: archive_path.clone(), + raw: (recording_plan.archive_raw && archive_path.is_some()) + .then(|| archive_path.as_ref().map(|path| path.with_extension("raw.wav"))) + .flatten(), + write_processed: recording_plan.archive_processed, + }; let started_at = Instant::now(); let level_progress = Arc::clone(&progress); let level_handler: Arc = Arc::new(move |level| { @@ -3154,7 +3163,7 @@ impl PendingAudioCapture for TauriPendingAudioCapture { .commit( consumer, level_handler, - archive_path.clone(), + archives.clone(), recording_plan.audio_processing, ) .map_err(map_recorder_error) diff --git a/openless-all/app/src-tauri/src/lib.rs b/openless-all/app/src-tauri/src/lib.rs index e0cf76be2..c385a1df8 100644 --- a/openless-all/app/src-tauri/src/lib.rs +++ b/openless-all/app/src-tauri/src/lib.rs @@ -560,6 +560,8 @@ macro_rules! app_invoke_handler_mobile { tauri::generate_handler![ $crate::commands::get_startup_snapshot, $crate::commands::get_settings, + $crate::commands::get_android_audio_effects_status, + $crate::commands::set_android_audio_monitoring, $crate::commands::get_settings_snapshot, $crate::commands::update_setting_fields, $crate::commands::get_default_style_system_prompts, diff --git a/openless-all/app/src-tauri/src/recorder.rs b/openless-all/app/src-tauri/src/recorder.rs index 40cf087d1..4611df06c 100644 --- a/openless-all/app/src-tauri/src/recorder.rs +++ b/openless-all/app/src-tauri/src/recorder.rs @@ -13,6 +13,25 @@ //! `drop`ped inside that thread. use std::collections::{HashMap, HashSet, VecDeque}; +#[cfg(target_os = "android")] +#[path = "recorder/android_capture.rs"] +pub(crate) mod android_capture; + +struct CaptureStream { + _cpal: Option, + #[cfg(target_os = "android")] + _android: Option, +} + +impl From for CaptureStream { + fn from(stream: cpal::Stream) -> Self { + Self { + _cpal: Some(stream), + #[cfg(target_os = "android")] + _android: None, + } + } +} use std::path::{Path, PathBuf}; use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; use std::sync::mpsc::{channel, sync_channel, Receiver, Sender, SyncSender, TrySendError}; @@ -82,17 +101,29 @@ pub(crate) fn voice_diagnostic( }); let writer = WRITER.get_or_init(|| { let (tx, rx) = sync_channel::(4096); - std::thread::Builder::new().name("voice-1003-diagnostics".into()).spawn(move || { - let path = std::env::temp_dir().join("openless-1003-3c02ffc1.ndjson"); - let Ok(mut file) = std::fs::OpenOptions::new().create(true).append(true).open(path) else { - return; - }; - for record in rx { - let Ok(mut line) = serde_json::to_vec(&record) else { continue; }; - line.push(b'\n'); - if file.write_all(&line).is_err() { break; } - } - }).ok().map(|_| tx) + std::thread::Builder::new() + .name("voice-1003-diagnostics".into()) + .spawn(move || { + let path = std::env::temp_dir().join("openless-1003-3c02ffc1.ndjson"); + let Ok(mut file) = std::fs::OpenOptions::new() + .create(true) + .append(true) + .open(path) + else { + return; + }; + for record in rx { + let Ok(mut line) = serde_json::to_vec(&record) else { + continue; + }; + line.push(b'\n'); + if file.write_all(&line).is_err() { + break; + } + } + }) + .ok() + .map(|_| tx) }); if let Some(writer) = writer { let record = serde_json::json!({ @@ -105,7 +136,9 @@ pub(crate) fn voice_diagnostic( "ts": std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH) .unwrap_or_default().as_millis().to_string(), }); - if writer.try_send(record).is_err() { DROPPED.fetch_add(1, Ordering::Relaxed); } + if writer.try_send(record).is_err() { + DROPPED.fetch_add(1, Ordering::Relaxed); + } } } #[cfg(any(not(target_os = "windows"), test))] @@ -262,6 +295,71 @@ impl WavArchiveWriter { } } +/// 调试录音要落哪几份盘（与 egui 侧同名同义）。 +/// +/// - `processed`：主档案 `.wav`，`write_processed` 时写降噪/增益之后的 +/// PCM，否则写原始 PCM（只勾「处理前」时用，回退/播放功能不变）。 +/// - `raw`：额外的 `.raw.wav`，只在同时要两份时创建，用于 A/B 对照。 +#[derive(Clone, Default)] +pub(crate) struct ArchiveTargets { + pub processed: Option, + pub raw: Option, + pub write_processed: bool, +} + +impl ArchiveTargets { + pub(crate) fn processed_only(path: Option) -> Self { + Self { + processed: path, + raw: None, + write_processed: true, + } + } + + fn canonical(&self) -> Option { + self.processed.clone().or_else(|| self.raw.clone()) + } +} + +/// 「处理前」音频的落盘通道：DSP 线程在处理每一帧之前先按这里落原始 PCM。 +#[derive(Clone)] +pub(crate) struct RawTap { + /// 额外的 `.raw.wav`（只在两份都要时存在）。 + raw_writer: Option>, + /// 主档案；`write_processed == false` 时原始 PCM 直接写进这里。 + canonical: Option>, + write_processed: bool, +} + +impl RawTap { + pub(crate) fn new( + canonical: Option>, + raw_writer: Option>, + write_processed: bool, + ) -> Self { + Self { + raw_writer, + canonical, + write_processed, + } + } + + fn append_raw(&self, samples: &[f32]) { + if self.raw_writer.is_none() && self.write_processed { + return; + } + let pcm = openless_core::pcm_f32_to_i16_le(samples); + if let Some(raw) = self.raw_writer.as_ref() { + raw.append(&pcm); + } + if !self.write_processed { + if let Some(canonical) = self.canonical.as_ref() { + canonical.append(&pcm); + } + } + } +} + #[derive(Clone)] struct ForwardTarget { consumer: Arc, @@ -281,9 +379,16 @@ struct CaptureSink { state: Mutex, delivery_gate: Mutex<()>, max_bytes: usize, + /// 处理前音频的落盘通道，由 `promote` 设置；DSP 线程每帧读一次。 + raw_tap: Arc>>>, } impl CaptureSink { + /// DSP 线程用它读「处理前」落盘通道；`promote` 时写入。 + fn raw_tap_slot(&self) -> Arc>>> { + Arc::clone(&self.raw_tap) + } + fn new(max_bytes: usize) -> Arc { let max_bytes = (max_bytes.max(2) & !1).min(STAGED_CAPTURE_MAX_BYTES.max(2)); Arc::new(Self { @@ -292,6 +397,7 @@ impl CaptureSink { }), delivery_gate: Mutex::new(()), max_bytes, + raw_tap: Arc::new(Mutex::new(None)), }) } @@ -307,7 +413,11 @@ impl CaptureSink { consumer: Arc, level_handler: Arc, archive: Option>, + raw_tap: Option>, ) -> Result<(), RecorderError> { + // 只要处理前时，主档案由 DSP 线程写原始 PCM，这里就不能再写处理后。 + let write_processed = raw_tap.as_ref().map_or(true, |tap| tap.write_processed); + *self.raw_tap.lock() = raw_tap; let _delivery = self.delivery_gate.lock(); let prefix = { let mut state = self.state.lock(); @@ -326,7 +436,7 @@ impl CaptureSink { *state = CaptureSinkState::Promoted(ForwardTarget { consumer, level_handler, - archive, + archive: if write_processed { archive } else { None }, }); prefix }; @@ -414,6 +524,8 @@ struct WarmCaptureSink { activation_fault_handler: Mutex)>>, activation_vad: Mutex, max_bytes: usize, + /// 处理前音频通道：与 `CaptureSink` 同样的机制，热麦克风路径也要能落原始 PCM。 + raw_tap: Arc>>>, } struct ActivationVadOutput { @@ -520,6 +632,7 @@ impl WarmCaptureSink { activation_fault_handler: Mutex::new(None), activation_vad: Mutex::new(ActivationVad::default()), max_bytes, + raw_tap: Arc::new(Mutex::new(None)), }) } @@ -531,7 +644,8 @@ impl WarmCaptureSink { } fn set_activation_consumer(&self, consumer: Option>) { - self.activation_consumer_during_capture.store(false, Ordering::Release); + self.activation_consumer_during_capture + .store(false, Ordering::Release); *self.activation_consumer.lock() = consumer; self.activation_vad.lock().reset(); } @@ -592,7 +706,10 @@ impl WarmCaptureSink { ) { let even_len = pcm.len() & !1; let incoming = &pcm[..even_len]; - let excess = buffer.len().saturating_add(incoming.len()).saturating_sub(max_bytes); + let excess = buffer + .len() + .saturating_add(incoming.len()) + .saturating_sub(max_bytes); let evictable_preroll = excess.min(*pre_roll_bytes); for _ in 0..(evictable_preroll / 2) { let _ = buffer.pop_front(); @@ -651,8 +768,10 @@ impl WarmCaptureSink { consumer: Arc, level_handler: Arc, archive: Option>, + raw_tap: Option>, ) -> Result<(), RecorderError> { let _delivery = self.delivery_gate.lock(); + *self.raw_tap.lock() = raw_tap; let (prefix, staged_level) = { let mut state = self.state.lock(); let WarmCaptureSinkState::Ring { claims, .. } = &mut *state; @@ -827,8 +946,8 @@ impl AudioConsumer for WarmCaptureSink { && self .activation_consumer_during_capture .load(Ordering::Acquire))) - .then(|| self.activation_consumer.lock().clone()) - .flatten(); + .then(|| self.activation_consumer.lock().clone()) + .flatten(); for target in targets { CaptureSink::deliver(&target, pcm); } @@ -890,9 +1009,20 @@ impl WarmStagedRecorder { mut self, consumer: Arc, level_handler: Arc, - audio_archive_path: Option, + archives: ArchiveTargets, ) -> Result<(WarmActiveCapture, Receiver, bool), RecorderError> { - let archive_path = audio_archive_path; + let archive_path = archives.canonical(); + let raw_writer = archives + .raw + .as_ref() + .filter(|_| archives.processed.is_some()) + .and_then(|path| match WavArchiveWriter::create(path) { + Ok(writer) => Some(Arc::new(writer)), + Err(error) => { + log::warn!("[recorder] raw archive create failed at {path:?}: {error}"); + None + } + }); let archive_writer = archive_path .as_ref() .and_then(|path| match WavArchiveWriter::create(path) { @@ -902,12 +1032,18 @@ impl WarmStagedRecorder { None } }); + let raw_tap = Some(Arc::new(RawTap::new( + archive_writer.clone(), + raw_writer, + archives.write_processed, + ))); let archive_active = archive_writer.is_some(); let result = self.hub.promote( self.press_id, consumer, level_handler, archive_writer.clone(), + raw_tap.clone(), ); let runtime_errors = match result { Ok(runtime_errors) => runtime_errors, @@ -1068,7 +1204,10 @@ impl WarmCaptureHub { pub(crate) fn clear_activation_fault_handler_if(&self, generation: u64) { let mut handler = self.sink.activation_fault_handler.lock(); - if handler.as_ref().is_some_and(|(current, _)| *current == generation) { + if handler + .as_ref() + .is_some_and(|(current, _)| *current == generation) + { *handler = None; } } @@ -1170,12 +1309,13 @@ impl WarmCaptureHub { let level_handler: Arc = Arc::new(move |level| { sink_for_level.publish_level(level); }); - let (recorder, runtime_errors, _) = Recorder::start_with_processing( + let (recorder, runtime_errors, _) = Recorder::start_with_archive_targets( device_name, consumer, level_handler, - None, + ArchiveTargets::default(), *self.processing_config.lock(), + Some(Arc::clone(&self.sink.raw_tap)), )?; let generation = self.stream_generation.fetch_add(1, Ordering::AcqRel) + 1; *self.recorder.lock() = Some(recorder); @@ -1232,12 +1372,13 @@ impl WarmCaptureHub { consumer: Arc, level_handler: Arc, archive: Option>, + raw_tap: Option>, ) -> Result, RecorderError> { let _lifecycle = self.lifecycle_gate.lock(); let (runtime_error_tx, runtime_error_rx) = channel(); if let Err(error) = self .sink - .promote(press_id, consumer, level_handler, archive) + .promote(press_id, consumer, level_handler, archive, raw_tap) { let fallback_allowed = self.runtime_fallback_claims.lock().remove(&press_id); if !fallback_allowed @@ -1559,6 +1700,26 @@ impl Recorder { level_handler: Arc, audio_archive_path: Option, processing: openless_core::AudioProcessingConfig, + ) -> Result<(Self, Receiver, bool), RecorderError> { + Self::start_with_archive_targets( + microphone_device_name, + consumer, + level_handler, + ArchiveTargets::processed_only(audio_archive_path), + processing, + None, + ) + } + + /// 由调用方决定主档案写哪份 PCM、是否额外写一份处理前的 `.raw.wav`， + /// 以及 AEC 的扬声器参考来源。 + pub fn start_with_archive_targets( + microphone_device_name: Option, + consumer: Arc, + level_handler: Arc, + archives: ArchiveTargets, + processing: openless_core::AudioProcessingConfig, + raw_tap_slot: Option>>>>, ) -> Result<(Self, Receiver, bool), RecorderError> { // Startup signal: the child thread reports the result via startup_tx once the Stream // is constructed. @@ -1575,15 +1736,36 @@ impl Recorder { // immediately and passed to the caller to decide whether history marks // has_audio_recording. Failure only logs a warning, no error; the main path continues. let archive_writer = - audio_archive_path.and_then(|path| match WavArchiveWriter::create(&path) { + archives + .canonical() + .and_then(|path| match WavArchiveWriter::create(&path) { + Ok(writer) => Some(Arc::new(writer)), + Err(err) => { + log::warn!("[recorder] wav archive create failed at {path:?}: {err}"); + None + } + }); + // 只有两份都要时才创建第二份档案。 + let raw_writer = archives + .raw + .as_ref() + .filter(|_| archives.processed.is_some()) + .and_then(|path| match WavArchiveWriter::create(path) { Ok(writer) => Some(Arc::new(writer)), Err(err) => { - log::warn!("[recorder] wav archive create failed at {path:?}: {err}"); + log::warn!("[recorder] raw archive create failed at {path:?}: {err}"); None } }); + let write_processed = archives.write_processed; let archive_active = archive_writer.is_some(); let archive_for_thread = archive_writer.clone(); + let raw_for_thread = raw_writer.clone(); + // AEC 的扬声器参考：与录音同生命周期，Drop 时停流。 + let far_end = (processing.dsp.aec_mode != openless_core::AecMode::Off) + .then(openless_core::FarEndQueue::default); + let far_end_for_thread = far_end.clone(); + let raw_tap_slot_for_thread = raw_tap_slot.clone(); let join_handle = match thread::Builder::new() .name("openless-recorder".into()) @@ -1593,6 +1775,10 @@ impl Recorder { consumer, level_handler, archive_for_thread, + raw_for_thread, + write_processed, + far_end_for_thread, + raw_tap_slot_for_thread, processing, state_for_thread, stop_for_thread, @@ -1701,12 +1887,14 @@ impl Recorder { let level_handler: Arc = Arc::new(move |level| { sink_for_level.publish_level(level); }); - let (recorder, runtime_errors, _) = Self::start_with_processing( + let raw_tap_slot = sink.raw_tap_slot(); + let (recorder, runtime_errors, _) = Recorder::start_with_archive_targets( microphone_device_name, consumer, level_handler, - None, + ArchiveTargets::default(), processing, + Some(raw_tap_slot), )?; Ok(StagedRecorder { recorder: Some(recorder), @@ -1728,9 +1916,9 @@ impl StagedRecorder { mut self, consumer: Arc, level_handler: Arc, - audio_archive_path: Option, + archives: ArchiveTargets, ) -> Result<(Recorder, Receiver, bool), RecorderError> { - let archive_path = audio_archive_path; + let archive_path = archives.canonical(); let archive_writer = archive_path .as_ref() .and_then(|path| match WavArchiveWriter::create(path) { @@ -1740,10 +1928,26 @@ impl StagedRecorder { None } }); + let raw_writer = archives + .raw + .as_ref() + .filter(|_| archives.processed.is_some()) + .and_then(|path| match WavArchiveWriter::create(path) { + Ok(writer) => Some(Arc::new(writer)), + Err(error) => { + log::warn!("[recorder] raw archive create failed at {path:?}: {error}"); + None + } + }); + let raw_tap = Some(Arc::new(RawTap::new( + archive_writer.clone(), + raw_writer, + archives.write_processed, + ))); let archive_active = archive_writer.is_some(); - if let Err(error) = self - .sink - .promote(consumer, level_handler, archive_writer.clone()) + if let Err(error) = + self.sink + .promote(consumer, level_handler, archive_writer.clone(), raw_tap) { if let Some(archive) = archive_writer { archive.finish(); @@ -1813,6 +2017,94 @@ pub fn list_input_devices() -> Result, RecorderError> { Ok(result) } +/// 打开「扬声器回环」参考采集，供 AEC 使用。 +/// +/// 走的是宿主默认**输出**设备：Windows 的 WASAPI 允许在 render 设备上开 capture +/// 流（loopback），Linux 的 PulseAudio/PipeWire 会把输出映射到 monitor 源，macOS +/// 走 CoreAudio 的同一设备。没有可用回环源时返回错误，调用方记一条 warn 后继续 +/// 录音（AEC 退化为无参考），绝不让整次录音失败。 +#[cfg(not(any(target_os = "android", target_os = "ios")))] +fn start_reference_stream( + stop_flag: &Arc, + queue: &openless_core::FarEndQueue, +) -> Result { + use cpal::traits::{DeviceTrait, HostTrait, StreamTrait}; + + let host = cpal::default_host(); + let device = host + .default_output_device() + .ok_or_else(|| RecorderError::EngineFailed("no default output device".into()))?; + let supported = device + .default_output_config() + .map_err(|error| RecorderError::EngineFailed(format!("default output config: {error}")))?; + let config = supported.config(); + let channels = usize::from(config.channels); + let rate = config.sample_rate.0; + let device_name = device.name().unwrap_or_default(); + let queue = queue.clone(); + let stop_for_callback = Arc::clone(stop_flag); + + macro_rules! make_reference_stream { + ($t:ty, $to_f32:expr) => {{ + let queue = queue.clone(); + let stop_for_callback = Arc::clone(&stop_for_callback); + device.build_input_stream( + &config, + move |data: &[$t], _: &cpal::InputCallbackInfo| { + if stop_for_callback.load(Ordering::Acquire) { + return; + } + let samples: Vec = data.iter().copied().map($to_f32).collect(); + queue.push(&samples, channels, rate); + }, + |error| log::warn!("[recorder] speaker reference stream error: {error}"), + None, + ) + }}; + } + + let stream = match supported.sample_format() { + SampleFormat::F32 => make_reference_stream!(f32, |sample: f32| sample), + SampleFormat::I16 => { + make_reference_stream!(i16, |sample: i16| sample as f32 / i16::MAX as f32) + } + SampleFormat::I32 => { + make_reference_stream!(i32, |sample: i32| sample as f32 / i32::MAX as f32) + } + SampleFormat::I8 => make_reference_stream!(i8, |sample: i8| sample as f32 / i8::MAX as f32), + SampleFormat::U8 => { + make_reference_stream!(u8, |sample: u8| (sample as f32 - 128.0) / 128.0) + } + other => { + return Err(RecorderError::EngineFailed(format!( + "unsupported speaker reference sample format: {other:?}" + ))) + } + } + .map_err(|error| { + RecorderError::EngineFailed(format!("build speaker reference stream: {error}")) + })?; + stream + .play() + .map_err(|error| RecorderError::EngineFailed(format!("play speaker reference: {error}")))?; + log::info!( + "[recorder] speaker reference capturing outputDevice={} rate={} channels={channels}", + device_name, + rate + ); + Ok(stream) +} + +#[cfg(any(target_os = "android", target_os = "ios"))] +fn start_reference_stream( + _stop_flag: &Arc, + _queue: &openless_core::FarEndQueue, +) -> Result { + Err(RecorderError::EngineFailed( + "speaker loopback is not available on mobile".into(), + )) +} + /// Audio thread body: build Stream -> report via startup_tx -> loop until stop_flag. /// `archiver` was already attempted by the caller on the synchronous path (Ok -> Some / /// failed -> None); this only threads it through to build_input_stream for the cpal callback. @@ -1821,18 +2113,27 @@ fn run_audio_thread( consumer: Arc, level_handler: Arc, archiver: Option>, + raw_archiver: Option>, + write_processed: bool, + far_end: Option, + raw_tap_slot: Option>>>>, processing: openless_core::AudioProcessingConfig, state: Arc, stop_flag: Arc, startup_tx: Sender>, runtime_error_tx: Sender, ) { - let (processor, frame_tx) = match AudioProcessingWorker::start( + let far_end_for_reference = far_end.clone(); + let (processor, frame_tx) = match AudioProcessingWorker::start_with_targets( processing, consumer, level_handler, archiver, + raw_archiver, Arc::clone(&state), + write_processed, + far_end, + raw_tap_slot, ) { Ok(worker) => worker, Err(error) => { @@ -1841,12 +2142,26 @@ fn run_audio_thread( } }; *state.processor_sender.lock() = Some(frame_tx.clone()); + // 扬声器回环参考：`cpal::Stream` 不是 Send，只能在音频线程里创建， + // 并随本线程结束一起释放（和麦克风流一样）。 + let _reference_stream = far_end_for_reference.as_ref().and_then(|queue| { + match start_reference_stream(&stop_flag, queue) { + Ok(stream) => Some(stream), + Err(error) => { + log::warn!( + "[recorder] speaker reference unavailable; continuing without AEC: {error}" + ); + None + } + } + }); let (stream, state) = match build_input_stream( microphone_device_name, frame_tx.clone(), Arc::clone(&state), Arc::clone(&stop_flag), runtime_error_tx.clone(), + processing, ) { Ok(s) => s, Err(err) => { @@ -1977,8 +2292,10 @@ fn run_audio_thread( // backend it calls AudioOutputUnitStop directly, terminating the callback synchronously. // The subsequent drop handles dispose / resource release. If pause fails, warn only and // do not block the drop. - if let Err(err) = stream.pause() { - log::warn!("[recorder] cpal Stream pause before drop failed: {err}"); + if let Some(cpal_stream) = &stream._cpal { + if let Err(err) = cpal_stream.pause() { + log::warn!("[recorder] cpal Stream pause before drop failed: {err}"); + } } drop(stream); log::info!("[recorder] cpal Stream dropped (mic released)"); @@ -2000,7 +2317,49 @@ fn build_input_stream( state: Arc, stop_flag: Arc, runtime_error_tx: Sender, -) -> Result<(cpal::Stream, Arc), RecorderError> { + _processing: openless_core::AudioProcessingConfig, +) -> Result<(CaptureStream, Arc), RecorderError> { + #[cfg(target_os = "android")] + if _processing.enhancement_enabled { + let config = _processing.dsp; + if config.android_aec_enabled + || config.android_noise_suppression_enabled + || config.android_agc_enabled + { + if microphone_device_name + .as_ref() + .is_some_and(|name| !name.trim().is_empty()) + { + android_capture::record_failure( + "System audio effects require the default Android input".into(), + ); + } else { + match android_capture::AndroidCapture::start( + config, + frame_tx.clone(), + Arc::clone(&state), + runtime_error_tx.clone(), + ) { + Ok(capture) => { + return Ok(( + CaptureStream { + _cpal: None, + _android: Some(capture), + }, + state, + )) + } + Err(error) => { + android_capture::record_failure(error.clone()); + if error.starts_with("busy:") { + return Err(RecorderError::EngineFailed(error)); + } + log::warn!("[android-audio] system effects unavailable; falling back to native capture: {error}"); + } + } + } + } + } let selecting_at = std::time::Instant::now(); let host = cpal::default_host(); let selected = select_input_device(&host, microphone_device_name.as_deref())?; @@ -2008,6 +2367,7 @@ fn build_input_stream( let selected_name = selected.name().ok(); let start = |device: &cpal::Device| { start_device_stream(device, &frame_tx, &state, &stop_flag, &runtime_error_tx) + .map(|(stream, state)| (stream.into(), state)) }; let initial_error = match start(&selected) { Ok(stream) => return Ok(stream), @@ -2054,6 +2414,24 @@ fn start_device_stream( stop_flag: &Arc, runtime_error_tx: &Sender, ) -> Result<(cpal::Stream, Arc), RecorderError> { + // Android DSP and neural denoisers share a 48 kHz capture contract. + // Keep native fallback for devices that cannot open this rate. + #[cfg(target_os = "android")] + if let Ok(ranges) = device.supported_input_configs() { + for range in ranges.filter(|range| { + range.min_sample_rate().0 <= 48_000 && range.max_sample_rate().0 >= 48_000 + }) { + let config = range.with_sample_rate(cpal::SampleRate(48_000)); + match start_native_config(device, config, frame_tx, state, stop_flag, runtime_error_tx) + { + Ok(stream) => return Ok(stream), + Err(RecorderError::PermissionDenied) => { + return Err(RecorderError::PermissionDenied) + } + Err(error) => log::warn!("[recorder] Android 48 kHz capture failed: {error}"), + } + } + } let mut default = None; let initial_error = match device.default_input_config() { Ok(config) => { @@ -2454,6 +2832,33 @@ impl AudioProcessingWorker { level_handler: Arc, archiver: Option>, state: Arc, + ) -> Result<(Self, AudioProcessingSender), RecorderError> { + Self::start_with_targets( + processing, + consumer, + level_handler, + archiver, + None, + state, + true, + None, + None, + ) + } + + /// `write_processed = false` 时主档案写原始 PCM；`raw_archiver` 是额外的 + /// 「处理前」档案（只在两份都要时给出）。`far_end` 是 AEC 的扬声器参考队列。 + #[allow(clippy::too_many_arguments)] + fn start_with_targets( + processing: openless_core::AudioProcessingConfig, + consumer: Arc, + level_handler: Arc, + archiver: Option>, + raw_archiver: Option>, + state: Arc, + write_processed: bool, + far_end: Option, + raw_tap_slot: Option>>>>, ) -> Result<(Self, AudioProcessingSender), RecorderError> { let (sender, receiver) = sync_channel::(AUDIO_PROCESSING_QUEUE_CAPACITY); @@ -2465,13 +2870,20 @@ impl AudioProcessingWorker { let join_handle = thread::Builder::new() .name("openless-audio-processor".into()) .spawn(move || { + #[cfg(target_os = "android")] + android_capture::prioritize_current_thread(); let mut processor = openless_core::AudioProcessor::new(processing); + if let Some(queue) = far_end.clone() { + processor.attach_far_end(queue); + } let mut enhancement_enabled = processing.enhancement_enabled; let deliver = |chunk: openless_core::audio::NormalizedPcmChunk| { state.note_first_pcm(true); consumer.consume_pcm_chunk(&chunk.pcm_i16_le); - if let Some(archiver) = &archiver { - archiver.append(&chunk.pcm_i16_le); + if write_processed { + if let Some(archiver) = &archiver { + archiver.append(&chunk.pcm_i16_le); + } } level_handler(chunk.level); update_peak(&state.peak_output_rms_milli, chunk.level); @@ -2481,6 +2893,27 @@ impl AudioProcessingWorker { let QueuedAudioProcessingMessage { message, _reservation } = queued; match message { AudioProcessingMessage::Frame(frame) => { + #[cfg(target_os = "android")] + processor.set_monitoring_enabled(android_capture::monitoring_enabled()); + // 「处理前」的原始帧：额外 raw 档案永远拿原始 PCM； + // 只要处理前、不要处理后时，主档案也写原始 PCM，回退与 + // 播放继续可用，且不会多出一份重复文件。 + if raw_archiver.is_some() || !write_processed { + let pcm = openless_core::pcm_f32_to_i16_le(&frame.samples); + if let Some(raw) = &raw_archiver { + raw.append(&pcm); + } + if !write_processed { + if let Some(canonical) = &archiver { + canonical.append(&pcm); + } + } + } + if let Some(slot) = raw_tap_slot.as_ref() { + if let Some(tap) = slot.lock().clone() { + tap.append_raw(&frame.samples); + } + } let queued_us = backlog.micros.load(Ordering::Acquire); if enhancement_enabled && queued_us >= AUDIO_PROCESSING_SHED_ENHANCEMENT_US { log::warn!("[recorder] disabling enhancement for this capture: processing backlog {} ms", queued_us / 1_000); @@ -2496,6 +2929,8 @@ impl AudioProcessingWorker { ) { deliver(chunk); } + #[cfg(target_os = "android")] + android_capture::publish_monitor(&processor); } AudioProcessingMessage::Flush(done) => { if let Some(chunk) = processor.finish() { @@ -3210,6 +3645,7 @@ mod tests { Arc::clone(&formal) as Arc, Arc::new(|_| {}), None, + None, ) .unwrap(); @@ -3233,6 +3669,7 @@ mod tests { Arc::clone(&formal) as Arc, Arc::new(|_| {}), None, + None, ) .unwrap(); @@ -3562,6 +3999,7 @@ mod tests { Arc::clone(&consumer) as Arc, Arc::new(move |level| levels_for_handler.lock().unwrap().push(level)), None, + None, ) .unwrap(); sink.consume_pcm_chunk(&[5, 6]); @@ -3584,7 +4022,12 @@ mod tests { sink.consume_pcm_chunk(&[1, 2]); sink.discard(); assert!(sink - .promote(consumer as Arc, Arc::new(|_| {}), None) + .promote( + consumer as Arc, + Arc::new(|_| {}), + None, + None + ) .is_err()); sink.consume_pcm_chunk(&[3, 4]); let is_discarded = matches!(&*sink.state.lock(), CaptureSinkState::Discarded); @@ -3607,6 +4050,7 @@ mod tests { consumer.clone(), Arc::new(move |level| levels_for_handler.lock().unwrap().push(level)), None, + None, ) .unwrap(); sink.consume_pcm_chunk(&[11, 12]); @@ -3628,8 +4072,14 @@ mod tests { sink.set_activation_consumer(None); sink.consume_pcm_chunk(&[5, 6]); - assert_eq!(consumer.chunks.lock().unwrap().as_slice(), &[vec![1, 2, 3, 4]]); - assert!(matches!(&*sink.state.lock(), WarmCaptureSinkState::Ring { .. })); + assert_eq!( + consumer.chunks.lock().unwrap().as_slice(), + &[vec![1, 2, 3, 4]] + ); + assert!(matches!( + &*sink.state.lock(), + WarmCaptureSinkState::Ring { .. } + )); } #[test] @@ -3639,7 +4089,7 @@ mod tests { sink.arm(19, false).unwrap(); sink.consume_pcm_chunk(&[5, 6]); let consumer = Arc::new(RecordingConsumer::default()); - sink.promote(19, consumer.clone(), Arc::new(|_| {}), None) + sink.promote(19, consumer.clone(), Arc::new(|_| {}), None, None) .unwrap(); assert_eq!(consumer.chunks.lock().unwrap().as_slice(), &[vec![5, 6]]); } @@ -3652,7 +4102,7 @@ mod tests { sink.consume_pcm_chunk(&[17, 18, 19, 20, 21, 22]); let consumer = Arc::new(RecordingConsumer::default()); - sink.promote(17, consumer.clone(), Arc::new(|_| {}), None) + sink.promote(17, consumer.clone(), Arc::new(|_| {}), None, None) .unwrap(); assert_eq!( @@ -3670,7 +4120,7 @@ mod tests { sink.consume_pcm_chunk(&[17, 18, 19, 20, 21, 22]); let consumer = Arc::new(RecordingConsumer::default()); - sink.promote(17, consumer.clone(), Arc::new(|_| {}), None) + sink.promote(17, consumer.clone(), Arc::new(|_| {}), None, None) .unwrap(); assert_eq!( @@ -3690,7 +4140,7 @@ mod tests { let first = Arc::new(RecordingConsumer::default()); let second = Arc::new(RecordingConsumer::default()); - sink.promote(18, second.clone(), Arc::new(|_| {}), None) + sink.promote(18, second.clone(), Arc::new(|_| {}), None, None) .unwrap(); sink.consume_pcm_chunk(&[5, 6]); @@ -3699,7 +4149,9 @@ mod tests { second.chunks.lock().unwrap().as_slice(), &[vec![1, 2, 3, 4], vec![5, 6]] ); - assert!(sink.promote(17, first, Arc::new(|_| {}), None).is_err()); + assert!(sink + .promote(17, first, Arc::new(|_| {}), None, None) + .is_err()); } fn complete_hub_operation(operation: impl FnOnce() + Send + 'static) { @@ -3768,6 +4220,7 @@ mod tests { Arc::new(RecordingConsumer::default()), Arc::new(|_| {}), None, + None, ) .unwrap(); } diff --git a/openless-all/app/src-tauri/src/recorder/android_capture.rs b/openless-all/app/src-tauri/src/recorder/android_capture.rs new file mode 100644 index 000000000..801810ef4 --- /dev/null +++ b/openless-all/app/src-tauri/src/recorder/android_capture.rs @@ -0,0 +1,340 @@ +//! Optional Android system preprocessing, attached to the actual recording +//! session. Inspired by MicYou's AudioRecord lifecycle; no playback capture. +use super::*; +use crate::android::jni::android::with_android_env; +use jni::{ + objects::{GlobalRef, JValue}, + JNIEnv, JavaVM, +}; +use std::time::Duration; + +static OWNER: AtomicBool = AtomicBool::new(false); +static STATUS: std::sync::Mutex> = std::sync::Mutex::new(None); +const RATE: u32 = 48_000; +const SAMPLES: usize = 960; // 20 ms; bounds JNI work and processing latency. +static MONITOR_UNTIL: AtomicU64 = AtomicU64::new(0); +static MONITOR_NEXT: AtomicU64 = AtomicU64::new(0); + +fn now_ms() -> u64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default() + .as_millis() as u64 +} + +pub fn set_monitoring(enabled: bool) { + MONITOR_UNTIL.store(if enabled { now_ms() + 2500 } else { 0 }, Ordering::Release); +} + +pub fn monitoring_enabled() -> bool { + now_ms() < MONITOR_UNTIL.load(Ordering::Acquire) +} + +pub fn publish_monitor(processor: &openless_core::AudioProcessor) { + let now = now_ms(); + if now < MONITOR_NEXT.load(Ordering::Relaxed) || !monitoring_enabled() { + return; + } + MONITOR_NEXT.store(now + 100, Ordering::Relaxed); + if let Some((raw, processed)) = processor.spectrums() { + crate::android::native_bridge::emit_audio_spectrum(raw, processed); + } +} +const EFFECTS: [&str; 3] = [ + "android/media/audiofx/AcousticEchoCanceler", + "android/media/audiofx/NoiseSuppressor", + "android/media/audiofx/AutomaticGainControl", +]; + +pub fn record_failure(error: String) { + *STATUS.lock().unwrap() = Some(serde_json::json!({"active": false, "error": error})); +} + +fn available(env: &mut JNIEnv, class: &str) -> bool { + let result = env + .call_static_method(class, "isAvailable", "()Z", &[]) + .and_then(|v| v.z()); + if result.is_err() { + let _ = env.exception_clear(); + } + result.unwrap_or(false) +} + +pub fn status() -> serde_json::Value { + let capabilities = with_android_env(|env, _| Ok(EFFECTS.map(|class| available(env, class)))) + .unwrap_or([false; 3]); + serde_json::json!({ + "aecAvailable": capabilities[0], "noiseSuppressionAvailable": capabilities[1], + "agcAvailable": capabilities[2], "runtime": STATUS.lock().unwrap().clone(), + }) +} + +pub fn prioritize_current_thread() { + let _ = with_android_env(|env, _| { + let result = env.call_static_method( + "android/os/Process", + "setThreadPriority", + "(I)V", + &[JValue::Int(-16)], + ); + if result.is_err() { + let _ = env.exception_clear(); + } + result.map(|_| ()).map_err(|e| e.to_string()) + }); +} + +struct Owner; +impl Drop for Owner { + fn drop(&mut self) { + OWNER.store(false, Ordering::Release); + } +} + +struct Record { + vm: JavaVM, + recorder: GlobalRef, + effects: Vec, + _owner: Owner, +} + +impl Drop for Record { + fn drop(&mut self) { + if let Ok(mut env) = self.vm.attach_current_thread() { + let _ = env.call_method(self.recorder.as_obj(), "stop", "()V", &[]); + let _ = env.exception_clear(); + for effect in &self.effects { + let _ = env.call_method(effect.as_obj(), "release", "()V", &[]); + let _ = env.exception_clear(); + } + let _ = env.call_method(self.recorder.as_obj(), "release", "()V", &[]); + let _ = env.exception_clear(); + } + *STATUS.lock().unwrap() = Some(serde_json::json!({"active": false, "error": null})); + } +} + +pub struct AndroidCapture { + stop: Arc, + done: Receiver<()>, +} + +impl AndroidCapture { + pub fn start( + config: openless_core::AudioDspConfig, + sender: AudioProcessingSender, + state: Arc, + errors: Sender, + ) -> Result { + OWNER + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .map_err(|_| { + "busy: previous Android AudioRecord teardown has not completed".to_string() + })?; + let owner = Owner; + let requested = [ + config.android_aec_enabled, + config.android_noise_suppression_enabled, + config.android_agc_enabled, + ]; + let record = with_android_env(|env, _| { + let result = (|| -> Result { + let min = env + .call_static_method( + "android/media/AudioRecord", + "getMinBufferSize", + "(III)I", + &[JValue::Int(RATE as i32), JValue::Int(16), JValue::Int(2)], + )? + .i()?; + if min <= 0 { + return Err(jni::errors::Error::NullPtr("AudioRecord minimum buffer")); + } + let recorder = env.new_object( + "android/media/AudioRecord", + "(IIIII)V", + &[ + JValue::Int(if requested[0] { 7 } else { 6 }), // communication / recognition + JValue::Int(RATE as i32), + JValue::Int(16), + JValue::Int(2), + JValue::Int(min.max((SAMPLES * 4) as i32)), + ], + )?; + let mut record = Record { + vm: env.get_java_vm()?, + recorder: env.new_global_ref(&recorder)?, + effects: Vec::new(), + _owner: owner, + }; + if env.call_method(&recorder, "getState", "()I", &[])?.i()? != 1 { + return Err(jni::errors::Error::NullPtr("AudioRecord initialization")); + } + let session = env + .call_method(&recorder, "getAudioSessionId", "()I", &[])? + .i()?; + for (class, enabled) in EFFECTS.into_iter().zip(requested) { + if !available(env, class) { + if enabled { + return Err(jni::errors::Error::NullPtr( + "requested Android audio effect unavailable", + )); + } + continue; + } + let signature = format!("(I)L{class};"); + let effect = env + .call_static_method(class, "create", &signature, &[JValue::Int(session)])? + .l()?; + if effect.is_null() { + if enabled { + return Err(jni::errors::Error::NullPtr( + "Android audio effect creation", + )); + } + continue; + } + // Own it before enabling so failed setup also releases it. + record.effects.push(env.new_global_ref(&effect)?); + let code = env + .call_method( + &effect, + "setEnabled", + "(Z)I", + &[JValue::Bool(u8::from(enabled))], + )? + .i()?; + let actual = env.call_method(&effect, "getEnabled", "()Z", &[])?.z()?; + if code != 0 || actual != enabled { + return Err(jni::errors::Error::NullPtr( + "Android audio effect rejected setting", + )); + } + } + env.call_method(&recorder, "startRecording", "()V", &[])?; + if env + .call_method(&recorder, "getRecordingState", "()I", &[])? + .i()? + != 3 + { + return Err(jni::errors::Error::NullPtr( + "Android recording did not start", + )); + } + Ok(record) + })(); + if result.is_err() { + let _ = env.exception_clear(); + } + result.map_err(|error| error.to_string()) + })?; + *STATUS.lock().unwrap() = Some( + serde_json::json!({"active": true, "aec": requested[0], "noiseSuppression": requested[1], "agc": requested[2], "error": null}), + ); + let stop = Arc::new(AtomicBool::new(false)); + let worker_stop = Arc::clone(&stop); + let (done_tx, done) = channel(); + state.accepting_audio.store(true, Ordering::Release); + thread::Builder::new() + .name("openless-android-audio".into()) + .spawn(move || { + let result = read_audio(&record, sender, &state, &worker_stop); + if let Err(error) = result { + if !worker_stop.load(Ordering::Acquire) { + let _ = errors.send(RecorderError::EngineFailed(format!( + "Android AudioRecord: {error}" + ))); + } + } + drop(record); + let _ = done_tx.send(()); + }) + .map_err(|error| error.to_string())?; + Ok(Self { stop, done }) + } +} + +fn read_audio( + record: &Record, + sender: AudioProcessingSender, + state: &StreamState, + stop: &AtomicBool, +) -> Result<(), String> { + let mut env = record + .vm + .attach_current_thread() + .map_err(|e| e.to_string())?; + let result = env.call_static_method( + "android/os/Process", + "setThreadPriority", + "(I)V", + &[JValue::Int(-16)], + ); + if result.is_err() { + let _ = env.exception_clear(); + } + let array = env + .new_short_array(SAMPLES as i32) + .map_err(|e| e.to_string())?; + let mut samples = vec![0i16; SAMPLES]; + let result = (|| -> Result<(), jni::errors::Error> { + while !stop.load(Ordering::Acquire) && state.accepting_audio.load(Ordering::Acquire) { + // Nonblocking reads keep stop independent of an OEM's blocking read. + let count = env + .call_method( + record.recorder.as_obj(), + "read", + "([SIII)I", + &[ + JValue::Object(array.as_ref()), + JValue::Int(0), + JValue::Int(SAMPLES as i32), + JValue::Int(1), + ], + )? + .i()?; + if count < 0 { + return Err(jni::errors::Error::NullPtr("AudioRecord read failed")); + } + if count == 0 { + thread::sleep(Duration::from_millis(2)); + continue; + } + env.get_short_array_region(&array, 0, &mut samples[..count as usize])?; + if stop.load(Ordering::Acquire) || !state.accepting_audio.load(Ordering::Acquire) { + break; + } + state.note_first_pcm(false); + *state.last_callback_time.lock() = Some(std::time::Instant::now()); + let message = AudioProcessingMessage::Frame(RawAudioFrame { + samples: samples[..count as usize] + .iter() + .map(|s| *s as f32 / 32768.0) + .collect(), + channels: 1, + input_sample_rate: RATE, + }); + if sender.try_send(message).is_err() { + return Err(jni::errors::Error::NullPtr( + "Android processing queue unavailable", + )); + } + } + Ok(()) + })(); + if result.is_err() { + let _ = env.exception_clear(); + } + result.map_err(|error| error.to_string()) +} + +impl Drop for AndroidCapture { + fn drop(&mut self) { + self.stop.store(true, Ordering::Release); + // Some OEMs block stop/release. Keep their owner gate held in that + // thread, but never make UI cancellation wait for native teardown. + if self.done.recv_timeout(Duration::from_millis(250)).is_err() { + log::warn!("[android-audio] AudioRecord teardown is still pending"); + } + } +} diff --git a/openless-all/app/src/i18n/de.ts b/openless-all/app/src/i18n/de.ts index 83ca7fdbd..cac6c2cfd 100644 --- a/openless-all/app/src/i18n/de.ts +++ b/openless-all/app/src/i18n/de.ts @@ -1575,6 +1575,35 @@ export const de: typeof zhCN = { lowLatencyDictationDesc: 'Hält einen Mikrofonstream bereit und bewahrt bis zu 500 ms im Speicher, damit Sprache direkt nach dem Kurzbefehl seltener verloren geht. Vor der Bestätigung wird Audio weder gespeichert noch hochgeladen; der Stromverbrauch steigt und die Mikrofonanzeige bleibt aktiv.', dsp: { + aec: 'System echo cancellation', + systemNoise: 'System noise suppression', + systemAgc: 'System automatic gain', + unavailable: 'Unavailable on this device', + androidEffectsHint: + 'Android system effects apply before the software chain. Avoid enabling the same effect twice. Unsupported startup falls back to normal capture.', + effectsActive: 'System effects are active for this recording.', + spectrum: 'Input / processed spectrum', + spectrumHint: + 'Shows the current enhanced recording only. This does not start a microphone, save audio, or upload it. Monitoring stops when this panel closes or goes to the background.', + spectrumLegend: 'Gray: input · Color: processed · 20 Hz–24 kHz', + spectrumWaiting: 'Start a recording with speech enhancement enabled to see the spectrum.', + presets: 'Equalizer presets', + choosePreset: 'Choose a preset', + preset_normal: 'Flat', + preset_brightVocal: 'Bright vocal', + preset_deepVoice: 'Deep voice', + preset_podcast: 'Podcast', + preset_vocalClarity: 'Vocal clarity', + preset_warmVocal: 'Warm vocal', + + aecOff: 'Aus', + aecLight: 'Leicht (wenig CPU)', + aecModel: 'Modell (stärker)', + aecModelHint: + 'Das Modell unterdrückt mehr Echo, kostet aber rund 3x CPU; auf älteren Geräten besser Leicht.', + aecHint: + 'Unterdrückt Lautsprecher-Rückkopplung. Braucht ein Referenzsignal; mit Kopfhörern aus lassen.', + buffer: 'Ausgabepuffer', noise: 'Noise reduction', dereverb: 'Comb filter', equalizer: 'Equalizer', @@ -1681,6 +1710,10 @@ export const de: typeof zhCN = { polishContextWindowLabel: 'Kontextfenster für Überarbeitung (Minuten)', polishContextWindowDesc: 'Verwendet überarbeitete Transkripte der letzten N Minuten als Gesprächskontext. 0 = deaktiviert.', + debugArchiveProcessedLabel: 'Verarbeitetes Audio behalten', + debugArchiveProcessedDesc: 'Die Version nach Rauschunterdrückung und Verstärkung, weiterhin in der bisherigen Aufnahmedatei.', + debugArchiveRawLabel: 'Auch Audio vor der Verarbeitung behalten', + debugArchiveRawDesc: 'Schreibt zusätzlich .raw.wav daneben. Ist nur dies aktiv, landet das Rohaudio in der Hauptdatei.', recordAudioForDebugLabel: 'Rohaufnahmen behalten (Diagnose)', recordAudioForDebugDesc: 'Speichert das rohe Mikrofonaudio als WAV, um Erkennungsprobleme zu untersuchen.', diff --git a/openless-all/app/src/i18n/en.ts b/openless-all/app/src/i18n/en.ts index cabdfbd89..a5da277a2 100644 --- a/openless-all/app/src/i18n/en.ts +++ b/openless-all/app/src/i18n/en.ts @@ -1545,6 +1545,35 @@ export const en: typeof zhCN = { lowLatencyDictationDesc: 'Keep a microphone stream ready and retain up to 500 ms in memory so speech immediately after the hotkey is less likely to be missed. Audio is not saved or uploaded until you confirm recording; this uses more power and keeps the microphone indicator active.', dsp: { + aec: 'System echo cancellation', + systemNoise: 'System noise suppression', + systemAgc: 'System automatic gain', + unavailable: 'Unavailable on this device', + androidEffectsHint: + 'Android system effects apply before the software chain. Avoid enabling the same effect twice. Unsupported startup falls back to normal capture.', + effectsActive: 'System effects are active for this recording.', + spectrum: 'Input / processed spectrum', + spectrumHint: + 'Shows the current enhanced recording only. This does not start a microphone, save audio, or upload it. Monitoring stops when this panel closes or goes to the background.', + spectrumLegend: 'Gray: input · Color: processed · 20 Hz–24 kHz', + spectrumWaiting: 'Start a recording with speech enhancement enabled to see the spectrum.', + presets: 'Equalizer presets', + choosePreset: 'Choose a preset', + preset_normal: 'Flat', + preset_brightVocal: 'Bright vocal', + preset_deepVoice: 'Deep voice', + preset_podcast: 'Podcast', + preset_vocalClarity: 'Vocal clarity', + preset_warmVocal: 'Warm vocal', + + aecOff: 'Off', + aecLight: 'Light (low CPU)', + aecModel: 'Model (stronger)', + aecModelHint: + 'The model cancels more echo but costs about 3x the CPU; prefer Light on older machines.', + aecHint: + 'Suppresses loudspeaker feedback while on speaker. It needs a speaker reference signal, so leave it off with headphones.', + buffer: 'Output buffer', noise: 'Noise reduction', dereverb: 'Comb filter', equalizer: 'Equalizer', @@ -1649,6 +1678,10 @@ export const en: typeof zhCN = { polishContextWindowLabel: 'Polish context window (minutes)', polishContextWindowDesc: 'Use the last N minutes of polished transcripts as multi-turn context; 0 = disabled.', + debugArchiveProcessedLabel: 'Keep processed audio', + debugArchiveProcessedDesc: 'The version after noise reduction and gain, stored in the existing recording file.', + debugArchiveRawLabel: 'Also keep pre-processing audio', + debugArchiveRawDesc: 'Writes .raw.wav next to it so you can compare. With only this enabled the raw audio goes into the main archive.', recordAudioForDebugLabel: 'Keep raw recording (debug)', recordAudioForDebugDesc: 'Save raw microphone audio as wav for diagnosing recognition issues.', diff --git a/openless-all/app/src/i18n/es.ts b/openless-all/app/src/i18n/es.ts index 0378847ea..7c57a418c 100644 --- a/openless-all/app/src/i18n/es.ts +++ b/openless-all/app/src/i18n/es.ts @@ -1566,6 +1566,35 @@ export const es: typeof zhCN = { lowLatencyDictationDesc: 'Mantén preparado el flujo del micrófono y conserva hasta 500 ms en memoria para reducir la pérdida de voz justo después del atajo. El audio no se guarda ni se sube antes de confirmar la grabación; consume más energía y mantiene activo el indicador del micrófono.', dsp: { + aec: 'System echo cancellation', + systemNoise: 'System noise suppression', + systemAgc: 'System automatic gain', + unavailable: 'Unavailable on this device', + androidEffectsHint: + 'Android system effects apply before the software chain. Avoid enabling the same effect twice. Unsupported startup falls back to normal capture.', + effectsActive: 'System effects are active for this recording.', + spectrum: 'Input / processed spectrum', + spectrumHint: + 'Shows the current enhanced recording only. This does not start a microphone, save audio, or upload it. Monitoring stops when this panel closes or goes to the background.', + spectrumLegend: 'Gray: input · Color: processed · 20 Hz–24 kHz', + spectrumWaiting: 'Start a recording with speech enhancement enabled to see the spectrum.', + presets: 'Equalizer presets', + choosePreset: 'Choose a preset', + preset_normal: 'Flat', + preset_brightVocal: 'Bright vocal', + preset_deepVoice: 'Deep voice', + preset_podcast: 'Podcast', + preset_vocalClarity: 'Vocal clarity', + preset_warmVocal: 'Warm vocal', + + aecOff: 'Desactivado', + aecLight: 'Ligero (poca CPU)', + aecModel: 'Modelo (más potente)', + aecModelHint: + 'El modelo cancela más pero cuesta unas 3x más CPU; en equipos antiguos usa Ligero.', + aecHint: + 'Suprime la realimentación del altavoz. Necesita una señal de referencia; con auriculares puedes dejarlo apagado.', + buffer: 'Búfer de salida', noise: 'Noise reduction', dereverb: 'Comb filter', equalizer: 'Equalizer', @@ -1672,6 +1701,10 @@ export const es: typeof zhCN = { polishContextWindowLabel: 'Ventana de contexto para mejorar texto (minutos)', polishContextWindowDesc: 'Usa las transcripciones mejoradas de los últimos N minutos como contexto de varias intervenciones. 0 = desactivado.', + debugArchiveProcessedLabel: 'Conservar el audio procesado', + debugArchiveProcessedDesc: 'La versión tras reducción de ruido y ganancia, en el archivo de grabación existente.', + debugArchiveRawLabel: 'Conservar también el audio sin procesar', + debugArchiveRawDesc: 'Escribe además .raw.wav junto a él. Si solo activas esto, el audio sin procesar va al archivo principal.', recordAudioForDebugLabel: 'Conservar grabación original (depuración)', recordAudioForDebugDesc: 'Guarda el audio original del micrófono en WAV para investigar problemas de reconocimiento.', diff --git a/openless-all/app/src/i18n/fr.ts b/openless-all/app/src/i18n/fr.ts index b9487f656..62fa0d9e9 100644 --- a/openless-all/app/src/i18n/fr.ts +++ b/openless-all/app/src/i18n/fr.ts @@ -1586,6 +1586,35 @@ export const fr: typeof zhCN = { lowLatencyDictationDesc: 'Garde un flux microphone prêt et conserve jusqu’à 500 ms en mémoire afin de réduire les mots perdus juste après le raccourci. Aucun audio n’est enregistré ni envoyé avant confirmation ; ce mode consomme davantage et maintient l’indicateur du microphone actif.', dsp: { + aec: 'System echo cancellation', + systemNoise: 'System noise suppression', + systemAgc: 'System automatic gain', + unavailable: 'Unavailable on this device', + androidEffectsHint: + 'Android system effects apply before the software chain. Avoid enabling the same effect twice. Unsupported startup falls back to normal capture.', + effectsActive: 'System effects are active for this recording.', + spectrum: 'Input / processed spectrum', + spectrumHint: + 'Shows the current enhanced recording only. This does not start a microphone, save audio, or upload it. Monitoring stops when this panel closes or goes to the background.', + spectrumLegend: 'Gray: input · Color: processed · 20 Hz–24 kHz', + spectrumWaiting: 'Start a recording with speech enhancement enabled to see the spectrum.', + presets: 'Equalizer presets', + choosePreset: 'Choose a preset', + preset_normal: 'Flat', + preset_brightVocal: 'Bright vocal', + preset_deepVoice: 'Deep voice', + preset_podcast: 'Podcast', + preset_vocalClarity: 'Vocal clarity', + preset_warmVocal: 'Warm vocal', + + aecOff: 'Désactivé', + aecLight: 'Léger (CPU réduit)', + aecModel: 'Modèle (plus fort)', + aecModelHint: + 'Le modèle annule davantage mais coûte environ 3x plus de CPU ; préférez Léger sur les machines anciennes.', + aecHint: + 'Réduit le retour du haut-parleur. Nécessite un signal de référence; inutile au casque.', + buffer: 'Tampon de sortie', noise: 'Noise reduction', dereverb: 'Comb filter', equalizer: 'Equalizer', @@ -1693,6 +1722,10 @@ export const fr: typeof zhCN = { polishContextWindowLabel: 'Fenêtre de contexte d’amélioration (minutes)', polishContextWindowDesc: 'Utilise les transcriptions améliorées des N dernières minutes comme contexte de plusieurs interventions. 0 = désactivé.', + debugArchiveProcessedLabel: 'Conserver l’audio traité', + debugArchiveProcessedDesc: 'La version après réduction de bruit et gain, toujours dans le fichier d’enregistrement existant.', + debugArchiveRawLabel: 'Conserver aussi l’audio avant traitement', + debugArchiveRawDesc: 'Écrit aussi .raw.wav à côté. Si seule cette option est active, l’audio brut va dans l’archive principale.', recordAudioForDebugLabel: 'Conserver l’enregistrement brut (débogage)', recordAudioForDebugDesc: 'Enregistre l’audio brut du microphone en WAV pour diagnostiquer les problèmes de reconnaissance.', diff --git a/openless-all/app/src/i18n/ja.ts b/openless-all/app/src/i18n/ja.ts index c1d09e207..db666bd0f 100644 --- a/openless-all/app/src/i18n/ja.ts +++ b/openless-all/app/src/i18n/ja.ts @@ -1533,6 +1533,35 @@ export const ja: typeof zhCN = { lowLatencyDictationDesc: 'マイクストリームを準備し、最大 500 ミリ秒をメモリに保持して、ホットキー直後の音声が欠落しにくくします。録音を確定するまで音声は保存・アップロードされません。電力を多く使い、マイク使用中の表示が続きます。', dsp: { + aec: 'System echo cancellation', + systemNoise: 'System noise suppression', + systemAgc: 'System automatic gain', + unavailable: 'Unavailable on this device', + androidEffectsHint: + 'Android system effects apply before the software chain. Avoid enabling the same effect twice. Unsupported startup falls back to normal capture.', + effectsActive: 'System effects are active for this recording.', + spectrum: 'Input / processed spectrum', + spectrumHint: + 'Shows the current enhanced recording only. This does not start a microphone, save audio, or upload it. Monitoring stops when this panel closes or goes to the background.', + spectrumLegend: 'Gray: input · Color: processed · 20 Hz–24 kHz', + spectrumWaiting: 'Start a recording with speech enhancement enabled to see the spectrum.', + presets: 'Equalizer presets', + choosePreset: 'Choose a preset', + preset_normal: 'Flat', + preset_brightVocal: 'Bright vocal', + preset_deepVoice: 'Deep voice', + preset_podcast: 'Podcast', + preset_vocalClarity: 'Vocal clarity', + preset_warmVocal: 'Warm vocal', + + aecOff: 'オフ', + aecLight: '軽量（低 CPU）', + aecModel: 'モデル（高精度）', + aecModelHint: + 'モデル版はより強く抑圧しますが CPU は約 3 倍です。古い端末では軽量版を推奨します。', + aecHint: + 'スピーカー使用時の回り込みを抑えます。参照信号が必要なので、ヘッドホンならオフで構いません。', + buffer: '出力バッファ', noise: 'Noise reduction', dereverb: 'Comb filter', equalizer: 'Equalizer', @@ -1637,6 +1666,10 @@ export const ja: typeof zhCN = { polishContextWindowLabel: '会話コンテキスト窓（分）', polishContextWindowDesc: '直近 N 分間の整文済み転写をマルチターン文脈として渡します。0 = 無効。', + debugArchiveProcessedLabel: '処理後の音声を保存', + debugArchiveProcessedDesc: 'ノイズ低減とゲイン適用後の音声で、従来の録音ファイルに保存されます。', + debugArchiveRawLabel: '処理前の音声も保存', + debugArchiveRawDesc: '同じフォルダに .raw.wav を追加保存します。これだけを有効にすると元音声がメインに保存されます。', recordAudioForDebugLabel: '元の録音を保持（デバッグ）', recordAudioForDebugDesc: '生のマイク音声を wav で保存し、認識問題の診断に利用。', audioRecordingMaxEntriesLabel: '元音声の保持件数', diff --git a/openless-all/app/src/i18n/ko.ts b/openless-all/app/src/i18n/ko.ts index dff252b94..43b745c29 100644 --- a/openless-all/app/src/i18n/ko.ts +++ b/openless-all/app/src/i18n/ko.ts @@ -1520,6 +1520,35 @@ export const ko: typeof zhCN = { lowLatencyDictationDesc: '마이크 스트림을 준비하고 최대 500ms를 메모리에 보관해 단축키 직후의 음성이 누락될 가능성을 줄입니다. 녹음을 확정하기 전에는 오디오를 저장하거나 업로드하지 않습니다. 전력 사용량이 늘고 마이크 표시가 계속 켜져 있습니다.', dsp: { + aec: 'System echo cancellation', + systemNoise: 'System noise suppression', + systemAgc: 'System automatic gain', + unavailable: 'Unavailable on this device', + androidEffectsHint: + 'Android system effects apply before the software chain. Avoid enabling the same effect twice. Unsupported startup falls back to normal capture.', + effectsActive: 'System effects are active for this recording.', + spectrum: 'Input / processed spectrum', + spectrumHint: + 'Shows the current enhanced recording only. This does not start a microphone, save audio, or upload it. Monitoring stops when this panel closes or goes to the background.', + spectrumLegend: 'Gray: input · Color: processed · 20 Hz–24 kHz', + spectrumWaiting: 'Start a recording with speech enhancement enabled to see the spectrum.', + presets: 'Equalizer presets', + choosePreset: 'Choose a preset', + preset_normal: 'Flat', + preset_brightVocal: 'Bright vocal', + preset_deepVoice: 'Deep voice', + preset_podcast: 'Podcast', + preset_vocalClarity: 'Vocal clarity', + preset_warmVocal: 'Warm vocal', + + aecOff: '끄기', + aecLight: '경량(저 CPU)', + aecModel: '모델(강화)', + aecModelHint: + '모델 버전은 더 강하게 제거하지만 CPU를 약 3배 사용합니다. 오래된 기기에서는 경량을 권장합니다.', + aecHint: + '스피커 사용 시 되먹임을 줄입니다. 참조 신호가 필요하므로 헤드폰이면 꺼 두세요.', + buffer: '출력 버퍼', noise: 'Noise reduction', dereverb: 'Comb filter', equalizer: 'Equalizer', @@ -1624,6 +1653,10 @@ export const ko: typeof zhCN = { polishContextWindowLabel: '대화 컨텍스트 윈도(분)', polishContextWindowDesc: '최근 N 분간 정리된 전사를 멀티턴 컨텍스트로 전달합니다. 0 = 비활성화.', + debugArchiveProcessedLabel: '처리된 오디오 저장', + debugArchiveProcessedDesc: '노이즈 감소와 게인 적용 후의 오디오로, 기존 녹음 파일에 저장됩니다.', + debugArchiveRawLabel: '처리 전 오디오도 저장', + debugArchiveRawDesc: '같은 폴더에 .raw.wav를 함께 저장합니다. 이것만 켜면 원본이 기본 파일에 저장됩니다.', recordAudioForDebugLabel: '원본 녹음 보관(디버그)', recordAudioForDebugDesc: '원시 마이크 오디오를 wav 로 저장하여 인식 문제 진단.', audioRecordingMaxEntriesLabel: '원본 녹음 보관 개수', diff --git a/openless-all/app/src/i18n/zh-CN.ts b/openless-all/app/src/i18n/zh-CN.ts index 6b92f58ea..09fc30a5e 100644 --- a/openless-all/app/src/i18n/zh-CN.ts +++ b/openless-all/app/src/i18n/zh-CN.ts @@ -1470,6 +1470,35 @@ export const zhCN = { lowLatencyDictationDesc: '保持麦克风流就绪，并在内存中保留最多 500 毫秒音频，减少按下热键后第一句话被漏识别的情况。确认录音前不会保存或上传音频；此模式会增加功耗并保持系统麦克风指示灯亮起。', dsp: { + aec: '系统回声消除', + systemNoise: '系统降噪', + systemAgc: '系统自动增益', + unavailable: '当前设备不支持', + androidEffectsHint: + 'Android 系统音效在软件处理链之前生效。建议避免重复开启同类效果。启动不支持时回退到常规采集。', + effectsActive: '本次录音已启用系统音效。', + spectrum: '输入／处理后频谱', + spectrumHint: + '仅显示当前增强录音，不会启动麦克风、保存或上传音频。关闭面板或进入后台后停止监测。', + spectrumLegend: '灰色：输入 · 彩色：处理后 · 20 Hz–24 kHz', + spectrumWaiting: '开启语音增强并开始录音后显示频谱。', + presets: '均衡器预设', + choosePreset: '选择预设', + preset_normal: '平直', + preset_brightVocal: '明亮人声', + preset_deepVoice: '低沉人声', + preset_podcast: '播客', + preset_vocalClarity: '清晰人声', + preset_warmVocal: '温暖人声', + + aecOff: '关闭', + aecLight: '轻量（省 CPU）', + aecModel: '模型（更强）', + aecModelHint: + '模型版抑制更强，但约是轻量版的 3 倍 CPU；老机器建议用轻量。', + aecHint: + '外放时压掉扬声器回授；需要扬声器参考信号，戴耳机可以关掉。', + buffer: '输出缓冲区', title: '语音增强设置', description: 'RNNoise 更轻量；PureVox 模型降噪耗时较高。设置在下一次录音生效。', noise: '降噪算法', @@ -1578,6 +1607,10 @@ export const zhCN = { '转写不超过 N 个字（不计空白）时直接插入原文，不调用润色模型。0 = 关闭。翻译模式始终润色。', polishContextWindowLabel: '对话上下文窗口（分钟）', polishContextWindowDesc: '把最近 N 分钟内已润色的转写作为多轮上下文，0 = 关闭。', + debugArchiveProcessedLabel: '保存处理后的音频', + debugArchiveProcessedDesc: '降噪与增益之后的版本，仍保存在原来的录制文件里。', + debugArchiveRawLabel: '另存一份处理前的音频', + debugArchiveRawDesc: '同目录下另存 .raw.wav 便于对照；只开这一项时，原始音频直接存进主档案。', recordAudioForDebugLabel: '保留原始录音（调试）', recordAudioForDebugDesc: '保存原始麦克风音频为 wav，便于排查识别问题。', audioRecordingMaxEntriesLabel: '原始录音保留条数', diff --git a/openless-all/app/src/i18n/zh-TW.ts b/openless-all/app/src/i18n/zh-TW.ts index 533136f34..c797f2f59 100644 --- a/openless-all/app/src/i18n/zh-TW.ts +++ b/openless-all/app/src/i18n/zh-TW.ts @@ -1472,6 +1472,35 @@ export const zhTW: typeof zhCN = { lowLatencyDictationDesc: '保持麥克風串流就緒，並在記憶體中保留最多 500 毫秒音訊，減少按下熱鍵後第一句話被漏辨識的情況。確認錄音前不會儲存或上傳音訊；此模式會增加耗電並保持系統麥克風指示燈亮起。', dsp: { + aec: '系統回聲消除', + systemNoise: '系統降噪', + systemAgc: '系統自動增益', + unavailable: '目前裝置不支援', + androidEffectsHint: + 'Android 系統音效在軟體處理鏈之前生效。建議避免重複開啟同類效果。啟動不支援時退回一般擷取。', + effectsActive: '本次錄音已啟用系統音效。', + spectrum: '輸入／處理後頻譜', + spectrumHint: + '僅顯示目前增強錄音，不會啟動麥克風、儲存或上傳音訊。關閉面板或進入背景後停止監測。', + spectrumLegend: '灰色：輸入 · 彩色：處理後 · 20 Hz–24 kHz', + spectrumWaiting: '開啟語音增強並開始錄音後顯示頻譜。', + presets: '均衡器預設', + choosePreset: '選擇預設', + preset_normal: '平直', + preset_brightVocal: '明亮人聲', + preset_deepVoice: '低沉人聲', + preset_podcast: '播客', + preset_vocalClarity: '清晰人聲', + preset_warmVocal: '溫暖人聲', + + aecOff: '關閉', + aecLight: '輕量（省 CPU）', + aecModel: '模型（更強）', + aecModelHint: + '模型版抑制更強，但約是輕量版的 3 倍 CPU；舊機器建議用輕量。', + aecHint: + '外放時壓掉揚聲器回授；需要揚聲器參考訊號，戴耳機可以關掉。', + buffer: '輸出緩衝區', title: '語音增強設定', description: 'RNNoise 更輕量；PureVox 模型降噪耗時較高。設定於下一次錄音生效。', noise: '降噪演算法', @@ -1566,6 +1595,10 @@ export const zhTW: typeof zhCN = { '轉寫不超過 N 個字（不計空白）時直接插入原文，不呼叫潤色模型。0 = 關閉。翻譯模式始終潤色。', polishContextWindowLabel: '對話上下文視窗（分鐘）', polishContextWindowDesc: '把最近 N 分鐘內已潤色的轉寫作為多輪上下文，0 = 關閉。', + debugArchiveProcessedLabel: '儲存處理後的音訊', + debugArchiveProcessedDesc: '降噪與增益之後的版本，仍存於原本的錄製檔案。', + debugArchiveRawLabel: '另存一份處理前的音訊', + debugArchiveRawDesc: '同一目錄下另存 .raw.wav 便於對照；只開這一項時，原始音訊直接存進主檔案。', recordAudioForDebugLabel: '保留原始錄音（除錯）', recordAudioForDebugDesc: '儲存原始麥克風音訊為 wav，便於排查識別問題。', audioRecordingMaxEntriesLabel: '原始錄音保留條數', diff --git a/openless-all/app/src/lib/audioDsp.ts b/openless-all/app/src/lib/audioDsp.ts index 9bce1b732..b4874df75 100644 --- a/openless-all/app/src/lib/audioDsp.ts +++ b/openless-all/app/src/lib/audioDsp.ts @@ -1,6 +1,11 @@ import type { AudioDspConfig } from './types'; export const defaultAudioDsp: AudioDspConfig = { + aecMode: 'off', + outputBufferMs: 300, + androidAecEnabled: false, + androidNoiseSuppressionEnabled: false, + androidAgcEnabled: false, noiseReduction: 'rnnoise', noiseStrength: 80, gainDb: 0, @@ -14,7 +19,25 @@ export const defaultAudioDsp: AudioDspConfig = { equalizerEnabled: false, equalizerPreampDb: 0, equalizerBandsDb: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], - processingOrder: ['noiseReduction', 'dereverb', 'equalizer', 'amplifier', 'agc', 'vad'], + processingOrder: [ + 'acousticEchoCancellation', + 'noiseReduction', + 'dereverb', + 'equalizer', + 'amplifier', + 'agc', + 'vad', + ], +}; + +// MicYou EqualizerPanel.vue, pinned upstream revision in micyou-upstream.json. +export const audioEqualizerPresets = { + normal: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], + brightVocal: [-2, -1, 0, 1, 2, 3, 4, 4, 3, 2], + deepVoice: [3, 4, 4, 3, 1, 0, -1, -2, -2, -1], + podcast: [2, 3, 2, 0, -1, -1, 1, 2, 3, 2], + vocalClarity: [-1, -1, 0, 1, 2, 3, 4, 3, 2, 1], + warmVocal: [1, 2, 3, 2, 1, 0, -1, -1, 0, 1], }; export const equalizerFrequencies = [ diff --git a/openless-all/app/src/lib/ipc/android-audio.ts b/openless-all/app/src/lib/ipc/android-audio.ts new file mode 100644 index 000000000..bc7bfb7f1 --- /dev/null +++ b/openless-all/app/src/lib/ipc/android-audio.ts @@ -0,0 +1,27 @@ +import { invokeOrMock } from './shared'; + +export interface AndroidAudioEffectsStatus { + aecAvailable: boolean; + noiseSuppressionAvailable: boolean; + agcAvailable: boolean; + runtime: null | { + active: boolean; + aec?: boolean; + noiseSuppression?: boolean; + agc?: boolean; + error: string | null; + }; +} + +export function getAndroidAudioEffectsStatus(): Promise { + return invokeOrMock('get_android_audio_effects_status', undefined, () => ({ + aecAvailable: false, + noiseSuppressionAvailable: false, + agcAvailable: false, + runtime: null, + })); +} + +export function setAndroidAudioMonitoring(enabled: boolean): Promise { + return invokeOrMock('set_android_audio_monitoring', { enabled }, () => undefined); +} diff --git a/openless-all/app/src/lib/ipc/mock-data.ts b/openless-all/app/src/lib/ipc/mock-data.ts index 29dbcc579..e30bd2922 100644 --- a/openless-all/app/src/lib/ipc/mock-data.ts +++ b/openless-all/app/src/lib/ipc/mock-data.ts @@ -145,6 +145,8 @@ export let mockSettings: UserPreferences = { autoUpdateCheck: true, historyMaxEntries: null, recordAudioForDebug: false, + debugArchiveProcessed: false, + debugArchiveRaw: false, audioRecordingMaxEntries: null, quickNoteExportDirectory: '', marketplaceBaseUrl: 'https://apic.openless.top', diff --git a/openless-all/app/src/lib/types.ts b/openless-all/app/src/lib/types.ts index e3f0f0e66..e78579c43 100644 --- a/openless-all/app/src/lib/types.ts +++ b/openless-all/app/src/lib/types.ts @@ -341,8 +341,21 @@ export interface StylePackRuntimeDiagnostics { } export type AudioDspStage = - 'noiseReduction' | 'dereverb' | 'equalizer' | 'amplifier' | 'agc' | 'vad'; + | 'acousticEchoCancellation' + | 'noiseReduction' + | 'dereverb' + | 'equalizer' + | 'amplifier' + | 'agc' + | 'vad'; export interface AudioDspConfig { + /** 回声消除算法：轻量 NLMS 省 CPU，aec7 模型抑制更强但约 3 倍开销。 */ + aecMode: 'off' | 'light' | 'model'; + /** 输出环形缓冲余量（100..1200 ms）。 */ + outputBufferMs: number; + androidAecEnabled?: boolean; + androidNoiseSuppressionEnabled?: boolean; + androidAgcEnabled?: boolean; noiseReduction: 'off' | 'rnnoise' | 'purevox' | 'spectral'; noiseStrength: number; gainDb: number; @@ -592,6 +605,10 @@ export interface UserPreferences { /** Whether to keep the raw microphone audio file (wav) per session, for debugging ASR misrecognition / mic sensitivity. * Default false. When on, it uses disk space and follows the same cleanup policy as historyRetentionDays. */ recordAudioForDebug: boolean; + /** 调试录音：追加保存「处理后」的音频（主档案 `.wav`）。 */ + debugArchiveProcessed: boolean; + /** 调试录音：额外保存「处理前」的 `.raw.wav`，用于 A/B 对照。 */ + debugArchiveRaw: boolean; /** Number of recent wav files kept in recordings/. null = follows the 200 hard cap; 1..=200 for user-defined values. * Decoupled from historyMaxEntries — many text entries with only the 5 most recent wavs is a valid combination. */ audioRecordingMaxEntries: number | null; diff --git a/openless-all/app/src/pages/settings/AndroidAudioEffects.tsx b/openless-all/app/src/pages/settings/AndroidAudioEffects.tsx new file mode 100644 index 000000000..9b052a25d --- /dev/null +++ b/openless-all/app/src/pages/settings/AndroidAudioEffects.tsx @@ -0,0 +1,198 @@ +import { useEffect, useState } from 'react'; +import { useTranslation } from 'react-i18next'; +import type { AudioDspConfig } from '../../lib/types'; +import { + getAndroidAudioEffectsStatus, + setAndroidAudioMonitoring, + type AndroidAudioEffectsStatus, +} from '../../lib/ipc/android-audio'; +import { isTauri } from '../../lib/ipc/shared'; +import { SettingRow, Toggle } from './shared'; + +type Spectrum = { raw: number[]; processed: number[] }; + +export function AndroidAudioEffects({ + value, + onChange, +}: { + value: AudioDspConfig; + onChange: (patch: Partial) => void; +}) { + const { t } = useTranslation(); + const label = (key: string, fallback: string) => + t(`settings.recording.dsp.${key}`, { defaultValue: fallback }); + const [status, setStatus] = useState(null); + const [monitoring, setMonitoring] = useState(false); + const [spectrum, setSpectrum] = useState(null); + const [error, setError] = useState(''); + useEffect(() => { + let disposed = false; + const refresh = () => { + if (document.hidden) return; + void getAndroidAudioEffectsStatus() + .then((next) => { + if (!disposed) setStatus(next); + }) + .catch(() => {}); + }; + refresh(); + const timer = window.setInterval(refresh, 2000); + return () => { + disposed = true; + window.clearInterval(timer); + }; + }, []); + useEffect(() => { + if (!monitoring || !isTauri) { + setSpectrum(null); + return; + } + let disposed = false; + let unlisten: (() => void) | undefined; + let timer: number | undefined; + const renew = () => { + if (document.hidden) setSpectrum(null); + void setAndroidAudioMonitoring(!document.hidden).catch((reason: unknown) => { + if (!disposed) setError(String(reason)); + }); + }; + void import('@tauri-apps/api/event') + .then(async ({ listen }) => { + const stop = await listen('android-audio-spectrum', ({ payload }) => { + if (disposed || document.hidden) return; + if (payload.raw.length === 64 && payload.processed.length === 64) setSpectrum(payload); + }); + if (disposed) { + stop(); + return; + } + unlisten = stop; + renew(); + timer = window.setInterval(renew, 1000); + }) + .catch((reason: unknown) => { + if (!disposed) setError(String(reason)); + }); + document.addEventListener('visibilitychange', renew); + return () => { + disposed = true; + unlisten?.(); + window.clearInterval(timer); + document.removeEventListener('visibilitychange', renew); + void setAndroidAudioMonitoring(false).catch(() => {}); + }; + }, [monitoring]); + const toggle = ( + field: 'androidAecEnabled' | 'androidNoiseSuppressionEnabled' | 'androidAgcEnabled', + title: string, + available: boolean, + ) => ( +
+ + { + if (available || value[field]) onChange({ [field]: !value[field] }); + }} + /> + +
+ ); + return ( +
+

+ {label( + 'androidEffectsHint', + 'Android system effects use the microphone session. They apply before the software chain; avoid enabling the same effect twice. Unsupported startup falls back to normal capture.', + )} +

+ {toggle( + 'androidAecEnabled', + label('aec', 'System echo cancellation'), + !!status?.aecAvailable, + )} + {toggle( + 'androidNoiseSuppressionEnabled', + label('systemNoise', 'System noise suppression'), + !!status?.noiseSuppressionAvailable, + )} + {toggle( + 'androidAgcEnabled', + label('systemAgc', 'System automatic gain'), + !!status?.agcAvailable, + )} + {status?.runtime?.error && ( +

+ {status.runtime.error} +

+ )} + {status?.runtime?.active && ( +

+ {label('effectsActive', 'System effects are active for this recording.')} +

+ )} + + { + setError(''); + setMonitoring(!monitoring); + }} + /> + + {monitoring && ( + <> + + {Array.from({ length: 64 }, (_, i) => { + const raw = Math.max(0, Math.min(1, spectrum?.raw[i] || 0)) * 76; + const processed = Math.max(0, Math.min(1, spectrum?.processed[i] || 0)) * 76; + return ( + + + + + ); + })} + + {label('spectrumLegend', 'Gray: input · Color: processed · 20 Hz–24 kHz')} + {!spectrum && ( +

+ {label( + 'spectrumWaiting', + 'Start a recording with speech enhancement enabled to see the spectrum.', + )} +

+ )} + + )} + {error &&

{error}

} +
+ ); +} diff --git a/openless-all/app/src/pages/settings/AudioDspSettings.tsx b/openless-all/app/src/pages/settings/AudioDspSettings.tsx index 971e63c1a..ab3becd32 100644 --- a/openless-all/app/src/pages/settings/AudioDspSettings.tsx +++ b/openless-all/app/src/pages/settings/AudioDspSettings.tsx @@ -1,7 +1,9 @@ import { useEffect, useState } from 'react'; import { useTranslation } from 'react-i18next'; import type { AudioDspConfig, AudioDspStage } from '../../lib/types'; -import { defaultAudioDsp, equalizerFrequencies } from '../../lib/audioDsp'; +import { audioEqualizerPresets, defaultAudioDsp, equalizerFrequencies } from '../../lib/audioDsp'; +import { isAndroid } from '../../lib/platform'; +import { AndroidAudioEffects } from './AndroidAudioEffects'; import { SelectLite } from '../../components/ui/SelectLite'; import { SettingRow, Toggle } from './shared'; @@ -13,6 +15,7 @@ export function AudioDspSettings({ onChange: (value: AudioDspConfig) => void; }) { const { t } = useTranslation(); + const [expanded, setExpanded] = useState(false); const [draft, setDraft] = useState({ ...defaultAudioDsp, ...value }); useEffect(() => setDraft({ ...defaultAudioDsp, ...value }), [value]); const label = (key: string, fallback: string) => @@ -53,6 +56,7 @@ export function AudioDspSettings({ ); const stageLabels: Record = { + acousticEchoCancellation: label('aec', 'Echo cancellation'), noiseReduction: label('noise', 'Noise reduction'), dereverb: label('dereverb', 'Comb filter'), equalizer: label('equalizer', 'Equalizer'), @@ -68,7 +72,10 @@ export function AudioDspSettings({ save({ processingOrder: order }); }; return ( -
+
setExpanded(event.currentTarget.open)} + > {label('title', 'Speech enhancement settings')} @@ -78,6 +85,31 @@ export function AudioDspSettings({ 'RNNoise is lightweight; PureVox offers model-based noise reduction with higher CPU use. Changes apply to the next recording.', )}

+ + save({ aecMode: v as AudioDspConfig['aecMode'] })} + options={[ + { value: 'off', label: label('aecOff', 'Off') }, + { value: 'light', label: label('aecLight', 'Light (low CPU)') }, + { value: 'model', label: label('aecModel', 'Model (stronger)') }, + ]} + /> + + {draft.aecMode === 'model' && + label( + 'aecModelHint', + 'The model cancels more echo but costs about 3x the CPU; prefer Light on older machines.', + )} + {slider('outputBufferMs', label('buffer', 'Output buffer'), 100, 1200, ' ms')} + {isAndroid() && expanded && } {draft.noiseReduction !== 'off' && slider('noiseStrength', label('strength', 'Noise reduction strength'), 0, 100, '%')} - {slider('gainDb', stageLabels.amplifier, -30, 30, ' dB')} + {slider('gainDb', stageLabels.amplifier, -50, 50, ' dB')} save({ agcEnabled: !draft.agcEnabled })} /> {draft.agcEnabled && ( <> - {slider('agcTarget', label('target', 'Target level'), 5, 90, '%')} + {slider('agcTarget', label('target', 'Target level'), 0, 100, '%')} {slider('agcAttack', label('attack', 'Attack speed'), 1, 100)} {slider('agcDecay', label('decay', 'Release speed'), 1, 100)} @@ -114,7 +146,7 @@ export function AudioDspSettings({ save({ vadEnabled: !draft.vadEnabled })} /> {draft.vadEnabled && - slider('vadThresholdDb', label('threshold', 'Speech gate threshold'), -80, -10, ' dB')} + slider('vadThresholdDb', label('threshold', 'Speech gate threshold'), -100, 0, ' dB')}
{label('tone', 'Tone and room adjustments')} @@ -134,6 +166,23 @@ export function AudioDspSettings({ {draft.equalizerEnabled && ( <> + + ({ + value: key, + label: label(`preset_${key}`, key), + })), + ]} + onChange={(key) => { + const bands = audioEqualizerPresets[key as keyof typeof audioEqualizerPresets]; + if (bands) save({ equalizerBandsDb: [...bands] }); + }} + /> + {slider('equalizerPreampDb', label('preamp', 'Equalizer preamp'), -20, 20, ' dB')}
- savePrefs({ ...prefs, recordAudioForDebug }); + const onRecordAudioForDebugChange = (recordAudioForDebug: boolean) => { + // 打开主开关时如果两项都没勾，默认勾上「处理后」，否则开了开关却什么都不写。 + const keepSomething = + prefs.debugArchiveProcessed || prefs.debugArchiveRaw || !recordAudioForDebug; + void savePrefs({ + ...prefs, + recordAudioForDebug, + debugArchiveProcessed: keepSomething ? prefs.debugArchiveProcessed : true, + }); + }; + const onDebugArchiveProcessedChange = (debugArchiveProcessed: boolean) => + void savePrefs({ ...prefs, debugArchiveProcessed }); + const onDebugArchiveRawChange = (debugArchiveRaw: boolean) => + void savePrefs({ ...prefs, debugArchiveRaw }); // Empty means unlimited, falling back to null → the backend applies its 200 default. const onAudioRecordingMaxEntriesChange = (raw: string) => { const trimmed = raw.trim(); @@ -112,6 +124,28 @@ export function DebugToolsSection() { + {prefs.recordAudioForDebug && ( + <> + + onDebugArchiveProcessedChange(!prefs.debugArchiveProcessed)} + /> + + + onDebugArchiveRawChange(!prefs.debugArchiveRaw)} + /> + + + )}
= ["snapshot", "library", "searchHistory", "historyDetail", "audioAsset", "vocabulary", "styleLibrary", "styleDetail", "capabilitySnapshot", "translationConfiguration", "marketplaceList", "marketplaceDetail", "listModels", "validate"] + let reads: Set = ["audioSettings", "snapshot", "library", "searchHistory", "historyDetail", "audioAsset", "vocabulary", "styleLibrary", "styleDetail", "capabilitySnapshot", "translationConfiguration", "marketplaceList", "marketplaceDetail", "listModels", "validate"] if let name = action["action"] as? String { let nested = (action["request"] as? [String: Any])?["command"] as? String let localRead = (name == "vocabularyBridge" && ["snapshot", "proposals"].contains(nested ?? "")) diff --git a/openless-all/ios/App/SettingsView.swift b/openless-all/ios/App/SettingsView.swift index 6d0a75f3f..90799f446 100644 --- a/openless-all/ios/App/SettingsView.swift +++ b/openless-all/ios/App/SettingsView.swift @@ -108,8 +108,12 @@ struct SettingsView: View { DisclosureGroup { SystemRecordingInstructions() } label: { SettingsLabel("在其他 App 中录音", symbol: "mic.badge.plus") } } } - if matches("语音识别 准备 识别方式 speech recognition") { + if matches("语音识别 准备 识别方式 语音增强 降噪 增益 均衡器 speech recognition audio enhancement noise gain equalizer") { Section("语音识别") { + NavigationLink { + NativeAudioSettingsView(core: controller.styleLibraryCore, blocked: controller.styleLibraryActivityBusy) + } label: { SettingsLabel("语音增强", symbol: "waveform.path") } + .accessibilityIdentifier("audioEnhancementSettings") AdaptiveRow(title: controller.speechPrepared ? "识别已就绪" : "准备语音识别", icon: "waveform") { Button(controller.preparing ? "准备中…" : "准备") { Task { await controller.prepareSpeech(allowDownload: false) } @@ -182,6 +186,149 @@ struct SettingsView: View { } } +private struct NativeAudioDsp: Codable { + var noiseReduction = "rnnoise" + var noiseStrength = 80 + var gainDb = 0 + var agcEnabled = true + var agcTarget = 35 + var agcAttack = 50 + var agcDecay = 50 + var vadEnabled = false + var vadThresholdDb = -50 + var dereverbLevel = 0 + var equalizerEnabled = false + var equalizerPreampDb = 0 + var equalizerBandsDb = Array(repeating: 0, count: 10) + var processingOrder = ["noiseReduction", "dereverb", "equalizer", "amplifier", "agc", "vad"] +} + +private struct NativeAudioSettings: Codable { + var enabled = false + var dsp = NativeAudioDsp() +} + +private struct NativeAudioSettingsView: View { + @EnvironmentObject private var interfacePreferences: InterfacePreferences + let core: CoreBridge? + let blocked: Bool + @State private var value = NativeAudioSettings() + @State private var loaded = false + @State private var saving = false + @State private var message: String? + private let presets: [(String, [Int])] = [ + ("平直", Array(repeating: 0, count: 10)), + ("明亮人声", [-2,-1,0,1,2,3,4,4,3,2]), + ("低沉人声", [3,4,4,3,1,0,-1,-2,-2,-1]), + ("播客", [2,3,2,0,-1,-1,1,2,3,2]), + ("清晰人声", [-1,-1,0,1,2,3,4,3,2,1]), + ("温暖人声", [1,2,3,2,1,0,-1,-1,0,1]) + ] + var body: some View { + Form { + if let message { + Section { Text(t(message)); Button("重新加载") { Task { await load() } }.disabled(saving || blocked) } + } + Group { + Section { + Toggle("语音增强", isOn: $value.enabled) + .accessibilityIdentifier("nativeAudioEnhancementToggle") + } footer: { + Text("设置保存后在下一次录音生效。RNNoise 较轻量，PureVox 使用更多计算资源。默认保持原有音频效果。") + } + if value.enabled { + Section("降噪与增益") { + Picker("降噪算法", selection: $value.dsp.noiseReduction) { + Text("关闭").tag("off"); Text("RNNoise").tag("rnnoise") + Text("PureVox 6").tag("purevox"); Text("频谱减法").tag("spectral") + }.accessibilityIdentifier("nativeAudioAlgorithmPicker") + control("降噪强度", \.noiseStrength, 0...100) + control("输入增益 dB", \.gainDb, -50...50) + Toggle("自动增益", isOn: $value.dsp.agcEnabled) + if value.dsp.agcEnabled { + control("目标电平 %", \.agcTarget, 0...100) + control("起音速度", \.agcAttack, 1...100) + control("释放速度", \.agcDecay, 1...100) + } + Toggle("语音门限", isOn: $value.dsp.vadEnabled) + if value.dsp.vadEnabled { control("门限 dB", \.vadThresholdDb, -100...0) } + } + Section { + control("梳状滤波 %", \.dereverbLevel, 0...100) + Toggle("十段均衡器", isOn: $value.dsp.equalizerEnabled) + if value.dsp.equalizerEnabled { + Menu("均衡器预设") { + ForEach(presets.indices, id: \.self) { i in + Button(t(presets[i].0)) { value.dsp.equalizerBandsDb = presets[i].1 } + } + } + control("前级增益 dB", \.equalizerPreampDb, -20...20) + ForEach(value.dsp.equalizerBandsDb.indices, id: \.self) { i in + VStack(alignment: .leading) { + Text("\(["31 Hz","63 Hz","125 Hz","250 Hz","500 Hz","1 kHz","2 kHz","4 kHz","8 kHz","16 kHz"][i]): \(value.dsp.equalizerBandsDb[i]) dB") + Slider(value: Binding(get: { Double(value.dsp.equalizerBandsDb[i]) }, set: { value.dsp.equalizerBandsDb[i] = Int($0.rounded()) }), in: -15...15, step: 1) + } + } + } + } header: { Text("音色与房间调整") } footer: { + Text("梳状滤波可能改变音色，建议从 0 开始。语音门限不会自动停止录音，过高会丢失轻声。") + } + Section("处理顺序") { + ForEach(value.dsp.processingOrder, id: \.self) { stage in + HStack { + Text(stageName(stage)) + Spacer() + Button { move(stage, -1) } label: { Image(systemName: "arrow.up") }.buttonStyle(.borderless) + .accessibilityLabel("上移 \(stageName(stage))") + Button { move(stage, 1) } label: { Image(systemName: "arrow.down") }.buttonStyle(.borderless) + .accessibilityLabel("下移 \(stageName(stage))") + } + } + } + Section { Button("恢复默认增强设置") { value.dsp = NativeAudioDsp() } } + } + }.disabled(!loaded || blocked || saving) + } + .navigationTitle("语音增强") + .toolbar { Button(t(saving ? "保存中…" : "保存")) { Task { await save() } }.disabled(!loaded || blocked || saving || core == nil).accessibilityIdentifier("nativeAudioSave") } + .task { await load() } + } + private func control(_ title: String, _ key: WritableKeyPath, _ range: ClosedRange) -> some View { + VStack(alignment: .leading) { + Text("\(t(title)): \(value.dsp[keyPath: key])") + Slider(value: Binding(get: { Double(value.dsp[keyPath: key]) }, set: { value.dsp[keyPath: key] = Int($0.rounded()) }), in: Double(range.lowerBound)...Double(range.upperBound), step: 1) + } + } + private func stageName(_ key: String) -> String { + t(["noiseReduction":"降噪", "dereverb":"梳状滤波", "equalizer":"均衡器", "amplifier":"输入增益", "agc":"自动增益", "vad":"语音门限"][key] ?? key) + } + private func t(_ key: String) -> String { interfacePreferences.localizer.text(key) } + private func move(_ stage: String, _ delta: Int) { + guard let i = value.dsp.processingOrder.firstIndex(of: stage), value.dsp.processingOrder.indices.contains(i + delta) else { return } + value.dsp.processingOrder.swapAt(i, i + delta) + } + private func load() async { + guard let core else { message = "音频设置暂不可用，请稍后重试。"; return } + saving = true + defer { saving = false } + do { + value = try JSONDecoder().decode(NativeAudioSettings.self, from: await core.control(["action":"audioSettings"])) + loaded = true; message = nil + } catch { message = "无法读取音频设置，请重试。" } + } + private func save() async { + guard let core, !blocked else { return } + saving = true + defer { saving = false } + do { + let dsp = try JSONSerialization.jsonObject(with: JSONEncoder().encode(value.dsp)) + let data = try await core.control(["action":"updateAudioSettings", "enabled":value.enabled, "dsp":dsp]) + value = try JSONDecoder().decode(NativeAudioSettings.self, from: data) + message = "已保存，下次录音生效。" + } catch { message = "保存失败，当前修改尚未应用。" } + } +} + private struct AccountSettingsDestination: View { @ObservedObject var controller: AppController @ObservedObject var productLibrary: ProductLibrary diff --git a/openless-all/ios/App/Speech/AudioPCM.swift b/openless-all/ios/App/Speech/AudioPCM.swift index a1993bd2a..b1e8333b0 100644 --- a/openless-all/ios/App/Speech/AudioPCM.swift +++ b/openless-all/ios/App/Speech/AudioPCM.swift @@ -60,6 +60,23 @@ nonisolated struct AudioPacket: Sendable { /// Owns the resampler off the UI and real-time audio callback. actor PCMWorker { private var converter: AVAudioConverter? + private var scratch: [AVAudioPCMBuffer] = [] + + private func outputBuffer(format: AVAudioFormat, capacity: AVAudioFrameCount) throws -> AVAudioPCMBuffer { + // Converter output capacity affects its pull requests at fractional + // ratios (e.g. 44.1 -> 16 kHz). Preserve the original capacity + // exactly, including flush; do not substitute a larger cached buffer. + if let output = scratch.first(where: { $0.format == format && $0.frameCapacity == capacity }) { + output.frameLength = 0 + return output + } + guard let output = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: capacity) else { throw VoiceFailure.format } + // Two entries cover alternating packet sizes without retaining an + // unbounded set of device callback capacities. + if scratch.count == 2 { scratch.removeFirst() } + scratch.append(output) + return output + } func convert(_ packet: AudioPacket) throws -> Data { let input = try packet.makeBuffer() let inputFormat = input.format @@ -75,31 +92,69 @@ actor PCMWorker { guard let converter else { throw VoiceFailure.format } let capacity = AVAudioFrameCount( ceil(Double(input.frameLength) * 16_000 / inputFormat.sampleRate) + 32) - guard let output = AVAudioPCMBuffer(pcmFormat: outputFormat, frameCapacity: capacity) else { - throw VoiceFailure.format - } - var supplied = false - var error: NSError? - let status = converter.convert(to: output, error: &error) { _, flag in - if supplied { - flag.pointee = .noDataNow - return nil + var cursor: AVAudioFrameCount = 0 + var invalidInput = false + var retainedInput: AVAudioPCMBuffer? + var pcm = Data() + // A filled output buffer does not imply that the converter has consumed + // the entire input packet. Drain .haveData before accepting the next + // packet, otherwise alternating packet sizes can silently lose frames. + for _ in 0..<8 { + let output = try outputBuffer(format: outputFormat, capacity: capacity) + var error: NSError? + let status = converter.convert(to: output, error: &error) { requested, flag in + if cursor == input.frameLength { + flag.pointee = .noDataNow + return nil + } + let frames = min(AVAudioFrameCount(requested), input.frameLength - cursor) + guard frames > 0 else { invalidInput = true; flag.pointee = .noDataNow; return nil } + flag.pointee = .haveData + if cursor == 0 && frames == input.frameLength { + cursor = input.frameLength + return input + } + // Respect the converter's requested frame count. Returning a whole + // large packet and marking it supplied loses its unconsumed suffix. + guard let slice = AVAudioPCMBuffer(pcmFormat: inputFormat, frameCapacity: frames) else { + invalidInput = true; flag.pointee = .noDataNow; return nil + } + slice.frameLength = frames + let source = UnsafeMutableAudioBufferListPointer(input.mutableAudioBufferList) + let target = UnsafeMutableAudioBufferListPointer(slice.mutableAudioBufferList) + let stride = Int(inputFormat.streamDescription.pointee.mBytesPerFrame) + let offset = Int(cursor) * stride + let count = Int(frames) * stride + guard stride > 0, source.count == target.count else { invalidInput = true; flag.pointee = .noDataNow; return nil } + for index in source.indices { + guard let from = source[index].mData, let to = target[index].mData, + offset + count <= Int(source[index].mDataByteSize), count <= Int(target[index].mDataByteSize) + else { invalidInput = true; flag.pointee = .noDataNow; return nil } + to.copyMemory(from: from.advanced(by: offset), byteCount: count) + } + cursor += frames + retainedInput = slice + return slice + } + withExtendedLifetime(retainedInput) {} + guard !invalidInput else { throw VoiceFailure.format } + if let error { throw error } + guard status != .error else { throw VoiceFailure.format } + if output.frameLength > 0, let bytes = output.int16ChannelData?[0] { + pcm.append(Data(bytes: bytes, count: Int(output.frameLength) * 2)) + } + if status == .inputRanDry || status == .endOfStream || output.frameLength == 0 { + guard cursor == input.frameLength else { throw VoiceFailure.format } + return pcm } - supplied = true - flag.pointee = .haveData - return input } - if let error { throw error } - guard status != .error else { throw VoiceFailure.format } - guard output.frameLength > 0, let bytes = output.int16ChannelData?[0] else { return Data() } - return Data(bytes: bytes, count: Int(output.frameLength) * 2) + throw VoiceFailure.format } func finish() throws -> [Data] { guard let converter else { return [] } var tail: [Data] = [] for _ in 0..<8 { - guard let output = AVAudioPCMBuffer(pcmFormat: converter.outputFormat, frameCapacity: 2048) - else { throw VoiceFailure.format } + let output = try outputBuffer(format: converter.outputFormat, capacity: 2048) var error: NSError? let status = converter.convert(to: output, error: &error) { _, flag in flag.pointee = .endOfStream diff --git a/openless-all/ios/CoreHost/src/control.rs b/openless-all/ios/CoreHost/src/control.rs index 54e4691c6..1e6b7e667 100644 --- a/openless-all/ios/CoreHost/src/control.rs +++ b/openless-all/ios/CoreHost/src/control.rs @@ -25,6 +25,8 @@ pub(crate) enum ControlAction { AccountSync { request: Value }, VocabularyBridge { request: Value }, Snapshot, + AudioSettings, + UpdateAudioSettings { enabled: bool, dsp: openless_core::AudioDspConfig }, StyleLibrary, StyleDetail { id: String }, SetStyleEnabled { id: String, enabled: bool }, @@ -194,6 +196,7 @@ impl Host { return; } let product_action = matches!(action, + ControlAction::AudioSettings | ControlAction::UpdateAudioSettings { .. } | ControlAction::AccountSync { .. } | ControlAction::VocabularyBridge { .. } | ControlAction::Library | ControlAction::SearchHistory { .. } | ControlAction::HistoryDetail { .. } | ControlAction::DeleteHistory { .. } | ControlAction::ClearLibraryHistory | ControlAction::AudioAsset { .. } | ControlAction::Vocabulary { .. } @@ -216,7 +219,7 @@ impl Host { // interval before Core has installed its own session. No start/save race. let admission = self.admission.lock().unwrap(); if admission.is_some() && !matches!(action, - ControlAction::Snapshot | ControlAction::Library | ControlAction::SearchHistory { .. } + ControlAction::AudioSettings | ControlAction::Snapshot | ControlAction::Library | ControlAction::SearchHistory { .. } | ControlAction::HistoryDetail { .. } | ControlAction::Vocabulary { .. } | ControlAction::StyleLibrary | ControlAction::StyleDetail { .. } | ControlAction::CapabilitySnapshot | ControlAction::TranslationConfiguration) { @@ -355,6 +358,21 @@ impl Host { } pub(super) async fn control(&self, action: ControlAction) -> Result { match action { + ControlAction::AudioSettings => { + let prefs = self.backend.get_preferences(); + Ok(json!({"enabled":prefs.audio_enhancement_enabled,"dsp":prefs.audio_dsp})) + } + ControlAction::UpdateAudioSettings { enabled, dsp } => { + let mut prefs = self.backend.get_preferences(); + prefs.audio_enhancement_enabled = enabled; + prefs.audio_dsp = dsp.normalized(); + prefs.audio_dsp.android_aec_enabled = false; + prefs.audio_dsp.android_noise_suppression_enabled = false; + prefs.audio_dsp.android_agc_enabled = false; + self.backend.update_settings(prefs, SettingsUpdateOptions::STRICT, &NoopSettingsRuntime)?; + let prefs = self.backend.get_preferences(); + Ok(json!({"enabled":prefs.audio_enhancement_enabled,"dsp":prefs.audio_dsp})) + } ControlAction::AccountSync { request } => self.account_sync.execute(request).await, ControlAction::VocabularyBridge { request } => vocabulary_bridge::execute(&self.backend, self.vocabulary_metadata_directory()?, request), ControlAction::Snapshot => self.control_snapshot().await, diff --git a/openless-all/ios/CoreHost/src/control_tests.rs b/openless-all/ios/CoreHost/src/control_tests.rs index 14616fdfb..3b63466ec 100644 --- a/openless-all/ios/CoreHost/src/control_tests.rs +++ b/openless-all/ios/CoreHost/src/control_tests.rs @@ -3,6 +3,72 @@ use super::*; use credentials::{IosCredentialStore, MemoryVault}; use openless_core::credentials::{CredentialKey, CredentialNamespace, SecretValue}; +#[test] +fn audio_settings_are_available_without_developer_mode_and_persist() { + let directory = tempfile::tempdir().unwrap(); + let storage = Arc::new(MemoryVault::default()); + let (handle, context) = open_at(directory.path(), false, storage.clone()); + let initial = result(control(handle, context, json!({"action":"audioSettings"}))); + assert_eq!(initial["enabled"], false); + let saved = result(control(handle, context, json!({"action":"updateAudioSettings","enabled":true,"dsp":{"noiseReduction":"off","gainDb":100,"androidAecEnabled":true}}))); + assert_eq!(saved["enabled"], true); + assert_eq!(saved["dsp"]["gainDb"], 50); + assert_eq!(saved["dsp"]["androidAecEnabled"], false); + ol_ios_destroy(handle); + let (handle, context) = open_at(directory.path(), false, storage); + assert_eq!(result(control(handle, context, json!({"action":"audioSettings"}))), saved); + ol_ios_destroy(handle); +} + +#[test] +fn native_capture_enhancement_and_stop_flush_match_shared_processor() { + #[derive(Default)] + struct Sink(Mutex>); + impl AudioConsumer for Sink { + fn consume_pcm_chunk(&self, pcm: &[u8]) { self.0.lock().unwrap().extend_from_slice(pcm); } + } + impl RecordingProgressSink for Sink { + fn publish_level(&self, _: u64, _: f32) -> Result<(), BackendError> { Ok(()) } + } + let directory = tempfile::tempdir().unwrap(); + let (handle, context) = open_at(directory.path(), false, Arc::new(MemoryVault::default())); + let owner = host(handle).unwrap(); + let id = SessionId::new(); + let config = openless_core::AudioProcessingConfig { + enhancement_enabled: true, + dsp: openless_core::AudioDspConfig { noise_reduction: openless_core::NoiseReduction::Off, gain_db: 6, agc_enabled: false, ..Default::default() }, + }; + let sink = Arc::new(Sink::default()); + owner.broker.captures.lock().unwrap().insert(id, Arc::new(Capture { + processing: Some(Mutex::new((openless_core::AudioProcessor::new(config), Vec::new()))), + consumer: sink.clone(), progress: sink.clone(), bytes: AtomicU64::new(0), archive: None, + })); + let mut reference = openless_core::AudioProcessor::new(config); + let mut expected = Vec::new(); + let pcm: Vec = (0..1001).flat_map(|i| (((i as f32 * 0.07).sin() * 3000.0) as i16).to_le_bytes()).collect(); + let session = CString::new(id.to_string()).unwrap(); + for bytes in pcm.chunks(640) { + assert_eq!(unsafe { ol_ios_feed_pcm(handle, session.as_ptr(), bytes.as_ptr(), bytes.len()) }, 0); + let samples: Vec = bytes.chunks_exact(2).map(|p| i16::from_le_bytes([p[0],p[1]]) as f32 / 32768.0).collect(); + if let Some(chunk) = reference.process(&samples, 1, 16_000) { expected.extend(chunk.pcm_i16_le); } + } + if let Some(chunk) = reference.finish() { expected.extend(chunk.pcm_i16_le); } + let task = owner.executor.spawn(Box::new(Recording { broker: owner.broker.clone(), id, archive: None }).stop()); + loop { + let request = next(context); + if request["kind"] == "nativeRequest" { + assert_eq!(request["operation"], "captureStop"); + reply(handle, &request, Value::Null); + break; + } + } + owner.executor.block_on(task).unwrap().unwrap(); + assert_eq!(*sink.0.lock().unwrap(), expected); + assert_eq!(expected.len(), pcm.len()); + assert_ne!(expected, pcm); + ol_ios_destroy(handle); +} + pub(super) fn open_at(directory: &std::path::Path, developer: bool, storage: Arc) -> (u64, u64) { static CONTEXT: AtomicU64 = AtomicU64::new(10_000); let context = CONTEXT.fetch_add(1, Ordering::Relaxed); @@ -250,6 +316,10 @@ fn management_is_rejected_during_start_admission_and_rewrite_requires_llm() { ); assert_eq!(busy["ok"], false); assert_eq!(busy["message"], "recording or processing is active"); + let audio = result(control(handle, context, json!({"action":"audioSettings"}))); + let busy_audio = control(handle, context, json!({"action":"updateAudioSettings","enabled":true,"dsp":audio["dsp"]})); + assert_eq!(busy_audio["ok"], false); + assert_eq!(busy_audio["message"], "recording or processing is active"); // Snapshot is allowed during capture and doesn't mutate settings. result(control(handle, context, json!({"action":"snapshot"}))); assert_eq!( diff --git a/openless-all/ios/CoreHost/src/lib.rs b/openless-all/ios/CoreHost/src/lib.rs index 3e6055de6..e66d4bf92 100644 --- a/openless-all/ios/CoreHost/src/lib.rs +++ b/openless-all/ios/CoreHost/src/lib.rs @@ -77,6 +77,7 @@ struct Callbacks { context: u64, } struct Capture { + processing: Option)>>, consumer: Arc, progress: Arc, bytes: AtomicU64, @@ -185,6 +186,7 @@ impl AudioRecorder for NativeRecorder { broker.captures.lock().unwrap().insert( id, Arc::new(Capture { + processing: context.recording.audio_processing.enhancement_enabled.then(|| Mutex::new((openless_core::AudioProcessor::new(context.recording.audio_processing), Vec::new()))), consumer, progress, bytes: AtomicU64::new(0), @@ -212,7 +214,16 @@ impl ActiveRecording for Recording { .broker .request("captureStop", self.id, Value::Null) .await; - self.broker.captures.lock().unwrap().remove(&self.id); + let capture = self.broker.captures.lock().unwrap().remove(&self.id); + if result.is_ok() { + if let Some(capture) = capture { + let tail = capture.processing.as_ref().and_then(|processing| processing.lock().unwrap().0.finish()); + if let Some(tail) = tail { + capture.consumer.consume_pcm_chunk(&tail.pcm_i16_le); + if let Some(archive) = &capture.archive { archive.append(&tail.pcm_i16_le); } + } + } + } if let Some(archive) = &self.archive { archive.finish(); } result.map(|_| ()) }) @@ -799,9 +810,19 @@ pub unsafe extern "C" fn ol_ios_feed_pcm( let capture = owner.broker.captures.lock().unwrap().get(&session).cloned(); let Some(capture) = capture else { return 1; }; let bytes = unsafe { std::slice::from_raw_parts(pcm,len) }; + let chunk = if let Some(processing) = &capture.processing { + let mut processing = processing.lock().unwrap(); + let (processor, samples) = &mut *processing; + samples.clear(); + samples.extend(bytes.chunks_exact(2).map(|p| i16::from_le_bytes([p[0], p[1]]) as f32 / 32768.0)); + Some(processor.process(samples, 1, 16_000)) + } else { None }; + let elapsed = capture.bytes.fetch_add(len as u64,Ordering::Relaxed).saturating_add(len as u64) * 1000 / 32_000; + if matches!(chunk, Some(None)) { return 0; } + let processed = chunk.flatten(); + let bytes = processed.as_ref().map(|chunk| chunk.pcm_i16_le.as_slice()).unwrap_or(bytes); if let Some(archive) = &capture.archive { archive.append(bytes); } capture.consumer.consume_pcm_chunk(bytes); - let elapsed = capture.bytes.fetch_add(len as u64,Ordering::Relaxed).saturating_add(len as u64) * 1000 / 32_000; let level = openless_core::external_audio::pcm_i16_le_rms(bytes); let _ = capture.progress.publish_level(elapsed,level); owner.broker.emit(json!({"kind":"level","sessionID":session.to_string(),"level":level,"elapsedMs":elapsed})); diff --git a/openless-all/ios/CoreHost/src/recording_maintenance_tests.rs b/openless-all/ios/CoreHost/src/recording_maintenance_tests.rs index a71bf0383..d90d10be9 100644 --- a/openless-all/ios/CoreHost/src/recording_maintenance_tests.rs +++ b/openless-all/ios/CoreHost/src/recording_maintenance_tests.rs @@ -229,6 +229,7 @@ fn idle_gate_never_scans_starting_capture_or_concurrent_management() { owner.broker.captures.lock().unwrap().insert( session, Arc::new(Capture { + processing: None, consumer: Arc::new(Sink), progress: Arc::new(Sink), bytes: AtomicU64::new(0), diff --git a/openless-all/ios/Resources/LocalizationSources.json b/openless-all/ios/Resources/LocalizationSources.json index 825542855..7c1bea1f2 100644 --- a/openless-all/ios/Resources/LocalizationSources.json +++ b/openless-all/ios/Resources/LocalizationSources.json @@ -6737,6 +6737,232 @@ "Keyboard/KeyboardViewController.swift" ], "translationSource": "Native iOS translation" + }, + "语音识别 准备 识别方式 语音增强 降噪 增益 均衡器 speech recognition audio enhancement noise gain equalizer": { + "callsites": [ + "App/SettingsView.swift:111" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "语音增强": { + "callsites": [ + "App/SettingsView.swift:115", + "App/SettingsView.swift:234", + "App/SettingsView.swift:292" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "平直": { + "callsites": [ + "App/SettingsView.swift:220" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "明亮人声": { + "callsites": [ + "App/SettingsView.swift:221" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "低沉人声": { + "callsites": [ + "App/SettingsView.swift:222" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "播客": { + "callsites": [ + "App/SettingsView.swift:223" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "清晰人声": { + "callsites": [ + "App/SettingsView.swift:224" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "温暖人声": { + "callsites": [ + "App/SettingsView.swift:225" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "设置保存后在下一次录音生效。RNNoise 较轻量，PureVox 使用更多计算资源。默认保持原有音频效果。": { + "callsites": [ + "App/SettingsView.swift:237" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "降噪与增益": { + "callsites": [ + "App/SettingsView.swift:240" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "降噪算法": { + "callsites": [ + "App/SettingsView.swift:241" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "频谱减法": { + "callsites": [ + "App/SettingsView.swift:243" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "降噪强度": { + "callsites": [ + "App/SettingsView.swift:245" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "输入增益 dB": { + "callsites": [ + "App/SettingsView.swift:246" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "自动增益": { + "callsites": [ + "App/SettingsView.swift:247", + "App/SettingsView.swift:303" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "目标电平 %": { + "callsites": [ + "App/SettingsView.swift:249" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "起音速度": { + "callsites": [ + "App/SettingsView.swift:250" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "释放速度": { + "callsites": [ + "App/SettingsView.swift:251" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "语音门限": { + "callsites": [ + "App/SettingsView.swift:253", + "App/SettingsView.swift:303" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "门限 dB": { + "callsites": [ + "App/SettingsView.swift:254" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "梳状滤波 %": { + "callsites": [ + "App/SettingsView.swift:257" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "十段均衡器": { + "callsites": [ + "App/SettingsView.swift:258" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "均衡器预设": { + "callsites": [ + "App/SettingsView.swift:260" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "前级增益 dB": { + "callsites": [ + "App/SettingsView.swift:265" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "音色与房间调整": { + "callsites": [ + "App/SettingsView.swift:273" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "梳状滤波可能改变音色，建议从 0 开始。语音门限不会自动停止录音，过高会丢失轻声。": { + "callsites": [ + "App/SettingsView.swift:274" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "处理顺序": { + "callsites": [ + "App/SettingsView.swift:276" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "恢复默认增强设置": { + "callsites": [ + "App/SettingsView.swift:288" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "保存中…": { + "callsites": [ + "App/SettingsView.swift:293" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "降噪": { + "callsites": [ + "App/SettingsView.swift:303" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "梳状滤波": { + "callsites": [ + "App/SettingsView.swift:303" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "均衡器": { + "callsites": [ + "App/SettingsView.swift:303" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "输入增益": { + "callsites": [ + "App/SettingsView.swift:303" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "音频设置暂不可用，请稍后重试。": { + "callsites": [ + "App/SettingsView.swift:311" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "无法读取音频设置，请重试。": { + "callsites": [ + "App/SettingsView.swift:317" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "已保存，下次录音生效。": { + "callsites": [ + "App/SettingsView.swift:327" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" + }, + "保存失败，当前修改尚未应用。": { + "callsites": [ + "App/SettingsView.swift:328" + ], + "translationSource": "Native iOS audio settings (2026-10-24)" } }, "infoPlistEntries": { diff --git a/openless-all/ios/Resources/en.lproj/Localizable.strings b/openless-all/ios/Resources/en.lproj/Localizable.strings index f681321ff..5991689dd 100644 --- a/openless-all/ios/Resources/en.lproj/Localizable.strings +++ b/openless-all/ios/Resources/en.lproj/Localizable.strings @@ -1015,3 +1015,42 @@ "中" = "ZH"; "展开候选" = "Expand candidates"; "收起候选" = "Collapse candidates"; + +// Native audio settings +"低沉人声" = "Deep voice"; +"保存中…" = "Saving..."; +"保存失败，当前修改尚未应用。" = "Save failed. Your changes have not been applied."; +"前级增益 dB" = "Preamp gain dB"; +"十段均衡器" = "Ten-band equalizer"; +"均衡器" = "Equalizer"; +"均衡器预设" = "Equalizer presets"; +"处理顺序" = "Processing order"; +"已保存，下次录音生效。" = "Saved. Applies to the next recording."; +"平直" = "Flat"; +"恢复默认增强设置" = "Reset enhancement settings"; +"播客" = "Podcast"; +"无法读取音频设置，请重试。" = "Unable to read audio settings. Please retry."; +"明亮人声" = "Bright vocal"; +"梳状滤波" = "Comb filter"; +"梳状滤波 %" = "Comb filter %"; +"梳状滤波可能改变音色，建议从 0 开始。语音门限不会自动停止录音，过高会丢失轻声。" = "The comb filter can change voice tone; start at 0. The speech gate does not stop recording, and a high threshold can remove quiet speech."; +"清晰人声" = "Vocal clarity"; +"温暖人声" = "Warm vocal"; +"目标电平 %" = "Target level %"; +"自动增益" = "Automatic gain"; +"设置保存后在下一次录音生效。RNNoise 较轻量，PureVox 使用更多计算资源。默认保持原有音频效果。" = "Saved settings apply to the next recording. RNNoise is lighter; PureVox uses more processing resources. Defaults preserve the existing audio behavior."; +"语音增强" = "Speech enhancement"; +"语音识别 准备 识别方式 语音增强 降噪 增益 均衡器 speech recognition audio enhancement noise gain equalizer" = "speech recognition prepare method audio enhancement noise gain equalizer"; +"语音门限" = "Speech gate"; +"起音速度" = "Attack speed"; +"输入增益" = "Input gain"; +"输入增益 dB" = "Input gain dB"; +"释放速度" = "Release speed"; +"门限 dB" = "Threshold dB"; +"降噪" = "Noise reduction"; +"降噪与增益" = "Noise reduction and gain"; +"降噪强度" = "Noise reduction strength"; +"降噪算法" = "Noise reduction algorithm"; +"音色与房间调整" = "Tone and room adjustments"; +"音频设置暂不可用，请稍后重试。" = "Audio settings are unavailable. Please retry later."; +"频谱减法" = "Spectral subtraction"; diff --git a/openless-all/ios/Resources/zh-Hans.lproj/Localizable.strings b/openless-all/ios/Resources/zh-Hans.lproj/Localizable.strings index a0e494059..ba99fe2e8 100644 --- a/openless-all/ios/Resources/zh-Hans.lproj/Localizable.strings +++ b/openless-all/ios/Resources/zh-Hans.lproj/Localizable.strings @@ -1015,3 +1015,42 @@ "中" = "中"; "展开候选" = "展开候选"; "收起候选" = "收起候选"; + +// Native audio settings +"低沉人声" = "低沉人声"; +"保存中…" = "保存中…"; +"保存失败，当前修改尚未应用。" = "保存失败，当前修改尚未应用。"; +"前级增益 dB" = "前级增益 dB"; +"十段均衡器" = "十段均衡器"; +"均衡器" = "均衡器"; +"均衡器预设" = "均衡器预设"; +"处理顺序" = "处理顺序"; +"已保存，下次录音生效。" = "已保存，下次录音生效。"; +"平直" = "平直"; +"恢复默认增强设置" = "恢复默认增强设置"; +"播客" = "播客"; +"无法读取音频设置，请重试。" = "无法读取音频设置，请重试。"; +"明亮人声" = "明亮人声"; +"梳状滤波" = "梳状滤波"; +"梳状滤波 %" = "梳状滤波 %"; +"梳状滤波可能改变音色，建议从 0 开始。语音门限不会自动停止录音，过高会丢失轻声。" = "梳状滤波可能改变音色，建议从 0 开始。语音门限不会自动停止录音，过高会丢失轻声。"; +"清晰人声" = "清晰人声"; +"温暖人声" = "温暖人声"; +"目标电平 %" = "目标电平 %"; +"自动增益" = "自动增益"; +"设置保存后在下一次录音生效。RNNoise 较轻量，PureVox 使用更多计算资源。默认保持原有音频效果。" = "设置保存后在下一次录音生效。RNNoise 较轻量，PureVox 使用更多计算资源。默认保持原有音频效果。"; +"语音增强" = "语音增强"; +"语音识别 准备 识别方式 语音增强 降噪 增益 均衡器 speech recognition audio enhancement noise gain equalizer" = "语音识别 准备 识别方式 语音增强 降噪 增益 均衡器 speech recognition audio enhancement noise gain equalizer"; +"语音门限" = "语音门限"; +"起音速度" = "起音速度"; +"输入增益" = "输入增益"; +"输入增益 dB" = "输入增益 dB"; +"释放速度" = "释放速度"; +"门限 dB" = "门限 dB"; +"降噪" = "降噪"; +"降噪与增益" = "降噪与增益"; +"降噪强度" = "降噪强度"; +"降噪算法" = "降噪算法"; +"音色与房间调整" = "音色与房间调整"; +"音频设置暂不可用，请稍后重试。" = "音频设置暂不可用，请稍后重试。"; +"频谱减法" = "频谱减法"; diff --git a/openless-all/ios/scripts/ReferencePCMWorker.swift b/openless-all/ios/scripts/ReferencePCMWorker.swift new file mode 100644 index 000000000..a25337fbd --- /dev/null +++ b/openless-all/ios/scripts/ReferencePCMWorker.swift @@ -0,0 +1,60 @@ +// Frozen pre-optimization oracle from f0744e10. Test-only. +import AVFoundation +import Foundation +actor ReferencePCMWorker { + private var converter: AVAudioConverter? + func convert(_ packet: AudioPacket) throws -> Data { + let input = try packet.makeBuffer() + let inputFormat = input.format + guard inputFormat.sampleRate > 0, input.frameLength > 0, + let outputFormat = AVAudioFormat( + commonFormat: .pcmFormatInt16, sampleRate: 16_000, channels: 1, interleaved: true) + else { + throw VoiceFailure.format + } + if converter?.inputFormat != inputFormat { + converter = AVAudioConverter(from: inputFormat, to: outputFormat) + } + guard let converter else { throw VoiceFailure.format } + let capacity = AVAudioFrameCount( + ceil(Double(input.frameLength) * 16_000 / inputFormat.sampleRate) + 32) + guard let output = AVAudioPCMBuffer(pcmFormat: outputFormat, frameCapacity: capacity) else { + throw VoiceFailure.format + } + var supplied = false + var error: NSError? + let status = converter.convert(to: output, error: &error) { _, flag in + if supplied { + flag.pointee = .noDataNow + return nil + } + supplied = true + flag.pointee = .haveData + return input + } + if let error { throw error } + guard status != .error else { throw VoiceFailure.format } + guard output.frameLength > 0, let bytes = output.int16ChannelData?[0] else { return Data() } + return Data(bytes: bytes, count: Int(output.frameLength) * 2) + } + func finish() throws -> [Data] { + guard let converter else { return [] } + var tail: [Data] = [] + for _ in 0..<8 { + guard let output = AVAudioPCMBuffer(pcmFormat: converter.outputFormat, frameCapacity: 2048) + else { throw VoiceFailure.format } + var error: NSError? + let status = converter.convert(to: output, error: &error) { _, flag in + flag.pointee = .endOfStream + return nil + } + if let error { throw error } + guard status != .error else { throw VoiceFailure.format } + if output.frameLength > 0, let bytes = output.int16ChannelData?[0] { + tail.append(Data(bytes: bytes, count: Int(output.frameLength) * 2)) + } + if status == .endOfStream || output.frameLength == 0 { return tail } + } + throw VoiceFailure.format + } +} diff --git a/openless-all/ios/scripts/audio-smoke.swift b/openless-all/ios/scripts/audio-smoke.swift index 418814d8c..82020a6b2 100644 --- a/openless-all/ios/scripts/audio-smoke.swift +++ b/openless-all/ios/scripts/audio-smoke.swift @@ -2,6 +2,24 @@ import AVFoundation import Foundation @main struct AudioSmoke { + static func packet(rate: Double, channels: AVAudioChannelCount, frames: AVAudioFrameCount, offset: Int, legacy: Bool) throws -> AudioPacket { + let format = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: rate, channels: channels, interleaved: false)! + let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: frames)! + buffer.frameLength = frames + for channel in 0.. 0.005 && power < 0.1, "invalid PCM amplitude") - print("PCM \(legacy ? "legacy copy" : "immutable") \(Int(rate))Hz × \(channels) → 16kHz mono: \(samples.count) samples") + print("PCM \(legacy ? "legacy copy" : "immutable") \(Int(rate))Hz × \(channels), variable=\(variablePackets) → 16kHz mono: \(samples.count) samples") combinations += 1 + } } } } diff --git a/openless-all/ios/scripts/test-audio.sh b/openless-all/ios/scripts/test-audio.sh index 7b31fa0de..39396d2ff 100755 --- a/openless-all/ios/scripts/test-audio.sh +++ b/openless-all/ios/scripts/test-audio.sh @@ -3,6 +3,6 @@ set -euo pipefail ROOT="$(cd "$(dirname "$0")/.." && pwd)" mkdir -p "$ROOT/.build/tests" xcrun swiftc -swift-version 6 -target "$(uname -m)-apple-macos14.0" -parse-as-library \ - "$ROOT/App/Speech/AudioPCM.swift" "$ROOT/scripts/audio-smoke.swift" \ + "$ROOT/App/Speech/AudioPCM.swift" "$ROOT/scripts/ReferencePCMWorker.swift" "$ROOT/scripts/audio-smoke.swift" \ -o "$ROOT/.build/tests/audio-smoke" "$ROOT/.build/tests/audio-smoke"