diff --git a/docs/research/issue-1264.md b/docs/research/issue-1264.md new file mode 100644 index 000000000..d9e4d8517 --- /dev/null +++ b/docs/research/issue-1264.md @@ -0,0 +1,98 @@ +# Issue #1264 Windows 语音快捷键联动调研报告 + +> 调研日期：2026-10-25 +> +> 范围：提交 `38fbe1c082e782e566e13769e4675a183ccc812a` 的 Windows/桌面端源码；调研时工作树与 upstream/beta 一致。仅静态调研，不实施功能，不运行构建、测试或打包。 + +## 需求与关联背景 + +[Issue #1264](https://github.com/Open-Less/openless/issues/1264) 要求：总开关开启后，新增“与语音快捷键联动”子选项；第一次点击录音快捷键开启持续本地唤醒监听，第二次点击临时关闭并释放麦克风，而不是反复进入设置修改总开关。维护者已[回复“oka”](https://github.com/Open-Less/openless/issues/1264#issuecomment-6387227197)，尚未补充启动、模式冲突或录音中切换的语义。调研时 Issue 仍 OPEN，未发现关联实现 PR。 + +- [#1240](https://github.com/Open-Less/openless/pull/1240) 于 2026-10-21 合并，为本需求提供 Windows 本地唤醒基础；其描述仍列出完成词残留、Windows `0xC0000374` 堆损坏来源未定位、超时按总录音时长等未验收问题。这些是继承风险，不属于本 Issue 应顺手扩大的修复范围。 +- [#1263](https://github.com/Open-Less/openless/pull/1263) 当前 OPEN，只补 ScrollLock 快捷键支持，明确不含语音唤醒开关联动；不是本需求的替代实现，也不是使用其他已支持快捷键的前置条件。 +- [#1235](https://github.com/Open-Less/openless/pull/1235) 当前 OPEN，增加 Toggle 模式 Enter 停止录音，会触及 Core 偏好、coordinator/hotkey_loops 和录音设置页，后续集成需组合回归。 + +## 1. 结论摘要 + +- **[已证实，现状]** 语音唤醒已有 Core 生命周期和 Windows runtime，但没有“与录音快捷键联动”的持久字段，也没有把普通录音快捷键改成“首次开启监听、再次关闭监听”的策略（`ports.rs:117-142`、`api.rs:3415-3432`）。[1] +- **[已证实，隐私]** 唤醒前固定使用本地 Sherpa，standby 音频不发送云端；唤醒后的正式正文走独立 ASR provider，可能是云端，不能把“本地唤醒”表述成“全流程本地”（`shared_types.rs:462-476`、`core_adapters.rs:1942-1957`、`api.rs:6364-6397`）。[2] +- **[已证实，风险]** `VoiceActivationHandle::stop()` 会取消 recognizer、清理 activation consumer，但随后可能按 low-latency 偏好保持 warm stream，不能保证 Issue 所说的“释放麦克风”（`recorder.rs:115-122`、`core_adapters.rs:2236-2274`）。[3] +- **[推断，方案]** 可复用现有 handle、generation、事件和 `WarmCaptureHub`，但必须增加 Core-owned linked policy、临时暂停状态和 effective warm-capture 约束。 + +## 2. 现状与本地/云端边界 + +`UserPreferences` 持久化总开关、唤醒/结束词、超时和设备名，设置变化会触发 runtime 重配（`commands/settings.rs:242-250,309-321`）；注释明确写明 standby audio never sent to a cloud provider（`shared_types.rs:462-476`）。[5] Windows runtime 解析固定 `SherpaOnnx` 与 `zipformer-bilingual-zh-en-streaming`，模型未安装就返回 ModelMissing；唤醒 recognizer 通过 `WarmCaptureHub` arm（`core_adapters.rs:1942-1957,2039-2048`）。[6] activation VAD 使用 RMS `0.015`、约 400 ms pre-roll 和 8 个 hangover chunks，静音块不直接送 recognizer（`recorder.rs:49-55,447-487`）。[7][17] 上述本地边界只适用于唤醒前音频，不代表唤醒后的正式听写和后续处理都离线；本轮未做运行时网络抓取验证。 + +唤醒信号经 generation 校验后，Core 调用 `handoff_to_dictation`；Windows 取消 wake ASR，并可创建本地 finish recognizer（`api.rs:3692-3758`、`core_adapters.rs:2115-2173`）。[8][18] 随后正式 dictation 通过 `resolve_dictation_asr_provider` 按当前偏好选 provider；共享云端 provider 仅在特定未配置条件下切到本地 fallback（`api.rs:6364-6397,9253-9264`）。[9][19] 因此“正文必须本地”是**待定义**，不能由现有语音唤醒开关推导。 + +启动阶段 Core 会先使旧 generation 失效，再停止旧 handle；只有偏好启用、host runtime 可用且 capture 能成功 arm 时才进入等待唤醒，旧回调会因 generation 不匹配被丢弃（`openless-all/app/crates/openless-core/src/api.rs:3415-3432,3692-3713`）。这说明 linked mode 应复用现有异步生命周期，而不是在 Tauri coordinator 另建一套状态机。设置保存会校验短语和超时，并在相关字段变化后触发 runtime 重配；临时快捷键状态则不能走持久化设置路径。 +## 3. Core 接缝、临时状态与设备释放 + +最合适的接缝是 `VoiceActivationRuntime::configure`、`VoiceActivationHandle::stop`/`handoff_to_dictation`、generation 和 `VoiceActivationStateChanged`。EventBus 已按 generation 保存最新状态，Tauri 已转发 `voice-activation:state-changed`（`ports.rs:117-142`、`events.rs:410-515`、`tauri_events.rs:213-215`）。[4][10][20] + +建议新增类似 `voiceActivationHotkeyLinked` 的**持久偏好**，但“当前因快捷键开启/暂停”只能是运行时状态，不能写回设置。首次完成按键仲裁的有效点击只 arm listener，第二次 press 进入临时暂停；普通 dictation 的 FIFO、generation 和 stale-event 防护继续由 Core 处理。Issue 没有规定只能 Toggle：Toggle 可以是首个实现选项，也可以另行定义 Auto 短按 toggle、Hold 按住监听，需先确认语义。 + +必须正视释放坑：`warm_capture_enabled` 的公式是 `voice_activation || (low_latency && !low_memory)`（`recorder.rs:115-122`）。`handle.stop()` 清 consumer、取消 wake/finish ASR 后，又以 `voice_activation=false` 按普通偏好配置 warm hub；low-latency 开启时仍可能预热麦克风（`core_adapters.rs:2236-2274`）。所以 stop 可复用，但不是硬释放保证；临时 pause 必须同时关闭 effective warm policy，并在需要时停止 native stream。 + +还要抑制自动 rearm：正式录音释放后存在 `configure_voice_activation()` 重配（`core_adapters.rs:3525-3535`）；结束词完成和错误恢复也会在持久 `voice_activation_enabled` 为真时重新 arm（`api.rs:3764-3774,3842-3850,3884-3890,4022-4055`）。[11][13][14][15] 状态机必须区分“持久启用”和“当前暂停”，所有 reconfigure/recovery 都检查后者。 +普通 dictation 的 provisional capture 还会在 warm 与 cold 两条路径之间切换：cold capture 会暂停 warm stream，录音结束再恢复；voice activation handoff 则使用带 pre-roll 的 staged warm capture。因而 Core 的 voice session gate 与 Tauri 的 WarmCaptureHub 是两层资源协调，不能只用一个 Busy 标志解释所有麦克风竞争。 + +## 4. 需定义语义、验证重点 +建议最小运行状态包括 LinkedDisabled、LinkedStarting、LinkedListeningForWake、LinkedStopping、LinkedPaused 和 LinkedUnavailable/Failed；它们与持久 `voice_activation_enabled` 分离。首次点击失败时应保留明确错误而不回退成普通 dictation，第二次点击若处于 handoff 或停止中应按 generation 防止重复 stop；重新启用必须由下一次明确点击或产品定义的恢复动作触发。 + +**[待定义]** 当前 Auto 用 350 ms 区分短按/长按，并有 debounce、cooldown、modifier grace（`hotkey_interpreter.rs:7-19`）。[16] 需明确四种模式的 linked 行为，以及 listener 在 Starting、Handoff、结束词收尾，或 Dictation、LessComputer、SelectionVoice、QA、TTS 活跃时应取消、排队、Busy 还是忽略。本次源码未证实 TTS 的麦克风抑制策略。`VoiceSessionGate` 只明确互斥 Dictation、LessComputer、SelectionVoice、QA（`voice_session.rs:5-79`）；standby listener 与 TTS 的关系仍待验证。[12] + +前端已有完整 phase 类型和状态事件通道，但设置页当前主要用事件判断模型缺失；linked mode 还需要把“已暂停但持久开启”与 Disabled、MicBusy、PermissionDenied 分开显示，避免用户误以为持久配置被清除。 +**[测试重点]** 现有单测已覆盖短语规范化、revision 去重、generation/session 防陈旧和 warm policy（`voice_activation.rs:428-565`、`recorder.rs:3244-3255`）。[22]新增用例应覆盖首次/二次点击、重复点击/stale release、暂停不持久化且不自动 rearm、low-latency 下真实 stream 释放、cold/warm 交替、设备不存在、权限拒绝、其他程序占用、模型缺失、唤醒后云端/本地 provider、结束词和超时恢复。资源释放与权限必须在 Windows 实机验证，静态单测不能替代。 +实机还应观察系统麦克风指示灯、设备切换后的实际占用、录音结束后的恢复时序，以及唤醒前 PCM 是否只到本地 recognizer；云端正文则应通过 provider 日志或网络策略单独验证，不能把状态事件当成隐私证明。 + +## 5. 推荐交互与改动边界 + +**推荐方案而非已实现行为：** 联动偏好默认关闭，保证旧用户行为不变；启用联动后，快捷键操作运行时监听状态，不反复写总开关。建议每次应用启动后先保持临时暂停，等待首次显式点击；是否恢复上次监听状态仍需产品确认。联动开启期间是否统一采用单击切换、覆盖原 Hold/Auto/DoubleClick 语义，应明确说明，不能暗中只支持 Toggle 而缩减需求。联动关闭时恢复原录音模式。 + +设置 UI 复用现有 Windows 语音唤醒内联子区域（`RecordingInputSection.tsx:617-665`），新增开关与常驻说明，不需要 Modal/Drawer/二次确认。[23] 显示“唤醒监听中 / 已临时暂停 / 正在切换 / 无法启动”等真实状态；Toast 只辅助反馈，权限、模型缺失等错误保留可操作的恢复入口。文案应写清“等待唤醒时仅本地处理；开始正式听写后按当前服务配置处理”。异步完成后才显示成功，不夺输入焦点；失败不悄悄回退为普通录音。此处为 UI 选型建议，尚未运行 UI 验证。 + +最小改动面不只是一个 TSX 开关：Core 偏好及 serde/wire 往返、前端类型/默认值和文案、Core hotkey dispatch 与监听有效状态、Tauri warm-capture 策略及设置重配入口、必要的偏好同步边界和回归测试。建议让 Core 统一拥有联动策略，复用现有 Windows runtime，暂不扩展到其他平台。持久联动偏好是否跨设备同步可沿用项目偏好分类；临时暂停态不能同步。 + +## 6. 远程验证要求 + +本轮只有文档变更，不触发构建。后续实施并推送代码时，按准确工作分支和提交 SHA 执行 GitHub Actions 的 `ci.yml`、`release-tauri.yml`（桌面全平台）和 `android-apk.yml`（至少 aarch64）；优先交付已就绪的桌面产物，不把其他 SHA 的成功结果移用到新代码。[24][25][26] + +现有 CI 的 Windows 完整 Tauri 后端测试使用 `--no-run`，只编译；实际执行的是独立 `backend-tests` 和 recorder 模块测试。完整 Core 测试另在 Linux 检查中执行。[27][28] 因此应把新增联动状态机尽量放到可独立执行的 Rust-only 用例中，再用 Windows 实机验收真实设备释放、快速切换和权限恢复；不能把编译成功、安装包生成或状态事件变化当成硬件生命周期通过。 + +额外回归必须包括：暂停后保存设置不会重新预热（`commands/settings.rs:221-236`）、low-latency 开启时仍能释放唤醒所持资源、旧恢复回调不会重新 arm；其他语音任务仍在使用麦克风时要说明占用来源，不能谎报整个应用已释放设备，也不能静默丢弃进行中的正文。[29] + +## 7. 证据边界 + +本报告完成的是源码与 GitHub 一手材料调研，只新增此 Markdown；未修改功能代码，未运行本地编译、测试、打包或依赖构建，未提交、推送或发布 Issue 评论。所有推荐交互和新增测试均待实施与验证，不把既有 PR 的未验收事项说成已经解决。 + +[1]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/ports.rs#L117-L142 +[2]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/core_adapters.rs#L1942-L1957 +[3]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/core_adapters.rs#L2236-L2274 +[4]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/events.rs#L410-L515 +[5]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/shared_types.rs#L462-L476 +[6]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/core_adapters.rs#L1942-L1957 +[7]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/recorder.rs#L49-L55 +[8]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/api.rs#L3692-L3758 +[9]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/api.rs#L6364-L6397 +[10]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/events.rs#L410-L515 +[11]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/api.rs#L3764-L3774 +[12]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/voice_session.rs#L5-L79 +[13]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/core_adapters.rs#L3525-L3535 +[14]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/api.rs#L4022-L4055 +[15]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/api.rs#L3842-L3890 +[16]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/hotkey_interpreter.rs#L7-L19 +[17]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/recorder.rs#L447-L487 +[18]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/core_adapters.rs#L2115-L2173 +[19]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/api.rs#L9253-L9264 +[20]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/tauri_events.rs#L213-L215 +[21]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/recorder.rs#L115-L122 +[22]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/voice_activation.rs#L428-L565 + +[23]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src/pages/settings/RecordingInputSection.tsx#L617-L665 +[24]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/.github/workflows/ci.yml#L10-L21 +[25]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/.github/workflows/release-tauri.yml#L31-L42 +[26]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/.github/workflows/android-apk.yml#L27-L41 +[27]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/.github/workflows/ci.yml#L723-L750 +[28]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/.github/workflows/check-linux-egui.yml#L108-L119 +[29]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/commands/settings.rs#L221-L236 diff --git a/openless-all/app/crates/openless-core/src/api.rs b/openless-all/app/crates/openless-core/src/api.rs index 85d00be1d..ab8d14ab2 100644 --- a/openless-all/app/crates/openless-core/src/api.rs +++ b/openless-all/app/crates/openless-core/src/api.rs @@ -2635,6 +2635,7 @@ pub struct OpenLessBackend { voice_activation_generation: Arc, voice_activation_handoff_generation: AtomicU64, voice_activation_configure_gate: tokio::sync::Mutex<()>, + voice_activation_link: Mutex, backend_lifecycle_gate: tokio::sync::Mutex<()>, backend_shutdown_requested: AtomicBool, voice_activation_session: Arc>>, @@ -3385,6 +3386,7 @@ impl OpenLessBackend { voice_activation_generation: Arc::new(AtomicU64::new(0)), voice_activation_handoff_generation: AtomicU64::new(0), voice_activation_configure_gate: tokio::sync::Mutex::new(()), + voice_activation_link: Mutex::new(crate::voice_activation_hotkey::VoiceActivationLink::default()), backend_lifecycle_gate: tokio::sync::Mutex::new(()), backend_shutdown_requested: AtomicBool::new(false), voice_activation_session: Arc::new(Mutex::new(None)), @@ -3409,12 +3411,171 @@ impl OpenLessBackend { Ok(()) } + fn voice_activation_intent(&self) -> (bool, bool, u64) { + let prefs = self.get_preferences(); + let linked = prefs.voice_activation_enabled && prefs.voice_activation_hotkey_linked + && self.deps.voice_activation_runtime.is_some(); + let (requested, revision) = { + let mut state = self.voice_activation_link.lock().expect("voice link lock poisoned"); + state.configure(linked); + (state.requested, state.revision) + }; + let paused = linked && !requested; + if let Some(runtime) = &self.deps.voice_activation_runtime { + runtime.set_capture_paused(revision, paused); + } + (prefs.voice_activation_enabled && !paused, paused, revision) + } + + fn pause_linked_voice_activation_after_failure(&self, generation: u64) { + let prefs = self.get_preferences(); + if !prefs.voice_activation_enabled || !prefs.voice_activation_hotkey_linked { + return; + } + let revision = { + let mut state = self.voice_activation_link.lock().expect("voice link lock poisoned"); + if self.voice_activation_generation.load(Ordering::Acquire) != generation { + return; + } + state.pause(); + state.revision + }; + if let Some(runtime) = &self.deps.voice_activation_runtime { + runtime.set_capture_paused(revision, true); + } + } + + async fn recover_voice_activation_failure(&self, generation: u64, error: BackendError) { + self.pause_linked_voice_activation_after_failure(generation); + match self.stop_voice_activation_if_current(generation, + crate::voice_activation::VoiceActivationStopReason::RuntimeFailure).await + { + Ok(true) => { + if self.voice_activation_intent().1 { + self.events.publish(None, BackendEventKind::VoiceActivationStateChanged( + crate::voice_activation::VoiceActivationStateSnapshot { + phase: voice_activation_failure_phase(error.code), + activation_generation: self.voice_activation_generation.load(Ordering::Acquire), + session_id: None, + message: Some(error.message), + })); + } else if self.voice_activation_is_requested() { + let _ = self.configure_voice_activation().await; + } + } + Ok(false) => {} + Err(error) => log::warn!("voice activation recovery stop failed: {error}"), + } + } + + fn voice_activation_is_requested(&self) -> bool { + self.voice_activation_intent().0 + } + + /// Accept clicks synchronously so native FIFO delivery never waits for ASR startup. + /// Some(false) consumes an edge; Some(true) schedules reconciliation. + pub fn accept_voice_activation_hotkey_edge(&self, edge: DictationHotkeyEdge) -> Option { + self.voice_activation_intent(); + let prefs = self.get_preferences(); + if !prefs.voice_activation_enabled || !prefs.voice_activation_hotkey_linked + || self.deps.voice_activation_runtime.is_none() + { + return None; + } + let (changed, paused, revision, generation) = { + let mut state = self.voice_activation_link.lock().expect("voice link lock poisoned"); + let changed = state.edge(edge); + let generation = if changed { + let handoff = self.voice_activation_handoff_generation.load(Ordering::Acquire); + if !state.requested { + state.finalize_session = self.voice_activation_session.lock() + .expect("voice activation session lock poisoned").as_ref().map(|s| s.session_id); + if handoff != 0 { + state.finalize_generation = Some(handoff); + } + } + // A handoff may already own staged body PCM before it has a + // session ID. Keep its ticket valid until that body can finish. + if handoff != 0 && state.finalize_generation == Some(handoff) { + self.voice_activation_generation.load(Ordering::Acquire) + } else { + self.voice_activation_generation.fetch_add(1, Ordering::AcqRel) + 1 + } + } else { + self.voice_activation_generation.load(Ordering::Acquire) + }; + (changed, !state.requested, state.revision, generation) + }; + if changed { + if let Some(runtime) = &self.deps.voice_activation_runtime { + runtime.set_capture_paused(revision, paused); + } + self.events.publish(None, BackendEventKind::VoiceActivationStateChanged( + crate::voice_activation::VoiceActivationStateSnapshot { + phase: if paused { crate::voice_activation::VoiceActivationPhase::Pausing } + else { crate::voice_activation::VoiceActivationPhase::Starting }, + activation_generation: generation, + session_id: None, + message: None, + })); + } + Some(changed) + } + + /// Runtime-only intent token for host feedback; never changes saved preferences. + pub fn voice_activation_hotkey_intent(&self) -> Option<(bool, u64)> { + let preferences = self.get_preferences(); + if !preferences.voice_activation_enabled || !preferences.voice_activation_hotkey_linked { + return None; + } + let state = self.voice_activation_link.lock().expect("voice link lock poisoned"); + Some((state.requested, state.revision)) + } + + pub async fn reconcile_voice_activation_hotkey(&self) -> Result<(), BackendError> { + let paused_session = self.voice_activation_link.lock() + .expect("voice link lock poisoned").finalize_session; + let session_id = paused_session.or_else(|| { + self.voice_activation_intent().1.then(|| self.voice_activation_session.lock() + .expect("voice activation session lock poisoned").as_ref().map(|s| s.session_id)).flatten() + }); + if let Some(session_id) = session_id { + // Recorder stop can reconfigure activation: never hold configure_gate here. + if let Err(error) = self.stop_dictation_session(session_id).await { + if !matches!(error.code, BackendErrorCode::InvalidState | BackendErrorCode::Busy) { + return Err(error); + } + } + } + self.configure_voice_activation().await + } + + async fn settle_stale_voice_dictation(&self, session_id: SessionId) { + let must_finalize = self.voice_activation_link.lock() + .expect("voice link lock poisoned").finalize_session == Some(session_id); + if must_finalize || self.voice_activation_intent().1 { + let _ = self.stop_dictation_session(session_id).await; + let _ = self.configure_voice_activation().await; + } else { + let _ = self.cancel_dictation(Some(session_id)).await; + } + } + /// Configure the opt-in local voice activation listener from the current /// preference snapshot. Reconfiguration replaces the previous native /// handle only after it has been stopped, so one process owns one mic. pub async fn configure_voice_activation(&self) -> Result<(), BackendError> { + self.voice_activation_intent(); let _configure_gate = self.voice_activation_configure_gate.lock().await; + let (effective, paused, revision) = self.voice_activation_intent(); let preferences = self.get_preferences(); + let pending_body = self.voice_activation_link.lock() + .expect("voice link lock poisoned").finalize_generation; + if pending_body.is_some_and(|generation| { + self.voice_activation_handoff_generation.load(Ordering::Acquire) == generation + }) { + return Ok(()); + } if self .voice_activation_session .lock() @@ -3447,7 +3608,7 @@ impl OpenLessBackend { .stop(crate::voice_activation::VoiceActivationStopReason::Disabled) .await; } - if !preferences.voice_activation_enabled + if !effective || self.backend_shutdown_requested.load(Ordering::Acquire) || !self.snapshot().running { @@ -3455,7 +3616,7 @@ impl OpenLessBackend { // activation was never enabled. Existing subscribers expect // BackendStarted to be the first startup event; later disable/error // transitions still publish Disabled below when a visible state exists. - let should_publish_disabled = had_previous_handle + let should_publish_disabled = paused || had_previous_handle || self .events .latest_voice_activation_state() @@ -3468,7 +3629,8 @@ impl OpenLessBackend { None, BackendEventKind::VoiceActivationStateChanged( crate::voice_activation::VoiceActivationStateSnapshot { - phase: crate::voice_activation::VoiceActivationPhase::Disabled, + phase: if paused { crate::voice_activation::VoiceActivationPhase::Paused } + else { crate::voice_activation::VoiceActivationPhase::Disabled }, activation_generation: generation, session_id: None, message: None, @@ -3528,14 +3690,22 @@ impl OpenLessBackend { let sink: Arc = Arc::new(CoreVoiceActivationSink { events: Arc::clone(&self.events), }); - match runtime.configure(config, sink).await { + let configured = if preferences.voice_activation_hotkey_linked && !config.wake_enabled { + Err(crate::ports::VoiceActivationError::InvalidConfig( + "enable the wake phrase before linking the recording shortcut".into())) + } else { + runtime.configure(config, sink).await + }; + match configured { Ok(handle) => { let stale_handle = { let mut slot = self .voice_activation_handle .lock() .expect("voice activation handle lock poisoned"); - if self.voice_activation_generation.load(Ordering::Acquire) != generation { + if self.voice_activation_generation.load(Ordering::Acquire) != generation + || self.voice_activation_link.lock().expect("voice link lock poisoned").revision != revision + { Some(handle) } else { *slot = Some(handle.into()); @@ -3565,7 +3735,11 @@ impl OpenLessBackend { Ok(()) } Err(error) => { + if self.voice_activation_generation.load(Ordering::Acquire) != generation { + return Ok(()); + } let error = map_voice_activation_error(error); + self.pause_linked_voice_activation_after_failure(generation); self.events.publish( None, BackendEventKind::VoiceActivationStateChanged( @@ -3760,22 +3934,22 @@ impl OpenLessBackend { { Ok(handoff) => handoff, Err(error) => { - log::warn!("voice activation handoff failed: {error}"); - match backend - .stop_voice_activation_if_current( - signal_generation, - crate::voice_activation::VoiceActivationStopReason::RuntimeFailure, - ) - .await - { - Ok(true) if backend.get_preferences().voice_activation_enabled => { - let _ = backend.configure_voice_activation().await; - } - Ok(_) => {} - Err(error) => { - log::warn!("voice activation recovery stop failed: {error}"); - } + let paused_handoff = { + let mut link = backend.voice_activation_link.lock() + .expect("voice link lock poisoned"); + if link.finalize_generation == Some(signal_generation) { + link.finalize_generation = None; + true + } else { false } + }; + if paused_handoff { + drop(_handoff_guard); + let _ = backend.configure_voice_activation().await; + return; } + log::warn!("voice activation handoff failed: {error}"); + backend.recover_voice_activation_failure(signal_generation, + map_voice_activation_error(error)).await; return; } }; @@ -3792,12 +3966,14 @@ impl OpenLessBackend { .await { Ok(session_id) => { - if backend + let must_finalize = backend.voice_activation_link.lock() + .expect("voice link lock poisoned").finalize_session == Some(session_id); + if must_finalize || backend .voice_activation_generation .load(Ordering::Acquire) != signal_generation { - let _ = backend.cancel_dictation(Some(session_id)).await; + backend.settle_stale_voice_dictation(session_id).await; return; } let committed = { @@ -3821,7 +3997,7 @@ impl OpenLessBackend { } }; if !committed { - let _ = backend.cancel_dictation(Some(session_id)).await; + backend.settle_stale_voice_dictation(session_id).await; return; } backend.events.publish( @@ -3838,21 +4014,7 @@ impl OpenLessBackend { } Err(error) => { log::warn!("voice dictation start failed: {error}"); - match backend - .stop_voice_activation_if_current( - signal_generation, - crate::voice_activation::VoiceActivationStopReason::RuntimeFailure, - ) - .await - { - Ok(true) if backend.get_preferences().voice_activation_enabled => { - let _ = backend.configure_voice_activation().await; - } - Ok(_) => {} - Err(error) => { - log::warn!("voice activation recovery stop failed: {error}"); - } - } + backend.recover_voice_activation_failure(signal_generation, error).await; } } })); @@ -3878,21 +4040,9 @@ impl OpenLessBackend { } => { let backend = Arc::clone(self); self.deps.task_spawner.spawn(Box::pin(async move { - match backend - .stop_voice_activation_if_current( - activation_generation, - crate::voice_activation::VoiceActivationStopReason::RuntimeFailure, - ) - .await - { - Ok(true) if backend.get_preferences().voice_activation_enabled => { - let _ = backend.configure_voice_activation().await; - } - Ok(_) => {} - Err(error) => { - log::warn!("voice activation recovery stop failed: {error}"); - } - } + backend.recover_voice_activation_failure(activation_generation, + BackendError::new(BackendErrorCode::Platform, + "voice activation microphone capture failed")).await; })); } VoiceActivationSignal::RuntimeStateChanged { .. } => {} @@ -3980,6 +4130,12 @@ impl OpenLessBackend { }, }); } + { + let mut link = self.voice_activation_link.lock().expect("voice link lock poisoned"); + if link.finalize_generation == Some(generation) { + link.finalize_session = Some(session_id); + } + } match self .start_reserved_dictation(reservation, DictationStartOptions::default()) .await @@ -4010,13 +4166,18 @@ impl OpenLessBackend { if !is_voice_session { return; } - let enabled = self.get_preferences().voice_activation_enabled; + { + let mut link = self.voice_activation_link.lock().expect("voice link lock poisoned"); + if link.finalize_session == Some(session_id) { + link.finalize_session = None; + link.finalize_generation = None; + } + } let _ = self .stop_voice_activation(crate::voice_activation::VoiceActivationStopReason::Cancelled) .await; - if enabled { - let _ = self.configure_voice_activation().await; - } + // Paused is a durable enabled-but-idle state, not Disabled. + let _ = self.configure_voice_activation().await; } async fn finish_voice_activation( @@ -4050,7 +4211,7 @@ impl OpenLessBackend { .is_some_and(|session| session.session_id == session_id); if still_voice_session { let _ = self.stop_voice_activation(reason).await; - if self.get_preferences().voice_activation_enabled { + if self.voice_activation_is_requested() { let _ = self.configure_voice_activation().await; } } @@ -5431,6 +5592,13 @@ impl OpenLessBackend { ) -> Result { use crate::hotkey_interpreter::HotkeyIntent; + if let Some(changed) = self.accept_voice_activation_hotkey_edge(edge) { + if changed { + self.reconcile_voice_activation_hotkey().await?; + } + return Ok(CliDispatchOutcome::Noop); + } + // Pressed and Released stay FIFO for toggle-like modes even though start/finalize await // native work. Hold-mode release deliberately bypasses the gate so a physical key-up can // cancel native startup instead of waiting for the recorder to become live. Combined also diff --git a/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/registry.rs b/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/registry.rs index 44e900442..8b2aeb779 100644 --- a/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/registry.rs +++ b/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/registry.rs @@ -207,6 +207,13 @@ pub const PREFERENCE_FIELDS: &[PreferenceField] = &[ shape: PreferenceShape::Boolean, reason: "local_only_microphone_capture", }, + PreferenceField { + rust_name: "voice_activation_hotkey_linked", + key: "voiceActivationHotkeyLinked", + class: PreferenceClass::Excluded, + shape: PreferenceShape::Boolean, + reason: "local_only_microphone_capture", + }, PreferenceField { rust_name: "voice_activation_wake_enabled", key: "voiceActivationWakeEnabled", diff --git a/openless-all/app/crates/openless-core/src/lib.rs b/openless-all/app/crates/openless-core/src/lib.rs index 1f363893f..2051099ec 100644 --- a/openless-all/app/crates/openless-core/src/lib.rs +++ b/openless-all/app/crates/openless-core/src/lib.rs @@ -80,6 +80,7 @@ pub mod testing; pub mod types; pub mod vocabulary; pub mod voice_activation; +mod voice_activation_hotkey; pub mod voice_edit_session; mod voice_session; diff --git a/openless-all/app/crates/openless-core/src/ports.rs b/openless-all/app/crates/openless-core/src/ports.rs index 66dfa69f4..089ce44ac 100644 --- a/openless-all/app/crates/openless-core/src/ports.rs +++ b/openless-all/app/crates/openless-core/src/ports.rs @@ -118,6 +118,9 @@ pub struct VoiceDictationHandoff { /// decides lifecycle and stale-event policy; the host decides how local ASR and /// the native capture stream are implemented. pub trait VoiceActivationRuntime: Send + Sync + 'static { + /// Apply runtime-only standby intent even before a recognizer exists. + fn set_capture_paused(&self, _revision: u64, _paused: bool) {} + fn configure( &self, config: VoiceActivationConfig, diff --git a/openless-all/app/crates/openless-core/src/shared_types.rs b/openless-all/app/crates/openless-core/src/shared_types.rs index ea06f2ee9..f4c82c4c0 100644 --- a/openless-all/app/crates/openless-core/src/shared_types.rs +++ b/openless-all/app/crates/openless-core/src/shared_types.rs @@ -462,6 +462,8 @@ pub struct UserPreferences { /// Opt-in local voice command listener. Standby audio is never sent to a cloud provider. #[serde(default)] pub voice_activation_enabled: bool, + #[serde(default)] + pub voice_activation_hotkey_linked: bool, #[serde(default = "default_true")] pub voice_activation_wake_enabled: bool, #[serde(default = "default_true")] @@ -1045,6 +1047,8 @@ struct UserPreferencesWire { microphone_device_name: String, #[serde(default)] voice_activation_enabled: bool, + #[serde(default)] + voice_activation_hotkey_linked: bool, #[serde(default = "default_true")] voice_activation_wake_enabled: bool, #[serde(default = "default_true")] @@ -1323,6 +1327,7 @@ impl Default for UserPreferencesWire { silence_auto_stop_seconds: prefs.silence_auto_stop_seconds, microphone_device_name: prefs.microphone_device_name, voice_activation_enabled: prefs.voice_activation_enabled, + voice_activation_hotkey_linked: prefs.voice_activation_hotkey_linked, voice_activation_wake_enabled: prefs.voice_activation_wake_enabled, voice_activation_finish_enabled: prefs.voice_activation_finish_enabled, voice_activation_wake_phrase: prefs.voice_activation_wake_phrase, @@ -1544,6 +1549,7 @@ impl<'de> Deserialize<'de> for UserPreferences { silence_auto_stop_seconds: wire.silence_auto_stop_seconds, microphone_device_name: wire.microphone_device_name, voice_activation_enabled: wire.voice_activation_enabled, + voice_activation_hotkey_linked: wire.voice_activation_hotkey_linked, voice_activation_wake_enabled: wire.voice_activation_wake_enabled, voice_activation_finish_enabled: wire.voice_activation_finish_enabled, voice_activation_wake_phrase: wire.voice_activation_wake_phrase, @@ -1943,6 +1949,7 @@ impl Default for UserPreferences { silence_auto_stop_seconds: default_silence_auto_stop_seconds(), microphone_device_name: String::new(), voice_activation_enabled: false, + voice_activation_hotkey_linked: false, voice_activation_wake_enabled: true, voice_activation_finish_enabled: true, voice_activation_wake_phrase: default_voice_activation_wake_phrase(), diff --git a/openless-all/app/crates/openless-core/src/voice_activation.rs b/openless-all/app/crates/openless-core/src/voice_activation.rs index 1faf28ded..aa5786f98 100644 --- a/openless-all/app/crates/openless-core/src/voice_activation.rs +++ b/openless-all/app/crates/openless-core/src/voice_activation.rs @@ -344,6 +344,8 @@ pub enum VoiceActivationSignal { #[serde(rename_all = "snake_case")] pub enum VoiceActivationPhase { Disabled, + Paused, + Pausing, Starting, ListeningForWake, WakeDetected, diff --git a/openless-all/app/crates/openless-core/src/voice_activation_hotkey.rs b/openless-all/app/crates/openless-core/src/voice_activation_hotkey.rs new file mode 100644 index 000000000..c7c1688b2 --- /dev/null +++ b/openless-all/app/crates/openless-core/src/voice_activation_hotkey.rs @@ -0,0 +1,123 @@ +//! Runtime-only intent and physical-click arbitration for linked voice activation. +use std::collections::{HashSet, VecDeque}; +use std::time::{Duration, Instant}; + +use crate::api::DictationHotkeyEdge; + +#[derive(Default)] +pub(crate) struct VoiceActivationLink { + linked: bool, + pub(crate) requested: bool, + pub(crate) revision: u64, + pub(crate) finalize_session: Option, + pub(crate) finalize_generation: Option, + pressed: Option<(u64, Instant)>, + consumed: VecDeque, + combined: HashSet, + last_click: Option, +} + +impl VoiceActivationLink { + pub(crate) fn configure(&mut self, linked: bool) { + if self.linked != linked { + self.linked = linked; + self.requested = false; + self.revision += 1; + self.pressed = None; + self.consumed.clear(); + self.combined.clear(); + self.last_click = None; + } + } + + pub(crate) fn pause(&mut self) { + self.requested = false; + self.revision += 1; + } + + pub(crate) fn edge(&mut self, edge: DictationHotkeyEdge) -> bool { + match edge { + DictationHotkeyEdge::Pressed { press_id, at } => { + if !self.consumed.contains(&press_id) { + // A fresh physical ID supersedes a release lost while the + // native shortcut recorder or registration was suspended. + if self.pressed.map(|press| press.0) != Some(press_id) { + self.pressed = Some((press_id, at)); + } + } + } + DictationHotkeyEdge::Combined { press_id, .. } => { + if self.combined.len() >= 64 { + self.combined.clear(); + } + self.combined.insert(press_id); + } + DictationHotkeyEdge::Released { press_id, at } => { + if self.pressed.map(|press| press.0) != Some(press_id) { + return false; + } + let (_, pressed_at) = self.pressed.take().expect("matched physical press"); + self.consumed.push_back(press_id); + if self.consumed.len() > 64 { + self.consumed.pop_front(); + } + if self.combined.remove(&press_id) || at < pressed_at { + return false; + } + if self.last_click.is_some_and(|last| { + at.saturating_duration_since(last) < Duration::from_millis(250) + }) { + return false; + } + self.last_click = Some(at); + self.requested = !self.requested; + self.revision += 1; + return true; + } + } + false + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_fresh_press_recovers_after_a_suppressed_release() { + let mut state = VoiceActivationLink::default(); + state.configure(true); + let at = Instant::now(); + state.edge(DictationHotkeyEdge::Pressed { press_id: 1, at }); + state.edge(DictationHotkeyEdge::Pressed { press_id: 2, at }); + assert!(state.edge(DictationHotkeyEdge::Released { press_id: 2, at })); + assert!(state.requested); + assert!(!state.edge(DictationHotkeyEdge::Released { press_id: 1, at })); + } + + #[test] + fn linked_clicks_are_paired_debounced_and_combination_safe() { + let mut state = VoiceActivationLink::default(); + state.configure(true); + let now = Instant::now(); + assert!(!state.requested); + assert!(!state.edge(DictationHotkeyEdge::Released { press_id: 1, at: now })); + assert!(!state.edge(DictationHotkeyEdge::Pressed { press_id: 1, at: now })); + assert!(!state.edge(DictationHotkeyEdge::Pressed { press_id: 1, at: now })); + assert!(state.edge(DictationHotkeyEdge::Released { press_id: 1, at: now })); + assert!(state.requested); + assert!(!state.edge(DictationHotkeyEdge::Released { press_id: 1, at: now })); + state.edge(DictationHotkeyEdge::Pressed { press_id: 2, at: now }); + assert!(!state.edge(DictationHotkeyEdge::Released { press_id: 2, at: now })); + let later = now + Duration::from_secs(1); + state.edge(DictationHotkeyEdge::Combined { press_id: 3, at: later }); + state.edge(DictationHotkeyEdge::Pressed { press_id: 3, at: later }); + assert!(!state.edge(DictationHotkeyEdge::Released { press_id: 3, at: later })); + state.edge(DictationHotkeyEdge::Pressed { press_id: 4, at: later }); + assert!(state.edge(DictationHotkeyEdge::Released { press_id: 4, at: later })); + assert!(!state.requested); + state.configure(false); + state.configure(true); + assert!(!state.requested); + } +} diff --git a/openless-all/app/crates/openless-core/tests/voice_activation_linked.rs b/openless-all/app/crates/openless-core/tests/voice_activation_linked.rs new file mode 100644 index 000000000..8886ba66e --- /dev/null +++ b/openless-all/app/crates/openless-core/tests/voice_activation_linked.rs @@ -0,0 +1,384 @@ +use std::future::Future; +use std::pin::Pin; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +use openless_core::testing::{FixtureDictationEngine, FixtureTextInserter, RecordingHostActions}; +use openless_core::*; +use tokio::sync::Semaphore; + +type Task = Pin + Send + 'static>>; + +#[derive(Default)] +struct Runtime { + configured: AtomicUsize, + stopped: Arc, + paused: Mutex<(u64, bool)>, + fail: AtomicBool, + delay: Mutex>>, + engine_start_delay: Mutex>>, + engine_finish_delay: Mutex>>, + handoff_delay: Arc>>>, + handoff_started: Arc, +} +impl VoiceActivationRuntime for Runtime { + fn set_capture_paused(&self, revision: u64, paused: bool) { + let mut policy = self.paused.lock().unwrap(); + if revision >= policy.0 { + *policy = (revision, paused); + } + } + fn configure( + &self, + _: VoiceActivationConfig, + _: Arc, + ) -> Task, VoiceActivationError>> { + self.configured.fetch_add(1, Ordering::SeqCst); + let stopped = Arc::clone(&self.stopped); + let fail = self.fail.load(Ordering::SeqCst); + let delay = self.delay.lock().unwrap().take(); + let handoff_delay = self.handoff_delay.clone(); + let handoff_started = self.handoff_started.clone(); + Box::pin(async move { + if let Some(delay) = delay { + delay.acquire_owned().await.unwrap().forget(); + } + if fail { + Err(VoiceActivationError::ModelMissing) + } else { + Ok(Box::new(Handle(stopped, handoff_delay, handoff_started)) as Box) + } + }) + } +} +struct Handle(Arc, Arc>>>, Arc); +impl VoiceActivationHandle for Handle { + fn handoff_to_dictation( + &self, + _: VoiceDictationHandoffRequest, + ) -> Task> { + let delay = self.1.lock().unwrap().take(); + self.2.store(true, Ordering::SeqCst); + Box::pin(async move { + if let Some(delay) = delay { delay.acquire_owned().await.unwrap().forget(); } + Ok(VoiceDictationHandoff { capture: Box::new(Capture) }) + }) + } + fn stop(&self, _: VoiceActivationStopReason) -> Task> { + self.0.fetch_add(1, Ordering::SeqCst); + Box::pin(async { Ok(()) }) + } +} +struct Capture; +impl PendingAudioCapture for Capture { + fn commit(self: Box, _: SessionId, _: Arc, + _: Arc, _: Arc) + -> Task, BackendError>> + { + Box::pin(async { Err(BackendError::new(BackendErrorCode::Unsupported, "fixture capture")) }) + } + fn discard(self: Box) -> Task> { + Box::pin(async { Ok(()) }) + } +} +struct Engine(FixtureDictationEngine, Arc); +impl DictationEngine for Engine { + fn start_with_audio_capture(&self, id: SessionId, context: Arc, + progress: Arc, capture: Box) + -> Task> + { + let start = self.0.start(id, context, progress); + let delay = self.1.engine_start_delay.lock().unwrap().take(); + Box::pin(async move { + if let Some(delay) = delay { delay.acquire_owned().await.unwrap().forget(); } + capture.discard().await?; + start.await + }) + } + fn start(&self, id: SessionId, context: Arc, progress: Arc) + -> Task> { self.0.start(id, context, progress) } + fn finish(&self, id: SessionId, progress: Arc) + -> Task> { + let finish = self.0.finish(id, progress); + let delay = self.1.engine_finish_delay.lock().unwrap().take(); + Box::pin(async move { + if let Some(delay) = delay { delay.acquire_owned().await.unwrap().forget(); } + finish.await + }) + } + fn update_context(&self, id: SessionId, context: Arc) + -> Task> { self.0.update_context(id, context) } + fn cancel(&self, id: SessionId) -> Task> { self.0.cancel(id) } +} +struct Directory(std::path::PathBuf); +impl Drop for Directory { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} +fn fixture() -> (Directory, Arc, Arc, FixtureDictationEngine) { + let dir = Directory(std::env::temp_dir().join(format!( + "openless-voice-link-{}-{}", + std::process::id(), + SessionId::new() + ))); + let runtime = Arc::new(Runtime::default()); + let engine = FixtureDictationEngine::successful("body", "body"); + let mut deps = BackendDependencies::unsupported(); + deps.voice_activation_runtime = Some(runtime.clone()); + deps.host_actions = Arc::new(RecordingHostActions::default()); + deps.text_inserter = Arc::new(FixtureTextInserter::with_outcome(InsertOutcome::Inserted)); + deps.dictation_engine = Arc::new(Engine(engine.clone(), runtime.clone())); + deps.task_spawner = Arc::new(TokioTaskSpawner); + let backend = Arc::new(OpenLessBackend::new(BackendConfig { + data_dir: dir.0.clone(), ..Default::default() + }, deps).unwrap()); + (dir, backend, runtime, engine) +} +fn edges(id: u64) -> [DictationHotkeyEdge; 2] { + let at = Instant::now() + Duration::from_secs(id); + [ + DictationHotkeyEdge::Pressed { press_id: id, at }, + DictationHotkeyEdge::Released { press_id: id, at: at + Duration::from_millis(10) }, + ] +} +async fn click(backend: &OpenLessBackend, id: u64) { + for edge in edges(id) { + backend.dispatch_dictation_hotkey_edge(edge).await.unwrap(); + } +} +fn enable(backend: &OpenLessBackend, mode: HotkeyMode) { + let mut prefs = backend.get_preferences(); + prefs.voice_activation_enabled = true; + prefs.voice_activation_hotkey_linked = true; + prefs.hotkey.mode = mode; + backend.repositories().preferences.set(prefs).unwrap(); +} + +#[test] +fn linked_preference_is_local_only_and_old_configs_default_off() { + use openless_core::cloud_sync_e2ee_documents::registry::{PREFERENCE_FIELDS, PreferenceClass}; + let old: UserPreferences = serde_json::from_str("{}").unwrap(); + assert!(!old.voice_activation_hotkey_linked); + let prefs = UserPreferences { voice_activation_hotkey_linked: true, ..old }; + let json = serde_json::to_value(&prefs).unwrap(); + assert_eq!(json["voiceActivationHotkeyLinked"], true); + let loaded: UserPreferences = serde_json::from_value(json).unwrap(); + assert!(loaded.voice_activation_hotkey_linked); + let field = PREFERENCE_FIELDS.iter().find(|f| f.key == "voiceActivationHotkeyLinked").unwrap(); + assert_eq!(field.class, PreferenceClass::Excluded); +} + +#[tokio::test] +async fn all_recording_modes_use_one_click_and_never_start_dictation() { + for mode in [HotkeyMode::Toggle, HotkeyMode::Hold, HotkeyMode::Auto, HotkeyMode::DoubleClick] { + let (_dir, backend, runtime, engine) = fixture(); + enable(&backend, mode); + backend.start().await.unwrap(); + assert_eq!(backend.latest_voice_activation_state().unwrap().phase, VoiceActivationPhase::Paused); + assert_eq!(runtime.configured.load(Ordering::SeqCst), 0); + click(&backend, 1).await; + assert_eq!(backend.latest_voice_activation_state().unwrap().phase, VoiceActivationPhase::ListeningForWake); + assert!(!runtime.paused.lock().unwrap().1); + click(&backend, 2).await; + assert_eq!(backend.latest_voice_activation_state().unwrap().phase, VoiceActivationPhase::Paused); + assert!(runtime.paused.lock().unwrap().1); + let mut prefs = backend.get_preferences(); + prefs.low_latency_dictation_enabled = true; + backend.repositories().preferences.set(prefs).unwrap(); + backend.configure_voice_activation().await.unwrap(); + assert_eq!(runtime.configured.load(Ordering::SeqCst), 1); + assert!(engine.actions().is_empty()); + assert!(backend.get_preferences().voice_activation_enabled); + assert!(backend.get_preferences().voice_activation_hotkey_linked); + backend.shutdown().await.unwrap(); + } +} + +#[tokio::test] +async fn pause_invalidates_inflight_start_without_waiting_for_the_model() { + let (_dir, backend, runtime, _) = fixture(); + enable(&backend, HotkeyMode::Hold); + backend.start().await.unwrap(); + let delay = Arc::new(Semaphore::new(0)); + *runtime.delay.lock().unwrap() = Some(delay.clone()); + for edge in edges(1) { backend.accept_voice_activation_hotkey_edge(edge); } + let start = { + let backend = backend.clone(); + tokio::spawn(async move { backend.reconcile_voice_activation_hotkey().await }) + }; + tokio::time::timeout(Duration::from_secs(5), async { + while runtime.configured.load(Ordering::SeqCst) == 0 { tokio::task::yield_now().await; } + }).await.unwrap(); + for edge in edges(2) { backend.accept_voice_activation_hotkey_edge(edge); } + assert!(runtime.paused.lock().unwrap().1); + delay.add_permits(1); + let _ = tokio::time::timeout(Duration::from_secs(5), start).await.unwrap().unwrap(); + backend.reconcile_voice_activation_hotkey().await.unwrap(); + assert_eq!(backend.latest_voice_activation_state().unwrap().phase, VoiceActivationPhase::Paused); + assert_eq!(runtime.stopped.load(Ordering::SeqCst), 1); + backend.shutdown().await.unwrap(); +} + +#[tokio::test] +async fn failed_start_stays_paused_and_the_next_click_retries() { + let (_dir, backend, runtime, engine) = fixture(); + enable(&backend, HotkeyMode::Auto); + backend.start().await.unwrap(); + runtime.fail.store(true, Ordering::SeqCst); + backend.accept_voice_activation_hotkey_edge(edges(1)[0]); + backend.accept_voice_activation_hotkey_edge(edges(1)[1]); + assert!(backend.reconcile_voice_activation_hotkey().await.is_err()); + assert_eq!(backend.latest_voice_activation_state().unwrap().phase, VoiceActivationPhase::Failed); + assert!(runtime.paused.lock().unwrap().1); + runtime.fail.store(false, Ordering::SeqCst); + click(&backend, 2).await; + assert_eq!(runtime.configured.load(Ordering::SeqCst), 2); + assert!(!runtime.paused.lock().unwrap().1); + assert!(engine.actions().is_empty()); + backend.shutdown().await.unwrap(); +} + +#[tokio::test] +async fn pausing_listener_does_not_stop_an_unrelated_explicit_recording() { + let (_dir, backend, _, engine) = fixture(); + enable(&backend, HotkeyMode::Toggle); + backend.start().await.unwrap(); + let id = backend.start_dictation().await.unwrap(); + click(&backend, 1).await; + click(&backend, 2).await; + assert_eq!(backend.snapshot().dictation.session_id, Some(id)); + assert_eq!(backend.snapshot().dictation.phase, DictationPhase::Recording); + assert_eq!(engine.actions().len(), 1); + backend.stop_dictation().await.unwrap(); + backend.shutdown().await.unwrap(); +} + +#[tokio::test] +async fn pause_finalizes_voice_started_body_without_cancelling_or_rearming() { + let (_dir, backend, runtime, engine) = fixture(); + enable(&backend, HotkeyMode::Toggle); + backend.start().await.unwrap(); + click(&backend, 1).await; + let generation = backend.latest_voice_activation_state().unwrap().activation_generation; + backend.handle_voice_activation_signal(VoiceActivationSignal::WakeDetected { + activation_generation: generation, + }); + tokio::time::timeout(Duration::from_secs(5), async { + while backend.snapshot().dictation.phase != DictationPhase::Recording { + tokio::task::yield_now().await; + } + }).await.unwrap(); + click(&backend, 2).await; + assert_eq!(backend.latest_voice_activation_state().unwrap().phase, VoiceActivationPhase::Paused); + assert_eq!(backend.list_history().unwrap()[0].final_text, "body"); + assert!(runtime.paused.lock().unwrap().1); + assert_eq!(runtime.configured.load(Ordering::SeqCst), 1); + assert!(!engine.actions().iter().any(|action| + matches!(action, openless_core::testing::FixtureEngineAction::Cancel(_)))); + backend.shutdown().await.unwrap(); +} + +async fn wait_for_phase(backend: &OpenLessBackend, phase: DictationPhase) { + tokio::time::timeout(Duration::from_secs(5), async { + while backend.snapshot().dictation.phase != phase { tokio::task::yield_now().await; } + }).await.unwrap(); +} + +#[tokio::test] +async fn pause_then_resume_during_voice_start_preserves_and_finishes_the_body() { + let (_dir, backend, runtime, engine) = fixture(); + enable(&backend, HotkeyMode::Hold); + backend.start().await.unwrap(); + click(&backend, 1).await; + let delay = Arc::new(Semaphore::new(0)); + *runtime.engine_start_delay.lock().unwrap() = Some(delay.clone()); + let generation = backend.latest_voice_activation_state().unwrap().activation_generation; + backend.handle_voice_activation_signal(VoiceActivationSignal::WakeDetected { + activation_generation: generation, + }); + tokio::time::timeout(Duration::from_secs(5), async { + while engine.actions().is_empty() { tokio::task::yield_now().await; } + }).await.unwrap(); + assert_eq!(backend.snapshot().dictation.phase, DictationPhase::Starting); + for edge in edges(2) { backend.accept_voice_activation_hotkey_edge(edge); } + assert!(runtime.paused.lock().unwrap().1); + for edge in edges(3) { backend.accept_voice_activation_hotkey_edge(edge); } + assert!(!runtime.paused.lock().unwrap().1); + let reconciliation = { + let backend = backend.clone(); + tokio::spawn(async move { backend.reconcile_voice_activation_hotkey().await }) + }; + delay.add_permits(1); + tokio::time::timeout(Duration::from_secs(5), reconciliation).await.unwrap().unwrap().unwrap(); + tokio::time::timeout(Duration::from_secs(5), async { + while backend.list_history().unwrap().is_empty() + || backend.latest_voice_activation_state().unwrap().phase != VoiceActivationPhase::ListeningForWake + { tokio::task::yield_now().await; } + }).await.unwrap(); + assert_eq!(backend.list_history().unwrap()[0].final_text, "body"); + assert!(!engine.actions().iter().any(|action| + matches!(action, openless_core::testing::FixtureEngineAction::Cancel(_)))); + backend.shutdown().await.unwrap(); +} + +#[tokio::test] +async fn pausing_an_already_finalizing_voice_body_restores_durable_paused_status() { + let (_dir, backend, runtime, _) = fixture(); + enable(&backend, HotkeyMode::Toggle); + backend.start().await.unwrap(); + click(&backend, 1).await; + let generation = backend.latest_voice_activation_state().unwrap().activation_generation; + backend.handle_voice_activation_signal(VoiceActivationSignal::WakeDetected { + activation_generation: generation, + }); + wait_for_phase(&backend, DictationPhase::Recording).await; + let delay = Arc::new(Semaphore::new(0)); + *runtime.engine_finish_delay.lock().unwrap() = Some(delay.clone()); + let finish = { + let backend = backend.clone(); + tokio::spawn(async move { backend.stop_dictation().await }) + }; + wait_for_phase(&backend, DictationPhase::Transcribing).await; + click(&backend, 2).await; + delay.add_permits(1); + tokio::time::timeout(Duration::from_secs(5), finish).await.unwrap().unwrap().unwrap(); + assert_eq!(backend.latest_voice_activation_state().unwrap().phase, VoiceActivationPhase::Paused); + assert_eq!(backend.list_history().unwrap()[0].final_text, "body"); + assert!(runtime.paused.lock().unwrap().1); + backend.shutdown().await.unwrap(); +} + +#[tokio::test] +async fn pause_preserves_a_staged_body_before_handoff_has_a_formal_session() { + let (_dir, backend, runtime, engine) = fixture(); + enable(&backend, HotkeyMode::Toggle); + backend.start().await.unwrap(); + click(&backend, 1).await; + let delay = Arc::new(Semaphore::new(0)); + *runtime.handoff_delay.lock().unwrap() = Some(delay.clone()); + let generation = backend.latest_voice_activation_state().unwrap().activation_generation; + backend.handle_voice_activation_signal(VoiceActivationSignal::WakeDetected { + activation_generation: generation, + }); + tokio::time::timeout(Duration::from_secs(5), async { + while !runtime.handoff_started.load(Ordering::SeqCst) { tokio::task::yield_now().await; } + }).await.unwrap(); + assert!(backend.snapshot().dictation.session_id.is_none()); + click(&backend, 2).await; + assert!(runtime.paused.lock().unwrap().1); + assert!(backend.voice_activation_generation_is_current(generation)); + assert_eq!(runtime.stopped.load(Ordering::SeqCst), 0); + delay.add_permits(1); + tokio::time::timeout(Duration::from_secs(5), async { + while backend.list_history().unwrap().is_empty() + || backend.latest_voice_activation_state().unwrap().phase != VoiceActivationPhase::Paused + { tokio::task::yield_now().await; } + }).await.unwrap(); + assert_eq!(backend.list_history().unwrap()[0].final_text, "body"); + assert!(!engine.actions().iter().any(|action| + matches!(action, openless_core::testing::FixtureEngineAction::Cancel(_)))); + assert_eq!(runtime.configured.load(Ordering::SeqCst), 1); + backend.shutdown().await.unwrap(); +} diff --git a/openless-all/app/e2e/settings-regressions.spec.ts b/openless-all/app/e2e/settings-regressions.spec.ts index d0bf5de97..323323c5b 100644 --- a/openless-all/app/e2e/settings-regressions.spec.ts +++ b/openless-all/app/e2e/settings-regressions.spec.ts @@ -288,3 +288,31 @@ test('voice activation opens model setup without leaving the multimodal pipeline await categories.getByRole('button', { name: 'AI 服务与模型', exact: true }).click(); await expect(models).toBeDisabled(); }); + +test('Windows wake shortcut linkage is opt-in, keyboard accessible and preserves recording mode', async ({ page }) => { + await page.addInitScript(() => { + Object.defineProperty(navigator, 'platform', { get: () => 'Win32' }); + Object.defineProperty(navigator, 'userAgent', { get: () => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)' }); + }); + await page.goto('http://127.0.0.1:5326/?os=win'); + await page.getByRole('button', { name: '设置', exact: true }).click(); + const dialog = page.getByRole('dialog', { name: '设置', exact: true }); + const categories = dialog.getByRole('navigation', { name: '设置分类' }); + await categories.getByRole('button', { name: '录音与输入', exact: true }).click(); + const linkage = dialog.getByRole('switch', { name: '与录音快捷键联动', exact: true }); + await expect(linkage).toHaveCount(0); + await dialog.getByRole('switch', { name: '语音唤醒听写', exact: true }).click(); + await expect(linkage).not.toBeChecked(); + const mode = await page.evaluate(async () => (await import('/src/lib/ipc/mock-data.ts')).mockSettings.hotkey.mode); + await linkage.focus(); + await page.keyboard.press('Space'); + await expect(linkage).toBeChecked(); + await expect(dialog.getByText('已临时暂停唤醒监听', { exact: true })).toBeVisible(); + expect(await page.evaluate(async () => (await import('/src/lib/ipc/mock-data.ts')).mockSettings.hotkey.mode)).toBe(mode); + await categories.getByRole('button', { name: 'AI 服务与模型', exact: true }).click(); + await categories.getByRole('button', { name: '录音与输入', exact: true }).click(); + await expect(linkage).toBeChecked(); + await linkage.click(); + await expect(linkage).not.toBeChecked(); + expect(await page.evaluate(async () => (await import('/src/lib/ipc/mock-data.ts')).mockSettings.hotkey.mode)).toBe(mode); +}); diff --git a/openless-all/app/harmony/ipc-audit.json b/openless-all/app/harmony/ipc-audit.json index d051d9e62..98805d575 100644 --- a/openless-all/app/harmony/ipc-audit.json +++ b/openless-all/app/harmony/ipc-audit.json @@ -1,8 +1,8 @@ { "scope": "Static shared IPC command coverage; excludes event subscriptions, dynamic commands and desktop-only calls outside src/lib/ipc.", - "total": 198, + "total": 199, "wired": 103, - "missing": 95, + "missing": 96, "commands": [ { "name": "accept_pending_correction", @@ -557,6 +557,13 @@ ], "implementation": "unimplemented" }, + { + "name": "get_voice_activation_state", + "calledFrom": [ + "src/lib/ipc/settings.ts" + ], + "implementation": "unimplemented" + }, { "name": "get_voice_edit_state", "calledFrom": [ diff --git a/openless-all/app/src-tauri/backend-tests/tests/capsule_feedback.rs b/openless-all/app/src-tauri/backend-tests/tests/capsule_feedback.rs new file mode 100644 index 000000000..000d6c17b --- /dev/null +++ b/openless-all/app/src-tauri/backend-tests/tests/capsule_feedback.rs @@ -0,0 +1,2 @@ +#[path = "../../src/capsule_feedback.rs"] +mod capsule_feedback; diff --git a/openless-all/app/src-tauri/backend-tests/tests/voice_activation_linked.rs b/openless-all/app/src-tauri/backend-tests/tests/voice_activation_linked.rs new file mode 100644 index 000000000..36a1be8dc --- /dev/null +++ b/openless-all/app/src-tauri/backend-tests/tests/voice_activation_linked.rs @@ -0,0 +1,3 @@ +// Execute the same framework-independent lifecycle regressions on Windows runners. +#[path = "../../../crates/openless-core/tests/voice_activation_linked.rs"] +mod linked; diff --git a/openless-all/app/src-tauri/src/capsule_feedback.rs b/openless-all/app/src-tauri/src/capsule_feedback.rs new file mode 100644 index 000000000..60ebbcd28 --- /dev/null +++ b/openless-all/app/src-tauri/src/capsule_feedback.rs @@ -0,0 +1,285 @@ +//! Noninteractive feedback on the existing shared capsule surface. +use openless_core::{ + CapsulePayload, CapsuleState, CapsuleStyle, DictationPhase, VoiceActivationPhase, + VoiceActivationStateSnapshot, +}; + +pub(crate) const CAPSULE_AUTO_HIDE_DELAY_MS: u64 = 2000; + +pub(crate) fn core_capsule_hide_delay(state: CapsuleState) -> Option { + match state { + CapsuleState::Done | CapsuleState::Error => Some(CAPSULE_AUTO_HIDE_DELAY_MS), + CapsuleState::Cancelled => Some(0), + _ => None, + } +} + +#[derive(Default)] +pub(crate) struct VoiceActivationFeedback { + pending: Option<(bool, u64)>, +} + +impl VoiceActivationFeedback { + pub(crate) fn request(&mut self, intent: Option<(bool, u64)>) { + self.pending = intent; + } + + pub(crate) fn complete( + &mut self, + state: &VoiceActivationStateSnapshot, + latest: Option<&VoiceActivationStateSnapshot>, + intent: Option<(bool, u64)>, + ) -> Option { + let (enabled, _) = self.pending?; + if self.pending != intent { + self.pending = None; + return None; + } + let latest = latest?; + use VoiceActivationPhase::*; + if matches!( + latest.phase, + Unavailable | PermissionDenied | MicBusy | Failed + ) || (enabled + && matches!( + latest.phase, + WakeDetected | HandingOff | ListeningForFinish | Finishing + )) + { + self.pending = None; + return None; + } + if state != latest { + return None; + } + if state.phase == if enabled { ListeningForWake } else { Paused } { + self.pending = None; + return Some(enabled); + } + None + } +} + +pub(crate) fn can_present_feedback( + capsule: Option, + dictation: DictationPhase, + other_owner: bool, +) -> bool { + !other_owner + && matches!( + dictation, + DictationPhase::Idle + | DictationPhase::Completed + | DictationPhase::Cancelled + | DictationPhase::Failed + ) + && !matches!( + capsule, + Some(CapsuleState::Recording | CapsuleState::Transcribing | CapsuleState::Polishing) + ) +} + +pub(crate) fn voice_activation_payload(enabled: bool, style: CapsuleStyle) -> CapsulePayload { + CapsulePayload { + session_id: None, + state: CapsuleState::Done, + level: 0.0, + elapsed_ms: 0, + message: Some( + if enabled { + "唤醒监听已开启" + } else { + "唤醒监听已暂停" + } + .into(), + ), + inserted_chars: None, + translation: false, + operating: false, + warming: false, + capsule_style: style, + selection_polish: false, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn state(phase: VoiceActivationPhase, generation: u64) -> VoiceActivationStateSnapshot { + VoiceActivationStateSnapshot { + phase, + activation_generation: generation, + session_id: None, + message: None, + } + } + + #[test] + fn only_a_successful_explicit_click_confirms_once() { + let mut feedback = VoiceActivationFeedback::default(); + let paused = state(VoiceActivationPhase::Paused, 1); + assert_eq!( + feedback.complete(&paused, Some(&paused), Some((false, 1))), + None + ); + for enabled in [true, false] { + let intent = Some((enabled, 2)); + feedback.request(intent); + let pending = state( + if enabled { + VoiceActivationPhase::Starting + } else { + VoiceActivationPhase::Pausing + }, + 2, + ); + assert_eq!(feedback.complete(&pending, Some(&pending), intent), None); + let done = state( + if enabled { + VoiceActivationPhase::ListeningForWake + } else { + VoiceActivationPhase::Paused + }, + 3, + ); + assert_eq!(feedback.complete(&done, Some(&done), intent), Some(enabled)); + assert_eq!(feedback.complete(&done, Some(&done), intent), None); + // An automatic rearm without another click cannot repeat the hint. + assert_eq!(feedback.complete(&pending, Some(&pending), intent), None); + assert_eq!(feedback.complete(&done, Some(&done), intent), None); + } + } + + #[test] + fn stale_states_and_rapid_intent_changes_do_not_confirm_old_requests() { + let mut feedback = VoiceActivationFeedback::default(); + feedback.request(Some((true, 2))); + let listening = state(VoiceActivationPhase::ListeningForWake, 2); + let pausing = state(VoiceActivationPhase::Pausing, 3); + assert_eq!( + feedback.complete(&listening, Some(&pausing), Some((false, 3))), + None + ); + feedback.request(Some((false, 3))); + assert_eq!( + feedback.complete(&listening, Some(&pausing), Some((false, 3))), + None + ); + let paused = state(VoiceActivationPhase::Paused, 4); + assert_eq!( + feedback.complete(&paused, Some(&paused), Some((false, 3))), + Some(false) + ); + feedback.request(Some((true, 5))); + assert_eq!( + feedback.complete(&listening, Some(&paused), Some((true, 5))), + None + ); + feedback.request(None); // Resync is not a new user operation. + assert_eq!( + feedback.complete(&listening, Some(&listening), Some((true, 5))), + None + ); + } + + #[test] + fn failure_or_wake_handoff_discards_a_delayed_enable_hint() { + for phase in [ + VoiceActivationPhase::Unavailable, + VoiceActivationPhase::PermissionDenied, + VoiceActivationPhase::MicBusy, + VoiceActivationPhase::Failed, + VoiceActivationPhase::WakeDetected, + VoiceActivationPhase::HandingOff, + VoiceActivationPhase::ListeningForFinish, + VoiceActivationPhase::Finishing, + ] { + let mut feedback = VoiceActivationFeedback::default(); + let intent = Some((true, 1)); + feedback.request(intent); + let next = state(phase, 2); + assert_eq!(feedback.complete(&next, Some(&next), intent), None); + let rearmed = state(VoiceActivationPhase::ListeningForWake, 3); + assert_eq!(feedback.complete(&rearmed, Some(&rearmed), intent), None); + } + } + + #[test] + fn pause_waits_for_body_finalization_before_confirming() { + let mut feedback = VoiceActivationFeedback::default(); + let intent = Some((false, 7)); + feedback.request(intent); + for phase in [ + VoiceActivationPhase::Finishing, + VoiceActivationPhase::Disabled, + ] { + let pending = state(phase, 8); + assert_eq!(feedback.complete(&pending, Some(&pending), intent), None); + } + let paused = state(VoiceActivationPhase::Paused, 9); + assert_eq!( + feedback.complete(&paused, Some(&paused), intent), + Some(false) + ); + } + + #[test] + fn active_capsule_owners_always_take_priority() { + assert!(can_present_feedback(None, DictationPhase::Idle, false)); + for phase in [ + DictationPhase::Starting, + DictationPhase::Recording, + DictationPhase::Transcribing, + DictationPhase::Polishing, + DictationPhase::Inserting, + ] { + assert!(!can_present_feedback( + Some(CapsuleState::Done), + phase, + false + )); + } + for capsule in [ + CapsuleState::Recording, + CapsuleState::Transcribing, + CapsuleState::Polishing, + ] { + assert!(!can_present_feedback( + Some(capsule), + DictationPhase::Idle, + false + )); + } + assert!(!can_present_feedback( + Some(CapsuleState::Done), + DictationPhase::Idle, + true + )); + } + + #[test] + fn both_capsule_styles_use_done_message_and_the_existing_two_second_hide() { + for style in [CapsuleStyle::Classic, CapsuleStyle::Siri] { + for enabled in [true, false] { + let payload = voice_activation_payload(enabled, style); + assert_eq!(payload.state, CapsuleState::Done); + assert_eq!(core_capsule_hide_delay(payload.state), Some(2000)); + assert_eq!( + payload.message.as_deref(), + Some(if enabled { + "唤醒监听已开启" + } else { + "唤醒监听已暂停" + }) + ); + assert_eq!(payload.capsule_style, style); + assert!(!payload.selection_polish && !payload.warming && !payload.operating); + assert!(payload.session_id.is_none()); + } + } + assert_eq!(core_capsule_hide_delay(CapsuleState::Recording), None); + assert_eq!(core_capsule_hide_delay(CapsuleState::Error), Some(2000)); + assert_eq!(core_capsule_hide_delay(CapsuleState::Cancelled), Some(0)); + } +} diff --git a/openless-all/app/src-tauri/src/commands/settings.rs b/openless-all/app/src-tauri/src/commands/settings.rs index 2b869abce..6c971cda6 100644 --- a/openless-all/app/src-tauri/src/commands/settings.rs +++ b/openless-all/app/src-tauri/src/commands/settings.rs @@ -8,6 +8,11 @@ pub fn get_settings(core: CoreState<'_>) -> UserPreferences { prefs } +#[tauri::command] +pub fn get_voice_activation_state(core: CoreState<'_>) -> Option { + core.latest_voice_activation_state() +} + #[tauri::command] pub fn get_default_style_system_prompts() -> StyleSystemPrompts { StyleSystemPrompts::default() @@ -241,6 +246,7 @@ pub(crate) fn sync_low_latency_capture_preferences(_prefs: &UserPreferences) {} fn voice_activation_preferences_changed(previous: &UserPreferences, next: &UserPreferences) -> bool { previous.voice_activation_enabled != next.voice_activation_enabled + || previous.voice_activation_hotkey_linked != next.voice_activation_hotkey_linked || previous.voice_activation_wake_enabled != next.voice_activation_wake_enabled || previous.voice_activation_finish_enabled != next.voice_activation_finish_enabled || previous.voice_activation_wake_phrase != next.voice_activation_wake_phrase diff --git a/openless-all/app/src-tauri/src/coordinator.rs b/openless-all/app/src-tauri/src/coordinator.rs index 9c2928a0e..74fa0651b 100644 --- a/openless-all/app/src-tauri/src/coordinator.rs +++ b/openless-all/app/src-tauri/src/coordinator.rs @@ -37,6 +37,9 @@ mod restore_runtime; pub(crate) mod selection_voice_session; pub(crate) mod voice_edit_session; use capsule_focus::*; +use crate::capsule_feedback::{ + core_capsule_hide_delay, VoiceActivationFeedback, CAPSULE_AUTO_HIDE_DELAY_MS, +}; pub(crate) use capsule_focus::{ capture_external_focus_target, capture_focus_target, capture_frontmost_app, restore_focus_target_if_possible, @@ -592,6 +595,7 @@ struct Inner { /// the old hint before the new payload or bails on a changed epoch, never /// emitting Idle after a new session. capsule_event_lock: Mutex<()>, + voice_activation_feedback: Mutex, /// Selection-polish lightweight hint is still showing or being processed. /// Older voice/QA auto-hide timers must yield during this window so they do /// not dismiss the selection-polish capsule early. @@ -759,6 +763,7 @@ impl Coordinator { last_capsule_state: Mutex::new(None), capsule_event_epoch: AtomicU64::new(0), capsule_event_lock: Mutex::new(()), + voice_activation_feedback: Mutex::new(VoiceActivationFeedback::default()), selection_polish_capsule_active: AtomicBool::new(false), qa_context: Arc::clone(&qa_context), shutdown: AtomicBool::new(false), @@ -897,6 +902,7 @@ impl Coordinator { last_capsule_state: Mutex::new(None), capsule_event_epoch: AtomicU64::new(0), capsule_event_lock: Mutex::new(()), + voice_activation_feedback: Mutex::new(VoiceActivationFeedback::default()), selection_polish_capsule_active: AtomicBool::new(false), qa_context: Arc::clone(&qa_context), shutdown: AtomicBool::new(false), @@ -977,6 +983,54 @@ impl Coordinator { let _ = self.present_core_capsule_at_revision(payload, None, captured_revision); } + pub(crate) fn reset_voice_activation_feedback(&self) { + self.inner.voice_activation_feedback.lock().request(None); + } + + pub(crate) fn present_voice_activation_feedback( + &self, + state: &openless_core::VoiceActivationStateSnapshot, + ) { + let latest = self.inner.backend.latest_voice_activation_state(); + let Some(enabled) = self.inner.voice_activation_feedback.lock().complete( + state, + latest.as_ref(), + self.inner.backend.voice_activation_hotkey_intent(), + ) else { + return; + }; + let preferences = self.inner.backend.get_preferences(); + if !preferences.show_capsule { + return; + } + let (epoch, revision) = { + let _guard = self.inner.capsule_event_lock.lock(); + let other_owner = self.selection_voice_owns_capsule() + || self.inner.selection_polish_capsule_active.load(Ordering::SeqCst) + || self.inner.vocab_card_visible.load(Ordering::SeqCst) + || self.inner.qa_context.is_panel_visible() + || self.inner.backend.less_computer_active_session().is_some() + || crate::recording_companion::claimed(); + if !crate::capsule_feedback::can_present_feedback( + *self.inner.last_capsule_state.lock(), + self.inner.backend.snapshot().dictation.phase, + other_owner, + ) { + return; + } + let Some(revision) = self.inner.host.capsule_feedback_revision() else { + return; + }; + (self.inner.capsule_event_epoch.load(Ordering::SeqCst), revision) + }; + let payload = crate::capsule_feedback::voice_activation_payload( + enabled, preferences.capsule_style, + ); + // The epoch/revision pair rejects a new shared-surface owner. Done + // reuses the existing two-second, epoch-safe, nonactivating outlet. + let _ = self.present_core_capsule_at_revision(payload, Some(epoch), revision); + } + /// Selection-voice claims the shared capsule from hotkey Start through /// session teardown so late dictation Done cannot overwrite Recording. pub(crate) fn selection_voice_owns_capsule(&self) -> bool { @@ -1948,29 +2002,13 @@ impl Coordinator { } } -const CAPSULE_AUTO_HIDE_DELAY_MS: u64 = 2000; - -/// Core terminal events describe semantics only; how long the native capsule -/// stays is the Tauri Host's job. Centralizing the mapping in a pure function -/// pins the done/error/cancel timings and lets tests run without real windows -/// and timers. -fn core_capsule_hide_delay(state: CapsuleState) -> Option { - match state { - CapsuleState::Done | CapsuleState::Error => Some(CAPSULE_AUTO_HIDE_DELAY_MS), - CapsuleState::Cancelled => Some(0), - _ => None, - } -} - fn schedule_capsule_idle(inner: &Arc, delay_ms: u64) { - let expected = inner.last_capsule_state.lock().as_ref().copied(); + let expected_epoch = inner.capsule_event_epoch.load(Ordering::SeqCst); let inner = Arc::clone(inner); let spawner = inner.host.clone(); spawner.spawn(async move { tokio::time::sleep(std::time::Duration::from_millis(delay_ms)).await; - if inner.last_capsule_state.lock().as_ref().copied() == expected { - hide_capsule_if_all_sessions_idle(&inner); - } + hide_capsule_if_all_sessions_idle(&inner, expected_epoch); }); } diff --git a/openless-all/app/src-tauri/src/coordinator/capsule_focus.rs b/openless-all/app/src-tauri/src/coordinator/capsule_focus.rs index c597cb068..c79f93efa 100644 --- a/openless-all/app/src-tauri/src/coordinator/capsule_focus.rs +++ b/openless-all/app/src-tauri/src/coordinator/capsule_focus.rs @@ -404,7 +404,7 @@ pub(super) fn selection_polish_capsule_epoch_is_current( /// Collapse path for old dictation/QA timers. It shares one short lock with all emits: it /// yields if Selection Polish is showing, and if a new voice/QA session emitted first, that /// emission is ordered before Idle by the lock. -pub(super) fn hide_capsule_if_all_sessions_idle(inner: &Arc) { +pub(super) fn hide_capsule_if_all_sessions_idle(inner: &Arc, expected_epoch: u64) { // Read session state first, then enter the capsule lock. The event epoch cancels this Idle // if any new payload arrived between the two reads. #[cfg(all(not(mobile), target_os = "windows"))] @@ -416,13 +416,12 @@ pub(super) fn hide_capsule_if_all_sessions_idle(inner: &Arc) { && inner.backend.less_computer_active_session().is_none() && selection_voice_idle; let selection_polish_active = inner.selection_polish_capsule_active.load(Ordering::SeqCst); - let observed_epoch = inner.capsule_event_epoch.load(Ordering::SeqCst); if !dictation_idle || selection_polish_active { return; } let _event_guard = inner.capsule_event_lock.lock(); - if inner.capsule_event_epoch.load(Ordering::SeqCst) == observed_epoch + if inner.capsule_event_epoch.load(Ordering::SeqCst) == expected_epoch && !inner.selection_polish_capsule_active.load(Ordering::SeqCst) { emit_capsule_with_context_locked(inner, CapsuleState::Idle, 0.0, 0, None, None, false); diff --git a/openless-all/app/src-tauri/src/coordinator/dictation_core.rs b/openless-all/app/src-tauri/src/coordinator/dictation_core.rs index f5b080ae3..17012a0f4 100644 --- a/openless-all/app/src-tauri/src/coordinator/dictation_core.rs +++ b/openless-all/app/src-tauri/src/coordinator/dictation_core.rs @@ -51,6 +51,20 @@ async fn dispatch( if !inner.backend.snapshot().running { inner.backend.start().await?; } + if let Some(changed) = inner.backend.accept_voice_activation_hotkey_edge(edge) { + if changed { + inner.voice_activation_feedback.lock().request( + inner.backend.voice_activation_hotkey_intent(), + ); + let backend = Arc::clone(&inner.backend); + inner.host.spawn(async move { + if let Err(error) = backend.reconcile_voice_activation_hotkey().await { + log::warn!("[voice-activation] linked hotkey failed: {error}"); + } + }); + } + return Ok(openless_core::CliDispatchOutcome::Noop); + } inner.backend.dispatch_dictation_hotkey_edge(edge).await } diff --git a/openless-all/app/src-tauri/src/coordinator/hotkey_loops.rs b/openless-all/app/src-tauri/src/coordinator/hotkey_loops.rs index 8799a6b6a..fdcdef90c 100644 --- a/openless-all/app/src-tauri/src/coordinator/hotkey_loops.rs +++ b/openless-all/app/src-tauri/src/coordinator/hotkey_loops.rs @@ -797,7 +797,7 @@ pub(super) fn less_computer_modifier_bridge_loop( handle_less_computer_modifier_pressed(&inner_cloned, press_id, at).await }); } - HotkeyEvent::Released { press_id, at } => { + HotkeyEvent::Released { press_id, at, .. } => { if inner.less_computer_press_generation.load(Ordering::SeqCst) != press_id { continue; } @@ -1457,7 +1457,9 @@ pub(super) fn combo_hotkey_bridge_loop(inner: Arc, rx: mpsc::Receiver, rx: mpsc::Receiver, rx: mpsc::Receiver, rx: mpsc::Receiver { + HotkeyEvent::Released { at, press_id, combined } => { + if combined { + // The abort bridge may lag this FIFO. Suppress linked clicks + // using the verdict captured before native key-up was sent. + inner_cloned.backend.accept_voice_activation_hotkey_edge( + openless_core::DictationHotkeyEdge::Combined { at, press_id }); + } if matches!( hotkey_runtime_target(&inner_cloned).dictation_mode, crate::types::HotkeyMode::Hold - ) { + ) && !(inner_cloned.backend.get_preferences().voice_activation_enabled + && inner_cloned.backend.get_preferences().voice_activation_hotkey_linked) + { let task_inner = Arc::clone(&inner_cloned); let _ = inner_cloned.host.spawn(async move { handle_released_edge(&task_inner, at, press_id).await; @@ -2545,6 +2559,7 @@ pub(crate) mod less_computer_test_support { last_capsule_state: Mutex::new(None), capsule_event_epoch: AtomicU64::new(0), capsule_event_lock: Mutex::new(()), + voice_activation_feedback: Mutex::new(VoiceActivationFeedback::default()), selection_polish_capsule_active: AtomicBool::new(false), qa_context: Arc::new(TauriQaHostContext::default()), shutdown: AtomicBool::new(false), @@ -2990,6 +3005,7 @@ pub(crate) mod windows_less_computer_tests { tx.send(HotkeyEvent::Released { at: released, press_id: 47, + combined: false, }) .unwrap(); drop(tx); diff --git a/openless-all/app/src-tauri/src/core_adapters.rs b/openless-all/app/src-tauri/src/core_adapters.rs index d3b383201..f130fee23 100644 --- a/openless-all/app/src-tauri/src/core_adapters.rs +++ b/openless-all/app/src-tauri/src/core_adapters.rs @@ -140,6 +140,8 @@ pub(crate) fn backend_dependencies( #[cfg(target_os = "windows")] { let prefs = preferences.get(); + warm_hub.set_capture_paused(0, + prefs.voice_activation_enabled && prefs.voice_activation_hotkey_linked); warm_hub.configure( crate::recorder::warm_capture_enabled( prefs.low_latency_dictation_enabled, @@ -1933,6 +1935,10 @@ impl TauriVoiceActivationRuntime { #[cfg(target_os = "windows")] impl openless_core::VoiceActivationRuntime for TauriVoiceActivationRuntime { + fn set_capture_paused(&self, revision: u64, paused: bool) { + self.warm_hub.set_capture_paused(revision, paused); + } + fn configure( &self, config: VoiceActivationConfig, diff --git a/openless-all/app/src-tauri/src/hotkey.rs b/openless-all/app/src-tauri/src/hotkey.rs index b04198e96..6e7eba79f 100644 --- a/openless-all/app/src-tauri/src/hotkey.rs +++ b/openless-all/app/src-tauri/src/hotkey.rs @@ -46,6 +46,8 @@ pub enum HotkeyEvent { Released { at: Instant, press_id: u64, + /// Native per-press verdict, independent of the asynchronous abort bridge. + combined: bool, }, // Combo abort is not in this enum: it goes through the separate `combo_abort` channel to avoid // being queued behind the Pressed → begin_session synchronous mic-open flow (see module docs). @@ -929,7 +931,8 @@ mod platform { ctx.shared.trigger_held.store(false, Ordering::SeqCst); let press_id = ctx.shared.trigger_press_id.swap(0, Ordering::SeqCst); if !tap_only_fn { - send_or_log(&ctx.tx, HotkeyEvent::Released { at: now, press_id }); + send_or_log(&ctx.tx, HotkeyEvent::Released { at: now, press_id, + combined: ctx.shared.trigger_companion_seen.load(Ordering::SeqCst) == press_id }); return; } @@ -946,7 +949,8 @@ mod platform { press_id, }, ); - send_or_log(&ctx.tx, HotkeyEvent::Released { at: now, press_id }); + send_or_log(&ctx.tx, HotkeyEvent::Released { at: now, press_id, + combined: ctx.shared.trigger_companion_seen.load(Ordering::SeqCst) == press_id }); } else { log::info!( "[hotkey] Fn 操作未触发听写（held_ms={}, companion_seen={companion_seen}）", @@ -1814,6 +1818,7 @@ mod platform { HotkeyEvent::Released { at: std::time::Instant::now(), press_id, + combined: ctx.shared.trigger_companion_seen.load(Ordering::SeqCst) == press_id, }, ); } @@ -2130,6 +2135,23 @@ mod platform { ); } + #[test] + fn windows_release_carries_combo_verdict_even_when_abort_bridge_has_not_drained() { + let (ctx, rx, abort_rx) = callback_context_with_combo(shared(HotkeyTrigger::RightControl)); + dispatch_keyboard_event(&ctx, VK_RCONTROL, WM_KEYDOWN); + let press_id = match rx.recv().unwrap() { + HotkeyEvent::Pressed { press_id, .. } => press_id, + other => panic!("expected press, got {other:?}"), + }; + dispatch_keyboard_event(&ctx, 0x41, WM_KEYDOWN); + dispatch_keyboard_event(&ctx, VK_RCONTROL, WM_KEYUP); + // Intentionally read Released before the independently queued Combined. + assert!(matches!(rx.recv().unwrap(), HotkeyEvent::Released { + press_id: released_id, combined: true, .. + } if released_id == press_id)); + assert_eq!(abort_rx.recv().unwrap().press_id, press_id); + } + #[test] fn windows_modifier_edges_are_deduped_from_mock_hook_events() { let shared = shared(HotkeyTrigger::RightControl); diff --git a/openless-all/app/src-tauri/src/lib.rs b/openless-all/app/src-tauri/src/lib.rs index e0cf76be2..4226d965c 100644 --- a/openless-all/app/src-tauri/src/lib.rs +++ b/openless-all/app/src-tauri/src/lib.rs @@ -263,6 +263,7 @@ macro_rules! app_invoke_handler_desktop { commands::set_capsule_language_menu, commands::get_settings, commands::get_settings_snapshot, + commands::get_voice_activation_state, commands::update_setting_fields, commands::get_default_style_system_prompts, commands::set_settings, @@ -561,6 +562,7 @@ macro_rules! app_invoke_handler_mobile { $crate::commands::get_startup_snapshot, $crate::commands::get_settings, $crate::commands::get_settings_snapshot, + $crate::commands::get_voice_activation_state, $crate::commands::update_setting_fields, $crate::commands::get_default_style_system_prompts, $crate::commands::set_settings, @@ -3595,6 +3597,7 @@ fn show_qa_window_no_activate(window: &tauri::WebviewWindow>>, runtime_fallback_claims: Mutex>, enabled: AtomicBool, + standby_policy: Mutex<(u64, bool)>, + standby_requested: AtomicBool, preview_active: AtomicBool, cold_capture_active: AtomicBool, pending_configuration: @@ -980,6 +982,8 @@ impl WarmCaptureHub { runtime_errors: Mutex::new(HashMap::new()), runtime_fallback_claims: Mutex::new(HashSet::new()), enabled: AtomicBool::new(false), + standby_policy: Mutex::new((0, false)), + standby_requested: AtomicBool::new(false), preview_active: AtomicBool::new(false), cold_capture_active: AtomicBool::new(false), pending_configuration: Mutex::new(None), @@ -1073,6 +1077,21 @@ impl WarmCaptureHub { } } + /// A linked pause overrides idle prewarming, never a promoted explicit recording. + pub(crate) fn set_capture_paused(&self, revision: u64, paused: bool) { + let _lifecycle = self.lifecycle_gate.lock(); + { + let mut policy = self.standby_policy.lock(); + if revision < policy.0 || *policy == (revision, paused) { + return; + } + *policy = (revision, paused); + } + let device = self.device_name.lock().clone(); + let processing = *self.processing_config.lock(); + self.configure_locked(self.standby_requested.load(Ordering::Acquire), device, processing); + } + pub(crate) fn configure( &self, enabled: bool, @@ -1080,6 +1099,7 @@ impl WarmCaptureHub { processing: openless_core::AudioProcessingConfig, ) { let _lifecycle = self.lifecycle_gate.lock(); + self.standby_requested.store(enabled, Ordering::Release); self.configure_locked(enabled, device_name, processing); } @@ -1089,6 +1109,8 @@ impl WarmCaptureHub { device_name: Option, processing: openless_core::AudioProcessingConfig, ) { + let paused = self.standby_policy.lock().1; + let enabled = enabled && !paused; let device_changed = *self.device_name.lock() != device_name; let config_changed = *self.processing_config.lock() != processing; if self.cold_capture_active.load(Ordering::Acquire) { @@ -1096,6 +1118,12 @@ impl WarmCaptureHub { return; } if (device_changed || config_changed || !enabled) && self.sink.has_claims() { + // A staged handoff already owns body PCM. Let it promote and finalize; + // invalidating it would lose the prefix and reopen a cold microphone. + if paused { + *self.pending_configuration.lock() = Some((enabled, device_name, processing)); + return; + } let staging_claims = self.sink.staging_claim_ids(); self.runtime_fallback_claims.lock().extend(staging_claims); *self.pending_configuration.lock() = Some((enabled, device_name, processing)); @@ -1152,7 +1180,9 @@ impl WarmCaptureHub { } fn ensure_started_locked(&self, device_name: Option) -> Result<(), RecorderError> { - if !self.enabled.load(Ordering::Acquire) { + if !self.enabled.load(Ordering::Acquire) + || (self.standby_policy.lock().1 && !self.sink.has_promoted_claims()) + { return Err(RecorderError::EngineFailed( "warm capture is disabled".into(), )); @@ -1323,6 +1353,9 @@ impl WarmCaptureHub { } loop { let lifecycle = self.lifecycle_gate.lock(); + if self.standby_policy.lock().1 { + return Err(RecorderError::EngineFailed("standby capture is paused".into())); + } let desired_processing = processing.unwrap_or(*self.processing_config.lock()); if *self.device_name.lock() != device_name || *self.processing_config.lock() != desired_processing @@ -3254,6 +3287,60 @@ mod tests { assert!(!warm_capture_enabled(false, true, false)); } + #[test] + fn linked_pause_overrides_prewarm_saved_configuration_and_cold_recovery() { + let hub = WarmCaptureHub::new(); + hub.set_capture_paused(1, true); + hub.configure(true, None, openless_core::AudioProcessingConfig::default()); + assert!(!hub.enabled.load(Ordering::Acquire)); + assert!(hub.recorder.lock().is_none()); + hub.begin_cold_capture().unwrap(); + hub.configure(true, None, openless_core::AudioProcessingConfig::default()); + hub.end_cold_capture(); + hub.resume_after_cold_capture(); + assert!(!hub.enabled.load(Ordering::Acquire)); + assert!(hub.recorder.lock().is_none()); + hub.set_capture_paused(0, false); + assert!(hub.standby_policy.lock().1); + assert!(hub.arm(99, None, openless_core::AudioProcessingConfig::default()).is_err()); + } + + #[test] + fn linked_pause_keeps_staged_voice_body_until_promotion_and_finalization() { + let hub = WarmCaptureHub::new(); + let key = voice_capture_key(9); + let consumer = Arc::new(RecordingConsumer::default()); + let body = [0x00, 0x40].repeat(160); + hub.sink.arm_with_limit(key, false, STAGED_CAPTURE_MAX_BYTES).unwrap(); + hub.sink.consume_pcm_chunk(&body); + hub.enabled.store(true, Ordering::Release); + hub.set_capture_paused(1, true); + hub.configure(true, None, openless_core::AudioProcessingConfig::default()); + assert!(hub.sink.staging_claim_ids().contains(&key)); + assert!(!hub.runtime_fallback_claims.lock().contains(&key)); + hub.sink.promote(key, consumer.clone(), Arc::new(|_| {}), None).unwrap(); + assert_eq!(consumer.chunks.lock().unwrap().concat(), body); + hub.release_claim(key); + assert!(!hub.enabled.load(Ordering::Acquire)); + assert!(hub.recorder.lock().is_none()); + } + + #[test] + fn linked_pause_preserves_promoted_recording_until_its_claim_releases() { + let hub = WarmCaptureHub::new(); + let consumer = Arc::new(RecordingConsumer::default()); + hub.sink.arm_with_limit(7, false, STAGED_CAPTURE_MAX_BYTES).unwrap(); + hub.sink.promote(7, consumer, Arc::new(|_| {}), None).unwrap(); + hub.enabled.store(true, Ordering::Release); + hub.set_capture_paused(1, true); + assert!(hub.sink.has_promoted_claims()); + hub.configure(true, None, openless_core::AudioProcessingConfig::default()); + assert!(hub.sink.has_promoted_claims()); + hub.release_claim(7); + assert!(!hub.enabled.load(Ordering::Acquire)); + assert!(hub.recorder.lock().is_none()); + } + #[test] fn activation_vad_skips_silence_and_preserves_speech_onset() { let sink = WarmCaptureSink::new(STAGED_CAPTURE_MAX_BYTES); diff --git a/openless-all/app/src-tauri/src/side_aware_combo.rs b/openless-all/app/src-tauri/src/side_aware_combo.rs index 859110c23..072f5d9ec 100644 --- a/openless-all/app/src-tauri/src/side_aware_combo.rs +++ b/openless-all/app/src-tauri/src/side_aware_combo.rs @@ -125,10 +125,11 @@ impl SideAwareComboState { fn release_active(&mut self) -> Option { let press_id = self.active_press_id.take()?; - self.companion_seen = false; + let combined = std::mem::take(&mut self.companion_seen); Some(HotkeyEvent::Released { at: Instant::now(), press_id, + combined, }) } diff --git a/openless-all/app/src-tauri/src/tauri_coordinator_host.rs b/openless-all/app/src-tauri/src/tauri_coordinator_host.rs index 86df885be..22951013d 100644 --- a/openless-all/app/src-tauri/src/tauri_coordinator_host.rs +++ b/openless-all/app/src-tauri/src/tauri_coordinator_host.rs @@ -2175,6 +2175,11 @@ impl TauriCoordinatorHost { /// Commit the replay frame before the webview/main-thread work is queued. /// The rail snapshot is an IPC replay source, so it must not wait for a /// window callback to become internally consistent. + pub(crate) fn capsule_feedback_revision(&self) -> Option { + let snapshot = self.capsule.snapshot.lock(); + snapshot.pending_payload_revision.is_none().then_some(snapshot.revision) + } + pub(crate) fn capsule_revision(&self) -> u64 { self.capsule.snapshot.lock().revision } diff --git a/openless-all/app/src-tauri/src/tauri_events.rs b/openless-all/app/src-tauri/src/tauri_events.rs index 25268c57b..533e013fd 100644 --- a/openless-all/app/src-tauri/src/tauri_events.rs +++ b/openless-all/app/src-tauri/src/tauri_events.rs @@ -211,6 +211,9 @@ async fn forward_legacy_event( emit_dictation_state(app, backend, snapshot) } BackendEventKind::VoiceActivationStateChanged(state) => { + if let Some(coordinator) = app.try_state::>() { + coordinator.present_voice_activation_feedback(&state); + } let _ = app.emit("voice-activation:state-changed", state); } BackendEventKind::TranscriptDelta(_) => {} @@ -821,6 +824,9 @@ fn present_qa_capsule( } async fn emit_resync(app: &AppHandle, backend: &OpenLessBackend, owners: &mut CapsuleOwners) { + if let Some(coordinator) = app.try_state::>() { + coordinator.reset_voice_activation_feedback(); + } emit_preferences(app, backend); let snapshot = backend.snapshot(); if let Some((owner, _)) = owners diff --git a/openless-all/app/src/i18n/de.ts b/openless-all/app/src/i18n/de.ts index 83ca7fdbd..c39647a5d 100644 --- a/openless-all/app/src/i18n/de.ts +++ b/openless-all/app/src/i18n/de.ts @@ -1690,6 +1690,15 @@ export const de: typeof zhCN = { voiceActivationDesc: 'Lokal auf eine Aktivierungsphrase warten, um das Diktat zu starten.', voiceActivationLocalOnly: 'Die Erkennung bleibt auf diesem Gerät; Bereitschaftsaudio wird weder hochgeladen noch archiviert.', + voiceActivationHotkeyLinked: "Aufnahme-Tastenkürzel verknüpfen", + voiceActivationHotkeyLinkedDesc: "Ein Klick schaltet die lokale Aktivierung statt der Aufnahme um. Nach dem Start pausiert. Pausieren beendet die Mikrofon-Vorwärmung, nicht andere Aufnahmen. Diktate nutzen weiterhin den gewählten Dienst.", + voiceActivationDisabled: "Sprachaktivierung deaktiviert", + voiceActivationPaused: "Aktivierung vorübergehend pausiert", + voiceActivationPausing: "Aktivierung wird gestoppt…", + voiceActivationStarting: "Aktivierung wird gestartet…", + voiceActivationListening: "Warten auf Aktivierungsphrase", + voiceActivationDictating: "Diktat oder Abschluss läuft", + voiceActivationError: "Aktivierung nicht verfügbar. Modell, Mikrofonberechtigung und Gerät prüfen.", voiceActivationMultimodalModels: 'Hier das Sprachaktivierungsmodell vorbereiten. Das Diktat nutzt weiterhin den aktuellen multimodalen Dienst.', voiceActivationModelMissing: 'Das lokale Zipformer-Modell ist nicht installiert.', diff --git a/openless-all/app/src/i18n/en.ts b/openless-all/app/src/i18n/en.ts index cabdfbd89..52a6f767b 100644 --- a/openless-all/app/src/i18n/en.ts +++ b/openless-all/app/src/i18n/en.ts @@ -1658,6 +1658,15 @@ export const en: typeof zhCN = { voiceActivationDesc: 'Listen locally for a wake phrase to start dictation.', voiceActivationLocalOnly: 'Recognition stays on this device; standby audio is not uploaded or archived.', + voiceActivationHotkeyLinked: "Link recording shortcut", + voiceActivationHotkeyLinkedDesc: "Click the recording shortcut to toggle wake listening instead of the usual recording gesture. Starts paused after launch. Pausing disables idle microphone prewarming, not other recordings. Wake listening is local; dictation still uses your current service.", + voiceActivationDisabled: "Voice activation disabled", + voiceActivationPaused: "Wake listening temporarily paused", + voiceActivationPausing: "Stopping wake listening…", + voiceActivationStarting: "Starting wake listening…", + voiceActivationListening: "Listening for the wake phrase", + voiceActivationDictating: "Dictating or finishing", + voiceActivationError: "Unable to start wake listening. Check the model, microphone permissions and device.", voiceActivationMultimodalModels: 'Prepare the voice activation model here. Dictation still uses your current multimodal service.', voiceActivationModelMissing: 'The local Zipformer model is not installed.', diff --git a/openless-all/app/src/i18n/es.ts b/openless-all/app/src/i18n/es.ts index 0378847ea..b470baabe 100644 --- a/openless-all/app/src/i18n/es.ts +++ b/openless-all/app/src/i18n/es.ts @@ -1682,6 +1682,15 @@ export const es: typeof zhCN = { voiceActivationDesc: 'Escucha localmente una frase de activación para iniciar el dictado.', voiceActivationLocalOnly: 'El reconocimiento permanece en este dispositivo; el audio de espera no se sube ni se archiva.', + voiceActivationHotkeyLinked: "Vincular atajo de grabación", + voiceActivationHotkeyLinkedDesc: "Pulsa el atajo para activar o pausar la escucha en vez de grabar. Inicia en pausa. Pausar desactiva el precalentamiento del micrófono, no otras grabaciones. La escucha es local; el dictado usa el servicio actual.", + voiceActivationDisabled: "Activación por voz desactivada", + voiceActivationPaused: "Escucha en pausa temporal", + voiceActivationPausing: "Deteniendo la escucha…", + voiceActivationStarting: "Iniciando la escucha…", + voiceActivationListening: "Esperando la frase de activación", + voiceActivationDictating: "Dictando o finalizando", + voiceActivationError: "No se puede iniciar la escucha. Revisa el modelo, los permisos y el micrófono.", voiceActivationMultimodalModels: 'Prepara aquí el modelo de activación por voz. El dictado sigue usando el servicio multimodal actual.', voiceActivationModelMissing: 'El modelo local Zipformer no está instalado.', diff --git a/openless-all/app/src/i18n/fr.ts b/openless-all/app/src/i18n/fr.ts index b9487f656..be254312b 100644 --- a/openless-all/app/src/i18n/fr.ts +++ b/openless-all/app/src/i18n/fr.ts @@ -1703,6 +1703,15 @@ export const fr: typeof zhCN = { voiceActivationDesc: 'Écouter localement une phrase d’activation pour démarrer la dictée.', voiceActivationLocalOnly: 'La reconnaissance reste sur cet appareil ; l’audio en veille n’est ni envoyé ni archivé.', + voiceActivationHotkeyLinked: "Lier le raccourci d’enregistrement", + voiceActivationHotkeyLinkedDesc: "Un appui active ou suspend la veille au lieu d’enregistrer. En pause au lancement. La pause coupe le préchauffage du micro, pas les autres enregistrements. La veille est locale ; la dictée utilise le service actuel.", + voiceActivationDisabled: "Activation vocale désactivée", + voiceActivationPaused: "Veille vocale temporairement suspendue", + voiceActivationPausing: "Arrêt de la veille…", + voiceActivationStarting: "Démarrage de la veille…", + voiceActivationListening: "En attente de la phrase d’activation", + voiceActivationDictating: "Dictée ou finalisation en cours", + voiceActivationError: "Veille indisponible. Vérifiez le modèle, les autorisations et le microphone.", voiceActivationMultimodalModels: 'Préparez ici le modèle d’activation vocale. La dictée utilise toujours votre service multimodal actuel.', voiceActivationModelMissing: 'Le modèle local Zipformer n’est pas installé.', diff --git a/openless-all/app/src/i18n/ja.ts b/openless-all/app/src/i18n/ja.ts index c1d09e207..07e5ea03b 100644 --- a/openless-all/app/src/i18n/ja.ts +++ b/openless-all/app/src/i18n/ja.ts @@ -1645,6 +1645,15 @@ export const ja: typeof zhCN = { voiceActivationDesc: '起動フレーズをローカルで待ち受けて音声入力を開始します。', voiceActivationLocalOnly: '認識はこの端末内で行われます。待機中の音声は送信・保存されません。', + voiceActivationHotkeyLinked: "録音ショートカットと連動", + voiceActivationHotkeyLinkedDesc: "録音操作の代わりに、1回押すとウェイク監視を開始・一時停止します。起動直後は一時停止します。停止中はマイクの待機予熱を無効にしますが、他の録音は中断しません。監視はローカルで、音声入力は現在のサービスを使用します。", + voiceActivationDisabled: "音声起動は無効です", + voiceActivationPaused: "ウェイク監視を一時停止中", + voiceActivationPausing: "ウェイク監視を停止しています…", + voiceActivationStarting: "ウェイク監視を開始しています…", + voiceActivationListening: "開始フレーズを待機中", + voiceActivationDictating: "音声入力または終了処理中", + voiceActivationError: "監視を開始できません。モデル、マイク権限、デバイスを確認してください。", voiceActivationMultimodalModels: 'ここで音声起動用モデルを準備します。音声入力には引き続き現在のマルチモーダルサービスを使用します。', voiceActivationModelMissing: 'ローカル Zipformer モデルがインストールされていません。', diff --git a/openless-all/app/src/i18n/ko.ts b/openless-all/app/src/i18n/ko.ts index dff252b94..c7ebfe846 100644 --- a/openless-all/app/src/i18n/ko.ts +++ b/openless-all/app/src/i18n/ko.ts @@ -1632,6 +1632,15 @@ export const ko: typeof zhCN = { voiceActivationDesc: '받아쓰기 시작 구문을 로컬에서 감지합니다.', voiceActivationLocalOnly: '인식은 이 기기에서만 수행되며 대기 오디오는 업로드하거나 보관하지 않습니다.', + voiceActivationHotkeyLinked: "녹음 단축키와 연동", + voiceActivationHotkeyLinkedDesc: "기존 녹음 동작 대신 한 번 눌러 깨우기 대기를 시작하거나 일시 중지합니다. 앱 시작 시에는 중지됩니다. 중지하면 대기 마이크 예열을 끄지만 다른 녹음은 중단하지 않습니다. 깨우기는 로컬에서 처리하며 받아쓰기는 현재 서비스를 사용합니다.", + voiceActivationDisabled: "음성 활성화 꺼짐", + voiceActivationPaused: "깨우기 대기 일시 중지됨", + voiceActivationPausing: "깨우기 대기 중지 중…", + voiceActivationStarting: "깨우기 대기 시작 중…", + voiceActivationListening: "깨우기 명령 대기 중", + voiceActivationDictating: "받아쓰기 또는 마무리 중", + voiceActivationError: "대기를 시작할 수 없습니다. 모델, 마이크 권한 및 장치를 확인하세요.", voiceActivationMultimodalModels: '여기에서 음성 활성화 모델을 준비합니다. 받아쓰기는 현재 멀티모달 서비스를 계속 사용합니다.', voiceActivationModelMissing: '로컬 Zipformer 모델이 설치되지 않았습니다.', diff --git a/openless-all/app/src/i18n/zh-CN.ts b/openless-all/app/src/i18n/zh-CN.ts index 6b92f58ea..0e428140c 100644 --- a/openless-all/app/src/i18n/zh-CN.ts +++ b/openless-all/app/src/i18n/zh-CN.ts @@ -1502,6 +1502,15 @@ export const zhCN = { '仅在 Windows 桌面端启用本地语音命令唤醒听写。待机音频只在内存中处理，不会上传或保存；会增加功耗并保持麦克风指示灯亮起。', voiceActivationLocalOnly: '本地识别：说“开始语音录入”开始，说“完成语音录入”结束。需要已安装本地 Zipformer 模型。', + voiceActivationHotkeyLinked: "与录音快捷键联动", + voiceActivationHotkeyLinkedDesc: "单击录音快捷键开启或暂停唤醒监听，替代原录音手势。启动应用后先暂停；暂停时关闭后台预热，其他正在录音的任务不受影响。等待唤醒仅本地处理，正式听写仍使用当前服务。", + voiceActivationDisabled: "已关闭语音唤醒", + voiceActivationPaused: "已临时暂停唤醒监听", + voiceActivationPausing: "正在停止唤醒监听…", + voiceActivationStarting: "正在启动唤醒监听…", + voiceActivationListening: "正在等待唤醒命令", + voiceActivationDictating: "正在听写或收尾", + voiceActivationError: "无法启动唤醒监听，请检查模型、麦克风权限和设备。", voiceActivationMultimodalModels: '此处准备语音唤醒模型，听写仍使用当前多模态服务。', voiceActivationModelMissing: '请先在“AI 服务与模型”的本地模型页准备 Zipformer。', voiceActivationDownloadZipformer: '前往 AI 服务与模型下载 Zipformer', diff --git a/openless-all/app/src/i18n/zh-TW.ts b/openless-all/app/src/i18n/zh-TW.ts index 533136f34..221c3e833 100644 --- a/openless-all/app/src/i18n/zh-TW.ts +++ b/openless-all/app/src/i18n/zh-TW.ts @@ -1573,6 +1573,15 @@ export const zhTW: typeof zhCN = { voiceActivationLabel: '語音啟用', voiceActivationDesc: '在本機等待喚醒詞以開始聽寫。', voiceActivationLocalOnly: '辨識只在此裝置上進行；待機音訊不會上傳或封存。', + voiceActivationHotkeyLinked: "與錄音快捷鍵聯動", + voiceActivationHotkeyLinkedDesc: "單擊錄音快捷鍵開啟或暫停喚醒監聽，取代原錄音手勢。啟動應用程式後先暫停；暫停時關閉背景預熱，其他錄音工作不受影響。等待喚醒僅在本機處理，正式聽寫仍使用目前服務。", + voiceActivationDisabled: "語音喚醒已關閉", + voiceActivationPaused: "喚醒監聽已暫停", + voiceActivationPausing: "正在停止喚醒監聽…", + voiceActivationStarting: "正在啟動喚醒監聽…", + voiceActivationListening: "正在等待喚醒命令", + voiceActivationDictating: "正在聽寫或完成處理", + voiceActivationError: "無法啟動喚醒監聽，請檢查模型、麥克風權限與裝置。", voiceActivationMultimodalModels: '在此準備語音喚醒模型，聽寫仍使用目前的多模態服務。', voiceActivationModelMissing: '請先在「AI 服務與模型」的本機模型頁準備 Zipformer。', voiceActivationDownloadZipformer: '前往 AI 服務與模型下載 Zipformer', diff --git a/openless-all/app/src/lib/ipc/mock-data.ts b/openless-all/app/src/lib/ipc/mock-data.ts index 29dbcc579..e89e406bf 100644 --- a/openless-all/app/src/lib/ipc/mock-data.ts +++ b/openless-all/app/src/lib/ipc/mock-data.ts @@ -52,6 +52,7 @@ export let mockSettings: UserPreferences = { lowMemoryBackgroundEnabled: false, audioEnhancementEnabled: false, voiceActivationEnabled: false, + voiceActivationHotkeyLinked: false, voiceActivationWakeEnabled: true, voiceActivationFinishEnabled: true, voiceActivationWakePhrase: '开始语音录入', diff --git a/openless-all/app/src/lib/ipc/settings.ts b/openless-all/app/src/lib/ipc/settings.ts index fc48e1ba1..dc7b790c1 100644 --- a/openless-all/app/src/lib/ipc/settings.ts +++ b/openless-all/app/src/lib/ipc/settings.ts @@ -1,3 +1,4 @@ +import type { VoiceActivationStateSnapshot } from '../voiceActivation'; import type { NativeCapsuleStatus, StyleSystemPrompts, UserPreferences } from '../types'; export type { UpdateChannel } from '../types'; import { @@ -73,3 +74,11 @@ export function getNativeCapsuleStatus(): Promise { error: null, })); } + +export function getVoiceActivationState(): Promise { + return invokeOrMock('get_voice_activation_state', undefined, () => + mockSettings.voiceActivationEnabled && mockSettings.voiceActivationHotkeyLinked + ? { phase: 'paused' as const, activationGeneration: 0 } + : null, + ); +} diff --git a/openless-all/app/src/lib/types.ts b/openless-all/app/src/lib/types.ts index e3f0f0e66..1f5f5b7d1 100644 --- a/openless-all/app/src/lib/types.ts +++ b/openless-all/app/src/lib/types.ts @@ -397,6 +397,7 @@ export interface UserPreferences { lowMemoryBackgroundEnabled: boolean; /** Enable local-only voice activation while idle (Windows desktop only). */ voiceActivationEnabled: boolean; + voiceActivationHotkeyLinked: boolean; voiceActivationWakeEnabled: boolean; voiceActivationFinishEnabled: boolean; voiceActivationWakePhrase: string; diff --git a/openless-all/app/src/lib/voiceActivation.test.ts b/openless-all/app/src/lib/voiceActivation.test.ts index 46513bb06..4edd5746a 100644 --- a/openless-all/app/src/lib/voiceActivation.test.ts +++ b/openless-all/app/src/lib/voiceActivation.test.ts @@ -1,5 +1,6 @@ import { isVoiceActivationModelMissing, + voiceActivationStatusKey, VOICE_ACTIVATION_MODEL_MISSING_MESSAGE, } from './voiceActivation'; @@ -29,6 +30,8 @@ assertEqual( ); for (const phase of [ 'disabled', + 'paused', + 'pausing', 'starting', 'listening_for_wake', 'listening_for_finish', @@ -41,3 +44,16 @@ for (const phase of [ } assertEqual(isVoiceActivationModelMissing(null), false, 'null state is ignored'); assertEqual(isVoiceActivationModelMissing(undefined), false, 'missing state is ignored'); + +for (const [phase, key] of [ + ['paused', 'voiceActivationPaused'], + ['pausing', 'voiceActivationPausing'], + ['listening_for_wake', 'voiceActivationListening'], + ['failed', 'voiceActivationError'], + ['permission_denied', 'voiceActivationError'], + ['mic_busy', 'voiceActivationError'], +] as const) { + if (voiceActivationStatusKey({ phase }) !== 'settings.recording.' + key) { + throw new Error('wrong persistent status for ' + phase); + } +} diff --git a/openless-all/app/src/lib/voiceActivation.ts b/openless-all/app/src/lib/voiceActivation.ts index 16207d98c..5048e4b4c 100644 --- a/openless-all/app/src/lib/voiceActivation.ts +++ b/openless-all/app/src/lib/voiceActivation.ts @@ -1,5 +1,7 @@ export type VoiceActivationPhase = | 'disabled' + | 'paused' + | 'pausing' | 'starting' | 'listening_for_wake' | 'wake_detected' @@ -28,3 +30,18 @@ export function isVoiceActivationModelMissing( snapshot?.phase === 'failed' && snapshot.message === VOICE_ACTIVATION_MODEL_MISSING_MESSAGE ); } + +export function voiceActivationStatusKey(snapshot: VoiceActivationStateSnapshot): string { + switch (snapshot.phase) { + case 'disabled': return 'settings.recording.voiceActivationDisabled'; + case 'paused': return 'settings.recording.voiceActivationPaused'; + case 'pausing': return 'settings.recording.voiceActivationPausing'; + case 'starting': + case 'wake_detected': + case 'handing_off': return 'settings.recording.voiceActivationStarting'; + case 'listening_for_wake': return 'settings.recording.voiceActivationListening'; + case 'listening_for_finish': + case 'finishing': return 'settings.recording.voiceActivationDictating'; + default: return 'settings.recording.voiceActivationError'; + } +} diff --git a/openless-all/app/src/pages/settings/RecordingInputSection.tsx b/openless-all/app/src/pages/settings/RecordingInputSection.tsx index 23373d44f..d9423c1e4 100644 --- a/openless-all/app/src/pages/settings/RecordingInputSection.tsx +++ b/openless-all/app/src/pages/settings/RecordingInputSection.tsx @@ -22,10 +22,11 @@ import { import { isTauri, listMicrophoneDevices, setDictationHotkey } from '../../lib/ipc'; import { isVoiceActivationModelMissing, + voiceActivationStatusKey, VOICE_ACTIVATION_MODEL_ALIAS, type VoiceActivationStateSnapshot, } from '../../lib/voiceActivation'; -import { getNativeCapsuleStatus } from '../../lib/ipc/settings'; +import { getNativeCapsuleStatus, getVoiceActivationState } from '../../lib/ipc/settings'; import { getPlatformCapabilities, getCachedPlatformCapabilities } from '../../lib/platform'; import type { NativeCapsuleStatus, @@ -85,6 +86,7 @@ export function RecordingInputSection({ const [microphoneDevicesLoaded, setMicrophoneDevicesLoaded] = useState(false); const [microphoneDevicesError, setMicrophoneDevicesError] = useState(null); const [voiceActivationModelMissing, setVoiceActivationModelMissing] = useState(false); + const [voiceActivationState, setVoiceActivationState] = useState(null); const voiceActivationStateObservedRef = useRef(false); const [displayModeRestartPending, setDisplayModeRestartPending] = useState(false); const [displayModeRestarting, setDisplayModeRestarting] = useState(false); @@ -162,6 +164,10 @@ export function RecordingInputSection({ 'voice-activation:state-changed', (event) => { voiceActivationStateObservedRef.current = true; + setVoiceActivationState((previous) => + (previous?.activationGeneration ?? -1) > (event.payload.activationGeneration ?? 0) + ? previous : event.payload, + ); setVoiceActivationModelMissing(isVoiceActivationModelMissing(event.payload)); }, ); @@ -170,8 +176,15 @@ export function RecordingInputSection({ return; } unlisten = stopListening; + const snapshot = await getVoiceActivationState(); + if (!cancelled && snapshot) { + setVoiceActivationState((previous) => + (previous?.activationGeneration ?? -1) >= (snapshot.activationGeneration ?? 0) + ? previous : snapshot, + ); + } } - void listenForVoiceActivationChanges(); + void listenForVoiceActivationChanges().catch(console.error); return () => { cancelled = true; unlisten?.(); @@ -204,6 +217,18 @@ export function RecordingInputSection({ if (!prefs?.voiceActivationEnabled) setVoiceActivationModelMissing(false); }, [prefs?.voiceActivationEnabled]); + useEffect(() => { + if (isTauri || os !== 'win') return; + let cancelled = false; + void getVoiceActivationState() + .then((snapshot) => { + if (!cancelled) setVoiceActivationState(snapshot); + }) + .catch(console.error); + return () => { cancelled = true; }; + // The confirmed preference snapshot also changes after an optimistic write. + }, [os, prefs]); + const saveKeyboardListAffectingPrefs = useCallback( async (nextPrefs: UserPreferences) => { try { @@ -627,6 +652,23 @@ export function RecordingInputSection({
{t('settings.recording.voiceActivationLocalOnly')}
+ + savePrefs({ ...prefs, voiceActivationHotkeyLinked: next })} + /> + + {prefs.voiceActivationHotkeyLinked && ( +
+ {voiceActivationState + ? t(voiceActivationStatusKey(voiceActivationState)) + : t('settings.recording.voiceActivationStarting')} + {voiceActivationState?.message && ` — ${voiceActivationState.message}`} +
+ )} {voiceActivationModelMissing && (