diff --git a/docs/research/issue-1238.md b/docs/research/issue-1238.md new file mode 100644 index 000000000..69ff1b704 --- /dev/null +++ b/docs/research/issue-1238.md @@ -0,0 +1,203 @@ +# Issue #1238:手改学词建议卡片靠近输入目标的调研 + +> 调研日期:2026-10-24。仅静态阅读源码、GitHub 元数据和官方 API 文档;没有运行应用、测试、编译或打包。 +> +> 工作分支 `HKLHaoBin/issue-1238` 的 HEAD 为 `1462581a0114ce43ec3822ccaaf20ad00c48c3a5`。主要研究基线固定为上游 `beta` 的 `38fbe1c082e782e566e13769e4675a183ccc812a`;下列源码引用均指向该提交,除非明确写为旧 HEAD。 + +## 1. 结论摘要 + +**可以实现,但不是把右下角的两个偏移量改成光标坐标那么简单。** 现有链路保存了插入目标身份、文本范围和建议 ID,却没有把“属于本次插入的屏幕位置”传到卡片展示端。[目标接口][target-port]、[观察接口][observation-port]、[建议结构][suggestion-types] + +推荐:**默认在本次输入目标附近显示非模态确认卡;优先可靠的文本插入光标矩形,失败时退到焦点输入框,再退到原目标屏幕的角落。** 卡片不抢焦点、不覆盖正在编辑的行,不持续追逐鼠标或每次按键;保留逐条确认、过期不入库。位置偏好“输入位置附近 / 屏幕角落”可以作为可选设置,但不能替代正确的数据关联与降级链路。 + +这是交互改进,不是已证实的识别或学习逻辑故障。原卡片源码注释明确说明右下角是为了避免挡住编辑行;改到输入附近时仍须解决遮挡问题,不能只提高可见性。[原设计与前端行为][card-ui] + +## 2. 需求、历史与研究基线 + +- [Issue #1238][issue] 报告 Windows 11、`2.0.1+build.20261016`、粘贴上屏、开启手改学词;微信中观察和卡片正常,问题是卡片离输入位置太远,短暂出现时容易错过。第一选择是靠近本次插入的光标/焦点元素,另一选择是提供位置偏好。 +- 调研时 issue 仍 OPEN,只有维护者“满足你。”的评论。查询 issue timeline 和按 `1238` 搜索 open/merged PR,未发现关联实现;这是本次查询结果,不是对所有分支的穷尽证明。[Issue][issue] +- Windows/Android 学词来自先前扩展工作。#1127 的关闭说明明确指出经集合 PR #1126 合入;#1128 本身 CLOSED 且无 merge commit,不能据此断言 Windows 学词未落地。[#1127][history-issue]、[#1126][history-release]、[#1128][history-pr] +- 当前分支明显早于 issue 报告版本。`git rev-list --left-right --count HEAD...38fbe1c0` 为 `2 / 132`,即当前分支独有 2 个提交、缺少上游 132 个提交。这里只 fetch 参考,未切换分支或合并。实施前应先有意识地处理基线差异,不能直接把旧 HEAD 当最新版本。 +- 旧 HEAD 前端是整卡固定 10 秒、建议变化重置 timer、到期全量 dismiss;固定上游已改成每条建议的绝对 deadline 和按 ID 过期。不要从旧文件重新实现旧超时机制。[旧前端][old-card-ui]、[上游前端][card-ui] + +## 3. 已确认的源码现状 + +### 3.1 固定角落的直接原因 + +`tauri_coordinator_host.rs:1568-1591` 的 `position_vocab_card(width, height, edge_margin)` 只读取**卡片所复用窗口的 `current_monitor()`**,将 monitor 的 physical size/position 除以 scale,然后用 `LogicalPosition` 设置右下角:横向减卡片宽度及 24 的边距,纵向减卡片高度及固定 80。既不接收光标/焦点元素,也不使用 monitor work area。[定位函数][position] + +当前不是“光标定位失败才回退”,而是**完全没有光标定位分支**。`current_monitor()` 也不能证明这就是本次输入目标所在屏幕。混合 DPI 与任务栏避让需随改动覆盖,但本次没有实机证据证明某个具体 DPI 组合已发生偏移。[定位函数][position] + +### 3.2 插入目标捕获不等于位置捕获 + +Core 的 `CapturedTextTarget` 只组合插入器与 host-context reader;`HostContextCapture` 只有应用标识和光标附近的文本上下文。`capture_target()` 注释要求同步路径只冻结身份,不应为定位加入慢 UIA 查询。[目标接口][target-port]、[插入与观察接口][observation-port] + +Tauri `capture_target()` 捕获 `SelectionInsertionTarget` 和 Windows IME target,留在插入器中,没有 caret rectangle 或屏幕 anchor。随后 `TauriEditObservationAdapter::arm()` 只传插入文本、时长和 `EditPair` callback,未把捕获的插入目标交给 watcher。[目标捕获][adapter-target]、[观察适配器][adapters] + +所以 HWND、process/thread ID、UIA identity 或 UTF-16 offset 都不能当屏幕坐标。“插入器持有目标”没有解决“建议卡知道原目标在哪里”。 + +### 3.3 Windows 有文本 caret 能力,没有卡片几何链 + +`host_document/windows.rs:529-613` 的观察器绑定 focused UIA element,拒绝密码框和不允许的进程,检查前台 HWND 与 focused element 是否仍一致,监听 Text/Value 变化,静默约 700ms 后读文本并分析插入文本范围。事件 handler 只标 dirty,观察和清理在 worker 中完成。[Windows 观察器][windows-observer] + +同文件 `626-699` 的另一条光标上下文读取路径尝试 `TextPattern2::GetCaretRange` 并检查 `isActive`;兼容路径是单个 collapsed selection。返回的是 **UIA 文本 range**,不是光标屏幕矩形,可参考但不能直接传给 `set_position`。[caret 查找][windows-caret] + +窗口关闭或切换控件已有观察保护,但 worker 检查、发布建议到主线程显示仍有间隔。新定位须处理自身快照失效,不能把“观察时有效”当成“显示时永远有效”。[观察器][windows-observer]、[展示末端检查][coordinator] + +### 3.4 建议拥有内容和时效,没有来源位置 + +`PendingCorrection` 只有 `id`、`expires_at_ms`、`pattern`、`replacement`,最多 5 条。学习设置默认观察 60 秒、展示 10 秒,规范化范围分别为 10–60 秒和 5–60 秒;不是硬编码“所有版本都展示几十秒”。[建议与设置][suggestion-types] + +Core sink 有 observation generation,生成建议再次核对 generation,分配 UUID、计算绝对 deadline、发布 `VocabularySuggestionsChanged`。但事件 session 参数为 `None`,payload 没有 observation/session/placement 关联键。`EditObservationSink::publish` 只返回 bool,也不把新 suggestion ID 交回 host。[生成建议][queue]、[arm/disarm][arm] + +**不能只增加 `suggestion_id -> anchor` 表就认为链路接通。** 还须让 host 知道建议属于哪个观察代次,或从 Core 获得创建结果;现有 bool callback 和纯建议列表没有这个关联。 + +### 3.5 卡片共用录音胶囊窗口 + +`coordinator.rs:66-187` 将卡片宽度设为 320 logical points,高度随条数变化。显示时缩小 capsule 窗口、开启点击命中、定位、失效布局缓存后显示;主线程末端检查听写 Idle 等活动会话状态,避免覆盖录音胶囊。关闭时隐藏并恢复 capsule hit-test。[展示生命周期][coordinator] + +事件桥重新读取当前 pending corrections,不直接使用可能迟到的事件 snapshot,以免已关闭的卡片复活。位置元数据也应遵循“读当前状态并校验代次”。[事件桥][event-bridge] + +上游前端逐条 accept/reject、按 deadline 到期,点击先本地隐藏,提交失败恢复;按钮 `onMouseDown.preventDefault()` 用于减少点击夺焦点。Windows 原生显示优先 `SW_SHOWNOACTIVATE`/`SWP_NOACTIVATE`,失败会回退普通 `window.show()`;不能仅凭静态代码宣称所有路径都绝不激活窗口。[前端][card-ui]、[原生显示][show-window] + +### 3.6 可复用能力与平台边界 + +- `capsule_mirrors.rs:27-99` 的 `MonitorRect` 已统一 virtual-desktop physical coordinates、工作区、scale 和 monitor identity;`256-275` 有 logical size 转目标屏 physical size、work-area clamp 的思路。应复用类型、转换与边界处理,**不是直接使用底部居中的 capsule placement 结果**。[多屏类型][monitors]、[placement][monitor-placement] +- macOS 的 cfg-gated helper 已有 focused-input/caret anchor 和 AX 范围读取,可参考平台接口;其坐标体系不能直接复制到 Windows。[macOS helper][macos-anchor] +- 本次聚焦 Windows 桌面建议卡,不要求新增 Android 悬浮窗功能,不改变识别模型、学词规则、批量接受策略或所有平台普通胶囊的位置。 + +## 4. 官方 API 核查:必须保留失败路径 + +### 4.1 UIA range 不保证能得到矩形 + +Microsoft `GetCaretRange` 返回零长度 range,`isActive` 表示文本控件是否拥有键盘焦点;为 false 时不能当成当前系统 caret。[GetCaretRange][ms-caret] + +`GetBoundingRectangles` 明确指出:degenerate/empty range、完全屏外、滚出可视区或被遮挡时会返回空数组。因此 **`GetCaretRange -> GetBoundingRectangles` 不能写成必定成功的通用方案**。可以尝试并验证结果,空结果必须正常降级。[GetBoundingRectangles][ms-rectangles] + +若用相邻字符的非空 range 估算位置,须明确只是近似,并验证文末、空行、多行、RTL、选区。本次未验证此算法,不列为首版必做。焦点元素矩形是粗定位,不能把整个编辑区中心冒充 caret。 + +### 4.2 Win32 caret 不能直接上屏 + +`GUITHREADINFO.rcCaret` 是相对 `hwndCaret` 的 client coordinates;`GetGUIThreadInfo` 说明其为所属窗口意义上的 logical coordinates,不会自动转换成调用线程的 DPI 模式。窗口失去激活时 handle 可能无效;文本方向和 padding 也影响实际插入点。[GUITHREADINFO][ms-gui-struct]、[GetGUIThreadInfo][ms-gui] + +可用此路径补充原生 edit control,但必须校验原目标、handle、矩形,显式完成客户区到屏幕及 DPI 语义转换,不能盲目乘一个 scale,也不能把当前线程 `GetCaretPos` 当跨进程目标 caret。 + +### 4.3 物理屏幕坐标与工作区 + +Microsoft 明确 UIA 不使用 logical coordinates,相关 bounding rectangle 是 physical。仓库锁定的 Tauri 2.12.4 中,Monitor position/size/work_area 也是 physical 类型。[UIA 缩放][ms-scaling]、[Tauri Monitor][tauri-monitor] + +推荐 Windows host 统一输出**带符号的 virtual-desktop physical rectangle**,用 anchor 所在屏幕的 scale 转换卡片 logical size/间隙,按该屏 work area 翻转、clamp,再设置 physical position。不得对整个多屏桌面坐标统一除以一个屏幕 scale;负坐标合法。Tauri 已有 `monitor_from_point` 及对应 position/size 类型接口。[Tauri 选屏][tauri-screen]、[定位接口][tauri-position] + +### 4.4 线程、权限与隐私 + +Microsoft 建议通用 UIA 客户端在无窗口的独立 MTA 线程调用 UIA,同线程注册/移除事件;跨线程 marshal 的元素受原 apartment 生命周期影响。[UIA threading][ms-threading] + +不要把 UIA COM object/text range 存进前端 payload 或全局位置缓存,不要在 Tauri 主线程同步读宿主几何。只传纯数据矩形、来源、时间与关联键,复用密码框/进程门禁。几何读取失败不应阻断插入或学词确认;无需注入目标进程、OCR、额外常驻全局监听或上传宿主文本。 + +## 5. 推荐接入点与实现边界 + +以下只是后续建议,未修改业务代码。 + +### 5.1 先贯通数据关联,再移动窗口 + +1. **冻结身份**:保留 session 开始时的目标捕获,不在该同步接口增加慢 UIA round-trip。 +2. **捕获位置**:插入完成、准备观察时在 worker 对照原目标捕获可选 caret/input rectangle。检测到有效手改且仍是同一目标时可更新一次,避免 60 秒内窗口移动后一直用旧坐标。快照包括目标身份、物理矩形、来源、时间和 observation 关联键。 +3. **贯通关联键**:扩展 adapter/sink/呈现元数据,让建议带可移植的 observation/session 关联键;平台几何可留 Tauri 短生命周期 registry。另一方式是 publish 返回结构化创建结果,再按 suggestion ID 建表。应结合其他 host 兼容性选最小改动,不能用全局“最近一次位置”代替关联。 +4. **显示前复核**:读取当前建议与同源位置,主线程窗口操作前检查 presentation generation、deadline、活动会话;旧建议/矩形/取消 session 不能覆盖新卡片。 +5. **清理**:accept/reject、过期、新 session、关闭学习、目标失效时清理对应位置。几何不写词典或云同步数据。 + +主要接入点:`ports.rs`、`api.rs`、必要的 shared/event metadata、`core_adapters.rs`、`host_document/windows.rs`、`coordinator.rs`、`tauri_coordinator_host.rs`。确定关联方式后再收敛实际文件,不预设全部都必须改。 + +### 5.2 定位与降级 + +- **优先**:同源且有效的 caret rectangle,下方留间隙,空间不足翻上方,必要时侧移。避开编辑行与已知输入法候选区域;无法可靠判定候选区域时不宣称完全无重叠。 +- **其次**:同源焦点输入元素的可见矩形。过大编辑区只作粗定位,必要时退角落,避免覆盖主编辑区域。 +- **再退**:原目标窗口所在屏幕的 work-area 角落。原目标关闭/失焦时不去读另一应用文本为旧建议“跟随”,可退角落或关闭该次呈现。 +- **最终**:原屏断开或窗口位置不可用时,按定义好的屏幕回退策略选可用屏并 clamp。鼠标只作选屏兜底,不能标为 caret 定位成功。 +- 每次显示/更新一批建议决定位置,不因每次按键重新移动卡片,避免用户追着确认按钮跑;必要更新也受同一 presentation generation 约束。 + +### 5.3 配置取舍 + +首版可先完成自动靠近输入目标与可靠角落降级。如同时加偏好,建议“输入位置附近(默认)/屏幕角落”,明确默认值、旧配置迁移及说明。普通胶囊“光标所在屏幕”只决定 monitor,不应复用成卡片 placement preference;文本插入光标与鼠标指针不能混为一谈。 + +偏好若进入全局 Preferences,还须覆盖持久化、前后端类型、设置页和国际化;不能为绕开这些成本就把原角落直接换成鼠标位置。 + +## 6. 非模态交互契约 + +- **需求与风险**:用户仍在另一应用编辑,确认一条纠错是否入词典;未经确认不写入,超时只是放弃本次建议,不加入拒绝名单。[现有语义][suggestion-types] +- **组件**:复用有操作的非模态确认卡,语义接近贴对象的简短 Popconfirm/建议气泡,不是纯 Tooltip 或“保存成功”Toast。无需 Modal、遮罩、焦点陷阱,不能因靠近光标而阻断输入。 +- **焦点**:自动出现不激活窗口、不移动输入焦点;点击记住/忽略后可继续在原输入框打字。原生 no-activate fallback 需单独实测。[显示路径][show-window] +- **动作**:“要记住这个改法吗”及 `原词 → 新词`,按钮明确为“记住 / 忽略”;保持逐条接受,不新增“全部接受”。提交失败恢复该项并给予可理解反馈,不把点击当保存成功。 +- **生命周期**:继续按每条绝对 deadline 过期,移动/追加不延长旧条目。新录音释放共享窗口给 capsule。外侧点击不阻止编辑,不全局劫持 Escape/Enter,以免吞掉宿主或输入法按键。[前端][card-ui]、[共享窗口][coordinator] +- **可访问性**:不假定原生不激活卡片能自然参与其他应用 Tab 顺序;显式键盘入口/替代确认路径需设计与实测,不能用强制夺焦点换取键盘可达。保留按钮可访问名称,长词可查看全文。 +- **时长**:最小范围是位置。延长展示、悬停暂停、待处理中心属于额外产品决策,须与 Core 绝对 deadline 一致;不能只停前端 timer,使按钮仍显示但后台已拒绝接受。 + +## 7. 实施后的验收清单 + +以下均**待 GitHub Actions 或桌面实机验证**,不是本次运行结果。 + +- 微信、原生编辑框、浏览器 textarea/contenteditable、Electron:插入后手改,建议靠近原目标;不支持 caret provider 时退输入框/角落,不阻断学词。 +- 100%/125%/150%/200%、混合 DPI、多屏左/上负坐标、任务栏移边:完整落在目标屏 work area、大小正确、边缘翻转。 +- 空框、文末/空行、多行、滚动、非折叠选区、RTL:无 `(0,0)` 假 caret,不把整控件当精确插入点,按约定回退。 +- 改词后切应用/控件、关窗口、拖窗口、断显示器:不跟到无关应用,不使用失效 anchor、不掉屏外。 +- 卡片出现/点击后继续输入:前台窗口与文本焦点不被意外改变;覆盖 no-activate 正常与 fallback 路径,候选框/编辑行不被严重遮挡。 +- 新听写:旧显示任务不覆盖新 capsule,卡片结束恢复尺寸、布局缓存与鼠标穿透。 +- 连续纠错、多条先后到期、接受/忽略与新事件交错:位置匹配当前来源,旧 timer/observer 不清掉或移动新卡片。 +- 密码框、禁用进程、关闭学习、权限不足、UIA 超时:隐私门禁保持、失败快速或降级,不阻塞 UI 线程、不额外记录文本。 +- 如加偏好:旧配置加载、切换生效时机、重启持久化、普通胶囊设置无回归。 +- 自动化重点:纯几何 flip/clamp、来源 ID 与清理、事件乱序、逐条 deadline 和失败恢复。不在本地编译或跑测试。 +- 后续代码推送后,按实际分支与准确 SHA 触发 GitHub Actions CI、桌面及 Android 产物构建;先提供桌面产物。本次无代码推送,未触发构建。 + +## 8. 完成状态与证据边界 + +已完成:issue/历史核查、固定上游源码链路、Windows/Tauri 官方定位及线程契约、实现接入点、交互与验收建议。 + +未执行:应用复现、具体微信版本的 UIA provider 探测、混合 DPI 实机验证、功能修改、配置实现、提交/推送/PR/评论、CI/产物构建。provider 是否提供精确 caret rectangle 仍须实测,不承诺所有宿主都能精确跟随。 + +[issue]: https://github.com/Open-Less/openless/issues/1238 +[history-issue]: https://github.com/Open-Less/openless/issues/1127#issuecomment-5912873585 +[history-release]: https://github.com/Open-Less/openless/pull/1126 +[history-pr]: https://github.com/Open-Less/openless/pull/1128 +[position]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/tauri_coordinator_host.rs#L1568-L1591 +[target-port]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/ports.rs#L34-L55 +[observation-port]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/ports.rs#L794-L860 +[suggestion-types]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/shared_types.rs#L257-L347 +[adapters]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/core_adapters.rs#L3834-L3866 +[adapter-target]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/core_adapters.rs#L4032-L4046 +[windows-observer]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/host_document/windows.rs#L529-L613 +[windows-caret]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/host_document/windows.rs#L626-L699 +[queue]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/api.rs#L2669-L2765 +[arm]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/crates/openless-core/src/api.rs#L7304-L7368 +[coordinator]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/coordinator.rs#L66-L187 +[event-bridge]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/tauri_events.rs#L467-L476 +[card-ui]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src/components/VocabSuggestionCard.tsx#L1-L199 +[old-card-ui]: https://github.com/Open-Less/openless/blob/1462581a0114ce43ec3822ccaaf20ad00c48c3a5/openless-all/app/src/components/VocabSuggestionCard.tsx#L23-L47 +[show-window]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/tauri_coordinator_host.rs#L511-L614 +[monitors]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/capsule_mirrors.rs#L27-L99 +[monitor-placement]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/capsule_mirrors.rs#L256-L275 +[macos-anchor]: https://github.com/Open-Less/openless/blob/38fbe1c082e782e566e13769e4675a183ccc812a/openless-all/app/src-tauri/src/lib.rs#L2438-L2621 +[ms-caret]: https://learn.microsoft.com/en-us/windows/win32/api/uiautomationclient/nf-uiautomationclient-iuiautomationtextpattern2-getcaretrange +[ms-rectangles]: https://learn.microsoft.com/en-us/windows/win32/api/uiautomationclient/nf-uiautomationclient-iuiautomationtextrange-getboundingrectangles +[ms-gui-struct]: https://learn.microsoft.com/en-us/windows/win32/api/winuser/ns-winuser-guithreadinfo +[ms-gui]: https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-getguithreadinfo +[ms-scaling]: https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-screenscaling +[ms-threading]: https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-threading +[tauri-monitor]: https://github.com/tauri-apps/tauri/blob/tauri-v2.12.4/crates/tauri/src/window/mod.rs#L55-L102 +[tauri-screen]: https://github.com/tauri-apps/tauri/blob/tauri-v2.12.4/crates/tauri/src/window/mod.rs#L1695-L1702 +[tauri-position]: https://github.com/tauri-apps/tauri/blob/tauri-v2.12.4/crates/tauri/src/window/mod.rs#L1923-L1968 + +## 9. 功能实施(2026-10-24) + +本节更新实施状态;上面的“未修改功能代码”描述的是调研阶段。 + +- 实施分支:`feat/issue-1238-vocab-card-anchor`,直接基于上游 `38fbe1c0`,不包含本地 beta 的独有提交。 +- Core 建议携带可选 observation ID,sink 暴露冻结的 session/observation 身份;手动入队不带来源,原有确认、去重和绝对到期语义保持不变。 +- Tauri managed state 只保留有界、短生命周期的 session 目标和纯数据位置;待交接目标不因 arm 前的 disarm 被清掉,旧代不能覆盖新代位置。 +- Windows 在现有 MTA 观察器中验证原始目标,尝试 UIA caret 矩形,空结果退焦点输入元素;几何失败不阻断正常学词。显示前再次轻量验证目标/窗口,失效时退源屏角落,不改抓新焦点。 +- 卡片使用目标屏物理坐标、DPI 与工作区,空间不足向上翻转;移动/缩放使用 `SWP_NOACTIVATE`。卡片点击处理 native `WM_MOUSEACTIVATE`,严格显示路径失败不回退 activating show,不改变普通胶囊策略。 +- 共享窗口的 card mode 停止常规 layout watcher;show/hide 按 presentation epoch 丢弃旧任务,末端重读建议和活动会话。 +- 新增 std-only 定位/registry/epoch 回归,并由独立 `backend-tests` 入口在 Windows Actions 实际执行。当前尚未运行这些测试或构建;待 Bugbot 审查后提交推送,使用该分支准确 SHA 触发 CI、桌面及 Android 产物。 +- 按追加需求,在“实验与拓展 → 手改学词/修改学习”现有子页增加 Windows 建议卡位置选择,不在普通设置新增入口。可选“输入位置附近”(默认)和“固定右下角”,后者跳过近端 anchor,仍使用原输入目标所在屏幕的工作区。旧配置缺少字段时默认靠近输入;恢复默认参数同时恢复此选项。更改后按现有规则结束观察、清除待确认建议,下次听写生效。 +- 不新增 Android UI,不改变其他平台的卡片定位。微信实际 provider 能力、候选框重叠和真实焦点行为仍是安装桌面包后的实机验收项目,不以纯逻辑测试代替。 + +审查记录:Review Bugbot 首轮两个 MEDIUM(重复 hide 清理、长听写 target 过期)已修复并补回归,复审明确返回 `No bugs found`;追加的位置选项审查也返回 `No bugs found`。首轮远程 CI/Android 完整日志揭示测试 fixture 漏初始化 epoch、Windows 专用目标校验未按平台编译两处问题,已补齐初始化并限定 Windows 编译;将复审后为新提交重新触发三项工作流。远程通过不替代上述实机验收。 diff --git a/openless-all/app/crates/openless-core/src/api.rs b/openless-all/app/crates/openless-core/src/api.rs index 3402ab525..f370d2660 100644 --- a/openless-all/app/crates/openless-core/src/api.rs +++ b/openless-all/app/crates/openless-core/src/api.rs @@ -2667,6 +2667,7 @@ struct HistoryProviderAttribution { } struct CoreEditObservationSink { + session_id: SessionId, settings: crate::shared_types::VocabularyLearningSettings, expected_generation: u64, generation: Arc, @@ -2676,6 +2677,14 @@ struct CoreEditObservationSink { } impl EditObservationSink for CoreEditObservationSink { + fn observation_id(&self) -> Option { + Some(self.expected_generation) + } + + fn session_id(&self) -> Option { + Some(self.session_id) + } + fn observation_duration(&self) -> std::time::Duration { std::time::Duration::from_secs(self.settings.observation_seconds as u64) } @@ -2751,6 +2760,7 @@ fn queue_pending_correction_state( } let suggestion = PendingCorrection { id: uuid::Uuid::new_v4().to_string(), + observation_id: generation.map(|(_, expected)| expected), expires_at_ms: now + i64::from(suggestion_seconds.clamp(5, 60)) * 1000, pattern, replacement, @@ -7342,6 +7352,7 @@ impl OpenLessBackend { } let expected_generation = self.edit_observation_generation.load(Ordering::Acquire); let sink = Arc::new(CoreEditObservationSink { + session_id, settings: self .get_preferences() .vocabulary_learning_settings @@ -9517,6 +9528,10 @@ mod tests { assert_eq!(prefs.vocabulary_learning_settings.observation_seconds, 10); assert_eq!(prefs.vocabulary_learning_settings.suggestion_seconds, 60); assert_eq!(prefs.vocabulary_learning_settings.max_phrase_chars, 32); + assert_eq!( + prefs.vocabulary_learning_settings.suggestion_position, + crate::shared_types::VocabularySuggestionPosition::NearInput + ); let restored: UserPreferences = serde_json::from_value(serde_json::to_value(&prefs).unwrap()).unwrap(); assert_eq!( @@ -9525,6 +9540,100 @@ mod tests { ); } + #[test] + fn vocabulary_suggestion_position_roundtrips_and_defaults_for_old_settings() { + use crate::shared_types::{VocabularyLearningSettings, VocabularySuggestionPosition}; + let old: VocabularyLearningSettings = serde_json::from_value(serde_json::json!({ + "observationSeconds": 20, "suggestionSeconds": 30, "maxPhraseChars": 16 + })) + .unwrap(); + assert_eq!( + old.suggestion_position, + VocabularySuggestionPosition::NearInput + ); + for (value, expected) in [ + ("nearInput", VocabularySuggestionPosition::NearInput), + ("bottomRight", VocabularySuggestionPosition::BottomRight), + ] { + let prefs: UserPreferences = serde_json::from_value(serde_json::json!({ + "vocabularyLearningSettings": { "suggestionPosition": value } + })) + .unwrap(); + assert_eq!( + prefs.vocabulary_learning_settings.suggestion_position, + expected + ); + let encoded = serde_json::to_value(&prefs).unwrap(); + assert_eq!( + encoded["vocabularyLearningSettings"]["suggestionPosition"], + value + ); + let restored: UserPreferences = serde_json::from_value(encoded).unwrap(); + assert_eq!( + restored.vocabulary_learning_settings.suggestion_position, + expected + ); + } + } + + #[test] + fn changing_only_vocabulary_card_position_invalidates_pending_and_stale_sink() { + use crate::shared_types::VocabularySuggestionPosition; + let (backend, _) = backend(); + let mut prefs = backend.get_preferences(); + prefs.vocabulary_learning_enabled = true; + backend + .update_settings( + prefs, + crate::SettingsUpdateOptions::STRICT, + &crate::NoopSettingsRuntime, + ) + .unwrap(); + let before = backend.get_preferences(); + let sink = CoreEditObservationSink { + session_id: SessionId::new(), + settings: before.vocabulary_learning_settings, + expected_generation: backend.edit_observation_generation.load(Ordering::Acquire), + generation: Arc::clone(&backend.edit_observation_generation), + typed_text: "ZIP".into(), + pending: Arc::clone(&backend.pending_corrections), + events: Arc::clone(&backend.events), + }; + let edit = crate::host_document::minimal_edit("ZIP", "VIP").unwrap(); + assert!(sink.publish(edit.clone())); + assert!(!backend.pending_corrections().is_empty()); + let mut prefs = before.clone(); + prefs.vocabulary_learning_settings.suggestion_position = + VocabularySuggestionPosition::BottomRight; + backend + .update_settings( + prefs, + crate::SettingsUpdateOptions::STRICT, + &crate::NoopSettingsRuntime, + ) + .unwrap(); + assert!(backend.pending_corrections().is_empty()); + assert!(!sink.publish(edit)); + let saved = backend.get_preferences(); + assert!(saved.vocabulary_learning_enabled); + assert_eq!( + saved.vocabulary_learning_settings.suggestion_position, + VocabularySuggestionPosition::BottomRight + ); + assert_eq!( + saved.vocabulary_learning_settings.observation_seconds, + before.vocabulary_learning_settings.observation_seconds + ); + assert_eq!( + saved.vocabulary_learning_settings.suggestion_seconds, + before.vocabulary_learning_settings.suggestion_seconds + ); + assert_eq!( + saved.vocabulary_learning_settings.max_phrase_chars, + before.vocabulary_learning_settings.max_phrase_chars + ); + } + #[test] fn vocabulary_learning_config_change_invalidates_pending_and_stale_sink() { let (backend, _) = backend(); @@ -9541,6 +9650,7 @@ mod tests { ) .unwrap(); let sink = CoreEditObservationSink { + session_id: SessionId::new(), settings: backend.get_preferences().vocabulary_learning_settings, expected_generation: backend.edit_observation_generation.load(Ordering::Acquire), generation: Arc::clone(&backend.edit_observation_generation), @@ -9589,6 +9699,7 @@ mod tests { fn observed_spelling_correction_saves_only_the_complete_confirmed_word() { let (backend, _) = backend(); let sink = CoreEditObservationSink { + session_id: SessionId::new(), settings: backend.get_preferences().vocabulary_learning_settings, expected_generation: backend.edit_observation_generation.load(Ordering::Acquire), generation: Arc::clone(&backend.edit_observation_generation), @@ -9601,6 +9712,8 @@ mod tests { )); let suggestions = backend.pending_corrections(); assert_eq!(suggestions.len(), 2); + assert_eq!(suggestions[0].observation_id, sink.observation_id()); + assert_eq!(suggestions[1].observation_id, sink.observation_id()); assert_eq!(suggestions[0].pattern, "ZIP"); assert_eq!(suggestions[0].replacement, "VIP"); assert_eq!(suggestions[1].pattern, "codx"); @@ -12063,6 +12176,8 @@ mod tests { .queue_pending_correction("扣的爱思".into(), "Codex".into()) .unwrap() .unwrap(); + assert_eq!(first.observation_id, None); + assert!(serde_json::to_value(&first).unwrap().get("observationId").is_none()); assert!(backend .queue_pending_correction("扣的爱思".into(), "Codex".into()) .unwrap() diff --git a/openless-all/app/crates/openless-core/src/ports.rs b/openless-all/app/crates/openless-core/src/ports.rs index 66dfa69f4..416c6d05c 100644 --- a/openless-all/app/crates/openless-core/src/ports.rs +++ b/openless-all/app/crates/openless-core/src/ports.rs @@ -841,6 +841,14 @@ pub trait TextInsertionSession: Send + Sync { /// document baseline. Keeping that decision here prevents macOS AX code from /// owning vocabulary policy or accepting a report from a stale generation. pub trait EditObservationSink: Send + Sync { + fn observation_id(&self) -> Option { + None + } + + fn session_id(&self) -> Option { + None + } + fn observation_duration(&self) -> std::time::Duration { std::time::Duration::from_secs(60) } diff --git a/openless-all/app/crates/openless-core/src/shared_types.rs b/openless-all/app/crates/openless-core/src/shared_types.rs index ddb570f96..03027a5a4 100644 --- a/openless-all/app/crates/openless-core/src/shared_types.rs +++ b/openless-all/app/crates/openless-core/src/shared_types.rs @@ -265,6 +265,9 @@ pub use crate::types::{CorrectionRule, RuleSource}; #[serde(rename_all = "camelCase")] pub struct PendingCorrection { pub id: String, + /// Host presentation correlation only; absent for manually queued suggestions. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub observation_id: Option, /// Absolute deadline shared by native and web confirmation surfaces. pub expires_at_ms: i64, /// The (wrong) pre-correction spelling. Only shown on the card so the @@ -319,6 +322,15 @@ pub const INSERT_FALLBACK_REASON_INSERT_FAILED: &str = "insertFailed"; /// having no reject list. pub const VOCAB_SUGGESTION_TTL_MS: u64 = 10_000; +/// Windows suggestion-card placement; other platforms retain their native presentation. +#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum VocabularySuggestionPosition { + #[default] + NearInput, + BottomRight, +} + /// Local-only tuning. Consent and sensitive-field protections remain separate. #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[serde(default, rename_all = "camelCase")] @@ -326,6 +338,7 @@ pub struct VocabularyLearningSettings { pub observation_seconds: u32, pub suggestion_seconds: u32, pub max_phrase_chars: u32, + pub suggestion_position: VocabularySuggestionPosition, } impl Default for VocabularyLearningSettings { @@ -334,6 +347,7 @@ impl Default for VocabularyLearningSettings { observation_seconds: 60, suggestion_seconds: 10, max_phrase_chars: 12, + suggestion_position: VocabularySuggestionPosition::default(), } } } @@ -344,6 +358,7 @@ impl VocabularyLearningSettings { observation_seconds: self.observation_seconds.clamp(10, 60), suggestion_seconds: self.suggestion_seconds.clamp(5, 60), max_phrase_chars: self.max_phrase_chars.clamp(2, 32), + suggestion_position: self.suggestion_position, } } } diff --git a/openless-all/app/src-tauri/backend-tests/tests/vocab_card_anchor.rs b/openless-all/app/src-tauri/backend-tests/tests/vocab_card_anchor.rs new file mode 100644 index 000000000..cb4ed7919 --- /dev/null +++ b/openless-all/app/src-tauri/backend-tests/tests/vocab_card_anchor.rs @@ -0,0 +1,3 @@ +// Exercise production geometry and observation ownership without Tauri or native ASR DLLs. +#[path = "../../src/vocab_card_placement.rs"] +mod vocab_card_placement; diff --git a/openless-all/app/src-tauri/src/coordinator.rs b/openless-all/app/src-tauri/src/coordinator.rs index 9c2928a0e..852b4fbc3 100644 --- a/openless-all/app/src-tauri/src/coordinator.rs +++ b/openless-all/app/src-tauri/src/coordinator.rs @@ -90,17 +90,22 @@ const VOCAB_CARD_EDGE_MARGIN: f64 = 24.0; /// So the window shrinks to the card's actual size while shown, and is /// restored when hidden. fn show_vocab_suggestion_card(inner: &Arc) { - let pending = inner.backend.pending_corrections(); - if pending.is_empty() { - return; - } + let epoch = inner.vocab_card_epoch.next(); let Some(capsule) = inner.host.capsule_window() else { return; }; - let height = VOCAB_CARD_CHROME_HEIGHT + VOCAB_CARD_ROW_HEIGHT * pending.len() as f64; let inner_for_main = Arc::clone(inner); let _ = capsule.run_on_main_thread(move |capsule| { let inner = inner_for_main; + if !inner.vocab_card_epoch.is_current(epoch) { + return; + } + let pending = inner.backend.pending_corrections(); + if pending.is_empty() { + return; + } + let height = VOCAB_CARD_CHROME_HEIGHT + VOCAB_CARD_ROW_HEIGHT * pending.len() as f64; + let observation_ids: Vec<_> = pending.iter().map(|item| item.observation_id).collect(); // Last gate: never show the card while dictation is not Idle. // // Upstream checks (observer generation, non-empty `pending_corrections`) @@ -125,17 +130,34 @@ fn show_vocab_suggestion_card(inner: &Arc) { #[cfg(not(mobile))] if let Err(e) = capsule.set_card_hit_test_mode() { log::warn!("[vocab-card] enabling card hit testing failed: {e}"); + hide_vocab_suggestion_card(&inner); + return; } + #[cfg(not(target_os = "windows"))] if let Err(e) = capsule.set_size(VOCAB_CARD_WIDTH, height) { log::warn!("[vocab-card] resize failed: {e}"); + hide_vocab_suggestion_card(&inner); + return; } - if let Err(e) = - capsule.position_vocab_card(VOCAB_CARD_WIDTH, height, VOCAB_CARD_EDGE_MARGIN) - { + if let Err(e) = capsule.position_vocab_card( + VOCAB_CARD_WIDTH, + height, + VOCAB_CARD_EDGE_MARGIN, + &observation_ids, + inner + .backend + .get_preferences() + .vocabulary_learning_settings + .suggestion_position, + ) { log::warn!("[vocab-card] position failed: {e}"); + hide_vocab_suggestion_card(&inner); + return; } if let Err(e) = capsule.refresh_card_hit_test() { log::warn!("[vocab-card] refreshing card hit testing failed: {e}"); + hide_vocab_suggestion_card(&inner); + return; } // Positioning ditto: the dedup cache in `maybe_position_capsule_bottom_center` // only tracks "monitor + translation state" and knows nothing about @@ -144,7 +166,14 @@ fn show_vocab_suggestion_card(inner: &Arc) { // and leave the capsule in the bottom-right corner the card moved it // to. capsule.invalidate_layout(); - capsule.show_for_recording(true); + if !capsule.show_vocab_card() { + log::warn!("[vocab-card] no-activate show failed; suppressing this presentation"); + let _ = capsule.hide(); + #[cfg(not(mobile))] + let _ = capsule.restore_capsule_hit_test_mode(); + inner.vocab_card_visible.store(false, Ordering::SeqCst); + inner.backend.dismiss_pending_corrections(); + } #[cfg(target_os = "macos")] capsule.restore_main_window_key_if_active(); }); @@ -160,14 +189,21 @@ fn show_vocab_suggestion_card(inner: &Arc) { /// the same window — the capsule flickers and the user thinks the hotkey is /// broken. fn hide_vocab_suggestion_card(inner: &Arc) { + let epoch = inner + .vocab_card_epoch + .request_hide(&inner.vocab_card_visible); inner.backend.dismiss_pending_corrections(); - if !inner.vocab_card_visible.swap(false, Ordering::SeqCst) { + let Some(epoch) = epoch else { return; - } + }; let Some(capsule) = inner.host.capsule_window() else { return; }; + let inner = Arc::clone(inner); let _ = capsule.run_on_main_thread(move |capsule| { + if !inner.vocab_card_epoch.is_current(epoch) { + return; + } // Hide before changing geometry: restoring size and position while the // window is visible can composite a frame with the card stretched wide // and flying across half the screen. @@ -184,6 +220,7 @@ fn hide_vocab_suggestion_card(inner: &Arc) { if let Err(e) = capsule.restore_capsule_geometry() { log::warn!("[vocab-card] restoring capsule geometry failed: {e}"); } + inner.vocab_card_visible.store(false, Ordering::SeqCst); }); } @@ -533,6 +570,7 @@ struct Inner { /// every dictation start would hide the capsule window and race /// `emit_capsule`'s show for the same window. vocab_card_visible: AtomicBool, + vocab_card_epoch: crate::vocab_card_placement::PresentationEpoch, hotkey: Mutex>, hotkey_status: Arc>, /// Webview fallback only: pairs one raw keydown/up edge with the same Core press id. @@ -728,6 +766,7 @@ impl Coordinator { overlay_qa_handoff: tokio::sync::Mutex::new(()), inserter: TextInserter::new(), vocab_card_visible: AtomicBool::new(false), + vocab_card_epoch: crate::vocab_card_placement::PresentationEpoch::default(), hotkey: Mutex::new(None), hotkey_status, window_hotkey_press_id: AtomicU64::new(0), @@ -868,6 +907,7 @@ impl Coordinator { overlay_qa_handoff: tokio::sync::Mutex::new(()), inserter: TextInserter::new(), vocab_card_visible: AtomicBool::new(false), + vocab_card_epoch: crate::vocab_card_placement::PresentationEpoch::default(), hotkey: Mutex::new(None), hotkey_status, window_hotkey_press_id: AtomicU64::new(0), diff --git a/openless-all/app/src-tauri/src/coordinator/hotkey_loops.rs b/openless-all/app/src-tauri/src/coordinator/hotkey_loops.rs index 8799a6b6a..8382b3b58 100644 --- a/openless-all/app/src-tauri/src/coordinator/hotkey_loops.rs +++ b/openless-all/app/src-tauri/src/coordinator/hotkey_loops.rs @@ -2515,6 +2515,7 @@ pub(crate) mod less_computer_test_support { overlay_qa_handoff: tokio::sync::Mutex::new(()), inserter: TextInserter::new(), vocab_card_visible: AtomicBool::new(false), + vocab_card_epoch: crate::vocab_card_placement::PresentationEpoch::default(), hotkey: Mutex::new(None), hotkey_status: Arc::new(Mutex::new(HotkeyStatus::default())), window_hotkey_press_id: AtomicU64::new(0), diff --git a/openless-all/app/src-tauri/src/core_adapters.rs b/openless-all/app/src-tauri/src/core_adapters.rs index b395a9efd..f79f1add5 100644 --- a/openless-all/app/src-tauri/src/core_adapters.rs +++ b/openless-all/app/src-tauri/src/core_adapters.rs @@ -343,7 +343,10 @@ pub(crate) fn backend_dependencies( dependencies.services.platform = Arc::new(TauriPlatformApi::new(Arc::clone(&app), hotkey_status)); dependencies.services.host_context = Arc::new(TauriHostContextAdapter::default()); - dependencies.services.edit_observation = Arc::new(TauriEditObservationAdapter::default()); + dependencies.services.edit_observation = Arc::new(TauriEditObservationAdapter { + app: Arc::clone(&app), + watcher: Mutex::new(None), + }); dependencies.services.experimental_asr = experimental_asr; dependencies.local_asr_runtime = Some(local_asr_runtime); #[cfg(not(mobile))] @@ -3843,7 +3846,67 @@ pub(crate) struct TauriTextInserter { } #[derive(Default)] +pub(crate) struct VocabPlacementState { + targets: Mutex< + crate::vocab_card_placement::TargetHandoff< + SessionId, + crate::selection::SelectionInsertionTarget, + >, + >, + anchors: Mutex, + observed_targets: Mutex>, +} + +impl VocabPlacementState { + fn remember_target( + &self, + session: SessionId, + target: crate::selection::SelectionInsertionTarget, + ) { + self.targets.lock().remember(session, target); + } + + fn take_target( + &self, + session: SessionId, + ) -> Option { + self.targets.lock().take(session) + } + + fn begin(&self, id: u64, target: Option<&crate::selection::SelectionInsertionTarget>) { + let mut targets = self.observed_targets.lock(); + targets.retain(|(key, _)| *key != id); + if let Some(target) = target { + targets.push((id, target.clone())); + } + if targets.len() > 8 { + targets.remove(0); + } + self.anchors.lock().begin(id); + } + + #[cfg(target_os = "windows")] + pub(crate) fn source_is_current(&self, id: u64) -> bool { + self.observed_targets + .lock() + .iter() + .find(|(key, _)| *key == id) + .is_some_and(|(_, target)| { + crate::selection::selection_insertion_target_is_current(target) + }) + } + + pub(crate) fn snapshot(&self, id: u64) -> Option { + self.anchors.lock().get(id) + } + + pub(crate) fn retain(&self, ids: &[u64]) { + self.anchors.lock().retain(ids); + } +} + struct TauriEditObservationAdapter { + app: AppHandleSlot, watcher: Mutex>, } @@ -3853,15 +3916,54 @@ impl EditObservationAdapter for TauriEditObservationAdapter { typed_text: String, sink: Arc, ) -> Result<(), BackendError> { + let app = self.app.lock().clone(); + let state = app + .as_ref() + .and_then(|app| app.try_state::()); + let observation_id = sink.observation_id(); + let target = sink + .session_id() + .and_then(|session| state.as_ref()?.take_target(session)); + if let (Some(state), Some(id)) = (state.as_ref(), observation_id) { + state.begin(id, target.as_ref()); + } + drop(state); + let update_app = app.clone(); *self.watcher.lock() = crate::host_document::watch_for_edits( typed_text, sink.observation_duration(), - move |edit| sink.publish(edit), + target, + move |edit, snapshot| { + if let (Some(app), Some(id)) = (update_app.as_ref(), observation_id) { + if let Some(state) = app.try_state::() { + let mut anchors = state.anchors.lock(); + match snapshot { + Some(snapshot) => { + anchors.update(id, snapshot); + } + None => anchors.invalidate(id), + } + } + } + sink.publish(edit) + }, + move || { + if let (Some(app), Some(id)) = (app.as_ref(), observation_id) { + if let Some(state) = app.try_state::() { + state.anchors.lock().invalidate(id); + } + } + }, ); Ok(()) } fn disarm(&self) { + if let Some(app) = self.app.lock().as_ref() { + if let Some(state) = app.try_state::() { + state.anchors.lock().stop(); + } + } *self.watcher.lock() = None; } } @@ -4066,6 +4168,12 @@ impl CoreTextInserter for TauriTextInserter { // whichever application happens to be focused at the end. let insertion_target = insertion_target .unwrap_or_else(crate::selection::capture_selection_insertion_target); + #[cfg(target_os = "windows")] + if let Some(handle) = app.lock().as_ref() { + if let Some(state) = handle.try_state::() { + state.remember_target(session_id, insertion_target.clone()); + } + } #[cfg(target_os = "macos")] let (app_handle, mut previous_input_source, streaming_ready) = { let app_handle = app.lock().clone().ok_or_else(|| { diff --git a/openless-all/app/src-tauri/src/host_document/mod.rs b/openless-all/app/src-tauri/src/host_document/mod.rs index e72221fd8..1a139b1d6 100644 --- a/openless-all/app/src-tauri/src/host_document/mod.rs +++ b/openless-all/app/src-tauri/src/host_document/mod.rs @@ -537,30 +537,46 @@ impl Drop for EditWatcher { /// /// `on_edit` is called on the observer thread, possibly multiple times. Returns `None` on any /// failure — failing to learn is acceptable; breaking typing is not. -pub fn watch_for_edits( +pub fn watch_for_edits( typed_text: String, lifetime: std::time::Duration, + target: Option, on_edit: F, + on_end: E, ) -> Option where - F: Fn(EditPair) -> bool + Send + Sync + 'static, + F: Fn(EditPair, Option) -> bool + + Send + + Sync + + 'static, + E: Fn() + Send + Sync + 'static, { + #[cfg(not(target_os = "windows"))] + let _ = (target, on_end); #[cfg(target_os = "macos")] { if typed_text.trim().is_empty() { return None; } - let stop = macos::spawn_edit_watcher(typed_text, lifetime, Box::new(on_edit))?; + let callback = move |edit| on_edit(edit, None); + let stop = macos::spawn_edit_watcher(typed_text, lifetime, Box::new(callback))?; Some(EditWatcher { stop }) } #[cfg(target_os = "windows")] { - windows::spawn_edit_watcher(typed_text, lifetime, Box::new(on_edit)) - .map(|stop| EditWatcher { stop }) + windows::spawn_edit_watcher( + typed_text, + lifetime, + target?, + Box::new(on_edit), + Box::new(on_end), + ) + .map(|stop| EditWatcher { stop }) } #[cfg(target_os = "android")] { - crate::android::edit_observation::arm(typed_text, lifetime, Box::new(on_edit)) + let callback = move |edit| on_edit(edit, None); + crate::android::edit_observation::arm(typed_text, lifetime, Box::new(callback)) .map(|generation| EditWatcher { generation }) } #[cfg(not(any(target_os = "macos", target_os = "windows", target_os = "android")))] diff --git a/openless-all/app/src-tauri/src/host_document/windows.rs b/openless-all/app/src-tauri/src/host_document/windows.rs index e95b2b5a7..72bbcd43b 100644 --- a/openless-all/app/src-tauri/src/host_document/windows.rs +++ b/openless-all/app/src-tauri/src/host_document/windows.rs @@ -1,6 +1,7 @@ //! Short-lived UIA observation. All registration and COM objects belong to one //! MTA worker; callbacks only mark text dirty and never read document contents. use super::EditPair; +use crate::vocab_card_placement::{AnchorSnapshot, Rect}; use openless_core::host_document::ObservedInsertion; use std::sync::{ atomic::{AtomicBool, Ordering}, @@ -23,7 +24,7 @@ use windows::Win32::{ UI::{Accessibility::*, WindowsAndMessaging::GetForegroundWindow}, }; -type Callback = Box bool + Send + Sync>; +type Callback = Box) -> bool + Send + Sync>; #[derive(Debug, Clone)] pub(super) struct VoiceEditTarget { @@ -377,6 +378,8 @@ pub(crate) fn insert_with_delivery_check( } struct Request { text: String, + target: crate::selection::SelectionInsertionTarget, + on_end: Box, window: isize, lifetime: Duration, stop: Arc, @@ -412,7 +415,9 @@ impl IUIAutomationPropertyChangedEventHandler_Impl for Changed_Impl { pub(super) fn spawn_edit_watcher( text: String, lifetime: Duration, + target: crate::selection::SelectionInsertionTarget, callback: Callback, + on_end: Box, ) -> Option> { if text.trim().is_empty() { return None; @@ -431,6 +436,7 @@ pub(super) fn spawn_edit_watcher( if !request.stop.load(Ordering::Acquire) { let _ = observe(&request); } + (request.on_end)(); } CoUninitialize(); }) @@ -443,6 +449,8 @@ pub(super) fn spawn_edit_watcher( .send(Request { text, lifetime, + target, + on_end, window: unsafe { GetForegroundWindow().0 as isize }, stop: stop.clone(), callback, @@ -526,7 +534,104 @@ unsafe fn read_text(element: &IUIAutomationElement) -> Result { Ok(text) } +fn physical_rect(rect: windows::Win32::Foundation::RECT) -> Option { + let rect = Rect { + left: rect.left, + top: rect.top, + right: rect.right, + bottom: rect.bottom, + }; + rect.valid().then_some(rect) +} + +unsafe fn caret_screen_rect(element: &IUIAutomationElement) -> Option { + use windows::Win32::System::Ole::{ + SafeArrayDestroy, SafeArrayGetElement, SafeArrayGetLBound, SafeArrayGetUBound, + }; + let CaretLookup::Found(range, _) = find_caret_range(element) else { + return None; + }; + let array = range.GetBoundingRectangles().ok()?; + if array.is_null() { + return None; + } + // A collapsed range can legitimately have no rectangles. Never expand it into + // unrelated text merely to manufacture a precise-looking caret position. + let rect = (|| { + let first = SafeArrayGetLBound(array, 1).ok()?; + let last = SafeArrayGetUBound(array, 1).ok()?; + if last.checked_sub(first)? != 3 { + return None; + } + let mut values = [0.0f64; 4]; + for (offset, value) in values.iter_mut().enumerate() { + let index = first + offset as i32; + SafeArrayGetElement(array, &index, value as *mut f64 as *mut _).ok()?; + } + Rect::from_xywh(values[0], values[1], values[2], values[3]) + })(); + let _ = SafeArrayDestroy(array); + rect +} + +unsafe fn capture_placement( + element: &IUIAutomationElement, + window: HWND, +) -> Option { + use windows::Win32::Foundation::RECT; + use windows::Win32::Graphics::Gdi::{ + GetMonitorInfoW, MonitorFromWindow, MONITORINFO, MONITOR_DEFAULTTONEAREST, + }; + use windows::Win32::UI::WindowsAndMessaging::{GetWindowRect, GetWindowThreadProcessId}; + let mut window_rect = RECT::default(); + GetWindowRect(window, &mut window_rect).ok()?; + let window_rect = physical_rect(window_rect)?; + let mut process_id = 0; + GetWindowThreadProcessId(window, Some(&mut process_id)); + if process_id == 0 { + return None; + } + let monitor = MonitorFromWindow(window, MONITOR_DEFAULTTONEAREST); + let mut info = MONITORINFO { + cbSize: std::mem::size_of::() as u32, + ..Default::default() + }; + let screen_rect = GetMonitorInfoW(monitor, &mut info) + .as_bool() + .then(|| physical_rect(info.rcMonitor)) + .flatten(); + let anchor = if element + .CurrentIsOffscreen() + .map(|value| value.as_bool()) + .unwrap_or(true) + { + None + } else { + caret_screen_rect(element).or_else(|| { + let rect = physical_rect(element.CurrentBoundingRectangle().ok()?)?; + // A whole document is not an insertion point. Large controls use the + // original monitor's corner instead of covering the editor itself. + let screen = screen_rect?; + let width = i64::from(rect.right) - i64::from(rect.left); + let height = i64::from(rect.bottom) - i64::from(rect.top); + (width > 0 && height * 2 < i64::from(screen.bottom) - i64::from(screen.top)) + .then_some(rect) + }) + }; + Some(AnchorSnapshot { + window: window.0 as usize, + process_id, + window_rect, + anchor, + screen_rect, + captured_at: Instant::now(), + }) +} + unsafe fn observe(request: &Request) -> Result<()> { + if !crate::selection::selection_insertion_target_is_current(&request.target) { + return Ok(()); + } let uia: IUIAutomation = CoCreateInstance(&CUIAutomation8, None, CLSCTX_INPROC_SERVER)?; let timeouts: IUIAutomation2 = uia.cast()?; timeouts.SetConnectionTimeout(200)?; @@ -537,6 +642,7 @@ unsafe fn observe(request: &Request) -> Result<()> { } let active = || -> Result { Ok(!request.stop.load(Ordering::Acquire) + && crate::selection::selection_insertion_target_is_current(&request.target) && GetForegroundWindow() == HWND(request.window as *mut _) && !element.CurrentIsPassword()?.as_bool() && uia @@ -593,7 +699,8 @@ unsafe fn observe(request: &Request) -> Result<()> { let current = read_text(&element)?; if !active()? || !anchor.observe(¤t, |edit| { - !request.stop.load(Ordering::Acquire) && (request.callback)(edit) + let snapshot = capture_placement(&element, HWND(request.window as *mut _)); + active().unwrap_or(false) && (request.callback)(edit, snapshot) }) { break; diff --git a/openless-all/app/src-tauri/src/lib.rs b/openless-all/app/src-tauri/src/lib.rs index e0cf76be2..5e8c57d79 100644 --- a/openless-all/app/src-tauri/src/lib.rs +++ b/openless-all/app/src-tauri/src/lib.rs @@ -3595,6 +3595,7 @@ fn show_qa_window_no_activate(window: &tauri::WebviewWindow( #[derive(Clone, Copy)] struct CapsuleHitTestEntry { previous_proc: isize, + no_activate: bool, control_left: i32, control_right: i32, control_top: i32, @@ -240,7 +241,7 @@ unsafe extern "system" fn capsule_hit_test_window_proc( use windows::Win32::Foundation::RECT; use windows::Win32::UI::WindowsAndMessaging::{ CallWindowProcW, DefWindowProcW, GetWindowRect, SetWindowLongPtrW, GWLP_WNDPROC, - HTTRANSPARENT, WM_NCDESTROY, WM_NCHITTEST, + HTTRANSPARENT, MA_NOACTIVATE, WM_MOUSEACTIVATE, WM_NCDESTROY, WM_NCHITTEST, }; if msg == WM_NCDESTROY { @@ -258,6 +259,15 @@ unsafe extern "system" fn capsule_hit_test_window_proc( return DefWindowProcW(hwnd, msg, wparam, lparam); } + if msg == WM_MOUSEACTIVATE + && capsule_hit_test_entries() + .lock() + .get(&(hwnd.0 as isize)) + .is_some_and(|entry| entry.no_activate) + { + return windows::Win32::Foundation::LRESULT(MA_NOACTIVATE as isize); + } + if msg == WM_NCHITTEST { let key = hwnd.0 as isize; let entry = capsule_hit_test_entries().lock().get(&key).copied(); @@ -297,6 +307,7 @@ unsafe extern "system" fn capsule_hit_test_window_proc( fn install_capsule_hit_test_proc( hwnd: windows::Win32::Foundation::HWND, rect: CapsuleHitTestRect, + no_activate: bool, ) -> tauri::Result<()> { use windows::Win32::Foundation::{GetLastError, SetLastError, ERROR_SUCCESS}; use windows::Win32::UI::WindowsAndMessaging::{SetWindowLongPtrW, GWLP_WNDPROC}; @@ -310,6 +321,7 @@ fn install_capsule_hit_test_proc( let key = hwnd.0 as isize; let mut entries = capsule_hit_test_entries().lock(); if let Some(entry) = entries.get_mut(&key) { + entry.no_activate = no_activate; entry.control_left = rect.left; entry.control_right = rect.right; entry.control_top = rect.top; @@ -333,6 +345,7 @@ fn install_capsule_hit_test_proc( key, CapsuleHitTestEntry { previous_proc: previous, + no_activate, control_left: rect.left, control_right: rect.right, control_top: rect.top, @@ -424,7 +437,7 @@ fn configure_capsule_hit_test( return Ok(()); }; set_capsule_input_region(hwnd, rect)?; - if let Err(error) = install_capsule_hit_test_proc(hwnd, rect) { + if let Err(error) = install_capsule_hit_test_proc(hwnd, rect, false) { clear_capsule_input_region(hwnd); return Err(error); } @@ -457,7 +470,7 @@ fn configure_card_hit_test( } let rect = capsule_card_hit_rect(client.right - client.left, client.bottom - client.top); clear_capsule_input_region(hwnd); - install_capsule_hit_test_proc(hwnd, rect) + install_capsule_hit_test_proc(hwnd, rect, true) } #[cfg(target_os = "windows")] @@ -1293,6 +1306,7 @@ impl TauriCapsuleWindow { #[cfg(not(mobile))] pub(crate) fn set_card_hit_test_mode(&self) -> tauri::Result<()> { + self.stop_layout_watch(); if crate::desktop_resources::active() { if let Some(coordinator) = self.app.try_state::>() { @@ -1316,6 +1330,20 @@ impl TauriCapsuleWindow { #[cfg(not(mobile))] pub(crate) fn restore_capsule_hit_test_mode(&self) -> tauri::Result<()> { + #[cfg(target_os = "windows")] + if let Some(window) = self.window() { + use raw_window_handle::{HasWindowHandle, RawWindowHandle}; + if let Ok(handle) = window.window_handle() { + if let RawWindowHandle::Win32(raw) = handle.as_raw() { + if let Some(entry) = capsule_hit_test_entries() + .lock() + .get_mut(&(raw.hwnd.get() as isize)) + { + entry.no_activate = false; + } + } + } + } self.state .hit_test_mode .store(HIT_TEST_MODE_CAPSULE, Ordering::SeqCst); @@ -1570,24 +1598,173 @@ impl TauriCapsuleWindow { width: f64, height: f64, edge_margin: f64, + observation_ids: &[Option], + position: openless_core::shared_types::VocabularySuggestionPosition, ) -> tauri::Result<()> { - let Some(window) = self.window() else { - return Ok(()); - }; - let Some(monitor) = window.current_monitor()? else { + #[cfg(target_os = "windows")] + { + use crate::capsule_mirrors::MonitorRect; + use crate::vocab_card_placement::{place_card, Rect, Screen}; + use windows::Win32::Foundation::{HWND, RECT}; + use windows::Win32::UI::WindowsAndMessaging::{ + GetForegroundWindow, GetWindowRect, GetWindowThreadProcessId, IsWindow, + }; + let Some(window) = self.window() else { + return Ok(()); + }; + let state = self + .app + .try_state::(); + let ids: Vec<_> = observation_ids.iter().flatten().copied().collect(); + if let Some(state) = state.as_ref() { + state.retain(&ids); + } + let id = observation_ids + .first() + .copied() + .flatten() + .filter(|id| observation_ids.iter().all(|other| *other == Some(*id))); + let snapshot = id.and_then(|id| state.as_ref()?.snapshot(id)); + let monitors: Vec<_> = self + .app + .available_monitors()? + .iter() + .map(MonitorRect::from_tauri) + .collect(); + let rect = |m: MonitorRect| Rect { + left: m.left, + top: m.top, + right: m.right, + bottom: m.bottom, + }; + let mut anchor = None; + let mut source_monitor = None; + if let Some(snapshot) = snapshot { + let hwnd = HWND(snapshot.window as *mut _); + let mut pid = 0; + let mut bounds = RECT::default(); + let valid_window = unsafe { + IsWindow(hwnd).as_bool() && { + GetWindowThreadProcessId(hwnd, Some(&mut pid)); + pid == snapshot.process_id + } + }; + if valid_window { + source_monitor = + crate::capsule_mirrors::monitor_rect_for_hwnd(&self.app, snapshot.window); + let unchanged = unsafe { + GetWindowRect(hwnd, &mut bounds).is_ok() && GetForegroundWindow() == hwnd + }; + let current = Rect { + left: bounds.left, + top: bounds.top, + right: bounds.right, + bottom: bounds.bottom, + }; + if position == openless_core::shared_types::VocabularySuggestionPosition::NearInput + && unchanged + && current == snapshot.window_rect + && id.is_some_and(|id| { + state + .as_ref() + .is_some_and(|state| state.source_is_current(id)) + }) + { + anchor = snapshot.anchor; + } + } + if source_monitor.is_none() { + source_monitor = snapshot + .screen_rect + .and_then(|screen| monitors.iter().copied().find(|m| rect(*m) == screen)); + } + } + let anchor_monitor = anchor.and_then(|a| { + let x = ((i64::from(a.left) + i64::from(a.right)) / 2) as i32; + let y = ((i64::from(a.top) + i64::from(a.bottom)) / 2) as i32; + crate::capsule_mirrors::monitor_at_point(&monitors, x, y) + }); + // A detached monitor cannot keep a stale near-input anchor alive. + if anchor_monitor.is_none() { + anchor = None; + } + let monitor = anchor_monitor + .or(source_monitor) + .or_else(|| { + self.app + .primary_monitor() + .ok() + .flatten() + .as_ref() + .map(MonitorRect::from_tauri) + }) + .or_else(|| monitors.first().copied()); + let Some(monitor) = monitor else { + return Ok(()); + }; + let screen = Screen { + bounds: rect(monitor), + work: Rect { + left: monitor.work_left, + top: monitor.work_top, + right: monitor.work_right, + bottom: monitor.work_bottom, + }, + scale: monitor.scale, + }; + if let Some(p) = place_card(screen, anchor, width, height, edge_margin) { + use raw_window_handle::{HasWindowHandle, RawWindowHandle}; + use windows::Win32::UI::WindowsAndMessaging::{ + SetWindowPos, SWP_NOACTIVATE, SWP_NOZORDER, + }; + let handle = window + .window_handle() + .map_err(|error| tauri::Error::Anyhow(anyhow::anyhow!("{error}")))?; + let RawWindowHandle::Win32(raw) = handle.as_raw() else { + return Err(tauri::Error::Anyhow(anyhow::anyhow!( + "card has no Win32 window" + ))); + }; + unsafe { + SetWindowPos( + HWND(raw.hwnd.get() as *mut _), + HWND::default(), + p.x, + p.y, + p.width as i32, + p.height as i32, + SWP_NOACTIVATE | SWP_NOZORDER, + ) + .map_err(|error| tauri::Error::Anyhow(error.into()))?; + } + } else { + return Err(tauri::Error::Anyhow(anyhow::anyhow!( + "no usable vocabulary card work area" + ))); + } return Ok(()); - }; - let scale = monitor.scale_factor(); - let size = monitor.size(); - let position = monitor.position(); - let monitor_width = size.width as f64 / scale; - let monitor_height = size.height as f64 / scale; - let monitor_x = position.x as f64 / scale; - let monitor_y = position.y as f64 / scale; - window.set_position(tauri::LogicalPosition::new( - monitor_x + monitor_width - width - edge_margin, - monitor_y + monitor_height - height - 80.0, - )) + } + #[cfg(not(target_os = "windows"))] + { + let _ = (observation_ids, position); + let Some(window) = self.window() else { + return Ok(()); + }; + let Some(monitor) = window.current_monitor()? else { + return Ok(()); + }; + let scale = monitor.scale_factor(); + let size = monitor.size(); + let position = monitor.position(); + let monitor_width = size.width as f64 / scale; + let monitor_height = size.height as f64 / scale; + let monitor_x = position.x as f64 / scale; + let monitor_y = position.y as f64 / scale; + window.set_position(tauri::LogicalPosition::new( + monitor_x + monitor_width - width - edge_margin, + monitor_y + monitor_height - height - 80.0, + )) + } } pub(crate) fn position_fallback_card(&self, width: f64, height: f64) -> tauri::Result<()> { @@ -1813,6 +1990,29 @@ impl TauriCapsuleWindow { } } + pub(crate) fn show_vocab_card(&self) -> bool { + if self.state.qa_voice_active.load(Ordering::SeqCst) { + return false; + } + #[cfg(target_os = "windows")] + { + let Some(window) = self.window() else { + return false; + }; + if !set_capsule_window_topmost_no_activate(&window, true) { + return false; + } + self.state.capsule_visible.store(true, Ordering::SeqCst); + self.sync_capsule_mirrors(); + true + } + #[cfg(not(target_os = "windows"))] + { + self.show_for_recording(true); + true + } + } + pub(crate) fn show_for_recording(&self, reassert_spaces: bool) { if self.state.qa_voice_active.load(Ordering::SeqCst) { let _ = self.hide(); return; } self.state.capsule_visible.store(true, Ordering::SeqCst); @@ -2040,6 +2240,7 @@ impl TauriCoordinatorHost { } pub(crate) fn bind(&self, app: AppHandle) { + app.manage(crate::core_adapters::VocabPlacementState::default()); #[cfg(not(mobile))] if crate::desktop_resources::active() { match crate::native_capsule::NativeCapsule::new(app.clone()) { diff --git a/openless-all/app/src-tauri/src/vocab_card_placement.rs b/openless-all/app/src-tauri/src/vocab_card_placement.rs new file mode 100644 index 000000000..1965b0c67 --- /dev/null +++ b/openless-all/app/src-tauri/src/vocab_card_placement.rs @@ -0,0 +1,415 @@ +//! Pure vocabulary-card geometry and observation ownership; no native handles are dereferenced here. + +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::time::{Duration, Instant}; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct Rect { + pub left: i32, + pub top: i32, + pub right: i32, + pub bottom: i32, +} + +impl Rect { + pub fn valid(self) -> bool { + self.right >= self.left + && self.bottom > self.top + && self.right.checked_sub(self.left).is_some() + && self.bottom.checked_sub(self.top).is_some() + } + + pub fn from_xywh(x: f64, y: f64, width: f64, height: f64) -> Option { + let values = [x, y, x + width, y + height]; + if width < 0.0 + || height <= 0.0 + || !values + .iter() + .all(|v| v.is_finite() && *v > i32::MIN as f64 && *v < i32::MAX as f64) + { + return None; + } + let rect = Self { + left: x.floor() as i32, + top: y.floor() as i32, + right: (x + width).ceil() as i32, + bottom: (y + height).ceil() as i32, + }; + rect.valid().then_some(rect) + } + + fn intersects(self, other: Self) -> bool { + self.left < other.right + && self.right >= other.left + && self.top < other.bottom + && self.bottom > other.top + } +} + +#[derive(Debug, Clone)] +pub(crate) struct AnchorSnapshot { + pub window: usize, + pub process_id: u32, + pub window_rect: Rect, + pub anchor: Option, + pub screen_rect: Option, + pub captured_at: Instant, +} + +#[derive(Default)] +pub(crate) struct AnchorRegistry { + active: Option, + entries: Vec<(u64, AnchorSnapshot)>, +} + +impl AnchorRegistry { + pub fn begin(&mut self, id: u64) { + self.active = Some(id); + } + + pub fn update(&mut self, id: u64, snapshot: AnchorSnapshot) -> bool { + if self.active != Some(id) { + return false; + } + self.entries.retain(|(key, _)| *key != id); + self.entries.push((id, snapshot)); + if self.entries.len() > 8 { + self.entries.remove(0); + } + true + } + + pub fn invalidate(&mut self, id: u64) { + if let Some((_, entry)) = self.entries.iter_mut().find(|(key, _)| *key == id) { + entry.anchor = None; + } + } + + pub fn stop(&mut self) { + self.active = None; + for (_, entry) in &mut self.entries { + entry.anchor = None; + } + } + + pub fn retain(&mut self, ids: &[u64]) { + let active = self.active; + self.entries.retain(|(id, entry)| { + (active == Some(*id) || ids.contains(id)) + && entry.captured_at.elapsed() <= Duration::from_secs(120) + }); + } + + pub fn get(&self, id: u64) -> Option { + self.entries + .iter() + .find(|(key, entry)| { + *key == id && entry.captured_at.elapsed() <= Duration::from_secs(120) + }) + .map(|(_, entry)| { + let mut snapshot = entry.clone(); + if snapshot.captured_at.elapsed() > Duration::from_secs(5) { + snapshot.anchor = None; + } + snapshot + }) + } +} + +/// Capacity-bounded handoff, deliberately independent of recording/inference duration. +pub(crate) struct TargetHandoff(Vec<(K, T)>); + +impl Default for TargetHandoff { + fn default() -> Self { + Self(Vec::new()) + } +} + +impl TargetHandoff { + pub fn remember(&mut self, key: K, target: T) { + self.0.retain(|(id, _)| *id != key); + self.0.push((key, target)); + if self.0.len() > 4 { + self.0.remove(0); + } + } + + pub fn take(&mut self, key: K) -> Option { + let index = self.0.iter().position(|(id, _)| *id == key)?; + Some(self.0.remove(index).1) + } +} + +/// Only vocabulary-card work uses this epoch; ordinary capsule events keep their own ownership. +#[derive(Default)] +pub(crate) struct PresentationEpoch(AtomicU64); + +impl PresentationEpoch { + pub fn next(&self) -> u64 { + self.0.fetch_add(1, Ordering::SeqCst).wrapping_add(1) + } + + pub fn request_hide(&self, visible: &AtomicBool) -> Option { + let epoch = self.next(); + // Keep ownership until the main-thread cleanup actually runs. A second + // hide must replace an obsolete cleanup, not cancel it and return early. + visible.load(Ordering::SeqCst).then_some(epoch) + } + + pub fn is_current(&self, epoch: u64) -> bool { + self.0.load(Ordering::SeqCst) == epoch + } +} + +#[derive(Clone, Copy)] +pub(crate) struct Screen { + pub bounds: Rect, + pub work: Rect, + pub scale: f64, +} + +#[derive(Debug, PartialEq, Eq)] +pub(crate) struct Placement { + pub width: u32, + pub height: u32, + pub x: i32, + pub y: i32, +} + +pub(crate) fn place_card( + screen: Screen, + anchor: Option, + logical_width: f64, + logical_height: f64, + logical_margin: f64, +) -> Option { + if !screen.bounds.valid() || screen.bounds.right == screen.bounds.left { + return None; + } + let work = if screen.work.valid() && screen.work.right > screen.work.left { + screen.work + } else { + screen.bounds + }; + if !screen.scale.is_finite() || screen.scale <= 0.0 || screen.scale > 16.0 { + return None; + } + let width = (logical_width * screen.scale).round(); + let height = (logical_height * screen.scale).round(); + if !width.is_finite() || !height.is_finite() || width < 1.0 || height < 1.0 { + return None; + } + let w = (width.min((work.right - work.left) as f64)) as i32; + let h = (height.min((work.bottom - work.top) as f64)) as i32; + let margin = (logical_margin.max(0.0) * screen.scale).round() as i32; + let gap = (12.0 * screen.scale).round() as i32; + let corner = ( + work.right.saturating_sub(w).saturating_sub(margin), + work.bottom.saturating_sub(h).saturating_sub(margin), + ); + let (x, y) = match anchor.filter(|rect| rect.valid() && screen.bounds.intersects(*rect)) { + Some(rect) if rect.bottom.saturating_add(gap).saturating_add(h) <= work.bottom => { + (rect.left, rect.bottom.saturating_add(gap)) + } + Some(rect) if rect.top.saturating_sub(gap).saturating_sub(h) >= work.top => { + (rect.left, rect.top.saturating_sub(gap).saturating_sub(h)) + } + _ => corner, + }; + Some(Placement { + width: w as u32, + height: h as u32, + x: x.clamp(work.left, work.right.saturating_sub(w).max(work.left)), + y: y.clamp(work.top, work.bottom.saturating_sub(h).max(work.top)), + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn rect(left: i32, top: i32, right: i32, bottom: i32) -> Rect { + Rect { + left, + top, + right, + bottom, + } + } + + fn screen() -> Screen { + Screen { + bounds: rect(-1920, 0, 0, 1080), + work: rect(-1880, 0, 0, 1040), + scale: 1.0, + } + } + + fn snapshot() -> AnchorSnapshot { + AnchorSnapshot { + window: 1, + process_id: 2, + window_rect: rect(-1800, 100, -100, 900), + anchor: Some(rect(-1000, 300, -1000, 320)), + screen_rect: Some(screen().bounds), + captured_at: Instant::now(), + } + } + + #[test] + fn zero_width_caret_is_valid_but_empty_or_nonfinite_geometry_is_not() { + assert!(Rect::from_xywh(-100.0, 20.0, 0.0, 20.0).is_some()); + assert!(Rect::from_xywh(0.0, 0.0, 1.0, 0.0).is_none()); + assert!(Rect::from_xywh(f64::NAN, 0.0, 1.0, 20.0).is_none()); + assert!(Rect::from_xywh(0.0, 0.0, -1.0, 20.0).is_none()); + } + + #[test] + fn places_below_caret_on_negative_coordinate_monitor() { + let placement = place_card(screen(), snapshot().anchor, 320.0, 108.0, 24.0).unwrap(); + assert_eq!((placement.x, placement.y), (-1000, 332)); + } + + #[test] + fn flips_above_input_near_bottom_and_clamps_right_edge() { + let placement = place_card( + screen(), + Some(rect(-100, 950, -80, 980)), + 320.0, + 108.0, + 24.0, + ) + .unwrap(); + assert_eq!((placement.x, placement.y), (-320, 830)); + } + + #[test] + fn fallback_corner_uses_work_area_not_monitor_bottom() { + let placement = place_card(screen(), None, 320.0, 108.0, 24.0).unwrap(); + assert_eq!((placement.x, placement.y), (-344, 908)); + } + + #[test] + fn scales_size_and_gap_by_the_target_monitor_only() { + for scale in [1.25, 1.5, 2.0] { + let monitor = Screen { scale, ..screen() }; + let placement = place_card(monitor, snapshot().anchor, 320.0, 108.0, 24.0).unwrap(); + assert_eq!(placement.width, (320.0 * scale).round() as u32); + assert_eq!(placement.y, 320 + (12.0 * scale).round() as i32); + } + } + + #[test] + fn offscreen_anchor_and_invalid_work_area_degrade_safely() { + let monitor = Screen { + work: rect(0, 0, 0, 0), + ..screen() + }; + let a = place_card(monitor, Some(rect(100, 100, 101, 120)), 320.0, 108.0, 24.0); + let b = place_card(monitor, None, 320.0, 108.0, 24.0); + assert_eq!(a, b); + } + + #[test] + fn card_larger_than_work_area_keeps_all_actions_on_screen() { + let monitor = Screen { + work: rect(-100, 0, 0, 100), + ..screen() + }; + let p = place_card(monitor, None, 320.0, 252.0, 24.0).unwrap(); + assert_eq!((p.width, p.height, p.x, p.y), (100, 100, -100, 0)); + } + + #[test] + fn stopped_or_previous_observer_cannot_overwrite_new_anchor() { + let mut registry = AnchorRegistry::default(); + registry.begin(1); + assert!(registry.update(1, snapshot())); + registry.stop(); + assert!(!registry.update(1, snapshot())); + registry.begin(2); + assert!(registry.update(2, snapshot())); + assert!(!registry.update(1, snapshot())); + registry.invalidate(1); + assert!(registry.get(2).unwrap().anchor.is_some()); + } + + #[test] + fn stopping_retains_original_screen_but_revokes_near_input_placement() { + let mut registry = AnchorRegistry::default(); + registry.begin(1); + registry.update(1, snapshot()); + registry.stop(); + let stored = registry.get(1).unwrap(); + assert!(stored.anchor.is_none()); + assert_eq!(stored.screen_rect, Some(screen().bounds)); + } + + #[test] + fn resolving_one_item_does_not_drop_other_items_from_the_same_observation() { + let mut registry = AnchorRegistry::default(); + registry.begin(1); + registry.update(1, snapshot()); + registry.begin(2); + registry.retain(&[1]); + assert!(registry.get(1).is_some()); + registry.retain(&[]); + assert!(registry.get(1).is_none()); + } + + #[test] + fn old_snapshot_falls_back_to_original_screen() { + let mut registry = AnchorRegistry::default(); + registry.begin(1); + let mut capture = snapshot(); + capture.captured_at -= Duration::from_secs(6); + registry.update(1, capture); + assert!(registry.get(1).unwrap().anchor.is_none()); + } + + #[test] + fn queued_show_and_hide_are_obsolete_even_after_a_new_session_returns_idle() { + let epoch = PresentationEpoch::default(); + let old_show = epoch.next(); + let old_hide = epoch.next(); + let next_session = epoch.next(); + assert!(!epoch.is_current(old_show)); + assert!(!epoch.is_current(old_hide)); + assert!(epoch.is_current(next_session)); + } + #[test] + fn duplicate_hide_keeps_a_live_cleanup_until_the_window_is_restored() { + let epoch = PresentationEpoch::default(); + let visible = AtomicBool::new(true); + let first = epoch.request_hide(&visible).unwrap(); + let second = epoch.request_hide(&visible).unwrap(); + assert!(!epoch.is_current(first)); + assert!(epoch.is_current(second)); + assert!(visible.load(Ordering::SeqCst)); + visible.store(false, Ordering::SeqCst); + assert!(epoch.request_hide(&visible).is_none()); + } + + #[test] + fn hiding_before_show_still_cancels_the_queued_show() { + let epoch = PresentationEpoch::default(); + let show = epoch.next(); + assert!(epoch.request_hide(&AtomicBool::new(false)).is_none()); + assert!(!epoch.is_current(show)); + } + + #[test] + fn target_handoff_is_consumed_by_session_not_by_elapsed_time() { + let mut targets = TargetHandoff::default(); + targets.remember(1, "original target"); + // No wall-clock expiry: long recording/polishing cannot evict the active session. + assert_eq!(targets.take(1), Some("original target")); + assert_eq!(targets.take(1), None); + let mut targets = TargetHandoff::default(); + for session in 1..=5 { + targets.remember(session, session); + } + assert_eq!(targets.take(1), None); + assert_eq!(targets.take(5), Some(5)); + } +} diff --git a/openless-all/app/src/i18n/de.ts b/openless-all/app/src/i18n/de.ts index 83ca7fdbd..6d247ac32 100644 --- a/openless-all/app/src/i18n/de.ts +++ b/openless-all/app/src/i18n/de.ts @@ -1357,6 +1357,11 @@ export const de: typeof zhCN = { vocabularyLearning: { title: 'Aus Korrekturen lernen', enabled: 'Lernen aus Korrekturen aktivieren', + suggestionPosition: 'Position der Vorschlagskarte', + positionNearInput: 'Nahe der Eingabe', + positionBottomRight: 'Unten rechts fixiert', + suggestionPositionHint: + 'Nur Windows: bevorzugt beim Textcursor oder Eingabefeld; andernfalls unten rechts auf dem ursprünglichen Eingabebildschirm. Die feste Position nutzt immer dessen Arbeitsbereich.', observationSeconds: 'Beobachtungsdauer', observationSecondsHint: 'Das aktuelle Eingabefeld nach dem Diktat höchstens 10–60 Sekunden beobachten.', diff --git a/openless-all/app/src/i18n/en.ts b/openless-all/app/src/i18n/en.ts index cabdfbd89..5a539892e 100644 --- a/openless-all/app/src/i18n/en.ts +++ b/openless-all/app/src/i18n/en.ts @@ -1333,6 +1333,11 @@ export const en: typeof zhCN = { vocabularyLearning: { title: 'Learn from corrections', enabled: 'Enable learning from corrections', + suggestionPosition: 'Suggestion card position', + positionNearInput: 'Near input', + positionBottomRight: 'Fixed bottom-right', + suggestionPositionHint: + 'Windows only: prefer the text caret or input field; if unavailable, fall back to the original input screen’s bottom-right corner. Fixed bottom-right always uses that screen’s work area.', observationSeconds: 'Observation duration', observationSecondsHint: 'Maximum time to observe the current field after dictation: 10–60 seconds.', diff --git a/openless-all/app/src/i18n/es.ts b/openless-all/app/src/i18n/es.ts index 0378847ea..4d30059ea 100644 --- a/openless-all/app/src/i18n/es.ts +++ b/openless-all/app/src/i18n/es.ts @@ -1350,6 +1350,11 @@ export const es: typeof zhCN = { vocabularyLearning: { title: 'Aprender de las correcciones', enabled: 'Activar aprendizaje de correcciones', + suggestionPosition: 'Posición de la tarjeta', + positionNearInput: 'Cerca de la entrada', + positionBottomRight: 'Fija abajo a la derecha', + suggestionPositionHint: + 'Solo Windows: cerca del cursor de texto o del campo; si no es posible, abajo a la derecha en la pantalla de entrada original. La posición fija siempre usa el área de trabajo de esa pantalla.', observationSeconds: 'Tiempo de observación', observationSecondsHint: 'Observar el campo actual durante un máximo de 10–60 segundos después del dictado.', diff --git a/openless-all/app/src/i18n/fr.ts b/openless-all/app/src/i18n/fr.ts index b9487f656..50419a4ee 100644 --- a/openless-all/app/src/i18n/fr.ts +++ b/openless-all/app/src/i18n/fr.ts @@ -1370,6 +1370,11 @@ export const fr: typeof zhCN = { vocabularyLearning: { title: 'Apprendre des corrections', enabled: 'Activer l’apprentissage des corrections', + suggestionPosition: 'Position de la carte', + positionNearInput: 'Près de la saisie', + positionBottomRight: 'Fixe en bas à droite', + suggestionPositionHint: + 'Windows uniquement : près du curseur de texte ou du champ ; sinon, en bas à droite de l’écran de saisie d’origine. La position fixe utilise toujours la zone de travail de cet écran.', observationSeconds: 'Durée d’observation', observationSecondsHint: 'Observer le champ actuel pendant 10 à 60 secondes au maximum après la dictée.', diff --git a/openless-all/app/src/i18n/ja.ts b/openless-all/app/src/i18n/ja.ts index c1d09e207..fd87a89d6 100644 --- a/openless-all/app/src/i18n/ja.ts +++ b/openless-all/app/src/i18n/ja.ts @@ -1321,6 +1321,11 @@ export const ja: typeof zhCN = { vocabularyLearning: { title: '修正から単語を学習', enabled: '修正からの学習を有効にする', + suggestionPosition: '提案カードの位置', + positionNearInput: '入力位置の近く', + positionBottomRight: '右下に固定', + suggestionPositionHint: + 'Windows のみ:テキストカーソルまたは入力欄の近くに表示し、位置を取得できない場合は元の入力先画面の右下に表示します。固定表示では常にその画面の作業領域を使います。', observationSeconds: '監視時間', observationSecondsHint: '音声入力後、現在の入力欄を監視する最大時間(10~60 秒)。', suggestionSeconds: '候補の表示時間', diff --git a/openless-all/app/src/i18n/ko.ts b/openless-all/app/src/i18n/ko.ts index dff252b94..4e4df67bd 100644 --- a/openless-all/app/src/i18n/ko.ts +++ b/openless-all/app/src/i18n/ko.ts @@ -1312,6 +1312,11 @@ export const ko: typeof zhCN = { vocabularyLearning: { title: '수정으로 단어 학습', enabled: '수정 학습 사용', + suggestionPosition: '제안 카드 위치', + positionNearInput: '입력 위치 근처', + positionBottomRight: '오른쪽 아래 고정', + suggestionPositionHint: + 'Windows 전용: 텍스트 커서나 입력란 근처에 표시하며, 위치를 알 수 없으면 원래 입력 화면의 오른쪽 아래에 표시합니다. 고정 위치는 항상 해당 화면의 작업 영역을 사용합니다.', observationSeconds: '관찰 시간', observationSecondsHint: '음성 입력 후 현재 입력란을 관찰하는 최대 시간입니다(10~60초).', suggestionSeconds: '제안 유지 시간', diff --git a/openless-all/app/src/i18n/vocabularyLearningPosition.test.ts b/openless-all/app/src/i18n/vocabularyLearningPosition.test.ts new file mode 100644 index 000000000..eae1c2046 --- /dev/null +++ b/openless-all/app/src/i18n/vocabularyLearningPosition.test.ts @@ -0,0 +1,30 @@ +import { de } from './de'; +import { en } from './en'; +import { es } from './es'; +import { fr } from './fr'; +import { ja } from './ja'; +import { ko } from './ko'; +import { zhCN } from './zh-CN'; +import { zhTW } from './zh-TW'; + +for (const [locale, messages] of Object.entries({ de, en, es, fr, ja, ko, zhCN, zhTW })) { + const learning = messages.settings.vocabularyLearning; + for (const key of [ + 'suggestionPosition', + 'suggestionPositionHint', + 'positionNearInput', + 'positionBottomRight', + ] as const) { + if (!learning[key].trim()) { + throw new Error(`${locale}: missing vocabulary learning ${key}`); + } + } + if (learning.positionNearInput === learning.positionBottomRight) { + throw new Error(`${locale}: vocabulary position choices need distinct labels`); + } + if (!learning.suggestionPositionHint.includes('Windows')) { + throw new Error(`${locale}: vocabulary position hint must state the Windows-only scope`); + } +} + +console.log('vocabulary learning position translations cover both Windows choices'); diff --git a/openless-all/app/src/i18n/zh-CN.ts b/openless-all/app/src/i18n/zh-CN.ts index 6b92f58ea..0d2932595 100644 --- a/openless-all/app/src/i18n/zh-CN.ts +++ b/openless-all/app/src/i18n/zh-CN.ts @@ -1276,6 +1276,11 @@ export const zhCN = { vocabularyLearning: { title: '手改学词', enabled: '启用手改学词', + suggestionPosition: '建议卡片位置', + positionNearInput: '输入位置附近', + positionBottomRight: '固定右下角', + suggestionPositionHint: + '仅 Windows:默认靠近文本光标或输入框;定位不可用时回退到原输入目标所在屏幕的右下角。固定右下角始终使用该屏幕的工作区。', observationSeconds: '观察时长', observationSecondsHint: '听写落字后观察当前输入框的最长时间,10–60 秒。', suggestionSeconds: '建议保留时长', diff --git a/openless-all/app/src/i18n/zh-TW.ts b/openless-all/app/src/i18n/zh-TW.ts index 533136f34..589369ba5 100644 --- a/openless-all/app/src/i18n/zh-TW.ts +++ b/openless-all/app/src/i18n/zh-TW.ts @@ -1276,6 +1276,11 @@ export const zhTW: typeof zhCN = { vocabularyLearning: { title: '手改學詞', enabled: '啟用手改學詞', + suggestionPosition: '建議卡片位置', + positionNearInput: '輸入位置附近', + positionBottomRight: '固定右下角', + suggestionPositionHint: + '僅 Windows:預設靠近文字游標或輸入框;無法定位時退回原輸入目標所在螢幕的右下角。固定右下角一律使用該螢幕的工作區。', observationSeconds: '觀察時長', observationSecondsHint: '聽寫輸入後觀察目前輸入框的最長時間,10–60 秒。', suggestionSeconds: '建議保留時長', diff --git a/openless-all/app/src/lib/ipc/mock-data.ts b/openless-all/app/src/lib/ipc/mock-data.ts index a9dea163f..3c967186a 100644 --- a/openless-all/app/src/lib/ipc/mock-data.ts +++ b/openless-all/app/src/lib/ipc/mock-data.ts @@ -138,7 +138,12 @@ export let mockSettings: UserPreferences = { windowsCompositionInput: false, cursorContextEnabled: false, vocabularyLearningEnabled: false, - vocabularyLearningSettings: { observationSeconds: 60, suggestionSeconds: 10, maxPhraseChars: 12 }, + vocabularyLearningSettings: { + observationSeconds: 60, + suggestionSeconds: 10, + maxPhraseChars: 12, + suggestionPosition: 'nearInput', + }, showOverviewActivityHeatmap: true, stackedRowLayout: false, conservativeLayout: false, diff --git a/openless-all/app/src/lib/types.ts b/openless-all/app/src/lib/types.ts index e3f0f0e66..7d98d07f2 100644 --- a/openless-all/app/src/lib/types.ts +++ b/openless-all/app/src/lib/types.ts @@ -124,6 +124,7 @@ export interface HostDocumentReadResult { /** A correction suggestion awaiting user confirmation (Tier2). The backend keeps it only in memory * and it is lost on restart — suggestions are ephemeral; repeating the same mistake regenerates one. */ export interface PendingCorrection { + observationId?: number; expiresAtMs: number; id: string; pattern: string; @@ -575,6 +576,7 @@ export interface UserPreferences { observationSeconds: number; suggestionSeconds: number; maxPhraseChars: number; + suggestionPosition: 'nearInput' | 'bottomRight'; }; /** Whether Overview shows the yearly activity heatmap card. Default true; turning it off only hides the card, activity counting continues. */ showOverviewActivityHeatmap: boolean; diff --git a/openless-all/app/src/pages/settings/VocabularyLearningSection.tsx b/openless-all/app/src/pages/settings/VocabularyLearningSection.tsx index 8742dfe5c..f64359f4a 100644 --- a/openless-all/app/src/pages/settings/VocabularyLearningSection.tsx +++ b/openless-all/app/src/pages/settings/VocabularyLearningSection.tsx @@ -1,12 +1,18 @@ import { useRef, useState } from 'react'; import { useTranslation } from 'react-i18next'; import { SelectLite } from '../../components/ui/SelectLite'; +import { detectOS } from '../../components/WindowChrome'; import type { UserPreferences } from '../../lib/types'; import { useHotkeySettings } from '../../state/HotkeySettingsContext'; import { Card } from '../_atoms'; import { btnGhostStyle, ExperimentalSectionTitle, SettingRow, Toggle } from './shared'; -const defaults = { observationSeconds: 60, suggestionSeconds: 10, maxPhraseChars: 12 }; +const defaults: UserPreferences['vocabularyLearningSettings'] = { + observationSeconds: 60, + suggestionSeconds: 10, + maxPhraseChars: 12, + suggestionPosition: 'nearInput', +}; const parameters = [ { key: 'observationSeconds', min: 10, max: 60, unit: 'seconds' }, { key: 'suggestionSeconds', min: 5, max: 60, unit: 'seconds' }, @@ -53,6 +59,33 @@ export function VocabularyLearningSection() { } /> + {detectOS() === 'win' && ( + + { + if (value !== 'nearInput' && value !== 'bottomRight') return; + void save((current) => ({ + ...current, + vocabularyLearningSettings: { + ...(current.vocabularyLearningSettings ?? defaults), + suggestionPosition: value, + }, + })); + }} + /> + + )} {parameters.map(({ key, min, max, unit }) => (