Skip to content

fix(windows): stop trusting recycled HWNDs in the capsule hit-test subclass - #1267

Open
lxcadoza993 wants to merge 1 commit into
Open-Less:betafrom
lxcadoza993:fix/recycled-hwnd-hit-test
Open

lxcadoza993 wants to merge 1 commit into
Open-Less:betafrom
lxcadoza993:fix/recycled-hwnd-hit-test

Conversation

@lxcadoza993

Copy link
Copy Markdown

What this fixes

install_capsule_hit_test_proc keeps its bookkeeping in a HashMap<isize, CapsuleHitTestEntry>
keyed by the numeric HWND value. Windows recycles HWND values: after a capsule/card window is
destroyed, the same handle value can be handed to an unrelated new window. Because the map is keyed
by value, a lookup during the new window's lifetime still finds the destroyed window's entry, and
three separate things go wrong:

  1. The fast path silently disables hit-testing on the new window. It sees the key present, writes
    the new geometry into the stale entry, and returns Ok(()) without installing anything. The new
    window therefore never carries our subclass procedure, so it never receives WM_NCHITTEST and the
    click-through region stops being honoured.
  2. WM_NCHITTEST hit-tests the wrong geometry — the live window is tested against the dead
    window's control rect.
  3. message forwarding can reach another window's previous_proc.
  4. WM_NCDESTROY removed the entry by key alone, so a recycled handle could retire a surviving
    window's entry, stranding it with our procedure installed and nothing recorded to forward to.

Why the fix looks like this

The only fact that reliably distinguishes "same window" from "recycled handle value" is whether our
own procedure is still installed on the handle
: a recycled HWND carries whatever procedure its new
owner installed. So every read of the map now proves that first, and an entry that fails the check is
dropped and re-installed rather than trusted.

unsafe fn capsule_hit_test_entry_matches(hwnd: HWND) -> bool {
    use windows::Win32::UI::WindowsAndMessaging::{GetWindowLongPtrW, GWLP_WNDPROC};
    let installed = unsafe { GetWindowLongPtrW(hwnd, GWLP_WNDPROC) };
    installed == capsule_hit_test_window_proc as *const () as usize as isize
}

Two supporting fixes:

  • install_capsule_hit_test_proc refuses to record itself as the predecessor. When the map has
    been wiped but the procedure is already on the window, SetWindowLongPtrW returns our own address;
    storing it would make every forwarded message recurse into this same procedure until the stack
    overflows. In that case no entry is recorded and DefWindowProcW takes the traffic.
  • WM_NCDESTROY only retires the entry this window owns.

I deliberately did not add a "creating thread id" check alongside this. Once we require that our
own procedure is on the handle, the thread id is redundant, and it would add a GetWindowThreadProcessId
syscall to the WM_NCHITTEST hover path, which runs per mouse move.

Reachability

On the current beta, desktop_resources::hide_capsule_window calls window.destroy() when desktop
low-memory mode is active, so capsule windows genuinely are destroyed and recreated — the recycled-handle
path is reachable, not theoretical. configure_capsule_hit_test / configure_card_hit_test are both
called on every style/geometry update.

Testing

New test reinstall_keeps_the_real_predecessor_and_never_self_wraps registers a real window class,
creates an HWND_MESSAGE window, and installs the subclass three times. It asserts that:

  • our procedure ends up installed on the handle;
  • a re-install refreshes geometry in place and keeps previous_proc pointing at the real class
    procedure rather than at ourselves;
  • with the map wiped but the procedure still installed, a further install records no entry
    (the self-install guard), so forwarding cannot recurse.

A real window is required here — the identity check reads the live GWLP_WNDPROC, which no mock can
stand in for.

Results: 1 passed; 0 failed. cargo check → 0 errors, warning count unchanged. cargo build --release → OK.

Heads-up: cargo test --lib does not run on Windows out of the box

Before any test executes, the test binary exits with 0xC0000139 (STATUS_ENTRYPOINT_NOT_FOUND).
The crate imports comctl32!TaskDialogIndirect, which only exists in Common-Controls v6; v6 is
bound solely through the /manifestdependency link arg that build.rs emits via
link_windows_common_controls_v6_manifest_dependency(). The cargo test harness links separately
and does not receive that manifest, so it loads against comctl32 v5 and fails at load time.

Workaround used to get the results above:

mt.exe -nologo -manifest cc6.manifest "-outputresource:<test-exe>;#1"

where cc6.manifest declares the Microsoft.Windows.Common-Controls 6.0.0.0 dependency. This is
pre-existing and independent of this change, but it means cargo test --lib currently gives the whole
repo zero coverage. Worth fixing separately (e.g. a rustflags link arg in .cargo/config.toml);
happy to send that as its own PR if you want it.

Scope — please read before merging as a crash fix

This is a robustness / correctness fix. It is not a fix for the 0xC0000374 heap-corruption
crashes. I traced that separately from two local minidumps and it does not implicate this code: the
subclass path never produced an invalid pointer to free, and the crash build did not even destroy
capsule windows (desktop_resources.rs came later, in 6336823). I have not claimed a root cause for
the crashes anywhere in this PR and do not want this merged under that impression.

If it's useful I can also post the heap-failure decode from the two dumps (FailureType 8
block_not_busy and 13 freelists_corruption, both from RtlpHeapHandleError) as a separate issue —
that data is what #1227/#1261 asked for, but it stops at "the two crashes are different shapes of the
same free-side corruption", which is not yet enough to name a culprit.

…bclass

`install_capsule_hit_test_proc` keys its bookkeeping on the numeric HWND, but
Windows recycles HWND values. Once a capsule/card window is destroyed and its
handle value is handed to a new window, a lookup by that key still finds the
dead window's entry, and three things go wrong:

- the fast path reuses the stale entry and returns early, so the *new* window
  never gets the subclass procedure installed -- hit-testing silently stops
  working for it;
- `WM_NCHITTEST` hit-tests the live window against the dead window's geometry;
- message forwarding can call a `previous_proc` that belonged to another window.

The only reliable identity check is whether *our own* procedure is still
installed on the handle: a recycled HWND carries whatever procedure its new
owner installed. Every read of the map now proves that first, and an entry that
fails the check is dropped and re-installed instead of trusted.

Two supporting fixes while in here:

- `install_capsule_hit_test_proc` now refuses to record itself as the
  predecessor. When the map has been wiped but the procedure is already on the
  window, `SetWindowLongPtrW` returns our own address; storing it would make
  every forwarded message recurse into this same procedure.
- `WM_NCDESTROY` only retires the entry this window owns.

Adds `reinstall_keeps_the_real_predecessor_and_never_self_wraps`, which uses a
real `HWND_MESSAGE` window because the identity check reads the live
`GWLP_WNDPROC` and cannot be mocked.

Note on running tests: `cargo test --lib` exits with `0xC0000139`
(STATUS_ENTRYPOINT_NOT_FOUND) on Windows before any test runs. The crate
imports `comctl32!TaskDialogIndirect`, which lives in Common-Controls v6, and
v6 is bound only through the `build.rs` `/manifestdependency` link arg -- the
cargo test harness links separately and does not get that manifest. Embedding a
v6 manifest in the test binary makes it run:

    mt.exe -nologo -manifest cc6.manifest "-outputresource:<test-exe>;Open-Less#1"

With that, the new test passes (1 passed; 0 failed), as does `cargo check`
(0 errors) and `cargo build --release`.
@H-Chris233 H-Chris233 added area:ui UI area bug Something isn't working needs-review Needs review P2 Medium priority platform:windows Windows-specific labels Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ui UI area bug Something isn't working needs-review Needs review P2 Medium priority platform:windows Windows-specific

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants