Keep native renderer E_OUTOFMEMORY from crashing Wisp and restart the HUD renderer

[Views2k]

Problem

Wisp Tools recorded a crash on 2026-10-17 (Wisp 2.6.12, native-analogue gauge, GPU render mode): OutOfMemoryException with HRESULT 0x8007000E (E_OUTOFMEMORY) on the background thread in AnalogHudRenderWorker.Run, 0.46 s after the game became active. The working set was 389 MB and the managed heap 68 MB, so this was not managed heap exhaustion; the same PC relaunched Wisp three minutes later and has run normally since.

The fatal code path:

1. Wisp.NativeRenderer.dll maps std::bad_alloc and failed Direct3D 11 / DirectComposition allocations (texture creation, composition surfaces) to E_OUTOFMEMORY, and CHECK_HR passes that code through.
2. DirectCompositionDevice raised every failed HRESULT through Marshal.ThrowExceptionForHR, which maps 0x8007000E to the runtime's OutOfMemoryException.
3. Both render workers catch Exception except OutOfMemoryException and StackOverflowException, so the native failure escaped the render thread, reached AppDomain.UnhandledException, and ended the process. Every other native HRESULT (for example DXGI_ERROR_DEVICE_HUNG) was recorded and reported through the existing status path.

The trigger is transient GPU/compositor allocation pressure while the game claims the GPU at launch.

Change

• NativeRendererException (a COMException carrying the HRESULT and the failing device operation) replaces Marshal.ThrowExceptionForHR for every native result in DirectCompositionDevice, including the motion channel. HResult is intact, so health breadcrumbs and _status(false, hr) reporting are unchanged. A native E_OUTOFMEMORY can no longer appear as OutOfMemoryException.
• Both workers handle it through their existing catch: AnalogHudRenderWorker records the stage as error with the HRESULT and failure phase and reports the status; DriftGaugeRenderWorker reports the status and now also records a health breadcrumb.
• Bounded recovery (NativeRendererRecovery): HudNativeHost and DriftGaugeWindow previously marked themselves failed permanently on any status failure, leaving the HUD blank until Wisp restarted. For E_OUTOFMEMORY only, they now release the renderer (and, for the HUD, its composition target) and create it again after 1 s, 2 s and 4 s, up to three attempts; a presented frame restores the budget. Any other HRESULT keeps the established unavailable state. A retired worker's late status callback is ignored (per-worker revision), so it cannot fail or restart its replacement.
• Test seam: an optional nativeFailureForTest callback on DirectCompositionDevice.Create (threaded through both worker constructors like the existing beforeDrawForTest) stands in for the native library on create, motion_channel and upload_texture. It is not consulted on the per-frame draw, present or wait paths.

Not changed: the legacy DirectCompositionAnalogHost path (dormant whenever HudNativeHost is attached, which every HUD window does), the crash-report exception-name allowlists, the native library, and the behaviour for non-transient HRESULTs.

Verification

• New DirectCompositionDeviceFailureTests (selected by the PR gate): a native stub returning E_OUTOFMEMORY from create and from texture upload produces a NativeRendererException with the HRESULT, a reported (false, 0x8007000E) status, a RendererFailed health breadcrumb (stage RendererCreate for the HUD worker) and a completed worker, through both AnalogHudRenderWorker and DriftGaugeRenderWorker on a real hidden HWND with the WARP device.
• New NativeRendererRecoveryTests: three growing retries for E_OUTOFMEMORY, none for DXGI_ERROR_DEVICE_HUNG, DXGI_ERROR_DEVICE_REMOVED or E_FAIL, and budget reset.
• HudNativeHostTests (nightly WPF runtime suite) now injects a transient 0x8007000E into the live shared HUD and checks that the old target is destroyed, the status does not become unavailable, and a new target and worker present again; the existing permanent-failure check follows unchanged.
• Locally (Release, SDK 8.0.425, same MSVC-built native renderer): see the checklist in the PR conversation for the exact commands and results.

Residual risks

• Retrying other transient-looking codes (for example DXGI_ERROR_DEVICE_REMOVED on hybrid iGPU/dGPU laptops) is deliberately out of scope; it would need separate evidence.
• The nightly suite is not part of the merge gate, so the HudNativeHostTests extension is verified by the local run noted below and the next nightly run.