Skip to content

bug(mxc): ETW audit consumer drops every event when wxc-exec emits none, with no warning #5585

Closed

Description

@ashwind-nvidia

User Story

I work on OpenShell's native Windows support and ran the MXC audit script (run-ocsf-audit.ps1) with etw_audit = true on a DGX Station GB300 (Windows 11 ARM64). I directly encountered this: the OCSF audit log was empty even though the sandbox launched and the OS emitted its audit events. This affects anyone who relies on the ETW to OCSF audit trail on a Windows build that does not emit events from the wxc-exec process.

Problem Statement

The ETW consumer attributes each Sandboxing-provider event to a sandbox by matching the PID and process start key of the wxc-exec the driver launched. That match is the only starting point. The first events that match teach the consumer the sandbox's identity, activity and correlation-vector keys, and the events from the shared OS broker process (an svchost) are then linked to the sandbox through those keys.

On some builds wxc-exec emits no events of its own. Nothing ever matches, so every broker event is buffered, ages out after a few seconds and is dropped. The drop is logged only at debug level, and the existing "events lost" counter covers only queue overflow, so the audit summary says "no dropped ETW records observed" while the OCSF log stays empty and the gateway log shows no warning.

The process the OS launches inside the sandbox is a child of the registered wxc-exec and is named by the ProcessLaunched event (processId), so the sandbox can be identified without needing an event from wxc-exec.

Separately, the released MXC emits different event names than the consumer looks up (it was written against a development build that MXC later renamed): SetUILimitsOnJob instead of ApplyUILimits, and SandboxLaunchDetails instead of CreateProcessInSandbox and Win32kLockdownApplied. Even once attribution works, three of the seven audit event types never fire. This reproduces on x64 as well.

run-ocsf-audit.ps1 also reports FAIL with "audit workload did not produce its artifact" even when the audit trail is complete. Its sandbox workload is a batch file the script writes into the shared folder before creating the sandbox, and the sandbox cannot run a batch file that the host created beforehand (the workload dies about 37 ms after starting; cmd reports "The process cannot access the file because it is being used by another process"). The sandbox can write new files, read the pre-existing file, and run a batch file it created itself. Running the workload inline avoids it.

Impact / Why This Matters

On the affected builds the Windows audit trail is silently empty: sandbox lifecycle, policy and process events are not recorded, and nothing tells the operator.

Acceptance Criteria

  • On a build where wxc-exec emits no events, the sandbox's OS events are still attributed to that sandbox and written to the OCSF log.
  • Attribution uses the launched process's parent and requires the registered process start key, so a recycled PID cannot be attributed to the wrong sandbox, and events from concurrent sandboxes are never mixed up.
  • Events that reach the consumer but can never be attributed are counted and produce a rate-limited warning at the default log level. Events the audit trail never writes do not trigger it on a healthy host.
  • The three audit event types the released MXC renamed (UI restrictions, win32k lockdown, process launch identity) fire: SetUILimitsOnJob is reported as "UI restrictions applied", and SandboxLaunchDetails yields the process-launch row (executable only, no arguments) and the win32k lockdown row.
  • run-ocsf-audit.ps1 reports verdict PASS when all seven event types fire: its workload runs and produces its artifact, including when -ShareDir contains spaces.
  • Behavior on builds where wxc-exec does emit events is unchanged.

Reproduction Steps

  1. On Windows ARM64 (build 28000.3231), in an elevated shell, run run-ocsf-audit.ps1 with etw_audit = true and one sandbox.
  2. Read the printed summary and the produced openshell-ocsf.<date>.log.
  3. Optionally capture the provider independently: logman create trace probe -p {f6ec123e-314e-400b-9e0a-151365e23083} 0xFFFFFFFFFFFFFFFF 0xFF -o probe.etl -ets, then decode with tracerpt, and run the gateway with --log-level info,mxc_etw=trace.

Actual: "0 of 7 expected event types fired", 0 OCSF events, empty OCSF log. The independent capture shows 16 provider events, all from one broker process and none from the registered wxc-exec. The consumer logs buffering uncorrelated MXC audit event for 11 of them and then dropping unattributed (aged out), at debug level only.

Expected: the events are attributed to the sandbox and written. On x64 (build 28120.3272) the same script gets 4 of 7 types and 5 events, because there wxc-exec itself emits 4 events (CreateProcessSecurityEnvironment and CreateProcessAsUserInsideSecurityEnvironment, two each) that anchor the attribution.

Environment

                 x64                       GB300 ARM64
Windows build    28120.3272                28000.3231
wxc-exec         1.0.0 and 1.0.1           1.0.1 (ARM64 and x64 builds)
events captured  20 (4 from wxc-exec)      16 (0 from wxc-exec)
audit result     4 of 7 types, 5 events    0 of 7 types, 0 events

OpenShell: windows branch at 5b9e5b9, native build, backend = "process_container". The wxc-exec version and the architecture of its binary make no difference (both were tested on each machine).

Logs

DEBUG mxc_etw: registered driver-owned MXC audit launch pid=6684 ...
DEBUG mxc_etw: buffering uncorrelated MXC audit event pid=11084 process_start_key=Some(...)
DEBUG mxc_etw: dropping unattributed (aged out) SandboxEngineCreateProcess (id=0) { ... } pid=11084

Not in scope

The underlying limitation that a sandbox cannot run a batch file created in its read-write folder before launch (it likely also affects the host-service e2e scenario) is not changed here; only the script's workload is. The Process Activity rows also record the actor as wxc-exec with the emitting PID, which is the broker's PID on builds where the broker emits the events. Both are tracked separately.

Metadata

Metadata

Assignees

No one assigned

    Labels

    os:windowsIssue affects Windows hostsstate:acceptedA maintainer decided OpenShell should pursue this issuestate:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions