Repository navigation
fix(mxc): attribute ETW audit events through the launched process - #5586
shailendra-nv merged 5 commits into
Conversation
On some Windows builds every Sandboxing provider event comes from the shared OS broker process and none from the wxc-exec the driver spawned. The consumer attributes events by matching that wxc-exec's PID and process start key, and uses the first events that match to learn the identity, activity and correlation-vector keys that link the broker's events to the sandbox. With no event from wxc-exec nothing ever matched: every event was buffered, aged out and dropped, the OCSF log stayed empty, and the only trace was a debug line. Anchor attribution on ProcessLaunched as well. It names the process the OS started inside the sandbox (processId). When that process is a child of a live registered wxc-exec, and the parent still carries the registered process start key, the event belongs to that sandbox and its keys are cross-linked, so the rest of the create burst attributes through the existing correlators. The parent lookup uses NtQueryInformationProcess; PROCESS_BASIC_INFORMATION needs the Win32_System_Kernel feature on the driver's windows dependency (no new crates, no lockfile change). The buffer now replays until a pass resolves nothing new, so a chain of keys (activity, then correlation vector, then identity) resolves in one drain. Make the loss visible. Events that reach the consumer but are discarded without ever being attributed are now counted, published to the session health (EtwSession::unattributed_event_count) and reported with a rate-limited warning at the default log level. Only events the emitter writes to the audit trail count, so the key-less BrokeredCreateProcessLaunchResult that every healthy host drops does not raise it. Tests: 93 of 93 openshell-driver-mxc tests pass on Windows x64 and on native ARM64, 17 of them new: the anchor and its guards (recycled PID, unreadable or unregistered parent, exited process, other event names), two concurrent sandboxes kept apart, a replay of the create burst captured on the GB300, multi-pass replay, drop counting, the reporter, and a live parent lookup. cargo fmt and clippy with the CI flags are clean on x64. Audit run (run-ocsf-audit.ps1) with etw_audit on: on a GB300 (ARM64, build 28000.3231, wxc-exec 1.0.1) it went from 0 of 7 event types and 0 OCSF events to 4 of 7 types and 5 events in 3 classes, with all 16 provider events coming from the broker; on x64 (build 28120.3272, wxc-exec 1.0.0) it stays at 4 of 7 types and 5 events and logs no warning. The runs used a copy of the audit script whose readiness probe connects to the port (Get-NetTCPConnection under-reports listeners on the test hosts) and a gateway log level of info,mxc_etw=trace for diagnostics. The remaining three types are missing because this OS emits different event names; that is addressed separately. Refs NVIDIA#5585 Signed-off-by: ashwind <ashwind@nvidia.com>
Three of the seven audit event types never fired because the released MXC emits different event names than the consumer looks up (the consumer was written against a development build that MXC later renamed). With attribution working, they were the only gap left in the audit trail. - SetUILimitsOnJob is the released name of ApplyUILimits: report it as "UI restrictions applied" (Config State Change [5019]). The older name still maps the same way. - SandboxLaunchDetails replaces CreateProcessInSandbox and Win32kLockdownApplied. Emit the process launch row (Process Activity [1007], executable only, arguments omitted as before) from its commandLine, and the win32k lockdown row (5019) when disallowWin32kSystemCalls is 1. Either row is skipped when its field is absent. disallowWin32kSystemCalls is attached to the row as an unmapped field. - Both events count as audited, so losing them raises the unattributed warning like the other mapped events. The mappings are pure functions (map_config_state_as, map_launch_details) so they are tested without the audit sink. Tests: 100 of 100 openshell-driver-mxc tests pass on Windows x64 and on native ARM64, 7 of them new: the UI-limits row, the older name, the launch and win32k rows from one event, no leaked command-line arguments, the win32k row only for flag 1, sparse events, and the audited-name contract. They use field values captured from a real SandboxLaunchDetails event. cargo fmt and clippy with the CI flags are clean on x64. Audit run (run-ocsf-audit.ps1, etw_audit on, one sandbox): on a GB300 (ARM64, build 28000.3231, wxc-exec 1.0.1) 4 of 7 types and 5 events became 7 of 7 types and 8 events, and on x64 (build 28120.3272, wxc-exec 1.0.0) 4 of 7 and 5 events became 7 of 7 and 8 events, in 3 OCSF classes, with no unattributed warning. The script's overall verdict stays FAIL because its sandbox workload does not produce its artifact, which is unrelated to the event types. Refs NVIDIA#5585 Signed-off-by: ashwind <ashwind@nvidia.com>
run-ocsf-audit.ps1 reported "audit workload did not produce its artifact" and verdict FAIL even when the audit trail was complete. Its sandbox workload was a batch file (audit-workload.cmd) that the script wrote into the shared folder before creating the sandbox. The sandbox cannot run a batch file the host created beforehand: the workload fails about 37 ms after it starts and cmd reports "The process cannot access the file because it is being used by another process". The same sandbox can write new files, read and copy the pre-existing file, and run a batch file it created itself; waiting before running the batch, turning ETW audit off, or using --no-tty instead of --detach made no difference. Run the workload inline instead: cmd /d /c "echo ... 1> hello.txt". It writes hello.txt relative to its working directory (the share), so the command needs no path and also works when -ShareDir contains spaces. The wait for the artifact, the verdict logic and the output are unchanged. This changes the test workload only. The underlying limitation (a sandbox cannot run a batch file that was in its read-write folder before launch) is not changed here; it is likely also why the host-service e2e scenario, which runs a host-created batch file, fails, which was not verified. Tests: run-ocsf-audit.ps1 with etw_audit on, one sandbox, on a GB300 (ARM64, build 28000.3231, wxc-exec 1.0.1) and on x64 (build 28120.3272, wxc-exec 1.0.0), with the two preceding commits: verdict PASS, exit code 0, 7 of 7 event types, workload output present and 8 OCSF events in 3 classes on both, no unattributed warning. Also PASS on x64 with a -ShareDir containing a space. Before this change the script reported FAIL on both with the workload artifact missing. The runs used a copy of the script whose readiness probe connects to the port (Get-NetTCPConnection under-reports listeners on the test hosts) and a gateway log level of info,mxc_etw=trace for diagnostics; the repository's probe is unchanged. Refs NVIDIA#5585 Signed-off-by: ashwind <ashwind@nvidia.com>
shailendra-nv
left a comment
There was a problem hiding this comment.
Requesting changes for the three inline findings: child PID reuse can misattribute an entire audit burst to another sandbox; unrelated system-wide provider traffic produces false audit-gap warnings; and shutdown leaves pending losses unreported.
Reviewed the complete diff at c360877 and the surrounding attribution/driver code. An isolated Rust probe using the PR's attribution implementation with injected parent lookup reproduced the child-PID reuse and unrelated-event counting issues, and confirmed that a normal final drain retains fresh unresolved events without counting them. I did not rerun the full native test suite or live MXC/ETW qualification.
Documentation follow-up: crates/openshell-driver-mxc/README.md:417–423 still says broker-emitted ARM64 events cannot be correlated and audit qualification fails. Update that limitation to match the resulting supported behavior.
Addresses review feedback on NVIDIA#5586. Child PID reuse: a queued ProcessLaunched event was attributed by looking up its child's parent when it was replayed. If that child had exited and Windows had reused the PID for another sandbox's child, the event (and the rest of its burst, through the activity id) was attributed to the wrong sandbox. The parent is now read once, when the event is first received, and kept with the event; a replay uses that snapshot. If the child is already gone at that point nothing can be proven and the event is not attributed. Foreign provider traffic: the Sandboxing provider is system-wide, so events from another gateway or consumer were counted as this gateway's audit gap, including on an idle gateway. Unresolved audited events are now counted as possible loss only if this gateway had a live sandbox while they waited (registered, retired or forgotten); otherwise they are counted separately as unowned and never warned about. The warning no longer claims a confirmed gap. Shutdown: the final warning ran before pending events were accounted for, so an unresolved event younger than the pending TTL was never counted. The consumer now replays once on disconnect, counts what is still unresolved, publishes the count, and only then forces the warnings. Tests: regressions for child PID reuse (and the legitimate late registration), an idle gateway receiving foreign events, a sandbox that came and went (retired and forgotten) while an event waited, and shutdown with a fresh unresolved event and with one that resolves on the final replay. Putting the live PID lookup back on replay fails the PID-reuse test. 108 driver tests pass on x64 and ARM64, clippy is clean on x64, and the real OCSF audit passes 7/7 on x64 and on a GB300 (ARM64) with no attribution warning. Signed-off-by: ashwind <ashwind@nvidia.com>
|
Thanks for the review. All three are fixed and pushed in 6e41ff7. PID reuse: the launched child's parent is now read once when the event first arrives and kept with it, so a replay no longer looks the PID up again. Your scenario is a regression test, and it fails with the old lookup. One small window remains between ETW emitting the event and us reading the parent, since the payload has no child start key. Foreign traffic: events that wait while this gateway has no live sandbox are now counted separately and never warned about. Otherwise they count as possible loss, and the warning no longer claims a confirmed gap. Shutdown: we replay once, count what's left, then force the warning. Tests pass on x64 and ARM64. |
shailendra-nv
left a comment
There was a problem hiding this comment.
Re-reviewed 6e41ff7. Requesting changes remains necessary:
- P1: the original child-PID attribution finding is still reproducible when reuse occurs before first consumer receipt; details are in the existing review thread. The snapshot fixes replay-time reuse but does not validate the child against the emitted event.
- P2: the new foreign-traffic classification can silently discard this gateway's own delayed events after sandbox exit; see the new inline finding.
The idle-gateway false-positive case and shutdown finalization are fixed. An isolated Rust harness using the current index and processor methods exercised six focused cases: it confirmed those fixes and legitimate late registration, and reproduced the two remaining defects. Windows parent lookup and OCSF emission were stubbed, so this is not a live MXC/ETW test or a full crate test run.
I have not requested /windows-ci or admitted the other CI yet because the review is not clean. Approval and merge remain gated on resolving these findings and passing the applicable checks. The README limitation noted in the previous review is also unchanged.
Addresses review feedback on NVIDIA#5586. Child PID reuse before the first receipt: the launched child's parent was read when the event reached the consumer, but an event arrives 1.2 to 1.9 s after it is emitted (measured on x64 and ARM64), so the PID can already have been reused by a child of another sandbox. The event names the child only by PID (no start key, no creation time), so the process now holding the PID must at least have existed when the event was emitted: one created later is refused, and nothing is attributed when either time is unknown. The trace is opened with raw timestamps and each event is placed on the wall clock by pairing the performance counter with the precise system time at receipt, instead of relying on ETW's conversion from the session's start. The launched child is created 5 to 11 ms before its event on both architectures; the bound allows 1 ms of clock noise. Its limits (creation-time resolution, a clock step during delivery) are documented where the constant is defined. Delayed delivery: classification used the time an event was received, so an event from a short-lived sandbox that was forgotten before the event arrived was called foreign traffic and never warned about. It now uses the time the event was emitted, and an unknown emission time counts as possible loss if this gateway ever had a sandbox. forget() stamps the sandbox's liveness only for a sandbox the index knew: the driver forgets it again on delete, which must not look like activity. README: the limitation that broker-emitted ARM64 events cannot be correlated and that audit qualification fails no longer holds; it now describes the supported behaviour, the bound, and how unattributed events are reported. Tests: regressions for PID reuse before the first receipt, the fail-closed cases, the slack boundary, a real child against a real raw QPC reading (so a wrong clock assumption fails), the conversion and its refusals, events received after forget, forgetting an unknown sandbox twice, an unknown emission time, and the raw-timestamp trace mode. Each fails when its fix is reverted (one reversed guard is redundant with the age conversion). 122 driver tests pass on Windows x64 and ARM64 in a clean and a configured environment, with 1 and 32 test threads and from a path with spaces; the real OCSF audit passes 7/7 on x64 and ARM64; fmt, workspace check and clippy pass on Linux. Signed-off-by: ashwind <ashwind@nvidia.com>
|
Thanks for the re-review. Both findings and the README are addressed in e332cc9. PID reuse: events reach the consumer 1.2 to 1.9 s after emission (measured), so a snapshot at receipt was too late. The process now holding the PID must have been created no later than the event was emitted (1 ms slack), otherwise it isn't attributed, including when either time is unknown. Timestamps are now raw QPC, converted at receipt. Delayed delivery: classification uses the emission time, and forget() only stamps for a sandbox it knew. Limits (creation-time resolution, a clock step during delivery) are noted in the code. Tests pass on x64 and ARM64, and the real audit passes 7/7 on both. |
|
/windows-ci |
|
/ok to test e332cc9 |
shailendra-nv
left a comment
There was a problem hiding this comment.
Re-reviewed the update from 6e41ff7 to e332cc9 and the affected attribution, buffering, shutdown accounting, tests, and documentation. Both remaining findings are addressed; no additional actionable findings.
Validation: 9 focused regression checks passed in an isolated Windows harness using the unchanged attribution/time/process code and the actual Windows APIs (log emission was stubbed). This is not a full driver-suite or live MXC audit run. Requested /windows-ci and standard CI for this exact head; approval and merge remain pending successful checks.
Windows ARM64 CI: ❌ failedCommit
Last 24 lines of the failed step |
There was a problem hiding this comment.
Re-reviewed e332cc9. The remaining attribution and delayed-delivery findings are fixed, and no additional actionable findings remain.
Validation: all Branch Checks passed; Windows x64 and ARM64 lint and test steps passed in run 44381667468; the 9 focused Windows regression checks also passed. The earlier review threads are resolved.
Issue #5585 now has state:accepted following the maintainer's explicit acceptance. GitHub's Issue Acceptance check still does not recognize the body link for this non-default target, and its Maintainer Approval check uses the main-branch roster. Verified that neither status is a required check for the windows target; the PR's review decision is APPROVED.
Summary
On some Windows builds every Sandboxing-provider event comes from the shared OS broker process and none from the
wxc-execthe driver launched. The ETW consumer could not attribute any of them, dropped them all silently, and the OCSF audit log stayed empty. This PR attributes events through the process thatProcessLaunchednames, makes unattributed drops visible, and maps the three events the released MXC renamed, so all seven audit event types fire, and makes the audit script's own test workload runnable so the script can report PASS.Related Issue
Closes #5585
Changes
ProcessLaunchedcarriesprocessId, the process the OS started inside the sandbox. When that process is a child of a live registeredwxc-exec, and the parent still carries the registered process start key, the event belongs to that sandbox and its activity id is cross-linked. The rest of the create burst then attributes through the existing identity/activity/correlation-vector correlators. The parent lookup usesNtQueryInformationProcess. A recycled PID is rejected because the parent's start key must equal the registered one, and the anchor applies only toProcessLaunched.EtwSession::unattributed_event_count) and reported with a rate-limited warning at the default log level. Only events the emitter writes to the audit trail count: the key-lessBrokeredCreateProcessLaunchResultthat every healthy host drops does not trigger it (an earlier version of this change did, and was corrected after a healthy x64 run raised the warning).PROCESS_BASIC_INFORMATIONneeds theWin32_System_Kernelfeature on the driver'swindowsdependency. No new crates and noCargo.lockchange.SetUILimitsOnJobnow produces the "UI restrictions applied" row (the olderApplyUILimitsstill works).SandboxLaunchDetails, which replacesCreateProcessInSandboxandWin32kLockdownApplied, produces the process-launch row (executable only, arguments omitted as before) fromcommandLine, and the win32k lockdown row whendisallowWin32kSystemCallsis1; either row is skipped when its field is absent. Both events count as audited for the unattributed warning. The mappings are pure functions so they are tested without the audit sink.run-ocsf-audit.ps1reported "audit workload did not produce its artifact" and FAIL even when the audit trail was complete: its sandbox workload was a batch file the script wrote into the shared folder before creating the sandbox, and the sandbox cannot run a batch file the host created beforehand (the workload dies about 37 ms after starting;cmdreports "The process cannot access the file because it is being used by another process"). The workload now runs inline (cmd /d /c "echo ... 1> hello.txt"), writing relative to its working directory (the share), so it needs no path and works when-ShareDircontains spaces. The artifact wait, verdict logic and output are unchanged.etw_consumer.rs, the driver'sCargo.tomlandrun-ocsf-audit.ps1change.Testing
openshell-driver-mxctests pass on Windows x64 and on native ARM64; 24 are new (17 for attribution, 7 for the renamed events). The attribution tests cover the anchor and its guards (recycled PID, unreadable or unregistered parent, exited process, other event names), two concurrent sandboxes kept apart, a replay of the create burst captured on the GB300, multi-pass replay, drop counting for age-out and overflow, that an event the audit never writes is not counted, the reporter's rate limiting, and a live parent lookup against a real child process. The renamed-event tests use field values captured from a realSandboxLaunchDetailsevent and cover the UI-limits row, the older name, the launch and win32k rows from one event, no leaked command-line arguments, the win32k row only for flag1, sparse events, and the audited-name contract.cargo fmt --checkandcargo clippy -p openshell-driver-mxc --all-targetswith the CI flags (-D warnings -A dead-code -A unused-imports -A clippy::unused-async) are clean on x64.run-ocsf-audit.ps1,etw_audit = true, one sandbox), fresh checkout ofwindowsat 5b9e5b9, before and after:After the first commit alone the GB300 reached 4 of 7 types and 5 events (same as x64); the second commit adds the three renamed types on both.
Audit script verdict,
run-ocsf-audit.ps1with the third commit: PASS (exit 0, 7 of 7 types, workload output present, 8 OCSF events in 3 classes, no unattributed warning) on the GB300 and on x64, and on x64 with a-ShareDircontaining a space. Without it the script reported FAIL on both with the workload artifact missing.Workload investigation (x64, one change at a time): the workload fails with the batch file; it still fails with ETW audit off, with
--no-ttyinstead of--detach, and after a 6 s delay; it works as an inline command, as a batch file created inside the sandbox, and copying the pre-existing batch file works (so the sandbox can read it, just not run it). The same result was confirmed on the GB300.On the GB300 all 16 provider events (captured independently with
logman/tracerpt) come from the broker and none fromwxc-exec, so the new anchor is what attributes them; the OCSF log went from empty to 4.9 KB. On x64, wherewxc-execemits 4 events itself, the result is unchanged and no warning is logged.wxc-exec1.0.0 vs 1.0.1 and its ARM64 vs x64 binary made no difference in the comparison runs.The audit runs used a copy of the script whose readiness probe connects to the port (
Get-NetTCPConnectionunder-reports listeners on these hosts) and a gateway log level ofinfo,mxc_etw=tracefor diagnostics; the repository copies are unchanged.Not tested: clippy on ARM64 (the pinned toolchain on the GB300 has no clippy component), more than one sandbox on a real host (covered by a unit test only), and the repository's CI for this PR (not run yet).
Not addressed in this PR
host-servicee2e scenario, which runs a host-created batch file, fails (not verified). To be raised with the MXC team.wxc-execwith the emitting PID, which is the broker's PID on builds where the broker emits the events. That was already the case on x64 and is not changed here.Checklist