Skip to content

fix(windows): end sandbox processes when the gateway is killed - #5299

Open
ashwind-nvidia wants to merge 11 commits into
NVIDIA:windowsfrom
ashwind-nvidia:fix/5298-gateway-kill-sandboxes/ashwind-nvidia
Open

ashwind-nvidia wants to merge 11 commits into
NVIDIA:windowsfrom
ashwind-nvidia:fix/5298-gateway-kill-sandboxes/ashwind-nvidia

Conversation

@ashwind-nvidia

@ashwind-nvidia ashwind-nvidia commented Nov 5, 2026 •

Copy link
Copy Markdown

Summary

With the MXC driver built into the gateway, hard-killing the gateway left its sandbox processes running with no owner, and they kept the gateway's sockets open so a new gateway could not bind the port. Two changes fix it: each sandbox generation now runs in its own kill-on-close job object owned by the MXC driver, so every process of that sandbox ends with the driver (and on stop/delete), and mio is updated to 1.2.1, which creates Windows sockets as non-inheritable so no child process can keep the gateway's port bound.

Related Issue

Closes #5298

Changes

  • Per-generation job in the driver. New openshell-driver-mxc job.rs (GenerationJob): a job object with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE. run_lifecycle_inner creates one per sandbox generation and spawns openshell-supervisor and wxc-exec inside it. Each process is created suspended, assigned to the job, then resumed, so it cannot start children outside the job. If a process cannot be assigned, the spawn fails and so does sandbox creation (fail closed). When the runtime pair exits (stop, delete, or either process dying) the job is terminated and dropped, which also ends the boundary and workload tree that Child::kill on the two direct children did not reach. If the driver's host process dies, Windows closes the handle and ends the tree. Because the ownership lives in the driver, it carries over when the driver moves out of the gateway.
  • Gateway-level job removed. openshell_gateway::process_lifetime::kill_children_on_exit() and its windows dependency are gone; the driver job replaces it.
  • mio 1.2.0 → 1.2.1 in Cargo.lock (only the mio entry changes: version and checksum; its dependencies are unchanged and tokio 1.52.1 requires mio ^1.2.0). mio 1.2.1 creates Windows sockets with WSA_FLAG_NO_HANDLE_INHERIT (1.2.0 used plain socket(), which leaves the handle inheritable and lets every spawned child keep a copy). On Windows I confirmed that with 1.2.0 tokio's listener, accepted and outbound sockets are all inheritable, and with 1.2.1 none are. Two Windows tests in openshell-server keep this pinned (the bound listener is not inheritable, and it still accepts connections).
  • E2E scenarios. run-mxc-e2e.ps1 gains two default scenarios. gateway-kill: create a sandbox, hard-kill the gateway, require every process under that gateway to end and a replacement gateway to bind the port. gateway-restart: run the gateway on a persisted SQLite database, send a real Ctrl-C, require a logged graceful shutdown, a clean exit and every sandbox process gone, then restart on the same database and require the record to still be listed, its workload not to be running, and the record to be deletable with its name reusable. It pins today's no-recovery behavior and needs updating when recovery is added.
  • Docs. WINDOWS.md, the driver README (new "Gateway exit and restart" section) and the build-openshell-mxc-windows skill now say that gateway exit ends workloads and that MXC has no restart recovery, unlike the Docker, Podman and VM drivers, which stop and recover sandboxes on a graceful restart.

Behavior: sandboxes end when the driver's host process exits, including a normal gateway restart. Before, they were left running without an owner. No restart recovery is added: after a restart on the same database the sandbox record is still listed with a stale phase (Provisioning after a graceful stop, Ready after a hard kill), nothing restarts the workload, and the record can be deleted and its name reused.

Testing

  • Checks appropriate to the affected code and behavior pass.

  • openshell-driver-mxc unit tests: 82 pass on Windows x64, including 5 new in job.rs: a spawned process runs after resume and terminate ends it; dropping the job ends the tree; a failed spawn is reported; assignment failure fails the spawn; hard-killing the process that owns the job ends the root and its grandchild (re-runnable by cargo test -p openshell-driver-mxc job::).

  • Windows full workspace (cargo nextest run --profile ci --workspace, CI exclusions): 6666 of 6666 pass.

  • cargo fmt --all -- --check, cargo metadata --locked, and Windows cargo clippy for openshell-driver-mxc, openshell-server, openshell-gateway with the CI flags (-D warnings -A dead-code -A unused-imports -A clippy::unused-async) are clean.

  • End-to-end, Windows 11 x64, three running sandboxes, fresh release build of this head: hard-killing the gateway leaves no sandbox process, the port can be bound, and a new gateway starts. Deleting the sandboxes one at a time removes exactly that sandbox's openshell-supervisor, wxc-exec, openshell-windows-sandbox and workload cmd.exe. Before the fix (earlier measurement), hard-killing left 9 processes and the port bound (os error 10048) in 5 of 5 runs.

  • New gateway-kill scenario: passes with this change (6 sandbox processes ended, port rebound) and fails against a gateway built without the job object (all 6, including the workload, survive).

  • Linux: the earlier head (6d3d2ce) passed cargo fmt, cargo check --locked, clippy -D warnings and the workspace nextest (8062 of 8079; the 17 failures only occur when the container runs as root and pass as a normal user). The later commits touch only Windows-only driver code, Cargo.lock (drops the gateway's windows edge) and docs/PowerShell; Linux was not re-run for them.

  • Graceful stop and restart, Windows x64: gateway-restart passes (Ctrl-C logged "Shutdown signal received", exit code 0, 6 sandbox processes ended, record persisted in Provisioning, not recovered, delete and re-create work). A graceful stop also ends the workloads against a gateway built without the job object, so this scenario documents graceful behavior; the job is what covers a hard kill.

  • ARM64, native: Windows 11 build 28000 on a DGX Station GB300, built with the repo's windows-msvc.ps1 build aarch64-pc-windows-msvc (all four binaries checked as ARM64 PE). openshell-driver-mxc unit tests 82 of 82 pass (including the 5 job tests); gateway-kill, gateway-restart and fs-rw pass; deleting three sandboxes one at a time removes exactly that sandbox's processes and leaves nothing at the end. The last two commits were applied to a checkout of the previous head as a patch, so the content equals this head.

Not tested: the full workspace nextest and clippy on ARM64, sandbox stop measured on its own (it shares the delete shutdown path), and the repository's CI for this PR (not run yet). The runner's port probe (GetActiveTcpListeners, #11) did not detect the listening gateway on the test host, so the e2e runs used a copy of the runner with a loopback connect probe; the repository's probe is unchanged.

Checklist

  • Follows Conventional Commits.
  • Commits are signed off (DCO).
  • Architecture docs updated (if applicable): no architecture change.

When the MXC driver runs inside the gateway, a hard-killed gateway left
its sandbox processes (wxc-exec, openshell-supervisor and
openshell-windows-sandbox) running with no owner. They also kept the
gateway's listening socket open, so the port stayed bound and a new
gateway failed to start with "Only one usage of each socket address"
(os error 10048).

There were two causes, so there are two changes:

- The gateway now puts itself in a kill-on-close job object before it
  spawns anything. Every sandbox process joins the job, and Windows ends
  them all when the gateway process exits for any reason, including a
  hard kill. If the job cannot be created the gateway logs a warning and
  continues.
- mio creates listening sockets with plain socket(), which leaves the
  handle inheritable, and child processes are started with handle
  inheritance enabled, so each sandbox process held a copy of the
  socket. The gateway, health and metrics listeners now clear the inherit
  flag right after bind.

Either change releases the port on its own; the job object also removes
the orphaned processes.

Tests: unit tests check that the process is in a kill-on-close job and
that a bound gateway listener is not inheritable and still accepts
connections. On a native Windows 11 x64 host with three running
sandboxes, hard-killing the gateway left 9 processes and an unbindable
port in 5 of 5 runs before this change. With it, no process survived,
the port could be bound and a new gateway started in 3 of 3 runs on the
final build, and in 5 of 5 runs on an earlier build that differed only by
two lint-driven edits. The existing MXC end-to-end scenarios give the
same results with and without the change.

Signed-off-by: ashwind <ashwind@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

All contributors have signed the DCO ✍️ ✅
Posted by the DCO Assistant Lite bot.

@ashwind-nvidia

Copy link
Copy Markdown
Author

I have read the DCO document and I hereby sign the DCO.

@shailendra-nv shailendra-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cleanup addresses the orphaned processes in today's in-process MXC setup. I'd like us to keep it compatible with the Windows repository split and make the shutdown behavior clear before carrying it forward. I've left comments on job ownership, lifecycle validation, and the socket fix. This is a code and design review; I haven't run this PR head on ARM64. Separately, linked issue #5298 still lacks state:accepted, and the ready PR has no area label; the existing acceptance and readiness checks need attention.

Comment thread crates/openshell-gateway/src/main.rs Resolved

@shailendra-nv shailendra-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes based on the comments in my earlier review. Please address job ownership so cleanup stays compatible with the standalone MXC driver, and document and validate the gateway shutdown/restart behavior, including ARM64 coverage. The existing inline comments also describe the possible Mio update to simplify the socket fix.

Addresses review feedback on NVIDIA#5299. mio 1.2.1 creates Windows sockets
with WSA_FLAG_NO_HANDLE_INHERIT (1.2.0 used plain socket(), which is
inheritable), so a sandbox process the gateway spawns can no longer keep
a copy of the gateway's listening socket and hold its port after a
gateway crash.

Update mio from 1.2.0 to 1.2.1 and remove the Win32 wrapper that cleared
the inherit flag after bind, together with the windows dependency it
added to openshell-server; windows is now a test-only dependency there.
The listener tests stay, so a regression to an inheritable socket still
fails the build.

Cargo.lock changes only the mio entry (version and checksum); its
dependency list is unchanged and tokio 1.52.1 requires mio ^1.2.0.

Tests: on Windows x64, bound_gateway_listener_is_not_inheritable and
listener_accepts_connections pass without the wrapper. With three
sandboxes running, hard-killing the gateway with the job object disabled
leaves the nine sandbox processes alive but the port can be bound and a
new gateway starts (2 of 2 runs), so the mio update alone frees the
port. With the job object enabled no process survives and the port can
be bound (3 of 3 runs). cargo fmt, cargo check --locked and clippy with
-D warnings pass for openshell-gateway and openshell-server on Linux.

Refs NVIDIA#5298

Signed-off-by: ashwind <ashwind@nvidia.com>
Addresses review feedback on NVIDIA#5299. The gateway-wide job object lived in
the gateway binary, so the cleanup was tied to whichever process hosts
the driver and would not survive moving the driver out of the gateway.
It also left stop and delete killing only the two direct children
(openshell-supervisor and wxc-exec), not the boundary and workload tree
beneath them.

Move the ownership into the MXC driver. run_lifecycle_inner creates one
job object with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE per sandbox
generation and spawns the supervisor and wxc-exec inside it. Each
process is created suspended, assigned to the job, then resumed, so it
cannot start children outside the job. If a process cannot be assigned
to its job, spawning fails and so does sandbox creation. After the
runtime pair exits (stop, delete, or either process dying) the job is
terminated and dropped, which ends the rest of the tree; if the driver's
host process dies, Windows closes the job handle and does the same.

Remove the gateway-level job (process_lifetime.rs, its call in main.rs
and the windows dependency it added to openshell-gateway). The
non-inheritable listener sockets from mio 1.2.1 are unchanged.

Tests: on Windows x64, cargo nextest for the workspace passes (6666 of
6666), the 82 openshell-driver-mxc tests pass, and cargo fmt and clippy
with -D warnings are clean. New tests in job.rs cover: a spawned process
runs after resume and terminate ends it; dropping the job ends the tree;
a failed spawn is reported; assignment failure fails the spawn;
hard-killing the process that owns the job ends the root and its
grandchild. With a fresh release gateway and three sandboxes,
hard-killing the gateway leaves no sandbox process, the port can be
bound and a new gateway starts. Deleting sandboxes one at a time removes
exactly that sandbox's supervisor, wxc-exec, boundary and workload
processes.

Refs NVIDIA#5298

Signed-off-by: ashwind <ashwind@nvidia.com>
Add a default process_container scenario to run-mxc-e2e.ps1 that checks
sandbox cleanup when the gateway dies without warning. It creates a
sandbox with a long-running marked workload, records the process tree
under that scenario's own gateway (other sandboxes on the host are never
counted or touched), hard-kills the gateway, and requires every recorded
process to end within 30 s and a replacement gateway to bind the same
port.

Document the scenario in the driver README and update the scenario-count
wording in the README and the build-openshell-mxc-windows skill.

Tests: on Windows x64 with the per-sandbox job, the scenario passes
(6 sandbox processes ended, port rebound). Against a gateway built
without the job object it fails, with all 6 processes, including the
workload, still running. The runner's existing port probe
(GetActiveTcpListeners) does not see the listener on the test machine
(issue NVIDIA#11), so both runs used a copy of the runner with a loopback
connect probe; the probe in the repository is unchanged.

Refs NVIDIA#5298

Signed-off-by: ashwind <ashwind@nvidia.com>
Addresses review feedback on NVIDIA#5299: cover a graceful gateway shutdown
and restart with persisted state, not only a hard kill, and record what
happens to the workload and the sandbox record.

Add a default process_container scenario to run-mxc-e2e.ps1. It runs the
gateway on a SQLite file, creates a sandbox with a long-running marked
workload, sends a real Ctrl-C, and requires the gateway to log a
graceful shutdown and exit and every recorded sandbox process to end. It
then restarts the gateway on the same database and requires the sandbox
record to still be listed, its workload not to be running (MXC has no
restart recovery: gateway_manages_lifecycle is false, so the gateway's
shutdown sweep and startup recovery are skipped), and the record to be
deletable with its name reusable. The scenario pins this no-recovery
behavior and needs updating when recovery is added.

Start-Gw takes an optional database URL (default stays in-memory) and
clears an inherited "ignore Ctrl-C" setting, which a session such as ssh
can pass to the gateway and which would otherwise stop it from ever
seeing a graceful-stop request. gateway-kill now shares the new
Wait-SandboxTree and Wait-TreeGone helpers with the new scenario.

Tests: on Windows x64 the scenario passes (6 sandbox processes ended on
Ctrl-C, gateway exit code 0, record persisted in phase Provisioning and
not recovered, delete and re-create work). gateway-kill, fs-rw and
fs-readonly still pass after the refactor. A graceful stop ends the
workloads with or without the per-generation job (checked against a
gateway built without it), so this scenario documents graceful behavior;
the job is what covers a hard kill. The runner's existing port probe
(GetActiveTcpListeners) does not see the listener on the test machine
(issue NVIDIA#11), so the runs used a copy of the runner with a loopback
connect probe; the probe in the repository is unchanged.

Refs NVIDIA#5298

Signed-off-by: ashwind <ashwind@nvidia.com>
Addresses review feedback on NVIDIA#5299: the Windows docs and build guidance
should say that gateway exit terminates workloads and that MXC does not
provide the stop-and-recover behavior of the Docker, Podman and VM
drivers.

Add a "Gateway exit and restart" section to the MXC driver README and
replace the one-line limits entries there and in WINDOWS.md. They state
that stop, delete, a graceful gateway stop and a gateway crash or hard
kill end every sandbox process, workloads included; that MXC reports
gateway_manages_lifecycle false, so the gateway skips its shutdown sweep
and startup recovery for it; and that after a restart on the same
database the sandbox record is still listed with a stale phase, no
workload runs, and the record can be deleted and its name reused. The
section says that behavior for a driver running as its own process is
not defined yet. Add matching guidance to the build-openshell-mxc-windows
skill, including how to report the behavior and that gateway-restart is
not evidence for the per-generation job.

The documented behavior was observed on Windows x64 with the
gateway-kill and gateway-restart E2E scenarios; ARM64 has not been run.

Refs NVIDIA#5298

Signed-off-by: ashwind <ashwind@nvidia.com>

@shailendra-nv shailendra-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 8305936. The three earlier requests have been addressed: the MXC driver owns a job per generation, Mio supplies the socket fix, and the lifecycle docs and restart scenarios are present, with native ARM64 results reported. This ownership fits the planned split better.

I found one remaining startup cleanup gap and a missing assertion in the graceful-restart test, detailed inline. I verified the launch ordering with an isolated native ARM64 Win32 experiment that pauses the owner between process creation and job assignment; I have not built or run the full updated gateway/E2E suite locally.

The external driver's behavior on gateway loss can remain a separate decision under NVIDIA/OpenShell-Windows#20; this PR should not need to add restart recovery. The existing issue-acceptance and maintainer-approval gates also still need attention.

Comment thread crates/openshell-driver-mxc/src/job.rs Resolved
Addresses review feedback on NVIDIA#5299: the gateway-restart scenario only
printed the gateway's exit code and treated the "Shutdown signal
received" log line as proof of a graceful stop. That line is logged
before cleanup is awaited, so a shutdown that began could still fail
(nonzero exit) while the job removed the processes and a replacement
gateway started, and the scenario would report a successful graceful
restart.

Decide the outcome in a small pure function, Get-GracefulStopFailure:
the gateway must exit with code 0 and have logged the shutdown message;
an unavailable exit code, a nonzero code (including 0xC000013A, the
Ctrl-C default-handler exit) or a missing message is a failure, and a
nonzero code is reported before the missing message. The scenario now
checks it right after the gateway exits, before the process tree check
and before the replacement gateway is started.

Cover the failing cases in tasks/scripts/test-mxc-remote-tests.ps1, which
loads the function from the runner by parsing it, like the existing pure
helper regressions. The decision only protects the scenario if it is
consulted at the right place, so the same script also checks the runner's
syntax tree: exactly one Send-CtrlC and one check, the check after Ctrl-C,
a throw on its result, and all of that before the replacement Start-Gw. Moving
or dropping the call fails the script.

Tests: test-mxc-remote-tests.ps1 passes (7 new checks: clean stop, nonzero
exit after the message, 0xC000013A, exit 0 without the message, missing
exit code, nonzero reported before the missing message, and the call order
in the runner) on Windows x64 and native ARM64. Removing the throw from a
copy of the runner makes the order check fail on both. The gateway-restart scenario still passes on both (gateway
exit code 0, replacement gateway starts). With the check told the exit
code was 1 (injected in a scratch copy of the runner), the scenario fails
with "gateway exited with code 1 after Ctrl-C; a graceful shutdown must
exit 0" and no replacement gateway is started, on both. The runs used a
scratch copy of the runner whose readiness probe connects to the port
(Get-NetTCPConnection under-reports listeners on the test hosts); the
probe in the repository is unchanged.

Refs NVIDIA#5298

Signed-off-by: ashwind <ashwind@nvidia.com>

@shailendra-nv shailendra-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for two reproducible issues: a hard kill during startup can leave a suspended child outside the generation job, and the new persisted-database scenario breaks when the checkout path contains spaces. Details and suggested fixes are in the inline comments.

Reviewed the full 12-file diff and relevant surrounding code at 8305936 against windows. The unchanged job module passed 6/6 test entries in an isolated Windows ARM64 harness using cached dependencies. The startup race was reproduced with an injected pause between creation and assignment; the database URL split was reproduced with an argument-capture executable. Full PR build, socket tests, and live MXC scenarios were not rerun.

Comment thread crates/openshell-driver-mxc/src/job.rs Resolved
Addresses review feedback on NVIDIA#5299. The driver used to create the
supervisor and wxc-exec suspended, assign them to the generation's
kill-on-close job, then resume them. If the gateway was hard-killed
between creation and assignment, that process was left running outside
any job.

Create each root process with CreateProcessW and
PROC_THREAD_ATTRIBUTE_JOB_LIST so it is a member of the job from its
first instruction; there is no window left. A handle list restricts
what the child inherits to its own stdio. The new child module owns
spawning (pipes, environment block, command-line quoting, exit watcher)
and replaces tokio::process for these processes.

Tests: unit tests cover born-in-job (with a negative control), a
hard-killed owner leaving no process, an unusable job failing the
spawn, and a stress test that hard-kills a host mid-spawn and counts
processes. Reinstating the old create-then-assign order makes the
stress test fail 6/6. 95 driver tests pass on x64 and ARM64, and the
gateway-kill, gateway-restart and fs-rw scenarios pass against real MXC
on both.

Refs NVIDIA#5298

Signed-off-by: ashwind <ashwind@nvidia.com>
Addresses review feedback on NVIDIA#5299. The gateway-restart scenario passes a
SQLite URL built from the checkout path, and Start-Process -ArgumentList
does not quote array elements, so a path containing a space was split into
two arguments and the gateway refused to start.

Build the argument string with the existing Quote-NativeArgument helper,
as the other native launches in this script already do.

Tests: from a checkout path containing a space, gateway-restart failed
before this change (the gateway rejected the split URL) and passes after.
It still passes from a path without spaces, and the script-level tests in
test-mxc-remote-tests.ps1 are unchanged.

Signed-off-by: ashwind <ashwind@nvidia.com>
@ashwind-nvidia

Copy link
Copy Markdown
Author

Thanks for the re-review. All three points are addressed and pushed (d3f40a2, 490571d, ab1aa4d).

Startup window: the supervisor and wxc-exec are now created directly inside the job with PROC_THREAD_ATTRIBUTE_JOB_LIST, so there is no separate assign step left. Handle inheritance is limited to the child's own stdio. Tests cover born-in-job (with a negative control) and a stress test that hard-kills a host mid-spawn and counts processes. Putting the old create-then-assign order back fails that test 6/6. Unit tests and gateway-kill, gateway-restart and fs-rw pass on x64 and on the GB300 (ARM64).

Exit code: gateway-restart now fails if the gateway's exit code is nonzero, before the restart, with a structural test of the check order.

Database URL: the gateway arguments now go through Quote-NativeArgument. Verified from a path with a space: it failed before and passes now.

@shailendra-nv shailendra-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed ab1aa4d, including the full updated diff and the new Windows process-launch implementation. The previous startup assignment gap, persisted database URL quoting issue, and graceful-shutdown exit-code assertion are addressed. Creation-time job membership and explicit stdio-only inheritance are now in place.

Two new P2 issues remain in child.rs: the named-pipe registration error path closes a handle after Tokio has already consumed and closed it, and the new environment-block test assumes the host has no other OPENSHELL_* variables. Details and reproductions are inline.

Validation on Windows ARM64: the unchanged child.rs and job.rs modules in an isolated harness produced 18/19 passing tests with the configured host environment; the environment-block assertion failed on inherited OPENSHELL_WXC_EXEC_PATH. With only that variable removed from the test process, 19/19 passed, including the mid-spawn owner-kill test. All 21 remote-script regression checks passed. An argument-capture probe confirmed the quoted database URL remains one argument with spaces. A separate Tokio 1.52.1 ownership probe confirmed from_raw_handle closes the pipe when registration fails. These are focused probes using cached dependencies, not a full PR build or live MXC E2E run.

Approval and merge remain on hold. The current OpenShell / PR Issue Acceptance check also fails because linked issue #5298 lacks state:accepted.

Addresses review feedback on NVIDIA#5299.

make_pipe closed the server end of a named pipe again when tokio failed
to register it. tokio takes ownership of the handle before registering
and closes it itself on failure, so this was a double close that could
hit an unrelated handle another thread had just been given. The error is
now propagated without a second close. A test seam (make_pipe_with)
covers both halves: the launcher must not touch the handle after a
failed registration, and tokio does close it.

The environment-block test assumed the host had no other OPENSHELL_*
variables. build_env_block now has a variant over an explicit host
environment, so the test is hermetic, and a second test checks that the
real host's variables are kept and sorted.

A review of the launcher found further defects, fixed here:
- start_kill reported "access denied" for a process that had just
  exited, before the exit watcher had published it. It now asks the
  process itself.
- The exit watcher reported a clean exit when waiting for the process or
  reading its exit code failed. It now reports an error.
- resolve_program tried a bare name before NAME.exe, searched the current
  directory through empty or relative PATH entries, ignored a PATH set
  for the new process, and gave a confusing failure for batch files. It
  now follows Windows: NAME.exe only, absolute PATH entries only, the new
  process's PATH, and a clear error for .bat and .cmd.
- CreateProcessW errors lost their kind (not found, access denied); the
  kind is kept and attribute errors carry context.
- The attribute list buffer is now pointer-aligned, raw_arg appends as
  documented, environment names that would corrupt the block are
  rejected, and the sort key upper-cases one UTF-16 unit at a time as
  Windows does.

Test hardening: the handle-inheritance test no longer depends on a file
lock (it raced with other tests' spawns and a cmd grandchild); it checks
the one child with DuplicateHandle and CompareObjectHandles. cmd.exe is
started with /d, the owner-kill test waits for the tree instead of a
fixed 700 ms, and the mid-spawn stress test fails if no generation was
ever started.

Tests: every new test fails when its fix is reverted, including the
mid-spawn stress test against the old create-then-assign order. 107
driver tests pass on Windows x64 and ARM64, in a clean and in a
configured environment (OPENSHELL_* set, empty and relative PATH
entries), with 1 and 32 test threads, and from a checkout path with
spaces. gateway-kill, gateway-restart and fs-rw pass against real MXC on
both.

Refs NVIDIA#5298

Signed-off-by: ashwind <ashwind@nvidia.com>
@ashwind-nvidia

Copy link
Copy Markdown
Author

Thanks, both are fixed in 6250418.

Double close: you're right, tokio closes the handle itself when registration fails (I checked the 1.52.1 source). The error is now propagated without a second close, and a test covers it.

Env test: it now runs against a synthetic host environment, so it no longer depends on OPENSHELL_* being unset.

I also had the launcher reviewed again and fixed a few more issues (a kill racing an exited process, the exit watcher reporting a clean exit when waiting failed, program lookup details, a flaky inheritance test). Tests pass on x64 and ARM64.

@shailendra-nv shailendra-nv added test:e2e Requires end-to-end coverage test:windows Run native Windows x64 and ARM64 lint/tests on PR mirrors labels Nov 18, 2026 — with ChatGPT Codex Connector

Copy link
Copy Markdown
Collaborator

/windows-ci

@github-actions

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/5299 does not exist yet. A maintainer needs to comment /ok to test 62504183d2d4ff16f0beebc8102c54db7f69ff57 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

Copy link
Copy Markdown
Collaborator

/ok to test 6250418

@sam-india-007

sam-india-007 commented Nov 18, 2026 •

Copy link
Copy Markdown

Windows ARM64 CI: ⌛ running

Commit bd4be8c, requested by @shailendra-nv

Copy link
Copy Markdown
Collaborator

Re-reviewed 6250418: both outstanding findings are fixed, and I found no additional actionable code issues. The two threads are resolved. All 31 tests from the exact updated child.rs/job.rs modules passed in an isolated Windows ARM64 harness with the configured host environment and 32 test threads; this is focused validation with cached dependencies, not a full workspace build.

Triggered /windows-ci, enabled test:windows and test:e2e, and admitted this SHA with /ok to test. Branch Checks passed. Windows x64/ARM64 and the remaining runtime suites are still running.

CI blocker: Linux x64 VM runtime build fails twice on the downloaded runtime's SHA-256. nix/pkgs/vm-runtime.nix is identical in this PR and its windows base (blob eb3819c09f7018532281372d2419cfca14c1a60b): expected sha256-kA6ZQz53geBp2zHx8tv+D8VcGkSrnC74Op/60H4fmck=, received sha256-mZwSUhs9IZ0pPLxzVs4AbRLdYWkUlHwP4lmfsbeiUGw=. The received value is already the x64 Linux pin on main, which also has updated ARM64 Linux/macOS pins. This is inherited CI drift, not introduced by this PR. The runtime pins need to be synchronized through the normal source update process before core E2E can pass.

Approval and merge remain pending successful CI.

Copy link
Copy Markdown
Collaborator

Final CI results for unchanged head 6250418:

  • Windows MSVC: x64 and ARM64 lint/tests both passed. Each architecture ran 6,691 tests: 6,691 passed, 43 skipped.
  • Branch Checks: passed, including Linux/macOS Rust checks/tests and SDK checks. Helm and the direct PR checks also passed.
  • Core E2E: failed solely because the VM runtime build hit the inherited checksum mismatch documented above. All Docker, Podman, and Kubernetes suites that ran passed; VM and dependent integration jobs could not run.

The updated runtime hashes are present in main commit a4c848a (the nix/pkgs/vm-runtime.nix portion). Synchronize the appropriate runtime pins into the Windows source, then rerun E2E; any source update will need validation at its new SHA.

No new code-review findings; all review threads are resolved. Issue #5298 is state:accepted. I have not approved or merged because the requested all-tests-passing condition is not met.

Signed-off-by: Shailendra Singh <shailendras@nvidia.com>

Copy link
Copy Markdown
Collaborator

/windows-ci

Copy link
Copy Markdown
Collaborator

/ok to test bd4be8c

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage test:windows Run native Windows x64 and ARM64 lint/tests on PR mirrors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(windows): sandbox processes outlive a killed gateway and keep its port bound

3 participants