Skip to content

fix(supervisor): reject unsupported live filesystem policy changes - #5327

Merged
shailendra-nv merged 4 commits into
NVIDIA:windowsfrom
nv-rahool:fix/5326-reject-live-filesystem-changes/nv-rahool
Nov 19, 2026
Merged

shailendra-nv merged 4 commits into
NVIDIA:windowsfrom
nv-rahool:fix/5326-reject-live-filesystem-changes/nv-rahool

Conversation

@nv-rahool

@nv-rahool nv-rahool commented Nov 6, 2026 •

Copy link
Copy Markdown

Summary

Backport of #5234 to windows. A live policy update that adds a filesystem path is accepted by the gateway, reported as loaded by the supervisor, and shown as effective, but the running workload never receives the new path. On Windows the MXC driver maps filesystem grants into the container only when the sandbox is created. After this change the supervisor compares the candidate's filesystem section with the one applied at startup and rejects a changed revision as failed, with "filesystem policy cannot be changed on a running sandbox; recreate the sandbox to apply filesystem changes".

Related Issue

Closes #5326

The windows counterpart of #5041, which #5234 closes on main.

CI prerequisite #5736 was split out and merged first. The merge preview against that updated windows base changes only the supervisor and policy documentation; the tested head remains 9dbf16901.

Changes

  • openshell-supervisor (lib.rs): add validate_live_filesystem_policy, the same check as fix(supervisor): reject live filesystem policy changes that cannot take effect #5234's ConfigRuntime::validate_live_filesystem_policy, except that it returns Err(miette::miette!(..)) like the other validators in this crate instead of miette::bail!. It compares include_workdir and the read_only / read_write paths as sets, so reordered paths still load. The poll loop runs it before reload_gateway_configuration_runtime and reports a mismatch as GatewayRuntimeReloadError::PolicyValidation. Rejection then follows the existing validation-failure handling: the revision is reported failed, and the configured policy_validation_failure_mode (default fail_closed) applies.
  • On windows the poll loop still lives in lib.rs (there is no config_runtime.rs), so the check is a free function that takes the applied policy (current_endpoint_policy) instead of a method.
  • Tests (ported from fix(supervisor): reject live filesystem policy changes that cannot take effect #5234, driven through run_policy_poll_loop_with_client because windows has no ConfigRuntime):
    • live_filesystem_changes_reject_polled_policy_without_advancing_active_revision: sandbox and global sources × added read-only path, added read-write path, and a filesystem section added where there was none; checks the failed status report, fail-closed versus retained policy, and the OPA generation.
    • live_filesystem_change_poll_reports_failure_then_accepts_network_only_repair: a rejected revision followed by a network-only revision with reordered paths loads. Unlike the main test, it sets the transparent TCP state as ready, because the windows reload path still checks it for the TCP fixture.
  • poll_rejects_first_tcp_expansion_and_reports_previous_policy_active: its second revision now keeps the first revision's filesystem section and changes only the network rules. It was built from a TCP-only policy with no filesystem section, so the new check rejected it before the TCP check ran. The gateway already refuses a sandbox policy update that drops the filesystem section. main no longer has the transparent TCP check, so fix(supervisor): reject live filesystem policy changes that cannot take effect #5234 did not need this.
  • Docs: the "Added filesystem paths" row in docs/how-it-works/policies/manage-policies.mdx, same text as fix(supervisor): reject live filesystem policy changes that cannot take effect #5234.
  • Follow-up (second commit): backport of fix(supervisor): reject live filesystem policy changes that cannot take effect #5234's dbf59381 ("preserve startup filesystem policy on retries"). The check now compares each candidate with the policy installed at startup (PolicyPollLoopContext::endpoint_policy) and first restores on the candidate only the proxy baseline paths that startup installed. Without it, a global policy, which keeps its raw gateway form, was compared with the enriched startup policy and rejected as a filesystem change. On windows the first poll never acknowledges a global policy, so this happened on the first poll. Adds global_policy_retry_uses_installed_filesystem_after_missing_initial_acknowledgement, driven through the poll loop like the other tests.

Testing

Maintainer verification (9dbf16901)

  • Branch Checks passed: the Linux x64, Linux ARM64, and macOS ARM64 Rust test jobs ran 24,001 passing test executions with 95 skipped. Rust lint/build checks and Python, Go, TypeScript, protobuf, and repository checks also passed.
  • GitHub Windows MSVC CI passed for x64 and ARM64. The external /windows-ci ARM64 run also passed on this exact commit; its full log confirms check, release build, workspace tests, unsupported-driver contracts, selective feature tests, artifact verification, and packaging.
  • Branch E2E passed, including all three VM-driver builds, both managed/external VM E2E suites, single-node and HA K3s conformance, and all other enabled container/integration jobs. The first single-node attempt passed installation and every conformance assertion, then hung during QEMU teardown and hit the job timeout. A targeted retry passed completely on the same commit; no harness changes were made.
  • Final pre-review status: 111 checks passed, 10 intentionally skipped, none pending or failed. Issue bug(supervisor): live filesystem policy changes are reported loaded but not applied on Windows #5326 is accepted and assigned to @nv-rahool.
  • Local pin verification passed: all three VM pins match official release asset digests and upstream main; K3s YAML parses, the existing 1.36.3+k3s1 version is retained, and the versioned installer serves bytes matching SHA-256 46177d4c99440b4c0311b67233823a8e8a2fc09693f6c89af1a7161e152fbfad. git diff --check passed.

Windows test executions (intentional reruns counted separately; filtered-out and compile-time-disabled tests excluded):

Suite Total Passed Skipped Failed
GitHub x64 workspace 6,704 6,661 43 0
GitHub ARM64 workspace 6,704 6,661 43 0
External ARM64 workspace 6,704 6,661 43 0
External ARM64 unsupported built-in drivers 1 1 0 0
External ARM64 default registry contract 1 1 0 0
External ARM64 selective feature matrix (four configurations) 8 8 0 0
Total 20,122 19,993 129 0

The broad local commit hook could not complete on native Windows ARM64 because the unrelated Python grpcio-tools==1.78.0 dependency failed to build. These focused CI-data commits used scoped verification under CONTRIBUTING.md; full validation runs in CI.

Approval was submitted for 9dbf16901. The subsequent maintainer-list check reports that this account is not listed in MAINTAINERS.md; the build and test runs above passed. The operator requested merging the separate pin PR and then this PR using the existing evidence, without an additional pin-PR CI run.

Original supervisor-change verification

  • Checks appropriate to the affected code and behavior pass
    • Linux (Rust 1.95.0): cargo test -p openshell-supervisor --lib passes 158/158; cargo fmt --all --check is clean; cargo clippy -p openshell-supervisor --all-targets --no-deps -- -D warnings is clean. With the new check disabled, both new tests fail.
    • Windows x64 (Windows Insider 28020.3382):
      • cargo test -p openshell-supervisor --lib passes 155/155.
      • cargo clippy -p openshell-supervisor --all-targets --no-deps -- -D warnings is clean.
      • mise run windows:build:x64 succeeds.
    • Through a gateway running the MXC driver on Windows x64 (process container, wxc-exec --probe reports baseContainerApiPresent: true), with a sandbox created with read_write: [A] whose workload keeps writing to A and tries to write to B:
      • Before: policy set --wait adding B printed "Policy version 2 loaded (active version: 2)" and exited 0; policy list showed version 2 as Loaded. 25 s later the workload was still running and still could not write to B.
      • After: the same command prints "Policy version 2 failed to load: filesystem policy cannot be changed on a running sandbox; recreate the sandbox to apply filesystem changes" and exits 1; policy list shows version 2 as Failed with that error. The supervisor logs the fail-closed transition (default fail_closed). A following network-only update with the filesystem section unchanged loads as version 3. Removing a path is still rejected by the gateway, and a new sandbox created with A and B can write to B.
    • Second commit:
      • Linux: cargo test -p openshell-supervisor --lib passes 159/159; fmt and clippy (--no-deps) are clean. With the first commit's comparison restored, the new test fails at the first poll (fail-closed).
      • Windows x64 (Windows Insider 28020.3382): cargo test -p openshell-supervisor --lib passes 156/156; cargo clippy -p openshell-supervisor --all-targets --no-deps -- -D warnings is clean; mise run windows:lint:x64 and mise run windows:build:x64 succeed.
      • Through a gateway with the MXC driver on Windows x64, with a sandbox under a global policy with read_write: [A] and one network rule. On this host the baseline paths /etc and /var/log exist (as C:\etc and C:\var\log), and the supervisor logged CONFIG:ENRICHED at startup.
        • With the first commit only, the supervisor logged CONFIG:FAIL_CLOSED ("filesystem policy cannot be changed…") 0.24 s after startup, and again after a network-only global update.
        • With both commits, the supervisor logged "Policy reloaded successfully (global)" for version 1 at startup and for version 2 after the network-only update. A third global update adding a read_write path still failed closed with the same message.
        • The sandbox-policy reproduction from bug(supervisor): live filesystem policy changes are reported loaded but not applied on Windows #5326 behaves as with the first commit alone: adding a path fails with the "recreate the sandbox" message, a network-only update loads, removing a path is refused by the gateway, and a new sandbox created with both paths can write to the added one.
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable): not added. The new unit tests run the supervisor's poll loop against a scripted gateway; fix(supervisor): reject live filesystem policy changes that cannot take effect #5234 adds no E2E test either.

Not tested: ARM64, the MXC end-to-end scenarios, and the full workspace test suite. Running cargo clippy -p openshell-supervisor --all-targets -- -D warnings without --no-deps on Windows stops in openshell-extension-core (transport.rs:185, unused async), which this change does not touch.

Note for review: after a rejected revision, sandbox get lists the stored candidate policy (including the added path) together with ConfigurationInvalid, while policy get shows the active policy. This is existing gateway behaviour and is unchanged here.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable): the policy docs row is updated; no architecture change.

Backport of NVIDIA#5234 to windows. The gateway accepts live policy updates
that only add filesystem paths, and the supervisor reported such a
revision as loaded after reloading the network policy alone. Filesystem
grants are applied when the sandbox starts (the MXC driver maps them
into the container at create), so the running workload never received
the added paths.

Compare each reload with the filesystem grants applied at startup and
reject a changed revision before replacing the running policy. The
rejection uses the existing policy validation failure handling, so the
revision is reported as failed and the configured failure mode applies.
Network-only reloads and reordered paths still load.

Closes NVIDIA#5326

Signed-off-by: Rahool Paliwal <rpaliwal@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Backport of the follow-up commit in NVIDIA#5234. Startup adds the proxy baseline
paths that exist on the host to the policy it installs, but a global
policy keeps its raw gateway form. The live filesystem check compared that
raw form with the enriched policy, so it rejected an unchanged filesystem
section. On windows the first poll never acknowledges a global policy, so
the check ran on the first poll and, with the default fail_closed mode,
blocked network traffic.

Compare each candidate with the policy installed at startup and first
restore on it only the baseline paths that startup installed.

Signed-off-by: Rahool Paliwal <rpaliwal@nvidia.com>
@nv-rahool

Copy link
Copy Markdown
Author

Pushed a second commit that backports #5234's follow-up dbf59381 (preserve startup filesystem policy on retries). On windows the issue it fixes is reached on every first poll of a global policy, because the first poll only acknowledges sandbox-sourced policies (initial_policy_ack_candidate). Whenever startup had added any proxy baseline path, the first commit failed such sandboxes closed. Details and test results are in the description.

@shailendra-nv shailendra-nv added test:e2e Requires end-to-end coverage test:windows Run native Windows x64 and ARM64 lint/tests on PR mirrors labels Nov 17, 2026 — with ChatGPT Codex Connector

Copy link
Copy Markdown
Collaborator

/ok to test 7520703

@github-actions

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/5327 does not exist yet. A maintainer needs to comment /ok to test 75207039106650c4dd896bbbd17c74d4b6f64058 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

Unaddressed breaking changes

Documented additive filesystem policy updates now fail the entire revision and may block network traffic. The PR acknowledges and documents the change, but this train disallows breaks.

1. policy — ACKED

openshell policy set and openshell policy update revisions that add filesystem paths changed from loading network changes while retaining existing runtime filesystem permissions to rejecting the entire revision.

  • Impact: CLI and API consumers receive a failed revision instead of success, and the default fail_closed mode blocks sandbox network traffic until a valid revision loads.
  • Action: Preserve the prior additive-filesystem revision behavior for this no-break train and defer whole-revision rejection to a break-allowing release with the existing recreate-the-sandbox guidance.
  • PR: fix(supervisor): reject unsupported live filesystem policy changes #5327
Review details
  • Execution: complete
  • Candidate: PR #5327 (ca94f4f7405af7b3d16ba07126b1310e4475f573)
  • Baseline: 5b9e5b91afb10afbce8d9be09e2513a7efc22a21 (5b9e5b91afb10afbce8d9be09e2513a7efc22a21)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

Copy link
Copy Markdown
Collaborator

/windows-ci

@sam-india-007

sam-india-007 commented Nov 18, 2026 •

Copy link
Copy Markdown

Windows ARM64 CI: ✅ passed

Commit 9dbf169, requested by @shailendra-nv, 1h 13m, full log

Step Result Duration
if ($env:PR_NUMBER) { Import-Module .\scripts\pr-ci\OpenShellPrCi.psm1; Assert-PrCiVariables -PrN... ✅ passed 0s
& .\scripts\setup-windows-arm64-runner.ps1 -VerifyOnly ✅ passed 1s
$env:PATH = "$(Split-Path -Parent $env:RUSTC_WRAPPER);$env:PATH"; $found = (Get-Command sccache -... ✅ passed 0s
New-Item -ItemType Directory -Force (Split-Path -Parent $env:SCCACHE_ERROR_LOG) | Out-Null; & $en... ✅ passed 2s
& $env:RUSTC_WRAPPER --zero-stats ✅ passed 0s
& .\scripts\checkout-openshell.ps1 -Ref $env:OPENSHELL_REF ✅ passed 4s
& .\scripts\set-openshell-version.ps1 -Version $env:OPENSHELL_VERSION ✅ passed 0s
Set-Location $env:OPENSHELL_CHECKOUT_DIR ✅ passed 0s
mise run --skip-tools windows:check:arm64 ✅ passed 4m 01s
mise run --skip-tools windows:build:arm64 ✅ passed 22m 28s
mise run --skip-tools windows:test:arm64 ✅ passed 21m 45s
mise run --skip-tools windows:test:unsupported:arm64 ✅ passed 23m 21s
mise run --skip-tools windows:artifacts ✅ passed 2s
& "$env:CI_PROJECT_DIR\scripts\package-windows-release.ps1" ✅ passed 40s

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

Unaddressed breaking changes

Documented additive filesystem policy updates now reject the entire revision and may block network traffic. The PR acknowledges and documents the change, but this train disallows breaks.

1. policy — ACKED

openshell policy set and openshell policy update revisions that add filesystem paths changed from loading network changes while retaining existing runtime filesystem permissions to rejecting the entire revision.

  • Impact: CLI and API consumers receive a failed revision instead of success, and the default fail_closed mode blocks sandbox network traffic until a valid revision loads.
  • Action: Preserve the prior additive-filesystem revision behavior for this no-break train and defer whole-revision rejection to a break-allowing release with the existing recreate-the-sandbox guidance.
  • PR: fix(supervisor): reject unsupported live filesystem policy changes #5327
Review details
  • Execution: complete
  • Candidate: PR #5327 (ca94f4f7405af7b3d16ba07126b1310e4475f573)
  • Baseline: 5b9e5b91afb10afbce8d9be09e2513a7efc22a21 (5b9e5b91afb10afbce8d9be09e2513a7efc22a21)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

Use the published VM runtime hashes already corrected on main by 59ddef6 and a4c848a so VM E2E builds can fetch their runtime dependencies.

Signed-off-by: Shailendra Singh <shailendras@nvidia.com>

Copy link
Copy Markdown
Collaborator

/ok to test 3df4002

Copy link
Copy Markdown
Collaborator

/windows-ci

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

Unaddressed breaking changes

Documented additive filesystem policy updates now reject the entire revision and may block network traffic. The PR provides migration guidance, but this no-break train does not permit the behavioral break.

1. policy — ACKED

openshell policy set and openshell policy update revisions that add filesystem paths changed from loading network changes while retaining existing filesystem permissions to rejecting the entire revision.

  • Impact: CLI and API consumers receive a failed revision instead of success, and default fail_closed mode blocks sandbox network traffic until a valid revision loads.
  • Action: Preserve additive-filesystem revision behavior on this no-break train and defer whole-revision rejection to a break-allowing release with the existing recreate-the-sandbox guidance.
  • PR: fix(supervisor): reject unsupported live filesystem policy changes #5327
Review details
  • Execution: complete
  • Candidate: PR #5327 (f6b5666a4226e9d82a54e88d2501a21ed1e9661b)
  • Baseline: 5b9e5b91afb10afbce8d9be09e2513a7efc22a21 (5b9e5b91afb10afbce8d9be09e2513a7efc22a21)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

Fetch the installer for the configured K3s v1.36.3+k3s1 release and verify its SHA-256, following the upstream fix in NVIDIA#5223.

Signed-off-by: Shailendra Singh <shailendras@nvidia.com>

Copy link
Copy Markdown
Collaborator

/ok to test 9dbf169

Copy link
Copy Markdown
Collaborator

/windows-ci

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

Unaddressed breaking changes

Documented additive filesystem policy updates now reject the entire revision and may block network traffic. The PR acknowledges and documents the change, but this train disallows breaks.

1. policy — ACKED

openshell policy set and openshell policy update revisions that add filesystem paths changed from loading network changes while retaining existing filesystem permissions to rejecting the entire revision.

  • Impact: CLI and API consumers receive a failed revision instead of success, and default fail_closed mode blocks sandbox network traffic until a valid revision loads.
  • Action: Preserve additive-filesystem revision behavior on this no-break train and defer whole-revision rejection to a break-allowing release with the existing recreate-the-sandbox guidance.
  • PR: fix(supervisor): reject unsupported live filesystem policy changes #5327
Review details
  • Execution: complete
  • Candidate: PR #5327 (db0f1631359b48fb29008294ff4ab56c8aacab7a)
  • Baseline: 5b9e5b91afb10afbce8d9be09e2513a7efc22a21 (5b9e5b91afb10afbce8d9be09e2513a7efc22a21)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@shailendra-nv shailendra-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the startup filesystem-policy comparison, proxy baseline normalization, rejection/recovery handling, and regression tests; no actionable findings. The VM-runtime hashes match upstream release digests, and the K3s installer is pinned to the existing configured version.

Verified this exact commit: Branch Checks, Windows x64/ARM64 CI, the external /windows-ci ARM64 check/build/test/package run, and full E2E all passed. The single-node K3s test assertions also passed in the initial run, which later timed out during VM teardown; a targeted retry passed completely. Detailed results and test counts are recorded in Testing.

shailendra-nv added a commit that referenced this pull request Nov 19, 2026
Refresh the three VM-runtime artifact hashes from upstream and pin the K3s installer to its configured release, preserving existing versions.

CI evidence from #5327 at 9dbf169 is attached to #5736. Both changed files match that tested source; the operator requested reusing those results without a standalone CI rerun.

Closes #5734

Signed-off-by: Shailendra Singh <shailendras@nvidia.com>
@shailendra-nv
shailendra-nv merged commit 8a93cab into NVIDIA:windows Nov 19, 2026
193 of 196 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage test:windows Run native Windows x64 and ARM64 lint/tests on PR mirrors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(supervisor): live filesystem policy changes are reported loaded but not applied on Windows

3 participants