Repository navigation
fix(supervisor): reject unsupported live filesystem policy changes - #5327
shailendra-nv merged 4 commits into
Conversation
Backport of NVIDIA#5234 to windows. The gateway accepts live policy updates that only add filesystem paths, and the supervisor reported such a revision as loaded after reloading the network policy alone. Filesystem grants are applied when the sandbox starts (the MXC driver maps them into the container at create), so the running workload never received the added paths. Compare each reload with the filesystem grants applied at startup and reject a changed revision before replacing the running policy. The rejection uses the existing policy validation failure handling, so the revision is reported as failed and the configured failure mode applies. Network-only reloads and reordered paths still load. Closes NVIDIA#5326 Signed-off-by: Rahool Paliwal <rpaliwal@nvidia.com>
Backport of the follow-up commit in NVIDIA#5234. Startup adds the proxy baseline paths that exist on the host to the policy it installs, but a global policy keeps its raw gateway form. The live filesystem check compared that raw form with the enriched policy, so it rejected an unchanged filesystem section. On windows the first poll never acknowledges a global policy, so the check ran on the first poll and, with the default fail_closed mode, blocked network traffic. Compare each candidate with the policy installed at startup and first restore on it only the baseline paths that startup installed. Signed-off-by: Rahool Paliwal <rpaliwal@nvidia.com>
|
Pushed a second commit that backports #5234's follow-up |
|
/ok to test 7520703 |
|
Label |
Compatibility review (advisory)Unaddressed breaking changes Documented additive filesystem policy updates now fail the entire revision and may block network traffic. The PR acknowledges and documents the change, but this train disallows breaks. 1. policy — ACKED
Review details
Advisory review; publication gates are unchanged. |
|
/windows-ci |
Windows ARM64 CI: ✅ passedCommit
|
Compatibility review (advisory)Unaddressed breaking changes Documented additive filesystem policy updates now reject the entire revision and may block network traffic. The PR acknowledges and documents the change, but this train disallows breaks. 1. policy — ACKED
Review details
Advisory review; publication gates are unchanged. |
|
/ok to test 3df4002 |
|
/windows-ci |
Compatibility review (advisory)Unaddressed breaking changes Documented additive filesystem policy updates now reject the entire revision and may block network traffic. The PR provides migration guidance, but this no-break train does not permit the behavioral break. 1. policy — ACKED
Review details
Advisory review; publication gates are unchanged. |
Fetch the installer for the configured K3s v1.36.3+k3s1 release and verify its SHA-256, following the upstream fix in NVIDIA#5223. Signed-off-by: Shailendra Singh <shailendras@nvidia.com>
|
/ok to test 9dbf169 |
|
/windows-ci |
Compatibility review (advisory)Unaddressed breaking changes Documented additive filesystem policy updates now reject the entire revision and may block network traffic. The PR acknowledges and documents the change, but this train disallows breaks. 1. policy — ACKED
Review details
Advisory review; publication gates are unchanged. |
shailendra-nv
left a comment
There was a problem hiding this comment.
Re-reviewed the startup filesystem-policy comparison, proxy baseline normalization, rejection/recovery handling, and regression tests; no actionable findings. The VM-runtime hashes match upstream release digests, and the K3s installer is pinned to the existing configured version.
Verified this exact commit: Branch Checks, Windows x64/ARM64 CI, the external /windows-ci ARM64 check/build/test/package run, and full E2E all passed. The single-node K3s test assertions also passed in the initial run, which later timed out during VM teardown; a targeted retry passed completely. Detailed results and test counts are recorded in Testing.
Refresh the three VM-runtime artifact hashes from upstream and pin the K3s installer to its configured release, preserving existing versions. CI evidence from #5327 at 9dbf169 is attached to #5736. Both changed files match that tested source; the operator requested reusing those results without a standalone CI rerun. Closes #5734 Signed-off-by: Shailendra Singh <shailendras@nvidia.com>
Summary
Backport of #5234 to
windows. A live policy update that adds a filesystem path is accepted by the gateway, reported as loaded by the supervisor, and shown as effective, but the running workload never receives the new path. On Windows the MXC driver maps filesystem grants into the container only when the sandbox is created. After this change the supervisor compares the candidate's filesystem section with the one applied at startup and rejects a changed revision as failed, with "filesystem policy cannot be changed on a running sandbox; recreate the sandbox to apply filesystem changes".Related Issue
Closes #5326
The
windowscounterpart of #5041, which #5234 closes onmain.CI prerequisite #5736 was split out and merged first. The merge preview against that updated
windowsbase changes only the supervisor and policy documentation; the tested head remains9dbf16901.Changes
openshell-supervisor(lib.rs): addvalidate_live_filesystem_policy, the same check as fix(supervisor): reject live filesystem policy changes that cannot take effect #5234'sConfigRuntime::validate_live_filesystem_policy, except that it returnsErr(miette::miette!(..))like the other validators in this crate instead ofmiette::bail!. It comparesinclude_workdirand theread_only/read_writepaths as sets, so reordered paths still load. The poll loop runs it beforereload_gateway_configuration_runtimeand reports a mismatch asGatewayRuntimeReloadError::PolicyValidation. Rejection then follows the existing validation-failure handling: the revision is reported failed, and the configuredpolicy_validation_failure_mode(defaultfail_closed) applies.windowsthe poll loop still lives inlib.rs(there is noconfig_runtime.rs), so the check is a free function that takes the applied policy (current_endpoint_policy) instead of a method.run_policy_poll_loop_with_clientbecausewindowshas noConfigRuntime):live_filesystem_changes_reject_polled_policy_without_advancing_active_revision: sandbox and global sources × added read-only path, added read-write path, and a filesystem section added where there was none; checks the failed status report, fail-closed versus retained policy, and the OPA generation.live_filesystem_change_poll_reports_failure_then_accepts_network_only_repair: a rejected revision followed by a network-only revision with reordered paths loads. Unlike themaintest, it sets the transparent TCP state as ready, because thewindowsreload path still checks it for the TCP fixture.poll_rejects_first_tcp_expansion_and_reports_previous_policy_active: its second revision now keeps the first revision's filesystem section and changes only the network rules. It was built from a TCP-only policy with no filesystem section, so the new check rejected it before the TCP check ran. The gateway already refuses a sandbox policy update that drops the filesystem section.mainno longer has the transparent TCP check, so fix(supervisor): reject live filesystem policy changes that cannot take effect #5234 did not need this.docs/how-it-works/policies/manage-policies.mdx, same text as fix(supervisor): reject live filesystem policy changes that cannot take effect #5234.dbf59381("preserve startup filesystem policy on retries"). The check now compares each candidate with the policy installed at startup (PolicyPollLoopContext::endpoint_policy) and first restores on the candidate only the proxy baseline paths that startup installed. Without it, a global policy, which keeps its raw gateway form, was compared with the enriched startup policy and rejected as a filesystem change. Onwindowsthe first poll never acknowledges a global policy, so this happened on the first poll. Addsglobal_policy_retry_uses_installed_filesystem_after_missing_initial_acknowledgement, driven through the poll loop like the other tests.Testing
Maintainer verification (
9dbf16901)/windows-ciARM64 run also passed on this exact commit; its full log confirms check, release build, workspace tests, unsupported-driver contracts, selective feature tests, artifact verification, and packaging.main; K3s YAML parses, the existing1.36.3+k3s1version is retained, and the versioned installer serves bytes matching SHA-25646177d4c99440b4c0311b67233823a8e8a2fc09693f6c89af1a7161e152fbfad.git diff --checkpassed.Windows test executions (intentional reruns counted separately; filtered-out and compile-time-disabled tests excluded):
The broad local commit hook could not complete on native Windows ARM64 because the unrelated Python
grpcio-tools==1.78.0dependency failed to build. These focused CI-data commits used scoped verification under CONTRIBUTING.md; full validation runs in CI.Approval was submitted for
9dbf16901. The subsequent maintainer-list check reports that this account is not listed inMAINTAINERS.md; the build and test runs above passed. The operator requested merging the separate pin PR and then this PR using the existing evidence, without an additional pin-PR CI run.Original supervisor-change verification
cargo test -p openshell-supervisor --libpasses 158/158;cargo fmt --all --checkis clean;cargo clippy -p openshell-supervisor --all-targets --no-deps -- -D warningsis clean. With the new check disabled, both new tests fail.cargo test -p openshell-supervisor --libpasses 155/155.cargo clippy -p openshell-supervisor --all-targets --no-deps -- -D warningsis clean.mise run windows:build:x64succeeds.wxc-exec --probereportsbaseContainerApiPresent: true), with a sandbox created withread_write: [A]whose workload keeps writing to A and tries to write to B:policy set --waitadding B printed "Policy version 2 loaded (active version: 2)" and exited 0;policy listshowed version 2 as Loaded. 25 s later the workload was still running and still could not write to B.policy listshows version 2 as Failed with that error. The supervisor logs the fail-closed transition (defaultfail_closed). A following network-only update with the filesystem section unchanged loads as version 3. Removing a path is still rejected by the gateway, and a new sandbox created with A and B can write to B.cargo test -p openshell-supervisor --libpasses 159/159; fmt and clippy (--no-deps) are clean. With the first commit's comparison restored, the new test fails at the first poll (fail-closed).cargo test -p openshell-supervisor --libpasses 156/156;cargo clippy -p openshell-supervisor --all-targets --no-deps -- -D warningsis clean;mise run windows:lint:x64andmise run windows:build:x64succeed.read_write: [A]and one network rule. On this host the baseline paths/etcand/var/logexist (asC:\etcandC:\var\log), and the supervisor loggedCONFIG:ENRICHEDat startup.CONFIG:FAIL_CLOSED("filesystem policy cannot be changed…") 0.24 s after startup, and again after a network-only global update.read_writepath still failed closed with the same message.Not tested: ARM64, the MXC end-to-end scenarios, and the full workspace test suite. Running
cargo clippy -p openshell-supervisor --all-targets -- -D warningswithout--no-depson Windows stops inopenshell-extension-core(transport.rs:185, unusedasync), which this change does not touch.Note for review: after a rejected revision,
sandbox getlists the stored candidate policy (including the added path) together withConfigurationInvalid, whilepolicy getshows the active policy. This is existing gateway behaviour and is unchanged here.Checklist