[Kevin Cho (chokevin)]

Every successfully resolved and provisioned NVIDIA nspawn node prepares a private driver root and generates a CDI specification from it, but NVIDIA Toolkit could still leave environment-variable device requests on its automatic legacy-injection path. That bypassed CDI container edits such as the host-matched nvidia-smi mount. NVIDIA setup now explicitly selects CDI runtime mode after successful spec generation on every supported host, while non-NVIDIA nodes remain no-ops and incomplete NVIDIA state still fails closed.

• Removes the host OS and version policy because the driver-root CDI contract is architectural rather than distro-specific.
• Runs nvidia-ctk config --set nvidia-container-runtime.mode=cdi --in-place only after the prepared driver-root CDI specification is generated successfully.
• Preserves fail-closed startup: CDI generation or runtime configuration failures stop NVIDIA setup rather than reporting success with a mismatched injection path.
• Keeps non-NVIDIA nodes as no-ops and retains the existing incomplete-state rejection before any NVIDIA setup commands run.
• Documents why Toolkit legacy injection is incompatible with the generated driver-root contract on all Unbounded NVIDIA nspawn nodes.

Evidence

This 12-second recording shows the unconditional CDI runtime-selection diff and the focused node-start regression result:

nvidia-cdi-invariant.mp4

Verification

• Passed: go test ./pkg/agent/goalstates -run '^TestResolveNvidiaHost' -count=1
• Passed: go test ./pkg/agent/phases/nodestart -run '^Test(ConfigureNVIDIAContainerRuntime|SetupNVIDIAProvisionedCapabilityControlsBehavior|ReconcileNVIDIANonGPUIsNoop)$' -count=1
• Passed: golangci-lint run -c .golangci.yaml ./pkg/agent/goalstates ./pkg/agent/phases/nodestart
• Passed: git diff --check
• Unable to verify broad package tests on macOS: existing Linux host-kernel probes, /private/var path canonicalization, and listener-state tests failed outside the changed behavior.
• Not run: live deployment of the patched agent; rollout remains fail-closed pending a separately reviewed installation and rollback path.

Risk

This changes NVIDIA Toolkit runtime selection for every NVIDIA nspawn node that completes driver-root CDI generation. A regression would surface during node-start runtime configuration or when GPU workloads request devices. Rollback is reverting this PR, which restores the prior host-gated or automatic runtime selection.

Review guide

1. pkg/agent/phases/nodestart/nvidia.go: successful CDI generation now always proceeds to CDI runtime configuration.
2. pkg/agent/phases/nodestart/nvidia_test.go: unconditional configuration and explicit failure coverage.
3. pkg/agent/goalstates/nvidia.go: removal of the distro-derived runtime-selection flag and OS-release plumbing.
4. pkg/agent/goalstates/nvidia_test.go: removal of host/version policy tests while preserving discovery and readiness coverage.
5. docs/content/reference/gpu/nvidia.md: driver-root and legacy-injection rationale for all NVIDIA nspawn nodes.

AI assistance: GitHub Copilot assisted with investigation, implementation, tests, evidence preparation, and PR drafting. The author verified the claims and final text.