[Kevin Cho (chokevin)]

Ubuntu 24.04 GPU nodes could generate a correct driver-root CDI specification but still route the device plugin's environment-variable request through NVIDIA Toolkit legacy injection. That bypassed CDI container edits such as the host-matched nvidia-smi mount, preventing Kubernetes GPU registration. Ubuntu 24.04 now explicitly selects CDI runtime mode, while unsupported and unverified host versions retain their existing automatic mode.

• Extends the existing Azure Linux 3 CDI-mode policy to Ubuntu 24.04, where the same auto-mode failure was observed.
• Keeps Azure Container Linux, Ubuntu 22.04, Fedora, RHEL-like, and unknown hosts on their existing behavior.
• Preserves fail-closed startup: an nvidia-ctk config failure still stops NVIDIA setup rather than reporting success with the wrong runtime mode.
• Documents why legacy injection is incompatible with the generated driver-root contract on the affected hosts.

Evidence

A 12-second recording shows the exact host-policy diff and the focused Ubuntu regression result. The video is attached below.

Verification

• Passed: go test ./pkg/agent/goalstates -run '^TestSupportedNVIDIAHostsUseCDIRuntime$' -count=1
• Passed: go test ./pkg/agent/phases/nodestart -run '^Test(ConfigureNVIDIAContainerRuntime|SetupNVIDIAProvisionedCapabilityControlsBehavior|ReconcileNVIDIANonGPUIsNoop)$' -count=1
• Passed: golangci-lint run -c .golangci.yaml ./pkg/agent/goalstates ./pkg/agent/phases/nodestart
• Passed: git diff --check
• Unable to verify broad package tests on macOS: existing Linux host-kernel probes, /private/var path canonicalization, and listener-state tests failed outside the changed behavior.
• Not run: live deployment of the patched agent; the affected environment remains fail-closed pending a separately reviewed installation and rollback path.

Risk

This changes NVIDIA Toolkit runtime selection for Ubuntu 24.04 GPU hosts. A regression would surface during node-start runtime configuration or when GPU workloads request devices. Rollback is reverting this commit, which restores toolkit auto mode on Ubuntu.

Review guide

1. pkg/agent/goalstates/nvidia.go: host/version policy boundary.
2. pkg/agent/goalstates/nvidia_test.go: positive Ubuntu 24.04 and negative host coverage.
3. pkg/agent/phases/nodestart/nvidia.go: existing fail-closed runtime configuration.
4. docs/content/reference/gpu/nvidia.md: driver-root and legacy-injection rationale.

AI assistance: GitHub Copilot assisted with investigation, implementation, tests, evidence preparation, and PR drafting. The author verified the claims and final text.

ubuntu24-cdi-fix.mp4