Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 25 additions & 5 deletions gpu-operator/life-cycle-policy.rst
Original file line number Diff line number Diff line change
Expand Up @@ -88,11 +88,12 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information.
:header-rows: 2

* - :rspan:`1` Component
- :cspan:`2` GPU Operator Version
- :cspan:`3` GPU Operator Version

* - v26.7.0
- v26.7.1
- v26.7.2
- v26.7.3

* - NVIDIA GPU Driver
- | `610.57.04 <https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-610-57-04/index.html>`_
Expand All @@ -109,68 +110,87 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information.
| `595.91.07 <https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-595-91-07/index.html>`_ (**D**)
| `580.196.07 <https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-580-196-07/index.html>`_
| `580.173.02 <https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-580-173-02/index.html>`_
- | `615.71.10 <https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-615-71-10/index.html>`_ (**R**)
| `615.71.09 <https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-615-71-09/index.html>`_
| `610.57.04 <https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-610-57-04/index.html>`_
| `595.112.07 <https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-595-112-07/index.html>`_
| `595.91.07 <https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-595-91-07/index.html>`_ (**D**)
| `580.196.07 <https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-580-196-07/index.html>`_
| `580.173.02 <https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-580-173-02/index.html>`_

* - NVIDIA Driver Manager for Kubernetes
- `v0.12.0 <https://ngc.nvidia.com/catalog/containers/nvidia:cloud-native:k8s-driver-manager>`__
- :cspan:`1` `v0.12.1 <https://ngc.nvidia.com/catalog/containers/nvidia:cloud-native:k8s-driver-manager>`__
- :cspan:`2` `v0.12.1 <https://ngc.nvidia.com/catalog/containers/nvidia:cloud-native:k8s-driver-manager>`__

* - NVIDIA Container Toolkit
- `1.20.0 <https://github.com/NVIDIA/nvidia-container-toolkit/releases>`__
- :cspan:`1` `1.20.1 <https://github.com/NVIDIA/nvidia-container-toolkit/releases>`__
- `1.20.2 <https://github.com/NVIDIA/nvidia-container-toolkit/releases>`__

* - NVIDIA Device Plugin for Kubernetes
- `0.20.0 <https://github.com/NVIDIA/k8s-device-plugin/releases>`__
- :cspan:`1` `0.20.1 <https://github.com/NVIDIA/k8s-device-plugin/releases>`__
- `0.20.1 <https://github.com/NVIDIA/k8s-device-plugin/releases>`__

* - DRA Driver for NVIDIA GPUs
- :cspan:`1` `v0.5.0 <https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/releases/tag/v0.5.0>`__
- `v${dra_version} <https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/releases/tag/v${dra_version}>`__
- :cspan:`1` `v${dra_version} <https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/releases/tag/v${dra_version}>`__

* - DCGM Exporter
- `v4.6.0-4.8.3 <https://github.com/NVIDIA/dcgm-exporter/releases>`__
- :cspan:`1` `v4.6.1-4.8.4 <https://github.com/NVIDIA/dcgm-exporter/releases>`__
- `v4.7.0-4.9.0 <https://github.com/NVIDIA/dcgm-exporter/releases>`__

* - Node Feature Discovery
- :cspan:`2` `v0.19.0 <https://github.com/kubernetes-sigs/node-feature-discovery/releases/>`__
- `v0.20.1 <https://github.com/kubernetes-sigs/node-feature-discovery/releases/>`__

* - | NVIDIA GPU Feature Discovery
| for Kubernetes
- `0.20.0 <https://github.com/NVIDIA/k8s-device-plugin/releases>`__
- :cspan:`1` `0.20.1 <https://github.com/NVIDIA/k8s-device-plugin/releases>`__
- `0.20.1 <https://github.com/NVIDIA/k8s-device-plugin/releases>`__

* - NVIDIA MIG Manager for Kubernetes
- `0.15.0 <https://github.com/NVIDIA/mig-parted/blob/main/CHANGELOG.md>`__
- :cspan:`1` `0.15.1 <https://github.com/NVIDIA/mig-parted/blob/main/CHANGELOG.md>`__
- `0.15.1 <https://github.com/NVIDIA/mig-parted/blob/main/CHANGELOG.md>`__

* - DCGM
- `4.6.0-1 <https://docs.nvidia.com/datacenter/dcgm/latest/release-notes/changelog.html>`__
- :cspan:`1` `4.6.1-1 <https://docs.nvidia.com/datacenter/dcgm/latest/release-notes/changelog.html>`__
- `4.7.0 <https://docs.nvidia.com/datacenter/dcgm/latest/release-notes/changelog.html>`__

* - Validator for NVIDIA GPU Operator
- v26.7.0
- v26.7.1
- v26.7.2
- v26.7.3

* - NVIDIA KubeVirt GPU Device Plugin
- :cspan:`2` `v1.6.0 <https://github.com/NVIDIA/kubevirt-gpu-device-plugin>`__
- `v1.6.0 <https://github.com/NVIDIA/kubevirt-gpu-device-plugin>`__

* - NVIDIA vGPU Device Manager
- `v0.5.0 <https://github.com/NVIDIA/vgpu-device-manager>`__
- :cspan:`1` `v0.5.1 <https://github.com/NVIDIA/vgpu-device-manager>`__
- `v0.5.1 <https://github.com/NVIDIA/vgpu-device-manager>`__

* - NVIDIA GDS Driver |gds|_
- :cspan:`2` `2.29.4 <https://github.com/NVIDIA/gds-nvidia-fs/releases>`__
- :cspan:`3` `2.29.4 <https://github.com/NVIDIA/gds-nvidia-fs/releases>`__

* - | NVIDIA Confidential Computing
| Manager for Kubernetes
- `v0.4.3 <https://github.com/NVIDIA/k8s-cc-manager/releases>`__
- :cspan:`1` `v0.4.4 <https://github.com/NVIDIA/k8s-cc-manager/releases>`__
- `v0.4.5 <https://github.com/NVIDIA/k8s-cc-manager/releases>`__

* - NVIDIA GDRCopy Driver
- :cspan:`2` `v2.6 <https://github.com/NVIDIA/gdrcopy/releases>`__
- :cspan:`3` `v2.6 <https://github.com/NVIDIA/gdrcopy/releases>`__

* - NVIDIA Kata Sandbox Device Plugin
- :cspan:`2` `v0.0.5 <https://github.com/NVIDIA/sandbox-device-plugin/releases>`__
- `v0.0.5 <https://github.com/NVIDIA/sandbox-device-plugin/releases>`__

.. _gds-open-kernel:

Expand Down
1 change: 1 addition & 0 deletions gpu-operator/nvidia-driver-configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,7 @@ The following table describes some of the fields in the custom resource.

* - ``gds.enabled``
- Specifies whether to enable GPUDirect Storage.
GDS is not supported with precompiled drivers or with ``kernelModuleType: proprietary``.
- ``false``

* - ``image``
Expand Down
103 changes: 102 additions & 1 deletion gpu-operator/release-notes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,105 @@ Refer to the :ref:`GPU Operator Component Matrix` for a list of software compone

----

.. _v26.7.3:

26.7.3
======

Breaking Changes
----------------

* DCGM Exporter 4.9.0 renames the default metrics to the DCGM 4.7 field names, and 11 of the renamed metrics change units.
For example, ``DCGM_FI_DEV_GPU_UTIL`` (percent) becomes ``DCGM_FI_DEV_GPU_UTIL_RATIO`` (0 to 1), and ``DCGM_FI_DEV_FB_USED`` (MiB) becomes ``DCGM_FI_DEV_FB_USED_BYTES`` (bytes).
The driver-version label on every metric changes from ``DCGM_FI_DRIVER_VERSION`` to ``DCGM_FI_SYSTEM_DRIVER_VERSION``.
Update dashboards, alerts, and queries that reference the previous metric or label names.
Refer to the `DCGM Exporter 4.9.0 release notes <https://github.com/NVIDIA/dcgm-exporter/releases/tag/4.9.0>`__ for the complete list.

New Features
------------

* Updated software component versions:

- NVIDIA Container Toolkit v1.20.2
- NVIDIA DCGM Exporter v4.7.0-4.9.0
- NVIDIA DCGM 4.7.0
- Node Feature Discovery v0.20.1
- NVIDIA Confidential Computing Manager for Kubernetes v0.4.5

* The Operator supports the DCGM 4.7 metric names in DCGM Exporter v4.7.0-4.9.0.
The default metrics file is ``/etc/dcgm-exporter/default-counters.csv``.
On OpenShift, the Operator's GPU monitoring rules support both the previous and renamed metrics.
The recorded ``accelerator_*`` metric names, values, and units are preserved for dashboards and alerts.
(`GPU Operator PR #3110 <https://github.com/NVIDIA/gpu-operator/pull/3110>`__)

* When CDI is enabled, the Operator rejects a device plugin ``DEVICE_LIST_STRATEGY`` environment variable that does not include ``cdi-annotations`` or ``cdi-cri``.
The Operator reports the error when reconciling the cluster policy, so you no longer discover the misconfiguration when GPU pods fail to start.
(`GPU Operator PR #3081 <https://github.com/NVIDIA/gpu-operator/pull/3081>`__)

Fixed Issues
------------

* Fixed an issue where upgrading from GPU Operator v26.3.x deleted existing NVIDIA driver daemon sets and removed driver pods from GPU nodes.
The Operator now waits until GPU nodes have driver owner labels and then upgrades the driver pods according to the driver upgrade policy.
(`GPU Operator PR #3157 <https://github.com/NVIDIA/gpu-operator/pull/3157>`__, `Issue #3155 <https://github.com/NVIDIA/gpu-operator/issues/3155>`__)

* Fixed an issue where host commands ran the container BusyBox implementation instead of the host binary.
This issue prevented the ``modprobe`` command from loading kernel-module soft dependencies.
Using the host binary ensures that dependencies, such as ``vfio_iommu_type1``, are loaded.
The specified module is required for KubeVirt GPU passthrough.
(`GPU Operator PR #3138 <https://github.com/NVIDIA/gpu-operator/pull/3138>`__)

* Fixed an issue where an ``NVIDIADriver`` resource with ``kernelModuleType: auto`` failed validation when GPUDirect Storage was enabled.
(`GPU Operator PR #3115 <https://github.com/NVIDIA/gpu-operator/pull/3115>`__, `Issue #3114 <https://github.com/NVIDIA/gpu-operator/issues/3114>`__)

* Fixed an issue where driver pods on RHEL hosts did not mount the host subscription files when you configured a custom driver repository with ``repoConfig``.
Driver pods also start when subscription paths are missing on RHEL, SLES, and SL Micro hosts.
(`GPU Operator PR #3109 <https://github.com/NVIDIA/gpu-operator/pull/3109>`__, `Issue #2817 <https://github.com/NVIDIA/gpu-operator/issues/2817>`__)

* Fixed an issue where container annotations from the NVIDIA Device Plugin did not reach the container runtime when CDI was enabled.
The Operator now configures the CDI annotation prefixes in the NVIDIA Container Toolkit container.
(`GPU Operator PR #3088 <https://github.com/NVIDIA/gpu-operator/pull/3088>`__)

* Fixed an issue where operands could start while the NVIDIA kernel module was loading or unloading.
Operands now wait until the kernel module finishes initializing.
(`GPU Operator PR #2959 <https://github.com/NVIDIA/gpu-operator/pull/2959>`__)

* Fixed an issue where DCGM Exporter could restart before completing initialization on nodes with many GPUs or large metrics configurations.
The startup probe waits up to five minutes for initialization to complete before liveness checks begin.
(`GPU Operator PR #3158 <https://github.com/NVIDIA/gpu-operator/pull/3158>`__)

Known Issues
------------

.. _v26.7.3-driver-manager-override:

* The default values use the same driver and driver manager container versions as the previous releases.
These defaults prevent a disruptive driver upgrade.
However, the following deployments need NVIDIA Driver Manager for Kubernetes v0.12.2:

- Red Hat OpenShift Container Platform in an air-gapped environment, to obtain the fix for uncompressed image layers that can prevent mirroring to Red Hat Quay with ``oc-mirror``.
Refer to `Issue #3166 <https://github.com/NVIDIA/gpu-operator/issues/3166>`__.
- Clusters that use the DRA Driver for NVIDIA GPUs.
- Nodes configured for GPU passthrough with ``vfio-pci``.

To upgrade the driver manager, specify ``--set driver.manager.version=v0.12.2`` with Helm, or set ``spec.driver.manager.version`` to ``v0.12.2`` in the cluster policy.
Changing the driver manager image triggers a :ref:`driver upgrade <gpu-driver-upgrades>` on each node and can evict GPU workloads even when the NVIDIA driver version is unchanged.

* On systems with NVLink 5 or later, GPU driver pods can prevent containerized DOCA-OFED updates from completing.
Refer to the :ref:`v26.7.2-known-issues` for details and the workaround.

* On NVIDIA B200 systems running OpenShift with standalone DCGM 4.7.0, DCGM Exporter 4.9.0 can enter ``CrashLoopBackOff`` when a custom metrics CSV includes ``DCGM_FI_PROF_NVLINK_TX_BYTES``.
The exporter fails to initialize the NVLink collector and reports ``This request is serviced by a module of DCGM that is not currently loaded``.
GPU metrics are unavailable while the exporter repeatedly restarts.
Do not add ``DCGM_FI_PROF_NVLINK_TX_BYTES`` to a custom metrics file.

* On OpenShift, enabling the NRI plugin through a cluster policy update before the NVIDIA driver and NVIDIA Container Toolkit finish initializing after a node reboot can leave standalone DCGM running without GPU access.
DCGM Exporter then enters ``CrashLoopBackOff``, and GPU metrics remain unavailable.
To work around this issue, use NVIDIA Driver Manager for Kubernetes v0.12.2.
Refer to the :ref:`Driver Manager image override instructions <v26.7.3-driver-manager-override>` for the settings and the driver-upgrade consequence.

----

.. _v26.7.2:

26.7.2
Expand Down Expand Up @@ -85,6 +184,8 @@ Fixed Issues
The ``spec.default`` and ``spec.driverType`` fields are now optional, with defaults of ``false`` and ``gpu``, respectively.
(`GPU Operator PR #3046 <https://github.com/NVIDIA/gpu-operator/pull/3046>`__)

.. _v26.7.2-known-issues:

Known Issues
------------

Expand All @@ -99,7 +200,7 @@ Known Issues
- Install the DOCA-OFED driver on the host and not use a containerized driver.
- Use the _inbox_ drivers provided by the operating system.
- These alternatives are not applicable to OpenShift Container Platform.

If neither of the preceding alternatives are viable, use this workaround.

As a workaround, use GPU Operator v26.7.2 or later and configure a non-default ``NVIDIADriver`` resource with the ``network.nvidia.com/operator.mofed.wait`` node label in its selector.
Expand Down
27 changes: 27 additions & 0 deletions gpu-operator/upgrade.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,33 @@ Prerequisites

$ kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged

- If CDI is enabled, review any ``DEVICE_LIST_STRATEGY`` override in ``devicePlugin.env`` before upgrading.
Remove legacy overrides to use the Operator's default CDI strategies, or explicitly set the value to ``cdi-cri,cdi-annotations``.
The Operator reports an error during cluster policy reconciliation if the override includes neither CDI strategy.
Refer to :doc:`cdi` for more information.

- Changing the Driver Manager image triggers a :ref:`driver upgrade <gpu-driver-upgrades>` and can evict GPU workloads even when the NVIDIA driver version is unchanged.
Review the target release's known issues in the :doc:`release-notes` before overriding the image.

.. _upgrading-dcgm-and-exporter:

*******************************
DCGM and DCGM Exporter Versions
*******************************

Refer to the :ref:`operator-component-matrix` for the default DCGM and DCGM Exporter versions.
If you override ``dcgm.version`` or ``dcgmExporter.version``, select compatible versions and upgrade both images together.
DCGM Exporter 4.9.0 requires a DCGM 4.7 or later host engine and cannot start against a standalone DCGM 4.6.x host engine.
This requirement applies when ``dcgm.enabled=true``, including the default OpenShift configuration, or when the exporter connects to a host engine installed on the node.
When upgrading a standalone DCGM deployment, the exporter can restart during the transition until the host engine has been upgraded.

DCGM Exporter renames the driver-version label from ``DCGM_FI_DRIVER_VERSION`` to ``DCGM_FI_SYSTEM_DRIVER_VERSION``.
Update dashboards, alerts, and queries that reference the previous label name; filters using that name can return no data after upgrading.

If you roll back these overrides, roll back both images together.
When rolling back the Operator, also restore compatible DCGM and DCGM Exporter versions.
Otherwise, OpenShift dashboards and alerts can lose GPU metrics because the older monitoring rules do not recognize the renamed metrics.


**********
Using Helm
Expand Down
2 changes: 1 addition & 1 deletion repo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ docs_root = "${root}/gpu-operator"
project = "gpu-operator"
name = "NVIDIA GPU Operator"
version = "26.7" # Update repo_docs.projects.openshift.version to match latest patch version maj.min.patch
source_substitutions = { minor_version = "26.7", version = "v26.7.2", recommended = "595.91.07", dra_version = "0.5.1", kata_version = "4.0.0" }
source_substitutions = { minor_version = "26.7", version = "v26.7.3", recommended = "595.91.07", dra_version = "0.5.1", kata_version = "4.0.0" }
copyright_start = 2020
sphinx_exclude_patterns = [
"life-cycle-policy.rst",
Expand Down
Loading