diff --git a/gpu-operator/life-cycle-policy.rst b/gpu-operator/life-cycle-policy.rst index 1fdcfea6a..9c50a7054 100644 --- a/gpu-operator/life-cycle-policy.rst +++ b/gpu-operator/life-cycle-policy.rst @@ -88,11 +88,12 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. :header-rows: 2 * - :rspan:`1` Component - - :cspan:`2` GPU Operator Version + - :cspan:`3` GPU Operator Version * - v26.7.0 - v26.7.1 - v26.7.2 + - v26.7.3 * - NVIDIA GPU Driver - | `610.57.04 `_ @@ -109,68 +110,87 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. | `595.91.07 `_ (**D**) | `580.196.07 `_ | `580.173.02 `_ + - | `615.71.10 `_ (**R**) + | `615.71.09 `_ + | `610.57.04 `_ + | `595.112.07 `_ + | `595.91.07 `_ (**D**) + | `580.196.07 `_ + | `580.173.02 `_ * - NVIDIA Driver Manager for Kubernetes - `v0.12.0 `__ - - :cspan:`1` `v0.12.1 `__ + - :cspan:`2` `v0.12.1 `__ * - NVIDIA Container Toolkit - `1.20.0 `__ - :cspan:`1` `1.20.1 `__ + - `1.20.2 `__ * - NVIDIA Device Plugin for Kubernetes - `0.20.0 `__ - :cspan:`1` `0.20.1 `__ + - `0.20.1 `__ * - DRA Driver for NVIDIA GPUs - :cspan:`1` `v0.5.0 `__ - - `v${dra_version} `__ + - :cspan:`1` `v${dra_version} `__ * - DCGM Exporter - `v4.6.0-4.8.3 `__ - :cspan:`1` `v4.6.1-4.8.4 `__ + - `v4.7.0-4.9.0 `__ * - Node Feature Discovery - :cspan:`2` `v0.19.0 `__ + - `v0.20.1 `__ * - | NVIDIA GPU Feature Discovery | for Kubernetes - `0.20.0 `__ - :cspan:`1` `0.20.1 `__ + - `0.20.1 `__ * - NVIDIA MIG Manager for Kubernetes - `0.15.0 `__ - :cspan:`1` `0.15.1 `__ + - `0.15.1 `__ * - DCGM - `4.6.0-1 `__ - :cspan:`1` `4.6.1-1 `__ + - `4.7.0 `__ * - Validator for NVIDIA GPU Operator - v26.7.0 - v26.7.1 - v26.7.2 + - v26.7.3 * - NVIDIA KubeVirt GPU Device Plugin - :cspan:`2` `v1.6.0 `__ + - `v1.6.0 `__ * - NVIDIA vGPU Device Manager - `v0.5.0 `__ - :cspan:`1` `v0.5.1 `__ + - `v0.5.1 `__ * - NVIDIA GDS Driver |gds|_ - - :cspan:`2` `2.29.4 `__ + - :cspan:`3` `2.29.4 `__ * - | NVIDIA Confidential Computing | Manager for Kubernetes - `v0.4.3 `__ - :cspan:`1` `v0.4.4 `__ + - `v0.4.5 `__ * - NVIDIA GDRCopy Driver - - :cspan:`2` `v2.6 `__ + - :cspan:`3` `v2.6 `__ * - NVIDIA Kata Sandbox Device Plugin - :cspan:`2` `v0.0.5 `__ + - `v0.0.5 `__ .. _gds-open-kernel: diff --git a/gpu-operator/nvidia-driver-configuration.rst b/gpu-operator/nvidia-driver-configuration.rst index 5121e7152..d7855f854 100644 --- a/gpu-operator/nvidia-driver-configuration.rst +++ b/gpu-operator/nvidia-driver-configuration.rst @@ -334,6 +334,7 @@ The following table describes some of the fields in the custom resource. * - ``gds.enabled`` - Specifies whether to enable GPUDirect Storage. + GDS is not supported with precompiled drivers or with ``kernelModuleType: proprietary``. - ``false`` * - ``image`` diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index 12f1f35f8..56992e3a6 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -33,6 +33,105 @@ Refer to the :ref:`GPU Operator Component Matrix` for a list of software compone ---- +.. _v26.7.3: + +26.7.3 +====== + +Breaking Changes +---------------- + +* DCGM Exporter 4.9.0 renames the default metrics to the DCGM 4.7 field names, and 11 of the renamed metrics change units. + For example, ``DCGM_FI_DEV_GPU_UTIL`` (percent) becomes ``DCGM_FI_DEV_GPU_UTIL_RATIO`` (0 to 1), and ``DCGM_FI_DEV_FB_USED`` (MiB) becomes ``DCGM_FI_DEV_FB_USED_BYTES`` (bytes). + The driver-version label on every metric changes from ``DCGM_FI_DRIVER_VERSION`` to ``DCGM_FI_SYSTEM_DRIVER_VERSION``. + Update dashboards, alerts, and queries that reference the previous metric or label names. + Refer to the `DCGM Exporter 4.9.0 release notes `__ for the complete list. + +New Features +------------ + +* Updated software component versions: + + - NVIDIA Container Toolkit v1.20.2 + - NVIDIA DCGM Exporter v4.7.0-4.9.0 + - NVIDIA DCGM 4.7.0 + - Node Feature Discovery v0.20.1 + - NVIDIA Confidential Computing Manager for Kubernetes v0.4.5 + +* The Operator supports the DCGM 4.7 metric names in DCGM Exporter v4.7.0-4.9.0. + The default metrics file is ``/etc/dcgm-exporter/default-counters.csv``. + On OpenShift, the Operator's GPU monitoring rules support both the previous and renamed metrics. + The recorded ``accelerator_*`` metric names, values, and units are preserved for dashboards and alerts. + (`GPU Operator PR #3110 `__) + +* When CDI is enabled, the Operator rejects a device plugin ``DEVICE_LIST_STRATEGY`` environment variable that does not include ``cdi-annotations`` or ``cdi-cri``. + The Operator reports the error when reconciling the cluster policy, so you no longer discover the misconfiguration when GPU pods fail to start. + (`GPU Operator PR #3081 `__) + +Fixed Issues +------------ + +* Fixed an issue where upgrading from GPU Operator v26.3.x deleted existing NVIDIA driver daemon sets and removed driver pods from GPU nodes. + The Operator now waits until GPU nodes have driver owner labels and then upgrades the driver pods according to the driver upgrade policy. + (`GPU Operator PR #3157 `__, `Issue #3155 `__) + +* Fixed an issue where host commands ran the container BusyBox implementation instead of the host binary. + This issue prevented the ``modprobe`` command from loading kernel-module soft dependencies. + Using the host binary ensures that dependencies, such as ``vfio_iommu_type1``, are loaded. + The specified module is required for KubeVirt GPU passthrough. + (`GPU Operator PR #3138 `__) + +* Fixed an issue where an ``NVIDIADriver`` resource with ``kernelModuleType: auto`` failed validation when GPUDirect Storage was enabled. + (`GPU Operator PR #3115 `__, `Issue #3114 `__) + +* Fixed an issue where driver pods on RHEL hosts did not mount the host subscription files when you configured a custom driver repository with ``repoConfig``. + Driver pods also start when subscription paths are missing on RHEL, SLES, and SL Micro hosts. + (`GPU Operator PR #3109 `__, `Issue #2817 `__) + +* Fixed an issue where container annotations from the NVIDIA Device Plugin did not reach the container runtime when CDI was enabled. + The Operator now configures the CDI annotation prefixes in the NVIDIA Container Toolkit container. + (`GPU Operator PR #3088 `__) + +* Fixed an issue where operands could start while the NVIDIA kernel module was loading or unloading. + Operands now wait until the kernel module finishes initializing. + (`GPU Operator PR #2959 `__) + +* Fixed an issue where DCGM Exporter could restart before completing initialization on nodes with many GPUs or large metrics configurations. + The startup probe waits up to five minutes for initialization to complete before liveness checks begin. + (`GPU Operator PR #3158 `__) + +Known Issues +------------ + +.. _v26.7.3-driver-manager-override: + +* The default values use the same driver and driver manager container versions as the previous releases. + These defaults prevent a disruptive driver upgrade. + However, the following deployments need NVIDIA Driver Manager for Kubernetes v0.12.2: + + - Red Hat OpenShift Container Platform in an air-gapped environment, to obtain the fix for uncompressed image layers that can prevent mirroring to Red Hat Quay with ``oc-mirror``. + Refer to `Issue #3166 `__. + - Clusters that use the DRA Driver for NVIDIA GPUs. + - Nodes configured for GPU passthrough with ``vfio-pci``. + + To upgrade the driver manager, specify ``--set driver.manager.version=v0.12.2`` with Helm, or set ``spec.driver.manager.version`` to ``v0.12.2`` in the cluster policy. + Changing the driver manager image triggers a :ref:`driver upgrade ` on each node and can evict GPU workloads even when the NVIDIA driver version is unchanged. + +* On systems with NVLink 5 or later, GPU driver pods can prevent containerized DOCA-OFED updates from completing. + Refer to the :ref:`v26.7.2-known-issues` for details and the workaround. + +* On NVIDIA B200 systems running OpenShift with standalone DCGM 4.7.0, DCGM Exporter 4.9.0 can enter ``CrashLoopBackOff`` when a custom metrics CSV includes ``DCGM_FI_PROF_NVLINK_TX_BYTES``. + The exporter fails to initialize the NVLink collector and reports ``This request is serviced by a module of DCGM that is not currently loaded``. + GPU metrics are unavailable while the exporter repeatedly restarts. + Do not add ``DCGM_FI_PROF_NVLINK_TX_BYTES`` to a custom metrics file. + +* On OpenShift, enabling the NRI plugin through a cluster policy update before the NVIDIA driver and NVIDIA Container Toolkit finish initializing after a node reboot can leave standalone DCGM running without GPU access. + DCGM Exporter then enters ``CrashLoopBackOff``, and GPU metrics remain unavailable. + To work around this issue, use NVIDIA Driver Manager for Kubernetes v0.12.2. + Refer to the :ref:`Driver Manager image override instructions ` for the settings and the driver-upgrade consequence. + +---- + .. _v26.7.2: 26.7.2 @@ -85,6 +184,8 @@ Fixed Issues The ``spec.default`` and ``spec.driverType`` fields are now optional, with defaults of ``false`` and ``gpu``, respectively. (`GPU Operator PR #3046 `__) +.. _v26.7.2-known-issues: + Known Issues ------------ @@ -99,7 +200,7 @@ Known Issues - Install the DOCA-OFED driver on the host and not use a containerized driver. - Use the _inbox_ drivers provided by the operating system. - These alternatives are not applicable to OpenShift Container Platform. - + If neither of the preceding alternatives are viable, use this workaround. As a workaround, use GPU Operator v26.7.2 or later and configure a non-default ``NVIDIADriver`` resource with the ``network.nvidia.com/operator.mofed.wait`` node label in its selector. diff --git a/gpu-operator/upgrade.rst b/gpu-operator/upgrade.rst index 5e0d6aaf6..0addd52bb 100644 --- a/gpu-operator/upgrade.rst +++ b/gpu-operator/upgrade.rst @@ -33,6 +33,33 @@ Prerequisites $ kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged +- If CDI is enabled, review any ``DEVICE_LIST_STRATEGY`` override in ``devicePlugin.env`` before upgrading. + Remove legacy overrides to use the Operator's default CDI strategies, or explicitly set the value to ``cdi-cri,cdi-annotations``. + The Operator reports an error during cluster policy reconciliation if the override includes neither CDI strategy. + Refer to :doc:`cdi` for more information. + +- Changing the Driver Manager image triggers a :ref:`driver upgrade ` and can evict GPU workloads even when the NVIDIA driver version is unchanged. + Review the target release's known issues in the :doc:`release-notes` before overriding the image. + +.. _upgrading-dcgm-and-exporter: + +******************************* +DCGM and DCGM Exporter Versions +******************************* + +Refer to the :ref:`operator-component-matrix` for the default DCGM and DCGM Exporter versions. +If you override ``dcgm.version`` or ``dcgmExporter.version``, select compatible versions and upgrade both images together. +DCGM Exporter 4.9.0 requires a DCGM 4.7 or later host engine and cannot start against a standalone DCGM 4.6.x host engine. +This requirement applies when ``dcgm.enabled=true``, including the default OpenShift configuration, or when the exporter connects to a host engine installed on the node. +When upgrading a standalone DCGM deployment, the exporter can restart during the transition until the host engine has been upgraded. + +DCGM Exporter renames the driver-version label from ``DCGM_FI_DRIVER_VERSION`` to ``DCGM_FI_SYSTEM_DRIVER_VERSION``. +Update dashboards, alerts, and queries that reference the previous label name; filters using that name can return no data after upgrading. + +If you roll back these overrides, roll back both images together. +When rolling back the Operator, also restore compatible DCGM and DCGM Exporter versions. +Otherwise, OpenShift dashboards and alerts can lose GPU metrics because the older monitoring rules do not recognize the renamed metrics. + ********** Using Helm diff --git a/repo.toml b/repo.toml index 2752d317e..1ed2b9ee3 100644 --- a/repo.toml +++ b/repo.toml @@ -176,7 +176,7 @@ docs_root = "${root}/gpu-operator" project = "gpu-operator" name = "NVIDIA GPU Operator" version = "26.7" # Update repo_docs.projects.openshift.version to match latest patch version maj.min.patch -source_substitutions = { minor_version = "26.7", version = "v26.7.2", recommended = "595.91.07", dra_version = "0.5.1", kata_version = "4.0.0" } +source_substitutions = { minor_version = "26.7", version = "v26.7.3", recommended = "595.91.07", dra_version = "0.5.1", kata_version = "4.0.0" } copyright_start = 2020 sphinx_exclude_patterns = [ "life-cycle-policy.rst",