From 02287252824f495b8498794499c707d2d272096e Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Tue, 10 Nov 2026 18:11:17 +0000 Subject: [PATCH 1/5] docs(gpu-operator): Add 26.7.3 release notes and component matrix column Signed-off-by: Mike McKiernan Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- gpu-operator/life-cycle-policy.rst | 31 +++++++++--- gpu-operator/release-notes.rst | 76 ++++++++++++++++++++++++++++++ repo.toml | 2 +- 3 files changed, 102 insertions(+), 7 deletions(-) diff --git a/gpu-operator/life-cycle-policy.rst b/gpu-operator/life-cycle-policy.rst index 1fdcfea6a..0090d1488 100644 --- a/gpu-operator/life-cycle-policy.rst +++ b/gpu-operator/life-cycle-policy.rst @@ -88,11 +88,12 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. :header-rows: 2 * - :rspan:`1` Component - - :cspan:`2` GPU Operator Version + - :cspan:`3` GPU Operator Version * - v26.7.0 - v26.7.1 - v26.7.2 + - v26.7.3 * - NVIDIA GPU Driver - | `610.57.04 `_ @@ -109,68 +110,86 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. | `595.91.07 `_ (**D**) | `580.196.07 `_ | `580.173.02 `_ + - | `615.71.10 `_ (**R**) + | `615.71.09 `_ + | `610.57.04 `_ + | `595.112.07 `_ + | `595.91.07 `_ (**D**) + | `580.196.07 `_ + | `580.173.02 `_ * - NVIDIA Driver Manager for Kubernetes - `v0.12.0 `__ - - :cspan:`1` `v0.12.1 `__ + - :cspan:`2` `v0.12.1 `__ * - NVIDIA Container Toolkit - `1.20.0 `__ - :cspan:`1` `1.20.1 `__ + - `1.20.2 `__ * - NVIDIA Device Plugin for Kubernetes - `0.20.0 `__ - :cspan:`1` `0.20.1 `__ + - `0.20.2 `__ * - DRA Driver for NVIDIA GPUs - :cspan:`1` `v0.5.0 `__ - - `v${dra_version} `__ + - :cspan:`1` `v${dra_version} `__ * - DCGM Exporter - `v4.6.0-4.8.3 `__ - :cspan:`1` `v4.6.1-4.8.4 `__ + - `v4.7.0-4.9.0 `__ * - Node Feature Discovery - :cspan:`2` `v0.19.0 `__ + - `v0.20.1 `__ * - | NVIDIA GPU Feature Discovery | for Kubernetes - `0.20.0 `__ - :cspan:`1` `0.20.1 `__ + - `0.20.2 `__ * - NVIDIA MIG Manager for Kubernetes - `0.15.0 `__ - :cspan:`1` `0.15.1 `__ + - `0.15.2 `__ * - DCGM - `4.6.0-1 `__ - :cspan:`1` `4.6.1-1 `__ + - `4.7.0 `__ * - Validator for NVIDIA GPU Operator - v26.7.0 - v26.7.1 - v26.7.2 + - v26.7.3 * - NVIDIA KubeVirt GPU Device Plugin - :cspan:`2` `v1.6.0 `__ + - `v1.6.1 `__ * - NVIDIA vGPU Device Manager - `v0.5.0 `__ - :cspan:`1` `v0.5.1 `__ + - `v0.5.2 `__ * - NVIDIA GDS Driver |gds|_ - - :cspan:`2` `2.29.4 `__ + - :cspan:`3` `2.29.4 `__ * - | NVIDIA Confidential Computing | Manager for Kubernetes - `v0.4.3 `__ - - :cspan:`1` `v0.4.4 `__ + - :cspan:`2` `v0.4.4 `__ * - NVIDIA GDRCopy Driver - - :cspan:`2` `v2.6 `__ + - :cspan:`3` `v2.6 `__ * - NVIDIA Kata Sandbox Device Plugin - :cspan:`2` `v0.0.5 `__ + - `v0.0.6 `__ .. _gds-open-kernel: diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index 12f1f35f8..2217fd79d 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -33,6 +33,82 @@ Refer to the :ref:`GPU Operator Component Matrix` for a list of software compone ---- +.. _v26.7.3: + +26.7.3 +====== + +New Features +------------ + +* Updated software component versions: + + - NVIDIA Container Toolkit v1.20.2 + - NVIDIA Device Plugin for Kubernetes v0.20.2 + - NVIDIA DCGM Exporter v4.7.0-4.9.0 + - NVIDIA DCGM 4.7.0 + - Node Feature Discovery v0.20.1 + - NVIDIA MIG Manager for Kubernetes v0.15.2 + - NVIDIA GPU Feature Discovery for Kubernetes v0.20.2 + - NVIDIA KubeVirt GPU Device Plugin v1.6.1 + - NVIDIA vGPU Device Manager v0.5.2 + - NVIDIA Kata Sandbox Device Plugin v0.0.6 + +* Updated the Go toolchain to 1.26.10 and the distroless base image to v4.2.1 to address CVEs. + +Improvements +------------ + +* The Operator supports the DCGM 4.7 metric names in DCGM Exporter v4.7.0-4.9.0. + The default metrics file is ``/etc/dcgm-exporter/default-counters.csv``. + (`GPU Operator PR #3110 `__) + +* When CDI is enabled, the Operator rejects a device plugin ``DEVICE_LIST_STRATEGY`` environment variable that does not include ``cdi-annotations`` or ``cdi-cri``. + The Operator reports the error when reconciling the cluster policy, so you no longer discover the misconfiguration when GPU pods fail to start. + (`GPU Operator PR #3081 `__) + +Fixed Issues +------------ + +* Fixed an issue where upgrading from GPU Operator v26.3.x deleted existing NVIDIA driver daemon sets and removed driver pods from GPU nodes. + The Operator now waits until GPU nodes have driver owner labels and then upgrades the driver pods according to the driver upgrade policy. + (`GPU Operator PR #3157 `__, `Issue #3155 `__) + +* Fixed an issue where container images for the GPU Operator and operands included uncompressed layers. + Mirroring the images to Red Hat Quay with ``oc-mirror`` failed in air-gapped environments. + (`Issue #3166 `__) + +* Fixed an issue where an ``NVIDIADriver`` resource with ``kernelModuleType: auto`` failed validation when GPUDirect Storage was enabled. + (`GPU Operator PR #3115 `__, `Issue #3114 `__) + +* Fixed an issue where driver pods on RHEL hosts did not mount the host subscription files when you configured a custom driver repository with ``repoConfig``. + Driver pods also start when subscription paths are missing on RHEL, SLES, and SL Micro hosts. + (`GPU Operator PR #3109 `__, `Issue #2817 `__) + +* Fixed an issue where container annotations from the NVIDIA Device Plugin did not reach the container runtime when CDI was enabled. + The Operator now configures the CDI annotation prefixes in the NVIDIA Container Toolkit container. + (`GPU Operator PR #3088 `__) + +* Fixed an issue where operands could start while the NVIDIA kernel module was loading or unloading. + Operands now wait until the kernel module finishes initializing. + (`GPU Operator PR #2959 `__) + +Known Issues +------------ + +* The default values use the same driver and driver manager container versions as the previous releases. + These defaults prevent a disruptive driver upgrade. + However, the following deployments need NVIDIA Driver Manager for Kubernetes v0.12.2: + + - Red Hat OpenShift Container Platform in an air-gapped environment. + - Clusters that use the DRA Driver for NVIDIA GPUs. + - Nodes configured for GPU passthrough with ``vfio-pci``. + + Upgrading the driver manager triggers a :ref:`driver upgrade ` on each node. + To upgrade the driver manager, specify ``--set driver.manager.version=v0.12.2`` with Helm, or set ``spec.driver.manager.version`` to ``v0.12.2`` in the cluster policy. + +---- + .. _v26.7.2: 26.7.2 diff --git a/repo.toml b/repo.toml index 2752d317e..1ed2b9ee3 100644 --- a/repo.toml +++ b/repo.toml @@ -176,7 +176,7 @@ docs_root = "${root}/gpu-operator" project = "gpu-operator" name = "NVIDIA GPU Operator" version = "26.7" # Update repo_docs.projects.openshift.version to match latest patch version maj.min.patch -source_substitutions = { minor_version = "26.7", version = "v26.7.2", recommended = "595.91.07", dra_version = "0.5.1", kata_version = "4.0.0" } +source_substitutions = { minor_version = "26.7", version = "v26.7.3", recommended = "595.91.07", dra_version = "0.5.1", kata_version = "4.0.0" } copyright_start = 2020 sphinx_exclude_patterns = [ "life-cycle-policy.rst", From 28a34942728b2a4b84e11db97f0f6e115d7f153a Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Tue, 10 Nov 2026 18:16:06 +0000 Subject: [PATCH 2/5] docs(gpu-operator): Remove trailing whitespace in release notes Signed-off-by: Mike McKiernan Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- gpu-operator/release-notes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index 2217fd79d..a854e9e0a 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -175,7 +175,7 @@ Known Issues - Install the DOCA-OFED driver on the host and not use a containerized driver. - Use the _inbox_ drivers provided by the operating system. - These alternatives are not applicable to OpenShift Container Platform. - + If neither of the preceding alternatives are viable, use this workaround. As a workaround, use GPU Operator v26.7.2 or later and configure a non-default ``NVIDIADriver`` resource with the ``network.nvidia.com/operator.mofed.wait`` node label in its selector. From 3f92de243d8fc3a27af295d4a1c6bc80afc9348f Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Fri, 13 Nov 2026 12:01:34 -0500 Subject: [PATCH 3/5] docs: Updates for the release Signed-off-by: Mike McKiernan --- gpu-operator/nvidia-driver-configuration.rst | 1 + gpu-operator/release-notes.rst | 29 +++++++++++++------- gpu-operator/upgrade.rst | 24 ++++++++++++++++ 3 files changed, 44 insertions(+), 10 deletions(-) diff --git a/gpu-operator/nvidia-driver-configuration.rst b/gpu-operator/nvidia-driver-configuration.rst index 5121e7152..d7855f854 100644 --- a/gpu-operator/nvidia-driver-configuration.rst +++ b/gpu-operator/nvidia-driver-configuration.rst @@ -334,6 +334,7 @@ The following table describes some of the fields in the custom resource. * - ``gds.enabled`` - Specifies whether to enable GPUDirect Storage. + GDS is not supported with precompiled drivers or with ``kernelModuleType: proprietary``. - ``false`` * - ``image`` diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index a854e9e0a..75c33acef 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -54,13 +54,10 @@ New Features - NVIDIA vGPU Device Manager v0.5.2 - NVIDIA Kata Sandbox Device Plugin v0.0.6 -* Updated the Go toolchain to 1.26.10 and the distroless base image to v4.2.1 to address CVEs. - -Improvements ------------- - * The Operator supports the DCGM 4.7 metric names in DCGM Exporter v4.7.0-4.9.0. The default metrics file is ``/etc/dcgm-exporter/default-counters.csv``. + On OpenShift, the Operator's GPU monitoring rules support both the previous and renamed metrics. + Existing metric names, values, and units are preserved for dashboards and alerts. (`GPU Operator PR #3110 `__) * When CDI is enabled, the Operator rejects a device plugin ``DEVICE_LIST_STRATEGY`` environment variable that does not include ``cdi-annotations`` or ``cdi-cri``. @@ -74,9 +71,11 @@ Fixed Issues The Operator now waits until GPU nodes have driver owner labels and then upgrades the driver pods according to the driver upgrade policy. (`GPU Operator PR #3157 `__, `Issue #3155 `__) -* Fixed an issue where container images for the GPU Operator and operands included uncompressed layers. - Mirroring the images to Red Hat Quay with ``oc-mirror`` failed in air-gapped environments. - (`Issue #3166 `__) +* Fixed an issue where host commands ran the container BusyBox implementation instead of the host binary. + This issue prevented the ``modprobe`` command from loading kernel-module soft dependencies. + Using the host binary ensures that dependencies, such as ``vfio_iommu_type1``, are loaded. + The specified module is required for KubeVirt GPU passthrough. + (`GPU Operator PR #3138 `__) * Fixed an issue where an ``NVIDIADriver`` resource with ``kernelModuleType: auto`` failed validation when GPUDirect Storage was enabled. (`GPU Operator PR #3115 `__, `Issue #3114 `__) @@ -93,6 +92,10 @@ Fixed Issues Operands now wait until the kernel module finishes initializing. (`GPU Operator PR #2959 `__) +* Fixed an issue where DCGM Exporter could restart before completing initialization on nodes with many GPUs or large metrics configurations. + The startup probe waits up to five minutes for initialization to complete before liveness checks begin. + (`GPU Operator PR #3158 `__) + Known Issues ------------ @@ -100,12 +103,16 @@ Known Issues These defaults prevent a disruptive driver upgrade. However, the following deployments need NVIDIA Driver Manager for Kubernetes v0.12.2: - - Red Hat OpenShift Container Platform in an air-gapped environment. + - Red Hat OpenShift Container Platform in an air-gapped environment, to obtain the fix for uncompressed image layers that can prevent mirroring to Red Hat Quay with ``oc-mirror``. + Refer to `Issue #3166 `__. - Clusters that use the DRA Driver for NVIDIA GPUs. - Nodes configured for GPU passthrough with ``vfio-pci``. - Upgrading the driver manager triggers a :ref:`driver upgrade ` on each node. To upgrade the driver manager, specify ``--set driver.manager.version=v0.12.2`` with Helm, or set ``spec.driver.manager.version`` to ``v0.12.2`` in the cluster policy. + Changing the driver manager image triggers a :ref:`driver upgrade ` on each node and can evict GPU workloads even when the NVIDIA driver version is unchanged. + +* On systems with NVLink 5 or later, GPU driver pods can prevent containerized DOCA-OFED updates from completing. + Refer to the :ref:`v26.7.2-known-issues` for details and the workaround. ---- @@ -161,6 +168,8 @@ Fixed Issues The ``spec.default`` and ``spec.driverType`` fields are now optional, with defaults of ``false`` and ``gpu``, respectively. (`GPU Operator PR #3046 `__) +.. _v26.7.2-known-issues: + Known Issues ------------ diff --git a/gpu-operator/upgrade.rst b/gpu-operator/upgrade.rst index 5e0d6aaf6..d831bf4eb 100644 --- a/gpu-operator/upgrade.rst +++ b/gpu-operator/upgrade.rst @@ -33,6 +33,30 @@ Prerequisites $ kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged +- If CDI is enabled, review any ``DEVICE_LIST_STRATEGY`` override in ``devicePlugin.env`` before upgrading. + Remove legacy overrides to use the Operator's default CDI strategies, or explicitly set the value to ``cdi-cri,cdi-annotations``. + The Operator reports an error during cluster policy reconciliation if the override includes neither CDI strategy. + Refer to :doc:`cdi` for more information. + +- Changing the Driver Manager image triggers a :ref:`driver upgrade ` and can evict GPU workloads even when the NVIDIA driver version is unchanged. + Review the target release's known issues in the :doc:`release-notes` before overriding the image. + +.. _upgrading-dcgm-and-exporter: + +******************************* +DCGM and DCGM Exporter Versions +******************************* + +Refer to the :ref:`operator-component-matrix` for the default DCGM and DCGM Exporter versions. +If you override ``dcgm.version`` or ``dcgmExporter.version``, select compatible versions and upgrade both images together. +DCGM Exporter 4.9.0 requires a DCGM 4.7 or later host engine and cannot start against a standalone DCGM 4.6.x host engine. +This requirement applies when ``dcgm.enabled=true``, including the default OpenShift configuration, or when the exporter connects to a host engine installed on the node. +When upgrading a standalone DCGM deployment, the exporter can restart during the transition until the host engine has been upgraded. + +If you roll back these overrides, roll back both images together. +When rolling back the Operator, also restore compatible DCGM and DCGM Exporter versions. +Otherwise, OpenShift dashboards and alerts can lose GPU metrics because the older monitoring rules do not recognize the renamed metrics. + ********** Using Helm From dc4cbbea1035fd643701c7a4459e5a984f7a02ec Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Fri, 13 Nov 2026 13:31:34 -0500 Subject: [PATCH 4/5] docs: More known issues Signed-off-by: Mike McKiernan --- gpu-operator/life-cycle-policy.rst | 15 ++++++++------- gpu-operator/release-notes.rst | 25 +++++++++++++++++++------ gpu-operator/upgrade.rst | 3 +++ 3 files changed, 30 insertions(+), 13 deletions(-) diff --git a/gpu-operator/life-cycle-policy.rst b/gpu-operator/life-cycle-policy.rst index 0090d1488..9c50a7054 100644 --- a/gpu-operator/life-cycle-policy.rst +++ b/gpu-operator/life-cycle-policy.rst @@ -130,7 +130,7 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. * - NVIDIA Device Plugin for Kubernetes - `0.20.0 `__ - :cspan:`1` `0.20.1 `__ - - `0.20.2 `__ + - `0.20.1 `__ * - DRA Driver for NVIDIA GPUs - :cspan:`1` `v0.5.0 `__ @@ -149,12 +149,12 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. | for Kubernetes - `0.20.0 `__ - :cspan:`1` `0.20.1 `__ - - `0.20.2 `__ + - `0.20.1 `__ * - NVIDIA MIG Manager for Kubernetes - `0.15.0 `__ - :cspan:`1` `0.15.1 `__ - - `0.15.2 `__ + - `0.15.1 `__ * - DCGM - `4.6.0-1 `__ @@ -169,12 +169,12 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. * - NVIDIA KubeVirt GPU Device Plugin - :cspan:`2` `v1.6.0 `__ - - `v1.6.1 `__ + - `v1.6.0 `__ * - NVIDIA vGPU Device Manager - `v0.5.0 `__ - :cspan:`1` `v0.5.1 `__ - - `v0.5.2 `__ + - `v0.5.1 `__ * - NVIDIA GDS Driver |gds|_ - :cspan:`3` `2.29.4 `__ @@ -182,14 +182,15 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. * - | NVIDIA Confidential Computing | Manager for Kubernetes - `v0.4.3 `__ - - :cspan:`2` `v0.4.4 `__ + - :cspan:`1` `v0.4.4 `__ + - `v0.4.5 `__ * - NVIDIA GDRCopy Driver - :cspan:`3` `v2.6 `__ * - NVIDIA Kata Sandbox Device Plugin - :cspan:`2` `v0.0.5 `__ - - `v0.0.6 `__ + - `v0.0.5 `__ .. _gds-open-kernel: diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index 75c33acef..b746d9bf3 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -38,21 +38,22 @@ Refer to the :ref:`GPU Operator Component Matrix` for a list of software compone 26.7.3 ====== +Breaking Changes +---------------- + +* DCGM Exporter 4.9.0 renames the driver-version label from ``DCGM_FI_DRIVER_VERSION`` to ``DCGM_FI_SYSTEM_DRIVER_VERSION``. + Update dashboards, alerts, and queries that reference the previous label name. + New Features ------------ * Updated software component versions: - NVIDIA Container Toolkit v1.20.2 - - NVIDIA Device Plugin for Kubernetes v0.20.2 - NVIDIA DCGM Exporter v4.7.0-4.9.0 - NVIDIA DCGM 4.7.0 - Node Feature Discovery v0.20.1 - - NVIDIA MIG Manager for Kubernetes v0.15.2 - - NVIDIA GPU Feature Discovery for Kubernetes v0.20.2 - - NVIDIA KubeVirt GPU Device Plugin v1.6.1 - - NVIDIA vGPU Device Manager v0.5.2 - - NVIDIA Kata Sandbox Device Plugin v0.0.6 + - NVIDIA Confidential Computing Manager v0.4.5 * The Operator supports the DCGM 4.7 metric names in DCGM Exporter v4.7.0-4.9.0. The default metrics file is ``/etc/dcgm-exporter/default-counters.csv``. @@ -99,6 +100,8 @@ Fixed Issues Known Issues ------------ +.. _v26.7.3-driver-manager-override: + * The default values use the same driver and driver manager container versions as the previous releases. These defaults prevent a disruptive driver upgrade. However, the following deployments need NVIDIA Driver Manager for Kubernetes v0.12.2: @@ -114,6 +117,16 @@ Known Issues * On systems with NVLink 5 or later, GPU driver pods can prevent containerized DOCA-OFED updates from completing. Refer to the :ref:`v26.7.2-known-issues` for details and the workaround. +* On NVIDIA B200 systems running OpenShift with standalone DCGM 4.7.0, DCGM Exporter 4.9.0 can enter ``CrashLoopBackOff`` when a custom metrics CSV includes ``DCGM_FI_PROF_NVLINK_TX_BYTES``. + The exporter fails to initialize the NVLink collector and reports ``This request is serviced by a module of DCGM that is not currently loaded``. + GPU metrics are unavailable while the exporter repeatedly restarts. + There is currently no workaround. + +* On OpenShift, enabling the NRI plugin through a cluster policy update before the NVIDIA driver and NVIDIA Container Toolkit finish initializing after a node reboot can leave standalone DCGM running without GPU access. + DCGM Exporter then enters ``CrashLoopBackOff``, and GPU metrics remain unavailable. + To work around this issue, use NVIDIA Driver Manager for Kubernetes v0.12.2. + Refer to the :ref:`Driver Manager image override instructions ` for the settings and the driver-upgrade consequence. + ---- .. _v26.7.2: diff --git a/gpu-operator/upgrade.rst b/gpu-operator/upgrade.rst index d831bf4eb..0addd52bb 100644 --- a/gpu-operator/upgrade.rst +++ b/gpu-operator/upgrade.rst @@ -53,6 +53,9 @@ DCGM Exporter 4.9.0 requires a DCGM 4.7 or later host engine and cannot start ag This requirement applies when ``dcgm.enabled=true``, including the default OpenShift configuration, or when the exporter connects to a host engine installed on the node. When upgrading a standalone DCGM deployment, the exporter can restart during the transition until the host engine has been upgraded. +DCGM Exporter renames the driver-version label from ``DCGM_FI_DRIVER_VERSION`` to ``DCGM_FI_SYSTEM_DRIVER_VERSION``. +Update dashboards, alerts, and queries that reference the previous label name; filters using that name can return no data after upgrading. + If you roll back these overrides, roll back both images together. When rolling back the Operator, also restore compatible DCGM and DCGM Exporter versions. Otherwise, OpenShift dashboards and alerts can lose GPU metrics because the older monitoring rules do not recognize the renamed metrics. From aef982ea5942036cd6e2e684e11323fe4a64d7c6 Mon Sep 17 00:00:00 2001 From: Mike McKiernan Date: Sat, 14 Nov 2026 17:15:11 +0000 Subject: [PATCH 5/5] docs(gpu-operator): Describe DCGM Exporter 4.9.0 metric renames as a breaking change List the metric renames and unit changes with the label rename, scope the preserved-names statement to the accelerator_* recording rules, add the NVLink custom-metrics workaround, and use the full cc-manager name. Signed-off-by: Mike McKiernan Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- gpu-operator/release-notes.rst | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index b746d9bf3..56992e3a6 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -41,8 +41,11 @@ Refer to the :ref:`GPU Operator Component Matrix` for a list of software compone Breaking Changes ---------------- -* DCGM Exporter 4.9.0 renames the driver-version label from ``DCGM_FI_DRIVER_VERSION`` to ``DCGM_FI_SYSTEM_DRIVER_VERSION``. - Update dashboards, alerts, and queries that reference the previous label name. +* DCGM Exporter 4.9.0 renames the default metrics to the DCGM 4.7 field names, and 11 of the renamed metrics change units. + For example, ``DCGM_FI_DEV_GPU_UTIL`` (percent) becomes ``DCGM_FI_DEV_GPU_UTIL_RATIO`` (0 to 1), and ``DCGM_FI_DEV_FB_USED`` (MiB) becomes ``DCGM_FI_DEV_FB_USED_BYTES`` (bytes). + The driver-version label on every metric changes from ``DCGM_FI_DRIVER_VERSION`` to ``DCGM_FI_SYSTEM_DRIVER_VERSION``. + Update dashboards, alerts, and queries that reference the previous metric or label names. + Refer to the `DCGM Exporter 4.9.0 release notes `__ for the complete list. New Features ------------ @@ -53,12 +56,12 @@ New Features - NVIDIA DCGM Exporter v4.7.0-4.9.0 - NVIDIA DCGM 4.7.0 - Node Feature Discovery v0.20.1 - - NVIDIA Confidential Computing Manager v0.4.5 + - NVIDIA Confidential Computing Manager for Kubernetes v0.4.5 * The Operator supports the DCGM 4.7 metric names in DCGM Exporter v4.7.0-4.9.0. The default metrics file is ``/etc/dcgm-exporter/default-counters.csv``. On OpenShift, the Operator's GPU monitoring rules support both the previous and renamed metrics. - Existing metric names, values, and units are preserved for dashboards and alerts. + The recorded ``accelerator_*`` metric names, values, and units are preserved for dashboards and alerts. (`GPU Operator PR #3110 `__) * When CDI is enabled, the Operator rejects a device plugin ``DEVICE_LIST_STRATEGY`` environment variable that does not include ``cdi-annotations`` or ``cdi-cri``. @@ -120,7 +123,7 @@ Known Issues * On NVIDIA B200 systems running OpenShift with standalone DCGM 4.7.0, DCGM Exporter 4.9.0 can enter ``CrashLoopBackOff`` when a custom metrics CSV includes ``DCGM_FI_PROF_NVLINK_TX_BYTES``. The exporter fails to initialize the NVLink collector and reports ``This request is serviced by a module of DCGM that is not currently loaded``. GPU metrics are unavailable while the exporter repeatedly restarts. - There is currently no workaround. + Do not add ``DCGM_FI_PROF_NVLINK_TX_BYTES`` to a custom metrics file. * On OpenShift, enabling the NRI plugin through a cluster policy update before the NVIDIA driver and NVIDIA Container Toolkit finish initializing after a node reboot can leave standalone DCGM running without GPU access. DCGM Exporter then enters ``CrashLoopBackOff``, and GPU metrics remain unavailable.