Skip to content

[Feature]: nvidia-dra-validator should be optionally running on some nodes and complete to release GPU claimed by it. #3056

Closed

Description

@davidandreoletti

Context:

Small cluster with 1 GPU. Both the nvidia gpu operator's dra-validation and other nvidia GPU workloads needing the same GPU.

Problem

The daemonset nvida-dra-validator aquires the GPU (via resource claim template) and runs in 2 steps:

    1. list available gpu on a node (GPU needed)
    1. print "all validations successful" every 86400s without requiring the GPU

The combination of Daemonset and step 2 prevents other GPU workloads from running despite the actual hardware/software GPU validation completed.

Activity

  1. added
    featureissue/PR that proposes a new feature or functionality
    needs-triageissue or PR has not been assigned a priority-px label
    on Oct 13, 2026
  2. changed the title [-][Feature]: nvidia-dra-validator should be optionally running on some nodes[/-] [+][Feature]: nvidia-dra-validator should be optionally running on some nodes and completed to release GPU claimed by it.[/+] on Oct 13, 2026
  3. changed the title [-][Feature]: nvidia-dra-validator should be optionally running on some nodes and completed to release GPU claimed by it.[/-] [+][Feature]: nvidia-dra-validator should be optionally running on some nodes and complete to release GPU claimed by it.[/+] on Oct 13, 2026
  4. francisguillier commented on Nov 5, 2026

    @francisguillier
    Contributor

    @davidandreoletti

    would it be possible to elaborate a little bit more the issue you are facing?

    When GPU operator with DRA integration is deployed, the following pods will be instantiated:

    gpu-operator              gpu-operator-node-feature-discovery-gc-7d4b864d68-jk4vv           1/1     Running   0          2m18s
    gpu-operator              gpu-operator-node-feature-discovery-master-8649684bd6-vmm4c       1/1     Running   0          2m18s
    gpu-operator              gpu-operator-node-feature-discovery-worker-lmbzj                  1/1     Running   0          2m18s
    gpu-operator              nvidia-dcgm-exporter-dra-msk2w                                    0/1     Running   0          108s
    gpu-operator              nvidia-dra-driver-controller-79bb7cd5c4-r6rms                     1/1     Running   0          118s
    gpu-operator              nvidia-dra-driver-kubelet-plugin-rn469                            2/2     Running   0          108s
    gpu-operator              nvidia-dra-validator-l24t5                                        1/1     Running   0          108s
    gpu-operator              nvidia-gpu-driver-ubuntu24.04-76bd69c5c8-6trzs                    1/1     Running   0          118s
    

    The nvidia-dra-validator pod has an init container (gpu-validation) that runs nvidia-smi -L and the main container (nvidia-dra-validator) that echos "all validations are successful" every 24 hours.

    The nvidia-dra-validator should not interfere with workloads requesting GPUs on the node, because its gpu-validation init container runs only once, when the NVIDIA GPU Operator is deployed.

  5. davidandreoletti commented on Nov 16, 2026

    @davidandreoletti
    Author

    Turns out, I misunderstood how dra valuation worked and in parallel had a set of pods which had GPU assignment failing due to the software's kubernetes operator not handling DRA resources well.

    It was wrong to have attributed the fault to the dra validation pod.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureissue/PR that proposes a new feature or functionalitylifecycle/frozenneeds-triageissue or PR has not been assigned a priority-px label

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions