You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Feature]: nvidia-dra-validator should be optionally running on some nodes and complete to release GPU claimed by it. #3056
changed the title [-][Feature]: nvidia-dra-validator should be optionally running on some nodes[/-][+][Feature]: nvidia-dra-validator should be optionally running on some nodes and completed to release GPU claimed by it.[/+]on Oct 13, 2026
changed the title [-][Feature]: nvidia-dra-validator should be optionally running on some nodes and completed to release GPU claimed by it.[/-][+][Feature]: nvidia-dra-validator should be optionally running on some nodes and complete to release GPU claimed by it.[/+]on Oct 13, 2026
The nvidia-dra-validator pod has an init container (gpu-validation) that runs nvidia-smi -L and the main container (nvidia-dra-validator) that echos "all validations are successful" every 24 hours.
The nvidia-dra-validator should not interfere with workloads requesting GPUs on the node, because its gpu-validation init container runs only once, when the NVIDIA GPU Operator is deployed.
Turns out, I misunderstood how dra valuation worked and in parallel had a set of pods which had GPU assignment failing due to the software's kubernetes operator not handling DRA resources well.
It was wrong to have attributed the fault to the dra validation pod.
Activity
[-][Feature]: nvidia-dra-validator should be optionally running on some nodes[/-][+][Feature]: nvidia-dra-validator should be optionally running on some nodes and completed to release GPU claimed by it.[/+][-][Feature]: nvidia-dra-validator should be optionally running on some nodes and completed to release GPU claimed by it.[/-][+][Feature]: nvidia-dra-validator should be optionally running on some nodes and complete to release GPU claimed by it.[/+]francisguillier commentedon Nov 5, 2026
@davidandreoletti
would it be possible to elaborate a little bit more the issue you are facing?
When GPU operator with DRA integration is deployed, the following pods will be instantiated:
The nvidia-dra-validator pod has an init container (gpu-validation) that runs
nvidia-smi -Land the main container (nvidia-dra-validator) that echos "all validations are successful" every 24 hours.The nvidia-dra-validator should not interfere with workloads requesting GPUs on the node, because its gpu-validation init container runs only once, when the NVIDIA GPU Operator is deployed.
davidandreoletti commentedon Nov 16, 2026
Turns out, I misunderstood how dra valuation worked and in parallel had a set of pods which had GPU assignment failing due to the software's kubernetes operator not handling DRA resources well.
It was wrong to have attributed the fault to the dra validation pod.