Skip to content

[Bug]: WITH_SHUTDOWN_HOST_GPU_CLIENTS for migManager ignored, because PR #2702 was not cherry-picked #3230

Open

Description

@erezzarum

Motivation

The WITH_SHUTDOWN_HOST_GPU_CLIENTS environment variable on migManager is ignored because PR #2702 was not cherry-picked into a release.

  • I would like to work on this issue.

Describe the bug

Not being able to set WITH_SHUTDOWN_HOST_GPU_CLIENTS=false correctly.

To Reproduce

Deploy GPU Operator with:

driver:
  enabled: false
toolkit:
  enabled: false
devicePlugin:
  enabled: true
nfd:
  enabled: true
gfd:
  enabled: true
mig:
  strategy: mixed
migManager:
  env:
    - name: WITH_SHUTDOWN_HOST_GPU_CLIENTS
      value: "false"

On nodes where the driver is already pre-installed, the WITH_SHUTDOWN_HOST_GPU_CLIENTS is always set to true, PR #2702 resolves it.

Expected behavior

Honor WITH_SHUTDOWN_HOST_GPU_CLIENTS

Environment (please provide the following information):

  • GPU Operator Version: v26.7.2
  • OS: Bottlerocket 1.65
  • Kernel Version: [e.g. 6.8.0-generic]
  • Container Runtime Version: [e.g. containerd 2.0.0]
  • Kubernetes Distro and Version: [e.g. K8s, OpenShift, Rancher, GKE, EKS]

Information to attach (optional if deemed irrelevant)

  • kubernetes pods status: kubectl get pods -n OPERATOR_NAMESPACE
  • kubernetes daemonset status: kubectl get ds -n OPERATOR_NAMESPACE
  • If a pod/ds is in an error state or pending state kubectl describe pod -n OPERATOR_NAMESPACE POD_NAME
  • If a pod/ds is in an error state or pending state kubectl logs -n OPERATOR_NAMESPACE POD_NAME --all-containers
  • Output from running nvidia-smi from the driver container: kubectl exec DRIVER_POD_NAME -n OPERATOR_NAMESPACE -c nvidia-driver-ctr -- nvidia-smi
  • containerd logs journalctl -u containerd > containerd.log

Activity

  1. added
    bugIssue/PR to expose/discuss/fix a bug
    needs-triageissue or PR has not been assigned a priority-px label
    on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIssue/PR to expose/discuss/fix a bugneeds-triageissue or PR has not been assigned a priority-px label

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions