Skip to content

ci: add helm chart-testing and unittest workflow - #2427

Open
giuliocalzo wants to merge 1 commit into
NVIDIA:mainfrom
giuliocalzo:ci/helm-chart-testing
Open

giuliocalzo wants to merge 1 commit into
NVIDIA:mainfrom
giuliocalzo:ci/helm-chart-testing

Conversation

@giuliocalzo

@giuliocalzo giuliocalzo commented May 5, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Add a helm GitHub Actions workflow that runs ct lint and helm-unittest against the gpu-operator chart on PRs that touch deployments/, plus a unit-test suite (with snapshots) covering the chart's core templates.

Why helm unittest?

The gpu-operator chart is the primary install surface for the operator and ships a large set of templates whose behavior depends heavily on values.yaml, conditional logic, and platform-specific branches (OpenShift, MIG, NRI/CDI, etc.). Today, regressions in those templates can only be caught by spinning up a cluster, which is slow and not part of every PR.

helm-unittest makes it cheap to lock that behavior down in CI:

  • Fast, hermetic feedback. Runs helm template in-process under a few hundred milliseconds, with no cluster, no network, no real Kubernetes API. Suitable for every PR.
  • Catches templating regressions early. Snapshot tests on rendered manifests fail loudly on unintended diffs (e.g. removed args, changed RBAC verbs, broken priorityClassName handling).
  • Documents values.yaml contracts. Each set: block is an executable example of a supported configuration (develMode, imagePullSecrets, custom DCGM/MIG/device-plugin configs, NRI + CDI validation, etc.), so the expected behavior of values is visible in the repo.
  • Validates failure paths too. failedTemplate assertions confirm that input validations (e.g. "the NRI Plugin cannot be enabled when CDI is disabled") actually fire, which ct lint alone does not exercise.
  • Cheap to maintain. Snapshots regenerate with helm unittest -u, so intentional template changes are an explicit, reviewable diff in the PR.

What's included

  • New workflow .github/workflows/helm.yaml with two jobs:
    • lint — helm/chart-testing-action (ct lint) against the chart.
    • unittest — installs helm-unittest and runs helm unittest deployments/gpu-operator.
    • Triggered only on PRs that change deployments/gpu-operator/**.
  • New helm-unittest suites under deployments/gpu-operator/tests/ covering: operator deployment, ClusterPolicy, ClusterRole/Binding, Role/Binding, ServiceAccount, NodeFeatureRules, NvidiaDriver, MIG/DCGM/device-plugin custom configs, cleanup and upgrade CRD hooks, OpenShift readonly-fs SCC, extra objects, and input validations. Snapshots are committed under __snapshot__/.
  • Snapshots don't record operand image versions. Renovate opens its values.yaml version bumps with GITHUB_TOKEN, and those PRs don't trigger this workflow. To keep the bumps from leaving stale snapshots on main, the ClusterPolicy, operator and NVIDIADriver suites load tests/values/pinned_versions.yaml, which sets every operand image version to pinned. A renders only pinned image versions test in each of those suites fails if a rendered image version isn't pinned, for example after a new operand is added.

Testing

  • helm unittest deployments/gpu-operator passes locally (17 suites, 64 tests, 24 snapshots) with helm-unittest v1.0.3 (pinned in CI) and v1.2.1.
  • With every image version in values.yaml bumped, all tests and snapshots still pass.
  • ct lint --all --validate-maintainers=false --chart-dirs deployments passes locally.

@ArangoGutierrez ArangoGutierrez left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding these. With helm unittest and ct lint running on every chart PR, template regressions like a broken NRI plugin guard in validations.yaml get caught before they ship.

This adds a helm-unittest suite for each chart template (kept out of the packaged chart through .helmignore) and a helm workflow that runs ct lint and helm unittest on PRs under deployments/gpu-operator. What is left is in the matchSnapshot cases, which have to keep passing across rebases and Renovate image bumps before they can gate PRs.

tests:
- it: should match snapshot
asserts:
- matchSnapshot: {}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The snapshots were recorded on the May base, and the default render has moved since: newer operand tags (driver-manager v0.12.1), the gpuclusters RBAC rules and the new container securityContext blocks. Run against main at 0fbefd5, these suites fail 12 tests and 11 snapshots across 7 suites, so this needs a rebase, helm unittest -u and a look at the snapshot diff before merging.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebased onto 0fbefd5 and regenerated the snapshots. Apart from the version churn, the snapshot diff is the upstream template changes since May:

  • nvidia.com_gpuclusters.yaml in the upgrade/cleanup CRD Job args.
  • The new pod and container securityContext and nodeSelector on the operator Deployment and the upgrade/cleanup Jobs, plus VALIDATOR_IMAGE on the operator.
  • GPUCluster/DRA, events and leases rules in the ClusterRole, and the deployments rule in the Role.
  • dcgm-exporter serviceMonitor.enabled: true and metricRelabelings.
  • default: true and upgradePolicy on the NVIDIADriver CR.

I also dropped the operator.initContainer test, because 39ae253 removed that block from the template. All 17 suites, 64 tests and 24 snapshots pass on helm-unittest v1.0.3 (the version CI pins) and on v1.2.1.

tests:
- it: should match snapshot
asserts:
- matchSnapshot: {}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default-values matchSnapshot cases here and in operator_test.yaml and nvidiadriver_test.yaml capture every operand image tag in values.yaml. Renovate bumps those tags daily with secrets.GITHUB_TOKEN, and PRs opened with that token do not start pull_request workflows (#3050 shows only DCO), so its bumps merge without updated snapshots and the next chart PR fails on a diff its author did not make. Pinning the versions through set: or a fixture values file, or asserting the specific fields instead of the whole object, would avoid that.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good catch. The ClusterPolicy, operator and NVIDIADriver suites now load tests/values/pinned_versions.yaml, which sets every operand image version to pinned. The operator, validator and node-status-exporter images already follow the suites' chart.appVersion. No snapshot records a version from values.yaml anymore. With every version in values.yaml bumped, all 24 snapshots still pass.

Adding an operand could reintroduce the problem, so each of those suites also has a renders only pinned image versions test. It matches every spec..version against pinned. In the operator suite it also checks every container image and *_IMAGE env value. An unpinned version then fails the test and names the leaked tag, instead of ending up in a snapshot.

@giuliocalzo
giuliocalzo force-pushed the ci/helm-chart-testing branch from 3e8fa1c to cf1d142 Compare October 25, 2026 17:58
@giuliocalzo
giuliocalzo requested a review from a team as a code owner October 25, 2026 17:58

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Devin Review

Comment on lines +1 to +3
suite: clusterpolicy
templates:
- clusterpolicy.yaml

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 DRA installation path lacks snapshot coverage

The suites cover ClusterPolicy but omit the alternative GPUCluster manifest and its cleanup hook. Changes to DRA installation manifests therefore lack the new snapshot checks.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

- name: Install helm-unittest plugin
run: |
echo "Installing helm-unittest plugin..."
helm plugin install https://github.com/helm-unittest/helm-unittest.git --verify=false --version v1.0.3 # 6f82a998e0b5461762ca959f87f5dd344af5e4eb

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟨 Unverified Helm plugin executes in CI

On every chart pull request, helm plugin install runs code from a mutable tag without verifying its signature. The commit SHA in the comment does not pin the installed code.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

@coderabbitai

coderabbitai Bot commented Oct 25, 2026

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 0fbefd5 and cf1d142.

Files ignored due to path filters: 16
  • deployments/gpu-operator/tests/__snapshot__/cleanup_crd_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/clusterpolicy_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/clusterrole_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/clusterrolebinding_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/dcgm_exporter_config_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/extra_objects_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/mig_config_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/nodefeaturerules_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/nvidiadriver_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/operator_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/plugin_config_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/readonlyfs_scc_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/role_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/rolebinding_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/serviceaccount_test.yaml.snap is excluded by !**/*.snap
  • deployments/gpu-operator/tests/__snapshot__/upgrade_crd_test.yaml.snap is excluded by !**/*.snap
Files selected for processing: 21
  • .github/workflows/helm.yaml
  • deployments/gpu-operator/.helmignore
  • deployments/gpu-operator/tests/cleanup_crd_test.yaml
  • deployments/gpu-operator/tests/clusterpolicy_test.yaml
  • deployments/gpu-operator/tests/clusterrole_test.yaml
  • deployments/gpu-operator/tests/clusterrolebinding_test.yaml
  • deployments/gpu-operator/tests/dcgm_exporter_config_test.yaml
  • deployments/gpu-operator/tests/extra_objects_test.yaml
  • deployments/gpu-operator/tests/mig_config_test.yaml
  • deployments/gpu-operator/tests/nodefeaturerules_test.yaml
  • deployments/gpu-operator/tests/nvidiadriver_test.yaml
  • deployments/gpu-operator/tests/operator_test.yaml
  • deployments/gpu-operator/tests/plugin_config_test.yaml
  • deployments/gpu-operator/tests/readonlyfs_scc_test.yaml
  • deployments/gpu-operator/tests/role_test.yaml
  • deployments/gpu-operator/tests/rolebinding_test.yaml
  • deployments/gpu-operator/tests/serviceaccount_test.yaml
  • deployments/gpu-operator/tests/upgrade_crd_test.yaml
  • deployments/gpu-operator/tests/validations_test.yaml
  • deployments/gpu-operator/tests/values/pinned_versions.yaml
  • deployments/gpu-operator/values.yaml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

Adds a pull request workflow that runs chart linting and Helm unit tests for changes under deployments/gpu-operator/**. Adds Helm tests for template rendering, configuration values, conditional resources, and validation behavior. Updates the chart ignore file and adds a pinned-version test fixture.

Priority: Low

Merge Risk: Minimal · up to cf1d1

The version assertion covers both driver and GDS; no outstanding issue identified here prevents merging after normal checks.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Adds a `helm` GitHub Actions workflow that runs `ct lint` and
`helm-unittest` against the gpu-operator chart on PRs that touch
`deployments/`, plus a unit-test suite (with snapshots) covering the
chart's core templates so chart regressions are caught in CI.

Workflow:
- `lint` job runs `ct lint` (with `--validate-maintainers=false` since
  the chart doesn't declare maintainers) using `CHART_DIR=deployments`.
- `unittest` job installs `helm-unittest v1.0.3` and runs
  `helm unittest deployments/gpu-operator`.
- Triggers only on PRs that change `deployments/gpu-operator/**`.

Tests cover: operator deployment, ClusterPolicy, RBAC, NodeFeatureRules,
NvidiaDriver, MIG/DCGM/device-plugin custom configs, cleanup and upgrade
CRD hooks, OpenShift readonly-fs SCC, extra objects, and input
validations. Snapshots are committed under `__snapshot__/`.

The ClusterPolicy, operator Deployment and NVIDIADriver suites load
`tests/values/pinned_versions.yaml`, which replaces every operand image
version with `pinned`. Renovate opens its values.yaml version bumps with
GITHUB_TOKEN, and PRs opened with that token do not trigger workflows,
so snapshots of the real versions would go stale on main and fail the
next unrelated chart PR. A "renders only pinned image versions" test in
each of those suites fails when a rendered image version is not pinned.

Also:
- Fix yamllint comment-spacing in `values.yaml` so `ct lint` passes.
- Add `tests/` to `.helmignore` so unittest fixtures and snapshots are
  excluded from the packaged chart.

Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
@giuliocalzo
giuliocalzo force-pushed the ci/helm-chart-testing branch from cf1d142 to 7463d1b Compare October 25, 2026 18:23

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants