Repository navigation
ci: add helm chart-testing and unittest workflow - #2427
giuliocalzo wants to merge 1 commit into
Conversation
a626e1a to
3e8fa1c
Compare
ArangoGutierrez
left a comment
There was a problem hiding this comment.
Thanks for adding these. With helm unittest and ct lint running on every chart PR, template regressions like a broken NRI plugin guard in validations.yaml get caught before they ship.
This adds a helm-unittest suite for each chart template (kept out of the packaged chart through .helmignore) and a helm workflow that runs ct lint and helm unittest on PRs under deployments/gpu-operator. What is left is in the matchSnapshot cases, which have to keep passing across rebases and Renovate image bumps before they can gate PRs.
| tests: | ||
| - it: should match snapshot | ||
| asserts: | ||
| - matchSnapshot: {} |
There was a problem hiding this comment.
The snapshots were recorded on the May base, and the default render has moved since: newer operand tags (driver-manager v0.12.1), the gpuclusters RBAC rules and the new container securityContext blocks. Run against main at 0fbefd5, these suites fail 12 tests and 11 snapshots across 7 suites, so this needs a rebase, helm unittest -u and a look at the snapshot diff before merging.
There was a problem hiding this comment.
Rebased onto 0fbefd5 and regenerated the snapshots. Apart from the version churn, the snapshot diff is the upstream template changes since May:
nvidia.com_gpuclusters.yamlin the upgrade/cleanup CRD Job args.- The new pod and container
securityContextandnodeSelectoron the operator Deployment and the upgrade/cleanup Jobs, plusVALIDATOR_IMAGEon the operator. - GPUCluster/DRA, events and leases rules in the ClusterRole, and the deployments rule in the Role.
- dcgm-exporter
serviceMonitor.enabled: trueandmetricRelabelings. default: trueandupgradePolicyon the NVIDIADriver CR.
I also dropped the operator.initContainer test, because 39ae253 removed that block from the template. All 17 suites, 64 tests and 24 snapshots pass on helm-unittest v1.0.3 (the version CI pins) and on v1.2.1.
| tests: | ||
| - it: should match snapshot | ||
| asserts: | ||
| - matchSnapshot: {} |
There was a problem hiding this comment.
The default-values matchSnapshot cases here and in operator_test.yaml and nvidiadriver_test.yaml capture every operand image tag in values.yaml. Renovate bumps those tags daily with secrets.GITHUB_TOKEN, and PRs opened with that token do not start pull_request workflows (#3050 shows only DCO), so its bumps merge without updated snapshots and the next chart PR fails on a diff its author did not make. Pinning the versions through set: or a fixture values file, or asserting the specific fields instead of the whole object, would avoid that.
There was a problem hiding this comment.
Thanks, good catch. The ClusterPolicy, operator and NVIDIADriver suites now load tests/values/pinned_versions.yaml, which sets every operand image version to pinned. The operator, validator and node-status-exporter images already follow the suites' chart.appVersion. No snapshot records a version from values.yaml anymore. With every version in values.yaml bumped, all 24 snapshots still pass.
Adding an operand could reintroduce the problem, so each of those suites also has a renders only pinned image versions test. It matches every spec..version against pinned. In the operator suite it also checks every container image and *_IMAGE env value. An unpinned version then fails the test and names the leaked tag, instead of ending up in a snapshot.
3e8fa1c to
cf1d142
Compare
| suite: clusterpolicy | ||
| templates: | ||
| - clusterpolicy.yaml |
There was a problem hiding this comment.
| - name: Install helm-unittest plugin | ||
| run: | | ||
| echo "Installing helm-unittest plugin..." | ||
| helm plugin install https://github.com/helm-unittest/helm-unittest.git --verify=false --version v1.0.3 # 6f82a998e0b5461762ca959f87f5dd344af5e4eb |
There was a problem hiding this comment.
|
No actionable comments were generated in the recent review.
|
Adds a `helm` GitHub Actions workflow that runs `ct lint` and `helm-unittest` against the gpu-operator chart on PRs that touch `deployments/`, plus a unit-test suite (with snapshots) covering the chart's core templates so chart regressions are caught in CI. Workflow: - `lint` job runs `ct lint` (with `--validate-maintainers=false` since the chart doesn't declare maintainers) using `CHART_DIR=deployments`. - `unittest` job installs `helm-unittest v1.0.3` and runs `helm unittest deployments/gpu-operator`. - Triggers only on PRs that change `deployments/gpu-operator/**`. Tests cover: operator deployment, ClusterPolicy, RBAC, NodeFeatureRules, NvidiaDriver, MIG/DCGM/device-plugin custom configs, cleanup and upgrade CRD hooks, OpenShift readonly-fs SCC, extra objects, and input validations. Snapshots are committed under `__snapshot__/`. The ClusterPolicy, operator Deployment and NVIDIADriver suites load `tests/values/pinned_versions.yaml`, which replaces every operand image version with `pinned`. Renovate opens its values.yaml version bumps with GITHUB_TOKEN, and PRs opened with that token do not trigger workflows, so snapshots of the real versions would go stale on main and fail the next unrelated chart PR. A "renders only pinned image versions" test in each of those suites fails when a rendered image version is not pinned. Also: - Fix yamllint comment-spacing in `values.yaml` so `ct lint` passes. - Add `tests/` to `.helmignore` so unittest fixtures and snapshots are excluded from the packaged chart. Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
cf1d142 to
7463d1b
Compare
Overview
Add a
helmGitHub Actions workflow that runsct lintandhelm-unittestagainst thegpu-operatorchart on PRs that touchdeployments/, plus a unit-test suite (with snapshots) covering the chart's core templates.Why helm unittest?
The
gpu-operatorchart is the primary install surface for the operator and ships a large set of templates whose behavior depends heavily onvalues.yaml, conditional logic, and platform-specific branches (OpenShift, MIG, NRI/CDI, etc.). Today, regressions in those templates can only be caught by spinning up a cluster, which is slow and not part of every PR.helm-unittestmakes it cheap to lock that behavior down in CI:helm templatein-process under a few hundred milliseconds, with no cluster, no network, no real Kubernetes API. Suitable for every PR.values.yamlcontracts. Eachset:block is an executable example of a supported configuration (develMode,imagePullSecrets, custom DCGM/MIG/device-plugin configs, NRI + CDI validation, etc.), so the expected behavior of values is visible in the repo.failedTemplateassertions confirm that input validations (e.g. "the NRI Plugin cannot be enabled when CDI is disabled") actually fire, whichct lintalone does not exercise.helm unittest -u, so intentional template changes are an explicit, reviewable diff in the PR.What's included
.github/workflows/helm.yamlwith two jobs:lint—helm/chart-testing-action(ct lint) against the chart.unittest— installshelm-unittestand runshelm unittest deployments/gpu-operator.deployments/gpu-operator/**.helm-unittestsuites underdeployments/gpu-operator/tests/covering: operator deployment, ClusterPolicy, ClusterRole/Binding, Role/Binding, ServiceAccount, NodeFeatureRules, NvidiaDriver, MIG/DCGM/device-plugin custom configs, cleanup and upgrade CRD hooks, OpenShift readonly-fs SCC, extra objects, and input validations. Snapshots are committed under__snapshot__/.values.yamlversion bumps withGITHUB_TOKEN, and those PRs don't trigger this workflow. To keep the bumps from leaving stale snapshots on main, the ClusterPolicy, operator and NVIDIADriver suites loadtests/values/pinned_versions.yaml, which sets every operand image version topinned. Arenders only pinned image versionstest in each of those suites fails if a rendered image version isn't pinned, for example after a new operand is added.Testing
helm unittest deployments/gpu-operatorpasses locally (17 suites, 64 tests, 24 snapshots) with helm-unittest v1.0.3 (pinned in CI) and v1.2.1.values.yamlbumped, all tests and snapshots still pass.ct lint --all --validate-maintainers=false --chart-dirs deploymentspasses locally.