Repository navigation
Open
Labels
area:buildCI/CD workflows, releases, build targets, packaging, and artifact publication.CI/CD workflows, releases, build targets, packaging, and artifact publication.topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.
Description
User Story
As a contributor waiting on Branch E2E, I want each compute driver to run the full conformance suite once per run, so that jobs that test one variant don't each spend 6–17 minutes re-running the whole suite.
Description
Every Docker, Podman, VM and Kubernetes job that runs directly on the runner runs the full conformance suite before its own tests, and the tmachine rows run it again. In run 44803367985 (77 tests):
| Driver | Full passes | Jobs |
|---|---|---|
| Kubernetes | 7 | kubernetes-e2e v1alpha1, v1beta1 and additional destination CA; workspace-managed; external-driver; tmachine k3s and k3s-ha |
| Docker | 5 | rust-docker, additional-ca, rust-docker-push; tmachine deb and deb-external-docker-driver |
| Podman | 5 | rust-podman-ci, additional-ca, external-driver; tmachine rootful and rootless |
| VM | 3 | rust-vm-managed, external-driver, additional-ca |
The variant jobs spend most of their time in the copy of the suite. kubernetes-e2e takes 21–25 min; 16.7–17.5 min of it is conformance on kind, and its own 39 Rust tests take 45 s.
Proposed change
- Choose one lane per driver, and per environment that really differs (for example k3s and k3s-ha), to run the full suite. Prefer the tmachine row where one exists.
- Other jobs run only the tests that exercise their difference, through a filter on the conformance program. Examples: the CA tests for additional-ca, a lifecycle smoke for push, external-driver and the Agent Sandbox v1alpha1 job.
- Write down which lane covers what, so moving tests in test(e2e): decide which e2e/rust tests to port and in what order #5297 doesn't lose coverage.
Context
- Parent: test: retire the old e2e test system onto testkit #5281. Related: test(e2e): decide which e2e/rust tests to port and in what order #5297 (moving host lanes onto testkit), ci(e2e): delete the host Podman lane after #5297 PR6 #5423, ci(e2e): show which environments run which test suites at each CI stage #5678 (show which environments run which suites).
- Open question: is any variant job meant to prove the whole suite passes under its variant? If yes, it keeps the full suite.
Definition of Done
- Each driver runs the full conformance suite on one lane per real environment difference, and other jobs run a stated subset.
- The choice of lanes and subsets is documented next to the job definitions or in
TESTING.md. - Branch E2E passes, and the PR records before/after times for the changed jobs.
+++ Agent notes
- Jobs are defined in
.github/workflows/branch-e2e.yml(docker-e2e, podman-e2e, vm-e2e, kubernetes-e2e and friends), withprebuilt-conformance: true/conformance: truepassed to thesetup-e2e-*actions. - Lane scripts that start the conformance program:
e2e/rust/e2e-{docker,podman,vm,kubernetes}.sh(e.g. "==> Running conformance suites against the Kubernetes gateway"). - Per-job times are from logs of run 44803367985 (job 136526209209 for v1alpha1).
+++
Metadata
Metadata
Assignees
Labels
area:buildCI/CD workflows, releases, build targets, packaging, and artifact publication.CI/CD workflows, releases, build targets, packaging, and artifact publication.topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.
Activity