Skip to content

test: add colocated core component acceptance suite - #841

Open
kvinwang wants to merge 397 commits into
nextfrom
codex/core-components-test-infrastructure
Open

test: add colocated core component acceptance suite#841
kvinwang wants to merge 397 commits into
nextfrom
codex/core-components-test-infrastructure

Conversation

@kvinwang

@kvinwang kvinwang commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

This PR adds the core-component acceptance suite and organizes the complete suite directly under test-suites/ so specifications, automation, fixtures, inventories, manifests, and retained audit evidence have one clear home.

  • Adds the deterministic dstack-test runner, dashboard, registry validation, fixture providers, and component harnesses.
  • Adds 373 acceptance cases covering guest OS, VMM, KMS, gateway, verifier, and cross-component integration behavior.
  • Colocates single-case entrypoints with their case.md specifications as run.py, run.sh, or run.cjs.
  • Keeps multi-case and location-sensitive harnesses in shared/automation/.
  • Distributes metadata across suite, chapter, section, and case directories; the runner discovers cases in directory-name order.
  • Uses each case-local metadata.json as the authoritative case-to-entrypoint binding.
  • Keeps fixture providers and images under shared/fixtures/, inventories under catalog/, environment manifests under manifests/, and retained historical evidence under audit/.
  • Adds suite authoring, execution, reporting, and methodology documentation.
  • Excludes product/runtime changes and source-local unit tests; those changes were split into dedicated PRs.

Layout

test-suites/
├── runner/
├── cases/
├── shared/
│   ├── automation/
│   └── fixtures/
├── catalog/
├── manifests/
└── audit/

Separation invariant

The PR targets next. Its changed paths are limited to:

  • REUSE.toml
  • docs/testing/**
  • test-suites/**

There are no product/runtime or source-local test changes under dstack/** or os/**.

Verification

  • Registry: 372 scripted / 372 promoted / 0 findings (373 total cases)
  • Runner unit tests: 40 passed
  • prek run --all-files: passed
  • reuse lint: passed
  • git diff --check origin/next...HEAD: passed

Rebase and tdxlab validation (2026-09-25)

Physical TDX (tdxlab, 4 workers, 372 scripted cases)

Tree PASS BLOCKED FAIL/ERROR
this branch on next (+ #1409 so images build) 351 7 14
this branch + #1405–#1414 365 7 0
this branch on next 635e7a2200 + #1414 + #1415 361 7 4 (KMS upgrade-matrix timeouts under load; all pass rerun alone)

The BLOCKED cases need an NVIDIA CC GPU on the TDX host or a Yocto image. The guest-side GPU cases were run on a GCP H100 CVM (a3-highgpu-1g) booting the candidate image and passed, apart from the reboot row that led to #1410.

Product fixes found while validating, sent separately

All of these are merged into next; #1409 and #1413 were closed in favour of #1415. tc-kms-onboard-005, which depends on #1406, is now promoted.

Local gates

Registry verification (373 total, 372 scripted, 372 promoted, 0 findings), 40 runner unit tests, reuse lint, prek run --all-files, and git diff --check origin/next...HEAD.

Rebased onto #1534 (2026-11-08)

The branch was rebased from its actual fork point 0fb3b24bbd94c18d4af2900b41cd82bcd1c0c284 onto #1534 (feat/kms-epochs-three-commits), rather than treating the current next tip as the source base.

Added tc-kms-rotation-013, a focused root/app key rotation lifecycle matrix covering keyring import, next-epoch generation, activation and serving, handover chain-hash checks, epoch-scoped authorization, restart recovery, disk epoch migration, and SealedEnv epoch binding. The matrix contains 14 focused Rust rows and rejects stale, forked, incomplete, or cross-epoch state.

First tdxlab rotation round: 14/14 rows passed on the tdxlab host on November 8, 2026, using Rust 1.92.0 and local Cargo targets. This round exercised the focused unit matrix; it did not boot a TDX VM or perform a live KMS deployment. Registry verification reports 373 scripted, 373 promoted, 374 total cases, with no findings. Prek passed for the added and modified files.

Forward-interval coverage audit (2026-11-08)

The earlier rotation-only addition was incomplete. This revision adds:

  • tc-kms-rotation-013: 14 Rust rows for keyring import/generation, activation and serving, handover validation, historical epochs, restart recovery, authorization selection, disk epoch migration, and SealedEnv epoch binding.
  • tc-kms-rotation-014: complete Foundry contract matrix for root/app epoch registration, chain hashes, activation boundaries, pending removal, evidence, owner checks, stale selectors, revocation, and upgrade compatibility.
  • tc-kms-rotation-015: auth-simple, auth-mock, auth-eth-bun, and auth-eth backend suites for operation- and epoch-scoped authorization.

The coverage audit is recorded in test-suites/audit/pr1534-forward-coverage-2026-11-08.md. The first tdxlab Rust round remains 14/14 PASS. tdxlab currently has neither Foundry nor the backend JavaScript dependencies, so contract and backend cases are intentionally not promoted or claimed as executed there.

Additional non-rotation coverage (2026-11-08)

Added tc-int-forward-016 for focused regressions from the forward interval outside key rotation: NUMA and extra-root-device measurement, VM annotations, passt/QMP behavior, guest container-name validation, RA-TLS key possession, TPM response bounds, and dstack-types serialization. The case deliberately fails on a zero-test result and records the live-vs-unit boundary.

The three newly added matrix cases (tc-kms-rotation-014, tc-kms-rotation-015, and tc-int-forward-016) remain unpromoted until their required Foundry, JavaScript, or full runtime prerequisites are available. The registry therefore reports them explicitly as pending rather than claiming execution.

tdxlab execution completed (2026-11-08)

After installing Foundry 1.9.0, Node.js 22.23.4, and the locked JavaScript dependencies on the tdxlab host:

  • tc-kms-rotation-013: 14/14 Rust rotation and migration rows passed.
  • tc-kms-rotation-014: 82/82 Foundry contract and upgrade rows passed (DstackKms.t.sol 42, DstackApp.t.sol 28, UpgradesWithPlugin.t.sol 12). Foundry artifacts were cleaned between suites.
  • tc-kms-rotation-015: auth-simple 33, auth-mock 15, auth-eth-bun 50, and auth-eth 22 tests passed under Node.js 22.23.4.
  • tc-int-forward-016: 11/11 focused non-rotation rows passed: NUMA layouts, extra root devices, VM annotations, passt/QMP, guest container-name validation, RA-TLS key possession, TPM response bounds, and dstack-types serialization.

The tdxlab host does not have a GPU or a live deployed KMS/Gateway chain in this round, so these results cover the deterministic contract, backend, Rust, and host-side matrices; live TDX VM lifecycle and deployed-chain rotation remain outside this execution round. The registry now reports 376 scripted, 376 promoted, 377 total cases, and zero findings.

Physical-TDX rotation migration result (2026-11-08)

The corrected physical-TDX harness passed tc-int-compatibil-008 on the latest run kms-e2e-20261108-fix18:

  • source KMS/OS: 0.6.0
  • KMS rotation onboarding: passed
  • same app VM and persistent disk retained
  • candidate OS image updated in place
  • reboot and disk/epoch migration: passed
  • VM identity preserved
  • observer started after migration
  • private material export: false

The run also passed the existing physical mixed KMS upgrade path once (tc-int-mixed-002, run kms-e2e-20261108-fix11). Earlier failures were retained as diagnostic runs while correcting test version pairing, signer pin isolation/normalization, Python SealedEnv dependencies, and the 0.6.0 image preparation.

Final 0.6.0 rotation migration verification (2026-11-08)

The latest physical-TDX run kms-e2e-20261108-fix17 passed tc-int-compatibil-008:

  • 0.6.0 KMS/OS source
  • root rotation onboarding to the candidate KMS
  • same VM and persistent disk retained
  • candidate OS updated in place
  • reboot completed with the migrated disk
  • VM identity preserved
  • observer and candidate app boot completed
  • no private material exported

The hardware preparation script now checks cryptography, eth_keys, eth_utils, and pyhpke before starting a build, so missing SealedEnv dependencies fail during preparation instead of during VM deployment.

Copilot AI review requested due to automatic review settings July 31, 2026 02:08
Copilot AI reviewed Jul 31, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@kvinwang kvinwang changed the title test: add core component acceptance infrastructure [STACKED on #840] test: add core component acceptance infrastructure Jul 31, 2026
@kvinwang
kvinwang force-pushed the codex/core-components-test-infrastructure branch from 7a21a97 to 313efb9 Compare July 31, 2026 03:34
@kvinwang kvinwang changed the title [STACKED on #840] test: add core component acceptance infrastructure test: add core component acceptance infrastructure Jul 31, 2026
@kvinwang
kvinwang changed the base branch from codex/core-components-product-fixes to master July 31, 2026 03:34
This was referenced Jul 31, 2026
Closed
Merged
Merged
Merged
Closed
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Closed
Merged
Merged
Merged
Merged
Closed
Merged
Merged
Merged
Merged
@kvinwang
test: match the public handover refusal changed in #1552
96a4f97 
Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: give simulated KMS clients a TDX lite attestation
373b1ac 
Since #1533, KMS key release verifies a TDX caller's OS image. The SDK
simulator replays a captured legacy quote whose image (e61be43d...) was
never published, so every simulated KMS client failed at the image
download.

Derive the KMS fixtures' simulator files with the candidate
dstack-tee-simulator instead: it computes the quote measurements from the
lite tdx_measurement document of the verifier fixture, the same way the
verifier does, and the captured RTMR3 events are replayed on top. The
verifier then takes the lite path and downloads nothing. The simulator
runs in a private user and mount namespace because the TDX ABI paths
under /sys are fixed.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: encode the KMS RPC protobuf length as a varint
6f0f01a 
The simulator vm_config is longer than 127 bytes, so the single-byte
length prefix rejected it before any request was sent.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: cover SealedEnv epochs across a KMS root rotation
dc46a1b 
tc-int-compatibil-009 seals an app's env at root epoch 0, rotates the KMS
to epoch 1, and checks on physical TDX that the guest:

- unseals with the historical key once and reseals to the current epoch,
  then fetches no historical key on later boots;
- rejects an envelope labeled above the current epoch before requesting
  any historical key;
- fetches the key a forged epoch label names and then fails to decrypt.

The epochs a boot requested come from the guest console, cross-checked
against the KMS app-key release counter. deploy_client also accepts a
SealedEnv deployment, whose epoch signer the VMM CLI verifies silently.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
kvinwang force-pushed the codex/core-components-test-infrastructure branch from e7743dc to fcc4e6c Compare November 11, 2026 14:03
@kvinwang
test: adapt KMS fixtures to the handover startup check
6332786 
Since #1564 a KMS refuses to start without a key handover path, and the
default config enables the admin listener with TLS at /etc/kms/certs.

- The isolated KMS fixtures serve admin TLS with their own RPC key and
  certificate, so their admin URLs are https.
- tc-kms-onboard-005 no longer adds admin TLS before disabling the public
  handover; the fixture already has it.
- tc-kms-startup-001 calls the target's admin listener over https.
- Drop the public_key_handover line this branch added to the full-stack
  KMS config; #1564 added the same key there.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
kvinwang force-pushed the codex/core-components-test-infrastructure branch from 06ba15a to 6332786 Compare November 11, 2026 14:03
Base automatically changed from feat/kms-epochs-combined to next November 11, 2026 14:11
@kvinwang
test: present a guest API token to the Worker listener
9e57f63 
The guest agent's external listener now grants nothing anonymously unless
the host lists scopes in sys-config. The Worker cases check that an
anonymous call is refused, then mint a token the way an app may: they
sign it with the internal issuer key from GetKey on the internal socket.
An unknown route on that listener is refused with 401 before dispatch,
so it no longer returns a pRPC error body.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: let physical guests pass KMS key release
de1e602 
- Key release verifies the guest's OS image, and the case-scoped KMS had
  no image to download. Link the candidate images from the image store
  into its image cache by digest.
- The VMM CLI seals env only to a pinned KMS signer; pin the signer the
  KMS serves, in a lease-scoped whitelist.
- Bind vm_root_path before the deploy so a failed deploy reports its
  error instead of an UnboundLocalError.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: build the ECDH decrypt probe with the SealedEnv dependencies
e6ce18e 
dstack-util's crypto module now also uses hpke, base64 and serde_json.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: render the dashboard probe with the scope-gated fields
ca1d7a8 
The dashboard model replaced the public_* switches with per-caller show_*
flags; the probe no longer compiled, so every check reported false.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: grant the sample guest API scopes and share token minting
25b7cd9 
- The simulator fixtures grant anonymous callers what the candidate's
  sample vmm.toml grants (dashboard, version, health, info), as a host
  deployed from that sample does.
- Move token minting into guest_api_token.py. The contract matrix and the
  robustness harness present a token on the Worker listener for the
  scopes the sample leaves closed (attest-app-key, logs, containers).
- An unknown method on that listener is refused with 401 before
  dispatch; the contract matrix checks that instead of a pRPC error body.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: expect the KMS to always verify OS images
aaa9b8c 
c25e974 removed [core.image] verify: the KMS always verifies, ignores a
leftover key, and GetMeta.os_image_verification always reports true.
Check that instead of mirroring the removed switch, and drop the ignored
key from the candidate KMS fixtures.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: plant measurement cache entries as their JSON files
32b4d35 
The verifier caches each measurement as measurements/<config>.json, so a
directory named after the config hash is never a cache entry and a
targeted clear leaves it in place.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: report steps from the KMS rotation unit matrices
3fb27c2 
The runner now rejects a PASS result without an all-PASS step, which
turned tc-kms-rotation-014 into INCOMPLETE. Report a step from the
rotation-013/014/015 harnesses. Also, for rotation-013, track the renamed
LUKS and SealedEnv tests and match each name exactly; for rotation-015,
keep a failing suite's output and report a summary matching its status.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: require at least the original upgrade-authority tests
0e8b418 
The suite grew from 8 to 14 tests; an exact count made the case fail on
every addition while it only needs to catch a filter that runs fewer.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: answer KMS authorization with an epoch-aware webhook
de0af00 
Since #1533 the KMS asks a webhook /authorizeApp for an AuthorizationResult
that carries the registered root epoch; a bootAuth-shaped reply is not
accepted, and an unregistered root epoch refuses to serve. The mock
authority now registers the KMS's own keys as root epoch 0 and answers in
that shape.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: drop the removed KMS compose from the locked-build check
7eaf2ff 
3937f527 removed dstack/kms/dstack-app/docker-compose.yaml; the dev
compose that replaced it builds no Rust, so three locked builds remain.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: report a step from the forward interval matrix
d557a02 
The runner rejects a PASS result without an all-PASS step. Count a row
as passed only when its test ran and passed, so a failing last row no
longer reports PASS.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: answer the upgrade-policy webhook in the epoch-aware shape
7fb4b88 
A current KMS asks its webhook /authorizeKms|App and accepts only an
AuthorizationResult; KMS 0.5.x still asks /bootAuth. Serve both from the
same policy and observation log.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: name the root epoch row in Manage.s.sol policy writes
33fb808 
Since #1561 the management scripts take ROOT_EPOCH or a range instead of
writing implicit rows.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: serve every matrix guest image to the KMS verifier
1a4b554 
The KMS now always verifies a guest's OS image, so a candidate KMS must
download the old guests' images too. Archive each matrix image once into
a shared cache and serve all of them.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: pin the KMS signer before deploying a Gateway
375935c 
The VMM CLI seals a SealedEnv-capable image's env only to a pinned KMS
epoch signer, as it already does for clients.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: run the auth-eth suite on Node 20 or newer
a8fe4a0 
vitest needs node:util.styleText; prefer the newest Node on PATH or from
fnm instead of the host default.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: publish dashboard metrics, logs and containers to lease callers
78e9b0c 
The guest API now grants anonymous callers only the scopes the host lists.
Lease VMMs extend the sample grant with the blocks the observability cases
read, a host decision equivalent to naming an issuer for them.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: register webhook-backed matrix KMS nodes as root epoch 0
9196257 
A KMS behind a webhook serves only a registered root epoch. The matrix
registers each webhook-backed candidate KMS, and an onboarded target, as
epoch 0 of its context; the fixture webhook reports it. Share the chain
hash with the upgrade-authority case.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: hash the SealedEnv envelope of a matrix client
646e120 
A SealedEnv deployment stores its envelope in configuration.sealed_env,
leaving encrypted_env empty.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
test: answer the current app epoch from the upgrade-policy webhook
0ff5221 
A current KMS reads GET /app/<id> before it serves an app's keys.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang kvinwang mentioned this pull request Nov 11, 2026
Open

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
+1 more reviewer
@github-advanced-security github-advanced-security[bot] github-advanced-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kvinwang @github-advanced-security