Repository navigation
feat(kms): add root/app key epochs, SealedEnv and epoch-scoped authorization - #1533
Merged
Merged
Conversation
kvinwang
force-pushed
the
feat/kms-epochs-combined
branch
4 times, most recently
from
November 8, 2026 02:57
51f8598 to
933ffb6
Compare
This was referenced Nov 8, 2026
kvinwang
force-pushed
the
feat/kms-epochs-combined
branch
2 times, most recently
from
November 8, 2026 03:34
4ed6d0d to
1de3c9e
Compare
kvinwang
force-pushed
the
feat/kms-epochs-combined
branch
from
November 8, 2026 07:26
1de3c9e to
96edd77
Compare
Describe recovery from KMS root-key exposure using a fleet-wide maintenance window: per-epoch roots registered on DstackKms, a single GetAppKey epoch selector (absent = epoch 0, -1 = latest), LUKS2 keyslot rotation with an epoch token, in-CVM env re-encryption, full-history KMS onboarding with a rotation mode, and a clean gateway reset. Signed-off-by: Kevin Wang <wy721@qq.com>
DstackKms keeps an append-only list of root epochs (CA and k256 public keys, the URL of the attestation proving their generation, and a keccak256 chain hash over all epochs so far) and a separately activated current epoch. Registration and activation are distinct steps so every KMS instance can import an epoch before it goes live. A pending epoch can be popped when the nodes holding it are lost before it is activated. Epoch 0 must be the kmsInfo keys when those are set. The attestation stays off chain: its report data binds the keys, so it needs no on-chain hash, and storing it would cost about 720 gas per byte. The KMS sends the root epoch it serves and its chain hash as rootEpoch/rootEpochHash in the boot info; every auth backend (auth-eth, auth-eth-bun, auth-simple, auth-mock, dstack-auth) denies a KMS that does not serve the current epoch with the registered keys. A KMS that predates root epochs reports none and is cut off once a later epoch is current. Info responses report the current epoch so a KMS can pick it at startup. dstack-types gains the shared chain hash computation and test vectors. Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Evidence updates are recoverable and mismatched evidence fails verification. Revert the unnecessary chain hash requirement for URL updates; retain it for activation and removal. Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
kvinwang
force-pushed
the
feat/kms-epochs-combined
branch
3 times, most recently
from
November 8, 2026 14:14
1a7dd23 to
a5457bf
Compare
Signed-off-by: Kevin Wang <wy721@qq.com>
The KMS stores every root epoch it holds (the newest at the existing paths, older ones under epochs/<n>/) and at startup serves the epoch the auth backend reports as current, provided its own keys produce the registered chain hash. Every boot check carries the served epoch and its chain hash so the backend can deny a stale or forked KMS; on a denial the KMS re-reads the current epoch and exits to restart onto it if it moved. GetAppKey takes an optional root_epoch (unset = current) so guests can fetch the keys of an older epoch to migrate, and responses carry the epoch served. Onboarding hands over the whole keyring and checks it against the chain. With rotate, the new KMS also generates the next epoch and attests it; the source must then be at the current epoch and pass the strict KMS authorization check. Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
A request may select any epoch up to the current one and is decided on that epoch's row, but the epoch-less setters only wrote rows from the current epoch on. After a rotation, removing an OS image, compose hash or device left older rows granting it. - DstackKms: drop the epoch-less add/remove setters for OS images, KMS aggregated MRs and KMS devices. Keep the *AtEpoch setters and add *AtEpochs(fromEpoch, toEpoch, value) range setters, so the caller names the rows and one transaction revokes a value from every row. - DstackApp: the IAppAuthBasicManagement setters and setAllowAnyDevice keep their ABI and interface ID but now write every row from 0 through the next epoch, so a removal is a full revocation. - Key root policy rows by registration instead of chain hash. A popped epoch re-registered with the same keys had the same chain hash and got its old rows back; every registration now starts with empty rows. Epoch 0 still uses the v0.5.x mappings. - Cap kms_serve at the current root epoch. Only the key handover operations may select a pending epoch. - Manage.s.sol KMS scripts take ROOT_EPOCH or FROM_ROOT_EPOCH and TO_ROOT_EPOCH. By default additions write the current through the latest epoch, and removals write every row. - Fix two stale DstackKms comments: evidence report data does not bind the epoch, and KMS authorization does not check the app ID. Signed-off-by: Kevin Wang <wy721@qq.com>
Disk and env migration fetch the keys of the recorded epochs, so an image not allowed in those rows stops boot. Say which root and app epoch it was. Signed-off-by: Kevin Wang <wy721@qq.com>
The symbolic checks cover properties no Forge test covers, such as epoch monotonicity and the legacy entry points refusing after rotation. Run each suite in its own job when dstack/kms/auth-eth changes, with Halmos and Forge pinned. Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
As in DstackKms, every DstackApp setter now names the app epoch rows it writes: *AtEpoch one row, and new *AtEpochs(fromEpoch, toEpoch, ...) a range up to the next epoch. The epoch-less addComposeHash, removeComposeHash, addDevice, removeDevice and setAllowAnyDevice are gone, and with them IAppAuthBasicManagement (0x8fd37527). The new IAppEpochManagement interface (0x853d2b79) lets tools tell current apps from those predating app epochs. App policies deployed from v0.5.9 and v0.6.0 keep working under the upgraded DstackKms: it authorizes them through isAppAllowed for app epoch 0 only. Manage.s.sol detects them by interface ID and calls their epoch-less setters, accepting only APP_EPOCH=0 or no epoch. For current policies, and for every KMS script, it requires an explicit epoch or range. Upgrading an old app keeps its allowlist as app epoch 0. New tests cover both releases against the upgraded KMS, before and after a root rotation, the interface IDs, and the upgrade. Formal verification: - The Halmos properties now cover symbolic ranges, full-range revocation on both sides, empty rows after pop and re-registration, epoch 0 rows being the legacy mappings, and legacy app policies limited to app epoch 0. - New TLA+ model EpochPolicy covers registration-keyed rows, range writes, pop and re-registration, legacy apps and their upgrade, and key release on the selected rows. Two mutation configs show the chain-hash keying and a legacy fallback that ignores the app epoch each break it. Signed-off-by: Kevin Wang <wy721@qq.com>
fix(kms-auth)!: make policy revocation cover every selectable epoch
Signed-off-by: Kevin Wang <wy721@qq.com>
…rt issuance Report a backend rejection before the epoch fields are checked, require attestation in GetAppKey and the key handover before the KMS self check reaches the auth backend, and add audit lines for SignCert and GetTempCaCert. Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
…checked logic The root_epoch and app_epoch tamper cases failed while generating the RA-TLS certificate for the historical key request, so they passed even without the epoch checks. Rename the test to what it checks, and report both epochs when a sealed env is ahead of the current ones; the error printed the root epoch even when the app epoch was the one exceeding. Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Add the deployment order, the GET /app/:appId backend requirement, the per-epoch setter semantics from #1561, the removed DstackKms setters and legacy KMS info setters, two-step ownership for v0.5.9 proxies, dstack-auth pending-epoch handover, and the reorg caveat of non-finalized block tags. Signed-off-by: Kevin Wang <wy721@qq.com>
…support Follow #1561: DstackApp loses its epoch-less setters and IAppAuthBasicManagement, reports IAppEpochManagement, and app policies predating app epochs keep working at app epoch 0. Signed-off-by: Kevin Wang <wy721@qq.com>
fix: #1533 review round 2 (dstack-auth handover, KMS tests, docs)
The temporary CA is not key material worth auditing, and it was the only caller without a boot info, so audit_key_release takes &BootInfo again. Signed-off-by: Kevin Wang <wy721@qq.com>
kms: drop the GetTempCaCert key release audit
With the public handover off by default, other KMS nodes can only onboard through the admin listener over TLS. A KMS without either would strand them, which shows up only when a node needs to join. Before serving, the KMS now requires core.onboard.public_key_handover = true, or the admin listener enabled with [core.admin.tls] key and certs, and names the missing setting otherwise. There is no opt-out, dev auth included. The onboarding phase is not checked. The built-in kms.toml now enables the admin listener with TLS on the RPC key and certificate. The admin listener fails closed, so a KMS on the default config needs an admin token to start. A KMS deployed as a CVM with the local key provider has no secret channel for its config: the app compose is measured and public. The new core.admin.auth_token_hash takes the token's hex SHA-256, so the compose carries only the hash: - dstack-app composes and entrypoint use the ADMIN_TOKEN_HASH the deploy scripts already compute, and the scripts publish the admin listener when KMS_ADMIN_ADDR is set. - dstackup generates a KMS admin token next to the VMM one and renders its hash. - tools/dev-stack.sh generates a local token. - The full-stack e2e KMS, shared with KMS 0.5.8, keeps the admin listener off and serves the public handover. Signed-off-by: Kevin Wang <wy721@qq.com>
Checked only before serving, a misconfigured node completed onboarding or bootstrap, getting or generating root keys, and only then failed. Check at startup instead, in every mode. Also document that an auth_token_hash, public in an app compose, needs a high-entropy random token: a weak one can be brute-forced offline from its hash. Signed-off-by: Kevin Wang <wy721@qq.com>
With the default config and no admin token, the handover check passed (admin enabled, TLS paths set) but the missing credential surfaced only when the admin listener was built, after onboarding or bootstrap. Build the admin listener setup right after the handover check, so such a node also fails before it gets or generates root keys. Signed-off-by: Kevin Wang <wy721@qq.com>
feat(kms)!: refuse to serve without a key handover path
Leechael
approved these changes
Nov 11, 2026
kvinwang
added a commit
that referenced
this pull request
Nov 11, 2026
Since #1533, KMS key release verifies a TDX caller's OS image. The SDK simulator replays a captured legacy quote whose image (e61be43d...) was never published, so every simulated KMS client failed at the image download. Derive the KMS fixtures' simulator files with the candidate dstack-tee-simulator instead: it computes the quote measurements from the lite tdx_measurement document of the verifier fixture, the same way the verifier does, and the captured RTMR3 events are replayed on top. The verifier then takes the lite path and downloads nothing. The simulator runs in a private user and mount namespace because the TDX ABI paths under /sys are fixed. Signed-off-by: Kevin Wang <wy721@qq.com>
kvinwang
added a commit
that referenced
this pull request
Nov 11, 2026
Since #1533 the KMS asks a webhook /authorizeApp for an AuthorizationResult that carries the registered root epoch; a bootAuth-shaped reply is not accepted, and an unregistered root epoch refuses to serve. The mock authority now registers the KMS's own keys as root epoch 0 and answers in that shape. Signed-off-by: Kevin Wang <wy721@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Register and rotate root/app key epochs, retain authorized historical keys, migrate disk headers and SealedEnv, and enforce epoch-scoped authorization while preserving temporary-CA compatibility.
This aggregates the linear split stack:
#1516 → #1519 → #1517 → #1520 → #1521 → #1528 → #1530 → #1532.
Restores all original feature commits after the previous stack rewrite omitted application epoch support. Includes the review fixes from #1534: optional selectors, grouped epoch fields, scoped operation names, consistent authorization methods, removal of legacy KMS setters, consolidated app epoch policy interface, and RPC injection/isolation regression coverage. No KMS contract-address equality check was added.
The final tree is identical to #1534 at
2c017875bf. It was independently compared against the original complete #1533/#1534 tree plus the intended review fixes; no unrelated feature changes remain.Validation on Linux x86_64 with Rust 1.92.0, Bun 1.4.4, Forge 1.8.4 and Solc 0.8.24: cert-client 7, KMS 75, local auth 9, Solidity 87, VMM SealedEnv 4, simple auth 33, mock auth 15, Bun Ethereum auth 50 and Jest Ethereum auth 22 tests passed. Guest-agent and dstack-util compile checks passed, including the earlier guest-key split independently. Solidity used pinned submodules and
forge test --offline --ffi -vv; Rust reused Cargo caches. Injection tests use loopback HTTP, a supplied vsock CID, synthetic payloads and temporary VM storage; no live TEE/chain or kernel AF_VSOCK identity test was performed. Tree equality, stack ancestry, Rust formatting and diff whitespace checks passed.GitHub currently locks the base of #1519 because it is a stack member. Its commits descend from #1516, but its PR base remains
next; both GraphQL and REST base updates were rejected.