kvinwang · Nov 11, 2026 · Nov 11, 2026 · Nov 11, 2026 · Nov 11, 2026 · Nov 11, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Upgrade notes
+
+- **Deployment order for key epochs:** upgrade the `DstackKms` and `DstackApp`
+  proxies and the auth backend, register the existing roots as root epoch 0
+  (`registerRootEpoch`), then roll out the KMS nodes, then guests. Keep root and
+  app epochs at 0 until every KMS and guest is upgraded. See
+  `dstack/kms/README.md` and `docs/deployment.md`.
+- **Auth backends must serve `GET /app/:appId`**, which reports an app's current
+  app epoch. The KMS reads it for `GetAppEnvEncryptPubKey` and
+  `GetSealedEnvPubKey`. `auth-eth` and `dstack-auth` serve it; custom webhook
+  backends must add it.
+
 ### Added
 
 - KMS root and application key epochs, epoch-scoped authorization, SealedEnv
@@ -36,12 +48,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   denies a KMS that reports no root epoch, as `/bootAuth/app` already did, so
   KMS binaries predating root epochs stop serving at activation. Upgrade every
   KMS before activating a new root epoch.
-- auth-eth, auth-eth-bun: policy is read at the `finalized` block by default
-  instead of `latest`, so on-chain changes, including revocations, take effect
-  only once finalized. Set `ETH_BLOCK_TAG=latest` (or `safe`) to override; an
-  OP-stack chain read through the Helios light client requires `latest`. The
-  server refuses to start if the RPC cannot serve the configured tag. In
-  auth-eth-bun, `ETH_FINALITY_CONFIRMATIONS` now counts back from that block.
+- auth-eth: policy is read at the `finalized` block by default instead of
+  `latest`, so on-chain changes, including revocations, take effect only once
+  finalized. Set `ETH_BLOCK_TAG=latest` (or `safe`) to override; an OP-stack
+  chain read through the Helios light client requires `latest`. Only
+  `finalized` protects policy from reorgs: with `latest` or `safe`, a reorg can
+  roll back an activation or a revocation the KMS has already acted on. The
+  server refuses to start if the RPC cannot serve the configured tag.
+  `ETH_FINALITY_CONFIRMATIONS` now counts back from that block.
+- DstackKms, DstackApp: material approvals are per epoch. A request is decided
+  on the row of the epoch it selects, so a value is revoked only once every row
+  allowing it is cleared. Material setters name their rows:
+  `*AtEpoch(epoch, value)` writes one epoch, `*AtEpochs(from, to, value)` a
+  range; `remove*AtEpochs(0, latest root epoch, value)` in `DstackKms` and
+  `remove*AtEpochs(0, currentAppEpoch, value)` in `DstackApp` revoke a value
+  everywhere. `DstackApp` reports its setters as the new ERC-165 interface
+  `IAppEpochManagement` (`0x853d2b79`). The `Manage.s.sol` scripts require
+  `ROOT_EPOCH` or `APP_EPOCH`, or a `FROM_*`/`TO_*` range.
+- App contracts deployed before app epochs (v0.5.x, v0.6.0) keep working with
+  the upgraded `DstackKms` and KMS, which authorize them for app epoch 0 only.
+  `Manage.s.sol` detects them by interface ID and calls their epoch-less
+  setters. Upgrading one to the current `DstackApp` keeps its compose hashes,
+  devices and `allowAnyDevice` as app epoch 0.
+- DstackKms, DstackApp: ownership transfers take two steps for proxies upgraded
+  from v0.5.9: `transferOwnership` nominates, and the new owner must call
+  `acceptOwnership`.
+- dstack-auth: `kms_export_keys` and `kms_import_keys` accept a registered
+  pending root epoch, so KMS nodes can onboard a new epoch before
+  `currentRootEpoch` is raised, as with the contract backend. Key-issuing
+  operations stay limited to the current epoch.
 - KMS: retain `GetTempCaCert`, temporary CA files and legacy handover fields so
   KMS nodes can be upgraded before existing OS images.
 
@@ -65,6 +100,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Removed
 
+- DstackKms: the epoch-less `addOsImageHash`, `removeOsImageHash`,
+  `addKmsAggregatedMr`, `removeKmsAggregatedMr`, `addKmsDevice` and
+  `removeKmsDevice`. Use the `*AtEpoch` or `*AtEpochs` forms; before any
+  rotation, `addOsImageHashAtEpoch(0, hash)` is the old call.
+- **Breaking:** DstackApp: the epoch-less `addComposeHash`, `removeComposeHash`,
+  `addDevice`, `removeDevice` and `setAllowAnyDevice`, and with them
+  `IAppAuthBasicManagement`: an upgraded app no longer reports interface ID
+  `0x8fd37527`, and calls to those functions revert. Use the `*AtEpoch` or
+  `*AtEpochs` forms (`IAppEpochManagement`); at app epoch 0,
+  `addComposeHashAtEpoch(0, hash)` is the old call.
+- DstackKms: `setKmsInfo`, `setKmsQuote` and `setKmsEventlog`. Root keys are
+  set with `registerRootEpoch`, and each epoch's attestation lives at its
+  `evidenceUrl`.
 - KMS: the `auth-mock` webhook backend. For development, use the built-in
   `[core.auth_api] type = "dev"`, which allows everything.
 - `auth-simple` (`dstack/kms/auth-simple`). Use `dstack-auth`, which reads the

diff --git a/docs/app-key-epochs.md b/docs/app-key-epochs.md
@@ -131,8 +131,9 @@ Direct legacy app checks use the current app epoch; legacy KMS app authorization
 still rejects nonzero app epochs.
 
 These material matrices are implemented by the Solidity policies. Config-file auth
-backends such as dstack-auth retain their own policy model and may reject handover
-of pending epochs.
+backends such as dstack-auth retain their own policy model; dstack-auth applies the
+same epoch limits, accepting a registered pending root epoch for KMS handover and
+only active epochs for key-issuing operations.
 
 ## Derivation and lifetime
 

diff --git a/docs/deployment.md b/docs/deployment.md
@@ -190,6 +190,15 @@ For adding Gateway, apps, and other config fields, see [dstack-auth Operations G
 
 For decentralized governance via smart contracts, see [On-Chain Governance](./onchain-governance.md).
 
+auth-eth reads policy at the block `ETH_BLOCK_TAG` names, `finalized` by
+default. Only `finalized` protects policy from reorgs. With `latest` (required
+for an OP-stack chain read through Helios) or `safe`, a reorg can revert both
+activations and revocations: a removed image or compose hash is authorized
+again, and a rolled-back root epoch activation makes every KMS restart onto the
+previous epoch. `ETH_FINALITY_CONFIRMATIONS=N` hides reorgs shallower than N
+blocks but delays every policy change, emergency revocations included, by N
+blocks. See [reorgs and the block tag](../dstack/kms/auth-eth/README.md#reorgs-and-the-block-tag).
+
 #### Getting OS Image Hash
 
 The OS image hash is in the `digest.txt` file inside the guest image tarball:
@@ -550,7 +559,7 @@ curl http://<new-kms>:9203/finish
 
 > **Admin authentication.** The KMS *admin* RPCs (for example `ClearImageCache`) are served on a dedicated `[core.admin]` listener behind the shared HTTP authenticator, just like the VMM and gateway: set `[core.admin] enabled = true` with an `auth_token` (or the `DSTACK_KMS_ADMIN_TOKEN` / `ADMIN_API_TOKEN` env vars), and clients send `Authorization: Bearer <token>` or `X-Admin-Token`. Enabled with neither `auth_token` nor `htpasswd_file` (and `insecure_no_auth = false`) fails closed — the KMS refuses to start. See [dstack-kms admin authentication](../dstack/kms/README.md#admin-api-authentication).
 >
-> **Key handover.** It is served as `Admin.GetKmsKey` on the `[core.admin]` listener, which needs `[core.admin.tls]` so it can verify the onboarding KMS's RA-TLS certificate. Pass the source admin URL (`https://`) and its token (`source_token`) in the onboarding request. The legacy public `KMS.GetKmsKey` is off by default; set `core.onboard.public_key_handover = true` on the source only when an onboarding KMS cannot use the admin listener, such as a 0.5.x KMS or an onboarding request without `source_token`.
+> **Key handover.** It is served as `Admin.GetKmsKey` on the `[core.admin]` listener, which needs `[core.admin.tls]` so it can verify the onboarding KMS's RA-TLS certificate. Pass the source admin URL (`https://`) and its token (`source_token`) in the onboarding request; the token is required unless the source enables the public handover. The legacy public `KMS.GetKmsKey` is off by default; set `core.onboard.public_key_handover = true` on the source only when an onboarding KMS cannot use the admin listener, such as a 0.5.x KMS or an onboarding request without `source_token`.
 
 #### Rotating KMS Root Keys
 

diff --git a/docs/dstack-auth-operations.md b/docs/dstack-auth-operations.md
@@ -112,7 +112,7 @@ Hex values are compared case-insensitively, with or without a `0x` prefix.
 | `apps.<appId>.allowAnyDevice` | No | If true, skip the device ID check for this app |
 | `apps.<appId>.currentAppEpoch` | No | Current app key epoch (default `0`), served by `GET /app/<appId>`. A non-zero epoch denies the legacy `/bootAuth/app` route. |
 | `rootEpochs` | No | KMS root public keys of each root epoch, indexed by epoch, `[{"caPubkey": "0x...", "k256Pubkey": "0x..."}]`. Epoch 0 is the original roots. Append only. `GET /` reports the current epoch as `rootEpoch` with its chain hash as `rootEpochHash`; the last entry is reported as `latestRootEpoch`, and entries beyond `currentRootEpoch` are pending activation. |
-| `currentRootEpoch` | No | Root epoch KMS instances serve (default 0). Raise it only once every KMS node holds that epoch. |
+| `currentRootEpoch` | No | Root epoch KMS instances serve (default 0). Key-issuing operations (`kms_serve`, `app_get_key`, `app_get_cert`) may request up to this epoch; KMS key handover (`kms_export_keys`, `kms_import_keys`) may request any registered epoch, so the other KMS nodes can onboard from the rotated one while the new epoch is pending. Raise it only once every KMS node holds that epoch. |
 
 ### Validation Logic
 

diff --git a/docs/kms-emergency-key-rotation-protocol.md b/docs/kms-emergency-key-rotation-protocol.md
@@ -1,6 +1,6 @@
 # KMS emergency key rotation protocol
 
-**Status:** Design draft; not implemented. Proposed fields require protobuf definitions.
+**Status:** Implemented.
 
 ## 1. Scope and assumptions
 
@@ -60,7 +60,7 @@ new `n`, whose OS image, KMS MR and device rows start empty because policy rows
 are keyed by the epoch's chain hash. Prepare them again before activating. A
 current epoch is never popped. `kmsInfo` follows the current epoch:
 registering epoch 0 and each activation copy its keys there and clear the legacy
-quote and event log, and `setKmsInfo` is refused once epochs exist. Epoch 0 must be the `kmsInfo` keys
+quote and event log; the legacy `setKmsInfo`, `setKmsQuote` and `setKmsEventlog` setters are removed. Epoch 0 must be the `kmsInfo` keys
 when those are set; a legacy `kmsInfo` quote and event log are converted off
 chain into the standard attestation encoding (`Attestation::from_tdx_quote`)
 before publishing. The KMS sends the epoch it serves and its chain hash (`rootEpoch`,
@@ -261,11 +261,11 @@ between the current epoch and L is refused, since its tail cannot be checked. Wi
 
 Every handover requires the recipient's KMS attestation: the source checks the
 attested RA-TLS client certificate on both `Admin.GetKmsKey` and the public
-`KMS.GetKmsKey`. Admin authorization (`source_token`) is recommended for joins
-and rotations alike, but not required: a rotation only produces a pending epoch
-N+1, which no node accepts until governance registers and activates it. Where
-possible, disable public legacy handover and reject incompatible legacy
-receivers without reducing the payload. The source's certificate must carry its
+`KMS.GetKmsKey`. Admin authorization (`source_token`) is required for joins
+and rotations alike, because the public `KMS.GetKmsKey` is off by default
+(`core.onboard.public_key_handover = false`). Enable it on a source only for
+receivers that cannot use the admin listener, such as 0.5.x KMS nodes, and
+reject incompatible legacy receivers without reducing the payload. The source's certificate must carry its
 OS image hash.
 Deployments that never registered epochs can still join at epoch 0.
 

diff --git a/docs/rc-testing-runbook.md b/docs/rc-testing-runbook.md
@@ -153,7 +153,8 @@ KMS first, then gateway, then the app. Each needs the previous one's identity.
    gramine-sealing-key-provider on the host, default `127.0.0.1:3443`). The
    simplest authorization that releases keys without a chain is
    `[core.auth_api] type = "dev"` with `gateway_app_id = "any"` — no extra
-   containers, unlike the on-chain compose which needs a Helios light client.
+   containers, unlike the on-chain compose, which runs auth-eth next to the KMS
+   and needs an Ethereum RPC endpoint.
 
 3. **Gateway as a CVM** with `--kms`. Internal ports are RPC 8000, admin 8001,
    WireGuard 51820/udp, proxy 443.

diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock
diff --git a/dstack/crates/dstack-auth/src/main.rs b/dstack/crates/dstack-auth/src/main.rs
@@ -386,13 +386,26 @@ fn authorize(
         },
     };
     let info = &request.boot_info;
+    // key handover may carry a registered but not yet active epoch, so the
+    // keyring can be distributed before activation; key-issuing operations
+    // stay on active epochs.
+    let handover = matches!(
+        request.operation,
+        Operation::KmsExportKeys | Operation::KmsImportKeys
+    );
     result.rejection_reason = if request.operation.is_kms() != is_kms {
         "invalid authorization operation".into()
     } else if current.is_none() && request.operation != Operation::KmsBootstrap {
         "root epoch not registered".into()
     } else if request
         .requested_root_epoch
-        .is_some_and(|epoch| epoch as usize > result.current_root_epoch)
+        .is_some_and(|epoch| epoch as usize > result.latest_root_epoch)
+    {
+        "requested root epoch is not registered".into()
+    } else if !handover
+        && request
+            .requested_root_epoch
+            .is_some_and(|epoch| epoch as usize > result.current_root_epoch)
     {
         "requested root epoch is not active".into()
     } else if request
@@ -1128,6 +1141,50 @@ mod tests {
         );
     }
 
+    #[test]
+    fn a_pending_root_epoch_is_authorized_for_kms_key_handover() {
+        // epoch 1 is registered (latest) but not current: pending
+        let al: Allowlist = serde_json::from_value(json!({
+            "kmsContractAddr": "0x1111111111111111111111111111111111111111",
+            "kms": {"mrAggregated": ["bb"], "allowAnyDevice": true},
+            "rootEpochs": [
+                {"caPubkey": "aa", "k256Pubkey": "bb"},
+                {"caPubkey": "cc", "k256Pubkey": "dd"}
+            ],
+            "currentRootEpoch": 0
+        }))
+        .unwrap();
+        let ask = |operation: &str, requested: u32| {
+            let request: AuthorizationRequest = serde_json::from_value(json!({
+                "operation": operation,
+                "appId": "11".repeat(20),
+                "mrAggregated": "bb",
+                "requestedRootEpoch": requested,
+            }))
+            .unwrap();
+            authorize(&request, true, &al).unwrap()
+        };
+
+        let current = ask("kms_export_keys", 0);
+        assert!(current.authorized, "{}", current.rejection_reason);
+        assert_eq!(current.current_root_epoch, 0);
+        assert_eq!(current.latest_root_epoch, 1);
+
+        for operation in ["kms_import_keys", "kms_export_keys"] {
+            let pending = ask(operation, 1);
+            assert!(
+                pending.authorized,
+                "{operation} at the pending epoch 1: {}",
+                pending.rejection_reason
+            );
+            assert!(!ask(operation, 2).authorized);
+        }
+
+        let serve = ask("kms_serve", 1);
+        assert!(!serve.authorized);
+        assert_eq!(serve.rejection_reason, "requested root epoch is not active");
+    }
+
     #[test]
     fn allows_only_a_kms_serving_the_current_root_epoch() {
         // Shared with the DstackKms contract tests.

diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs
@@ -3327,16 +3327,16 @@ impl Stage1<'_> {
                 bail!("sealed env encapsulated key must be 32 bytes");
             }
             let payload = [envelope.enc, envelope.ciphertext].concat();
-            let current = self.keys.root_epoch();
-            if envelope.root_epoch > current || envelope.app_epoch > self.keys.app_epoch() {
+            let (current_root, current_app) = (self.keys.root_epoch(), self.keys.app_epoch());
+            if envelope.root_epoch > current_root || envelope.app_epoch > current_app {
                 bail!(
-                    "sealed env epoch {} exceeds current epoch {current}",
-                    envelope.root_epoch
+                    "sealed env epoch (root {}, app {}) exceeds current epoch (root {current_root}, app {current_app})",
+                    envelope.root_epoch,
+                    envelope.app_epoch
                 );
             }
             let historical;
-            let secret = if envelope.root_epoch == current
-                && envelope.app_epoch == self.keys.app_epoch()
+            let secret = if envelope.root_epoch == current_root && envelope.app_epoch == current_app
             {
                 current_secret
             } else {
@@ -3351,17 +3351,10 @@ impl Stage1<'_> {
                 &sealed_env_info_at_epochs(envelope.root_epoch, envelope.app_epoch),
                 &payload,
             )?;
-            if envelope.root_epoch == self.keys.root_epoch()
-                && envelope.app_epoch == self.keys.app_epoch()
-            {
+            if envelope.root_epoch == current_root && envelope.app_epoch == current_app {
                 return Ok((plaintext, Vec::new()));
             }
-            let sealed = seal_env_at_epochs(
-                current_secret,
-                self.keys.root_epoch(),
-                self.keys.app_epoch(),
-                &plaintext,
-            )?;
+            let sealed = seal_env_at_epochs(current_secret, current_root, current_app, &plaintext)?;
             return Ok((plaintext, sealed));
         }
         let ciphertext = &self.shared.encrypted_env;
@@ -4539,7 +4532,7 @@ mod sealed_env_file_tests {
     }
 
     #[tokio::test]
-    async fn sealed_env_binds_both_epochs_and_rejects_mislabeled_versions() {
+    async fn sealed_env_at_the_current_epochs_opens_and_binds_both_epochs() {
         let dir = tempfile::tempdir().unwrap();
         let args = SetupArgs {
             work_dir: dir.path().into(),
@@ -4555,17 +4548,14 @@ mod sealed_env_file_tests {
         let (opened, update) = current.decrypt_env().await.unwrap();
         assert_eq!(opened, plaintext);
         assert!(update.is_empty());
-        for (field, value) in [
-            ("version", 2),
-            ("root_epoch", 2),
-            ("app_epoch", 1),
-            ("app_epoch", 3),
-        ] {
-            let mut tampered: Value = serde_json::from_slice(&sealed).unwrap();
-            tampered[field] = value.into();
-            current.shared.sealed_env = Some(serde_json::to_vec(&tampered).unwrap());
-            assert!(current.decrypt_env().await.is_err());
-        }
+        let mut tampered: Value = serde_json::from_slice(&sealed).unwrap();
+        tampered["version"] = 2.into();
+        current.shared.sealed_env = Some(serde_json::to_vec(&tampered).unwrap());
+        let err = current.decrypt_env().await.unwrap_err();
+        assert!(
+            err.to_string().contains("unsupported sealed env version"),
+            "{err}"
+        );
         let envelope: SealedEnv = serde_json::from_slice(&sealed).unwrap();
         assert_eq!(envelope.version, 1);
         let payload = [envelope.enc.clone(), envelope.ciphertext.clone()].concat();

diff --git a/dstack/kms/Cargo.toml b/dstack/kms/Cargo.toml
@@ -53,6 +53,7 @@ moka = { workspace = true, features = ["future"] }
 
 [dev-dependencies]
 cc-eventlog.workspace = true
+mock-attestation = { path = "../crates/mock-attestation" }
 
 [features]
 default = []
diff --git a/dstack/kms/README.md b/dstack/kms/README.md
@@ -390,7 +390,8 @@ still bind the root identity; a valid quote alone is not a KMS policy decision.
 `--root-ca` additionally pins the CLI connection before sending the request.
 
 Onboarding with an admin token uses a self-issued client certificate. Tokenless
-onboarding obtains the source temp CA first so pre-0.6 sources remain usable.
+onboarding, which needs a source with `core.onboard.public_key_handover = true`,
+obtains the source temp CA first so pre-0.6 sources remain usable.
 Both paths check the source attestation captured from the key-carrying connection
 against local KMS policy before importing anything. Retries and reconnects repeat
 verification on the response connection; no assumption relies on one TCP session