kvinwang · Nov 7, 2026 · Nov 8, 2026 · Nov 8, 2026 · Nov 8, 2026 · Nov 8, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- KMS root and application key epochs, epoch-scoped authorization, SealedEnv
+  HPKE encryption, automatic environment migration, and epoch-aware SDK key
+  selection.
+
 ### Changed
 
 - KMS: retain `GetTempCaCert`, temporary CA files and legacy handover fields so

diff --git a/docs/app-key-epochs.md b/docs/app-key-epochs.md
@@ -0,0 +1,73 @@
+# Application key epochs
+
+Application owners can rotate their app's derived keys without rotating KMS root
+keys or other apps. This extends SealedEnv (#1528); the PR dependency order is
+#1519 → #1517 → #1520 → #1521 → #1528 → application epochs.
+
+## Authorization and rotation
+
+`checkAppAuthorization` returns `currentAppEpoch` together with the current root
+epoch and its chain hash. `requestedRootEpoch` and `requestedAppEpoch` are
+independent `OptionalEpoch` values: omitted selects current, explicit zero selects
+historical epoch zero. Future epochs are denied through `rejectionReason`.
+`checkKmsAuthorization` remains separate and uses app epoch zero.
+
+`DstackApp.currentAppEpoch` starts at zero. Its owner calls
+`advanceAppEpoch(expectedCurrentEpoch)` to increment it. The expected value
+prevents a stale or retried transaction from advancing twice. Epochs cannot be
+rolled back and are limited to uint32. `AdvanceAppEpoch` in `Manage.s.sol` exposes
+this operation; `GetAppEpoch` in `Query.s.sol` reads it. Upgrade the app policy
+implementation before advancing, and upgrade the KMS/auth backend and guests
+before using nonzero epochs. Legacy policies without the ERC-165 `IAppEpoch`
+capability remain at zero; failures of an advertised getter propagate.
+
+The Ethereum auth backends query the policy through the KMS contract. The simple
+backend reads `apps[appId].currentAppEpoch` (default zero); operators must preserve
+its monotonicity. Public encryption-key lookup uses `GET /app/:appId` to read this
+value. Legacy boot authorization and legacy environment public-key issuance are
+rejected once the app epoch is nonzero, preventing old clients from silently
+continuing with epoch-zero keys.
+
+## Derivation and lifetime
+
+Every key is identified by `(root_epoch, app_epoch)`. App epoch zero keeps existing derived keys and app-key signatures. Nonzero app epochs append
+`"app-epoch-v1:" || app_epoch[4, big-endian]` to the existing derivation context
+for disk, environment, SealedEnv, app CA and secp256k1 keys.
+
+The root signature for a nonzero app secp256k1 key signs Keccak256 of
+`"dstack-kms-issued-v2:" || app_id[20] || app_epoch[4, big-endian] || public_key[33]`.
+Relying parties must verify this format against the trusted root key for the
+returned root epoch. Epoch zero keeps the original `dstack-kms-issued` format.
+
+Restart CVMs after advancing the policy. A running CVM continues to use its boot
+key pair; omitted guest-agent GetKey selectors mean that CVM's boot epochs, not a
+fresh chain lookup. Explicit historical requests select each dimension
+independently and cache by the complete pair. Historical access still requires
+current app authorization. Supplied `trusted_ca` always pins the KMS connection,
+regardless of epoch selection; historical responses cannot replace that pin.
+
+The guest migrates disk headers and environment ciphertext if either dimension
+changes and rejects future epochs in either dimension. LUKS tokens record both
+epochs (missing app epoch means zero). Rekeying replaces the keyslot wrapping key;
+it does not rotate the volume encryption key or erase host snapshots. Once a disk
+header carries an epoch token, older guest images reject that header as malformed;
+reverting the VM image therefore requires migrating the disk with a compatible
+image or restoring a compatible header backup. App rotation isolates future app
+keys from an exposed old derived key, but cannot recover from compromise of the
+KMS root or revoke already disclosed secrets.
+
+## SDK selection
+
+Both selectors are optional; specify zero explicitly to recover an old key.
+For example, Python `client.get_key("storage", "secp256k1", root_epoch=3,
+app_epoch=0)` selects exactly that pair. JavaScript uses
+`client.getKey("storage", "secp256k1", 3, 0)`. Rust provides
+`get_key_at_epochs(..., Some(3), Some(0))`; Go provides `GetKeyAtEpochs` with
+pointer selectors. Responses report both epochs. Existing root-only helpers
+continue to use the CVM's current app epoch.
+
+Sealing helpers also accept app epochs: `seal_env_at_epochs` (Rust),
+`SealEnvAtEpochs` (Go), and the optional final argument of `sealEnv` / `seal_env`
+(JavaScript / Python). See [SealedEnv](sealed-env.md) for the wire and signature
+formats. The VMM UI and CLI use the epochs returned with the encryption key. SealedEnv
+uses a single version-1 format that always includes both epochs.
diff --git a/docs/onboarding.md b/docs/onboarding.md
@@ -329,3 +329,9 @@ registered-root requirement as the other backends. Missing registration denies k
 release, and invalid epoch configuration returns HTTP 503. Legacy `/bootAuth/*`
 routes remain available for bootstrap. Local compose/device allowlists apply to
 all permitted historical roots; they are not the on-chain per-epoch matrices.
+
+Application entries also accept `currentAppEpoch` (default zero). Keep this value
+monotonic, restart the application after advancing it, and retain permissions for
+historical keys needed during migration. `/app/:appId` exposes the current app
+epoch; legacy app boot authorization is denied once it is nonzero. The Rust
+backend uses the same normalized app ID lookup for policy and epoch selection.
diff --git a/docs/sealed-env.md b/docs/sealed-env.md
@@ -44,7 +44,8 @@ that epoch's P-256 CA private scalar using HKDF-SHA256:
 - info: the raw 20-byte app ID.
 
 `GetAppKey` returns it as `sealed_env_key` alongside the legacy key. The legacy
-key derivation remains unchanged. This PR reserves app epochs and accepts only zero.
+key derivation remains unchanged at app epoch zero. `GetAppEnvEncryptPubKey`
+rejects nonzero app epochs; use `GetSealedEnvPubKey` after app rotation.
 
 The public-key response signs Keccak256 of this unambiguous byte sequence:
 
@@ -77,10 +78,12 @@ with nonempty legacy `encrypted_env` while switching to an older image. Empty en
 fields preserve the existing format. Guest `UpdateSealedEnv` uploads also require
 a capable running image.
 
-SealedEnv has one format, version 1, with both epoch fields always present.
-This PR accepts app epoch zero only; the field is reserved for application key
-rotation without changing this format. Signature and HPKE contexts always include
-both epochs, including zero. Changing either epoch invalidates authentication.
+SealedEnv has one format, version 1, with both epoch fields always present. The
+receiver-key KDF appends `"app-epoch-v1:" || app_epoch[4, big-endian]` to its info
+for nonzero app epochs; at zero its info is just the app ID. Signature and HPKE
+contexts always include both epochs, including zero. Changing either epoch
+invalidates the signature or ciphertext. See [application epochs](app-key-epochs.md)
+for owner operations.
 
 The browser UI receives the key through the same-origin VMM API and does not
 independently verify the response signer or freshness. Its trust boundary includes

diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock
diff --git a/dstack/cert-client/Cargo.toml b/dstack/cert-client/Cargo.toml
@@ -23,6 +23,7 @@ tokio.workspace = true
 x509-parser = { workspace = true, features = ["verify"] }
 
 [dev-dependencies]
+prpc.workspace = true
 ra-rpc = { workspace = true, features = ["rocket", "client"] }
 mock-attestation = { path = "../crates/mock-attestation" }
 hex.workspace = true

diff --git a/dstack/cert-client/src/lib.rs b/dstack/cert-client/src/lib.rs
@@ -86,10 +86,14 @@ pub async fn request_app_keys(
         .build()
         .into_client()?;
     let requested_root_epoch = request.root_epoch;
+    let requested_app_epoch = request.app_epoch;
     let response = KmsClient::new(client).get_app_key(request).await?;
     if requested_root_epoch.is_some_and(|epoch| epoch != response.root_epoch) {
         bail!("KMS returned a different root epoch than requested");
     }
+    if requested_app_epoch.is_some_and(|epoch| epoch != response.app_epoch) {
+        bail!("KMS returned a different app epoch than requested");
+    }
     let cert = server_cert
         .lock()
         .map_err(|_| anyhow::anyhow!("KMS certificate lock poisoned"))?;
@@ -168,24 +172,28 @@ impl CertRequestClient {
         }
     }
 
-    /// The app keys of an older KMS root epoch, from the KMS that served the
-    /// current ones.
-    pub async fn get_app_key(&self, root_epoch: u32) -> Result<AppKeyResponse> {
+    /// Fetch app keys from the startup KMS, using its current epochs for omitted selectors.
+    pub async fn get_app_key(
+        &self,
+        root_epoch: Option<u32>,
+        app_epoch: Option<u32>,
+    ) -> Result<AppKeyResponse> {
         let CertRequestClient::Kms { client, vm_config } = self else {
             bail!("only KMS-provided keys have older root epochs");
         };
         let response = client
             .get_app_key(GetAppKeyRequest {
                 api_version: 1,
                 vm_config: vm_config.clone(),
-                root_epoch: Some(root_epoch),
+                root_epoch,
+                app_epoch,
             })
             .await?;
-        if response.root_epoch != root_epoch {
-            bail!(
-                "KMS returned keys of root epoch {} instead of {root_epoch}",
-                response.root_epoch
-            );
+        if root_epoch.is_some_and(|epoch| epoch != response.root_epoch) {
+            bail!("KMS returned a different root epoch than requested");
+        }
+        if app_epoch.is_some_and(|epoch| epoch != response.app_epoch) {
+            bail!("KMS returned a different app epoch than requested");
         }
         Ok(response)
     }
@@ -463,13 +471,54 @@ mod tests {
         )
     }
 
-    #[rocket::post("/GetAppKey", data = "<_request>")]
+    #[test]
+    fn generated_epoch_selectors_preserve_omission_and_explicit_zero() {
+        use prpc::Message;
+
+        let default = GetAppKeyRequest::default();
+        assert_eq!((default.root_epoch, default.app_epoch), (None, None));
+        for json in ["{}", r#"{"api_version":1,"vm_config":""}"#] {
+            let request: GetAppKeyRequest = serde_json::from_str(json).unwrap();
+            assert_eq!((request.root_epoch, request.app_epoch), (None, None));
+        }
+        for root_epoch in [None, Some(0), Some(7)] {
+            for app_epoch in [None, Some(0), Some(9)] {
+                let request = GetAppKeyRequest {
+                    root_epoch,
+                    app_epoch,
+                    ..Default::default()
+                };
+                let json = serde_json::to_vec(&request).unwrap();
+                let from_json: GetAppKeyRequest = serde_json::from_slice(&json).unwrap();
+                let from_proto =
+                    GetAppKeyRequest::decode(request.encode_to_vec().as_slice()).unwrap();
+                for decoded in [from_json, from_proto] {
+                    assert_eq!(
+                        (decoded.root_epoch, decoded.app_epoch),
+                        (root_epoch, app_epoch)
+                    );
+                }
+            }
+        }
+        let explicit: GetAppKeyRequest =
+            serde_json::from_str(r#"{"root_epoch":0,"app_epoch":0}"#).unwrap();
+        assert_eq!(
+            (explicit.root_epoch, explicit.app_epoch),
+            (Some(0), Some(0))
+        );
+    }
+
+    #[rocket::post("/GetAppKey", data = "<request>")]
     fn direct_app_keys(
-        _request: String,
+        request: String,
         _cert: rocket::mtls::Certificate<'_>,
-        state: &rocket::State<(Arc<Mutex<String>>, std::sync::atomic::AtomicUsize)>,
+        state: &rocket::State<(Arc<Mutex<String>>, Arc<Mutex<Vec<GetAppKeyRequest>>>)>,
     ) -> rocket::response::content::RawJson<String> {
-        state.1.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
+        state
+            .1
+            .lock()
+            .unwrap()
+            .push(serde_json::from_str(&request).unwrap());
         rocket::response::content::RawJson(state.0.lock().unwrap().clone())
     }
 
@@ -543,12 +592,10 @@ mod tests {
         let (tx, rx) = tokio::sync::oneshot::channel();
         let tx = Mutex::new(Some(tx));
         let response_body = Arc::new(Mutex::new(serde_json::to_string(&response).unwrap()));
+        let requests = Arc::new(Mutex::new(Vec::<GetAppKeyRequest>::new()));
         let server = rocket::custom(figment)
             .attach(ra_rpc::ratls_client_verifier::RaTlsClientAuth::fairing())
-            .manage((
-                response_body.clone(),
-                std::sync::atomic::AtomicUsize::new(0),
-            ))
+            .manage((response_body.clone(), requests.clone()))
             .mount("/", rocket::routes![direct_app_keys])
             .attach(rocket::fairing::AdHoc::on_liftoff("port", move |rocket| {
                 if let Some(tx) = tx.lock().unwrap().take() {
@@ -597,6 +644,7 @@ mod tests {
             api_version: 1,
             vm_config: String::new(),
             root_epoch: None,
+            app_epoch: None,
         };
         let result = request_app_keys(
             url.clone(),
@@ -609,13 +657,16 @@ mod tests {
         .unwrap();
         assert_eq!(result.ca_cert, response.ca_cert);
         assert_eq!(result.root_epoch, 1);
+        let received = requests.lock().unwrap().last().unwrap().clone();
+        assert_eq!((received.root_epoch, received.app_epoch), (None, None));
 
         let other_ca = kms_root().pem_cert;
         for requested_root_epoch in [None, Some(0), Some(1)] {
             let request = || GetAppKeyRequest {
                 api_version: 1,
                 vm_config: String::new(),
                 root_epoch: requested_root_epoch,
+                app_epoch: None,
             };
             let selected = AppKeyResponse {
                 root_epoch: requested_root_epoch.unwrap_or(1),
@@ -670,6 +721,62 @@ mod tests {
             .await
             .is_err());
         }
+        assert_eq!(requests.lock().unwrap().len(), 10);
+
+        let cert = client_cert();
+        let client = CertRequestClient::Kms {
+            client: kms_client(
+                url.clone(),
+                root.pem_cert.clone(),
+                cert.cert_pem,
+                cert.key_pem,
+                verifier.clone(),
+            )
+            .unwrap(),
+            vm_config: "selector-test".into(),
+        };
+        for root_epoch in [None, Some(0), Some(1)] {
+            for app_epoch in [None, Some(0), Some(2)] {
+                let selected = AppKeyResponse {
+                    root_epoch: root_epoch.unwrap_or(3),
+                    app_epoch: app_epoch.unwrap_or(4),
+                    ..response.clone()
+                };
+                *response_body.lock().unwrap() = serde_json::to_string(&selected).unwrap();
+                let result = client.get_app_key(root_epoch, app_epoch).await.unwrap();
+                assert_eq!(
+                    (result.root_epoch, result.app_epoch),
+                    (selected.root_epoch, selected.app_epoch)
+                );
+                let received = requests.lock().unwrap().last().unwrap().clone();
+                assert_eq!(
+                    (received.root_epoch, received.app_epoch),
+                    (root_epoch, app_epoch)
+                );
+                assert_eq!(received.api_version, 1);
+                assert_eq!(received.vm_config, "selector-test");
+
+                for (wrong_root, wrong_app, message) in [
+                    (selected.root_epoch + 1, selected.app_epoch, "root epoch"),
+                    (selected.root_epoch, selected.app_epoch + 1, "app epoch"),
+                ] {
+                    let mismatched = AppKeyResponse {
+                        root_epoch: wrong_root,
+                        app_epoch: wrong_app,
+                        ..selected.clone()
+                    };
+                    *response_body.lock().unwrap() = serde_json::to_string(&mismatched).unwrap();
+                    let result = client.get_app_key(root_epoch, app_epoch).await;
+                    if (message == "root epoch" && root_epoch.is_some())
+                        || (message == "app epoch" && app_epoch.is_some())
+                    {
+                        assert!(result.unwrap_err().to_string().contains(message));
+                    } else {
+                        assert!(result.is_ok());
+                    }
+                }
+            }
+        }
         *response_body.lock().unwrap() = serde_json::to_string(&response).unwrap();
         let production = Arc::new(AttestationVerifier::new_prod(None).unwrap());
         assert!(
@@ -678,15 +785,8 @@ mod tests {
                 .is_err()
         );
         shutdown.notify();
-        let server = task.await.unwrap().unwrap();
-        assert_eq!(
-            server
-                .state::<(Arc<Mutex<String>>, std::sync::atomic::AtomicUsize)>()
-                .unwrap()
-                .1
-                .load(std::sync::atomic::Ordering::SeqCst),
-            11
-        );
+        task.await.unwrap().unwrap();
+        assert_eq!(requests.lock().unwrap().len(), 38);
     }
 
     #[test]