diff --git a/CHANGELOG.md b/CHANGELOG.md index 8319bdc58..89a44ccd5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,26 +9,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- KMS: retain `GetTempCaCert`, temporary CA files and legacy handover fields so + KMS nodes can be upgraded before existing OS images. + - dstack-util, cert-client: guests mint self-issued RA-TLS client certificates for KMS calls instead of fetching the temp CA and minting from it; the KMS authenticates the quote inside the certificate, not its issuer. - The KMS root CA now comes from `GetMeta`, over the same unauthenticated - connection `GetTempCaCert` used, so the trust posture is unchanged: relying - parties bind the KMS through the `key-provider` launch event (RTMR3/PCR14) - against the on-chain `kmsInfo` or a pinned key, and `verify_key_provider_id` - also enforces it in-guest when the app pins `key_provider_id`. A guest built - from this change needs KMS 0.6.0 or later; older KMS releases pin the temp CA - and refuse the certificate at the TLS handshake + Guests fetch the root CA with the key response and bind it to the attested + RPC certificate, without a metadata preflight. Provider pins and the + `key-provider` launch event (RTMR3/PCR14) retain the root identity binding. + The new flow requires an attested KMS that accepts self-issued certificates. - dstack-types: `KeyProvider::Kms` drops `tmp_ca_key` and `tmp_ca_cert`, so `.appkeys.json` no longer carries them. Files written by older images still parse, but out-of-tree readers that require those fields fail on new files ### Fixed -- kms: onboarding from a source older than 0.6.0 (no `source_token`) now - verifies the source's attestation on the connection that returns the root - keys, and checks it against local policy before using them. It used to - verify the source only on the earlier `GetTempCaCert` connection +- kms: onboarding with or without `source_token` verifies the source's + attestation on the connection that returns the root keys and checks local + policy before using them. Legacy sources requiring temp-CA-issued clients + must be upgraded. ## [0.6.3] - 2026-11-01 diff --git a/docs/deployment.md b/docs/deployment.md index ec67cdedd..3621228ed 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -131,9 +131,13 @@ Production KMS requires: | Custom | Your own authorization logic | Implement webhook interface | All auth servers implement the same webhook interface: -- `GET /` - Health check -- `POST /bootAuth/app` - App boot authorization -- `POST /bootAuth/kms` - KMS boot authorization + +- `GET /` - Health check and current root epoch +- `POST /authorizeApp` - Application key and certificate authorization +- `POST /authorizeKms` - KMS self-authorization and key handover + +The legacy `/bootAuth/app` and `/bootAuth/kms` routes remain available for older +clients and KMS bootstrap/onboarding checks. #### Using auth-simple (Config-Based) @@ -470,9 +474,9 @@ Additional KMS instances can onboard from an existing KMS to share the same root **How it works:** 1. New KMS starts in onboard mode (empty `auto_bootstrap_domain`) -2. New KMS generates an RA-TLS client certificate with its TDX quote: self-issued when given a `source_token`, or minted from the temp CA fetched with `GetTempCaCert` for a source older than 0.6.0 -3. New KMS calls `GetKmsKey` over mTLS: `Admin.GetKmsKey` with the token, or the legacy public `KMS.GetKmsKey` without one -4. Source KMS verifies attestation via `bootAuth/kms` webhook +2. New KMS generates an RA-TLS client certificate with its attestation: self-issued with an admin token, or signed by the source temporary CA for tokenless legacy onboarding +3. New KMS calls `GetKmsKey` over mTLS: `Admin.GetKmsKey` with the token, or the explicitly enabled public `KMS.GetKmsKey` without one +4. Source KMS verifies the attestation and authorizes handover via `/authorizeKms` 5. If approved, source KMS returns root keys 6. New KMS verifies the source's attestation on that same connection and checks it against its own policy before using the keys 7. Both KMS instances now derive identical keys @@ -513,6 +517,54 @@ curl http://:9203/finish > > **Key handover.** It is served as `Admin.GetKmsKey` on the `[core.admin]` listener, which needs `[core.admin.tls]` so it can verify the onboarding KMS's RA-TLS certificate. Pass the source admin URL and its token (`source_token`) in the onboarding request. `core.onboard.public_key_handover = true` keeps the legacy public `KMS.GetKmsKey`. +#### Rotating KMS Root Keys + +KMS root keys are versioned by epoch; epoch 0 is the original roots. Every KMS +keeps all epochs so CVMs can migrate from any of them, and serves the epoch +governance made current. See the [rotation design](./kms-emergency-key-rotation-protocol.md) +for the full procedure; the KMS steps are: + +1. If epochs were never registered, publish the attestation the KMS produced at + bootstrap (kept in `bootstrap-info.json` on the bootstrapped node) and + register the current root public keys as epoch 0: + `DstackKms.registerRootEpoch(caPubkey, k256Pubkey, evidenceUrl)`, via + `script/Manage.s.sol:RegisterRootEpoch`, or a `rootEpochs` entry in the + auth-simple config. Epoch 0 is current once registered, and must be the + `kmsInfo` keys when those are set. A deployment whose `kmsInfo` holds a + legacy `quote` and `eventlog` instead of an attestation converts them with + `dstack-kms legacy-attestation --quote --eventlog ` and publishes + the result first: from then on `kmsInfo` follows the current epoch's keys and + its quote and event log are cleared. +2. Onboard a fresh KMS with `"rotate": true` (the **Rotate** checkbox in the + onboard UI) from a KMS at the current epoch N-1. It imports epochs 0..N-1, + generates epoch N, and returns the new public keys with an attestation whose + report data is + `keccak256("dstack-kms-generated-keys-v2:{N};{ca_pubkey_hex};{k256_pubkey_hex};")`. +3. Verify the attestation, publish it, and register epoch N with its URL. The + cluster keeps serving N-1. +4. Onboard the other KMS nodes from the rotated one; they import 0..N and keep + serving N-1. +5. Activate epoch N (`ActivateRootEpoch`, or `currentRootEpoch` for + auth-simple). Each KMS restarts onto N on its next uncached authorization + check; a KMS that does not hold N waits + until it is onboarded again. + If the nodes holding N are lost before N is activated, drop it with + `PopRootEpoch` (or remove the `rootEpochs` entry) and rotate again; nodes + that imported the dropped N must be onboarded again. + +Each `authorizeKms` or `authorizeApp` result carries the current epoch and its chain hash, +including when authorization is denied. KMS verifies these against its keyring; +there is no cached serving epoch. An optional `requestedRootEpoch` selects historical +keys, and omission selects the returned current epoch. Requests for future +epochs are denied. No registered epoch returns a rejection with +`rejectionReason: "Root epoch not registered"` and a zero hash. + +RPC certificates still require a listener restart when their root CA changes; +callers retry after the certificate is refreshed. Register epoch 0 before normal +key release. Bootstrap keeps using scoped KMS authorization so it can generate +the first roots before registration. See the +[authorization protocol](../dstack/kms/README.md#authorization-protocol). + --- ## Deploying Apps diff --git a/docs/onboarding.md b/docs/onboarding.md index 28b216ff2..ff948d2e3 100644 --- a/docs/onboarding.md +++ b/docs/onboarding.md @@ -312,3 +312,20 @@ dstack logs ``` The KMS VM ID is printed by `dstackup install` when it creates or reuses the KMS CVM. + +### Upgrading the local KMS to root epochs + +Upgrade the bundled `dstack-auth` together with the KMS. Before normal key release, +set `kmsContractAddr` in the host's `auth-allowlist.json` to the KMS's attested app ID +and register the verified bootstrap CA/k256 public keys in `rootEpochs` (epoch zero). +Use the bootstrap evidence to verify these keys; do not trust an unauthenticated +metadata response. Set `currentRootEpoch` to the active registered epoch (initially +zero). See [root registration and rotation](deployment.md#rotating-kms-root-keys). +The local installer does not register these keys automatically; an older installed +KMS can continue using the legacy boot routes while this migration is prepared. + +The Rust backend implements `/authorizeKms` and `/authorizeApp` with the same +registered-root requirement as the other backends. Missing registration denies key +release, and invalid epoch configuration returns HTTP 503. Legacy `/bootAuth/*` +routes remain available for bootstrap. Local compose/device allowlists apply to +all permitted historical roots; they are not the on-chain per-epoch matrices. diff --git a/docs/security/public-security-reports.md b/docs/security/public-security-reports.md index d04b40628..373299a7c 100644 --- a/docs/security/public-security-reports.md +++ b/docs/security/public-security-reports.md @@ -59,7 +59,7 @@ These issues were filed as concrete vulnerability reports, security audit findin | [#616](https://github.com/Dstack-TEE/dstack/issues/616) Host-controlled Docker registry mirror enables image substitution attacks | Closed | Not a production vulnerability | Registry mirrors are untrusted transport. Digest-pinned image references and measured compose configuration protect against substitution. No code fix was applied | | [#617](https://github.com/Dstack-TEE/dstack/issues/617) Guest agent exposes raw private keys to all local processes | Closed | Not a production vulnerability | dstack treats a CVM as one application trust domain. It does not provide per-container key isolation inside the same measured application. No code fix was applied | | [#618](https://github.com/Dstack-TEE/dstack/issues/618) Disk encryption disableable via kernel cmdline, not measured in RTMR | Closed | Not a production vulnerability | The kernel command line is measured into RTMR2, so changing `dstack.storage_encrypted=false` changes attestation evidence. No code fix was applied | -| [#619](https://github.com/Dstack-TEE/dstack/issues/619) KMS `get_temp_ca_cert` returns temp CA private key without authentication | Closed | Duplicate | The report duplicates the private advisory response for the temp CA bootstrap flow. [#1106](https://github.com/Dstack-TEE/dstack/pull/1106) removes the reason the temp CA existed - the KMS no longer pins it - so callers can be migrated off it; the RPC itself is retained until guests and KMS-to-KMS onboarding are | +| [#619](https://github.com/Dstack-TEE/dstack/issues/619) KMS `get_temp_ca_cert` returns temp CA private key without authentication | Closed | Duplicate | The report duplicates the private advisory response for the temp CA bootstrap flow. [#1106](https://github.com/Dstack-TEE/dstack/pull/1106) removes the reason the temp CA existed - the KMS no longer pins it - so callers can be migrated off it; the RPC and temporary CA material remain available for existing OS images and legacy onboarding while new guests use self-issued certificates | ## Related security roadmap and hardening diff --git a/docs/security/security-best-practices.md b/docs/security/security-best-practices.md index 5948b2c28..e3f61a666 100644 --- a/docs/security/security-best-practices.md +++ b/docs/security/security-best-practices.md @@ -111,7 +111,7 @@ Development settings are intentionally easy to audit, but they are not productio - The KMS contract pins a concrete gateway app id. Do not use `gateway_app_id = "any"` for production traffic. - TEE quotes are evaluated by deployment policy, including TCB status and expected OS/application measurements. -The KMS TLS listener verifies client certificates by the attestation they carry rather than by an issuer CA, so it needs no `rpc.tls.mutual` section. It still accepts connections without a client certificate, because bootstrap and public metadata endpoints must be reachable before a client has an RA-TLS certificate. Current guests present self-issued certificates, but `GetTempCaCert` remains in use by older guest images and by KMS-to-KMS onboarding without a `source_token`, which mint their client certificates from that CA; it returns temp CA private material, so treat it as bootstrap-sensitive. +The KMS TLS listener verifies client certificates by the attestation they carry rather than by an issuer CA, so it needs no `rpc.tls.mutual` section. It still accepts connections without a client certificate, because bootstrap and public metadata endpoints must be reachable before a client has an RA-TLS certificate. Guests and onboarding clients use self-issued certificates. The temporary-CA RPC remains available for old OS images and tokenless legacy onboarding; its public key material authenticates no caller. App key release and KMS key handover still require verified caller attestation from the RA-TLS client certificate. Certificate signing verifies the CSR signature and embedded attestation before signing. diff --git a/docs/security/security-model.md b/docs/security/security-model.md index b3d43ed3b..ccf70629f 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -425,7 +425,7 @@ The KMS Rocket TLS listener permits connections without a client certificate bec App key release and KMS key handover require verified caller attestation from the RA-TLS client certificate. Certificate signing verifies the CSR signature and the attestation embedded in the CSR before signing. -The unauthenticated or non-client-certificate surface includes bootstrap and temp-CA bootstrap material retrieval, env-encryption public-key retrieval, metadata, health, and metrics behavior documented for operators. `GetTempCaCert` returns temp CA private material and remains in use by older guest images and by KMS-to-KMS onboarding without a `source_token`, which mint their client certificates from that CA; operators must treat it as bootstrap-sensitive rather than harmless public metadata. +The unauthenticated or non-client-certificate surface includes bootstrap, env-encryption public-key retrieval, metadata, health, and metrics behavior documented for operators. Clients create self-issued certificates locally. The temporary-CA RPC and its public key material remain available for legacy clients. Guest key retrieval verifies the actual response connection's attestation and checks that the returned CA signed its RPC certificate. Provider pins and launch measurement continue to bind the selected KMS root identity; attestation alone does not authorize an arbitrary CVM as the expected KMS. KMS-to-KMS root-key handover is served by `Admin.GetKmsKey` on the authenticated admin listener, which additionally requires a verified RA-TLS client certificate. diff --git a/docs/tutorials/kms-cvm-deployment.md b/docs/tutorials/kms-cvm-deployment.md index 24c68287e..2b6de47a0 100644 --- a/docs/tutorials/kms-cvm-deployment.md +++ b/docs/tutorials/kms-cvm-deployment.md @@ -73,8 +73,6 @@ Inside the CVM at `/etc/kms/certs/`: | `root-ca.key` | Root CA signing key (P256 ECDSA) | | `rpc.crt` | TLS certificate for RPC server | | `rpc.key` | RPC server private key | -| `tmp-ca.crt` | Temp CA, served by the legacy `GetTempCaCert` bootstrap RPC | -| `tmp-ca.key` | Temp CA private key, served alongside it | | `root-k256.key` | Ethereum signing key (secp256k1) | | `bootstrap-info.json` | Public keys and TDX attestation quote | diff --git a/docs/tutorials/troubleshooting-kms-deployment.md b/docs/tutorials/troubleshooting-kms-deployment.md index 7282db94d..ac38cbb4d 100644 --- a/docs/tutorials/troubleshooting-kms-deployment.md +++ b/docs/tutorials/troubleshooting-kms-deployment.md @@ -303,7 +303,7 @@ sudo systemctl restart qgsd ### GetMeta Returns "Connection refused" on Port 9200 -**Symptom:** KMS responds to `GetTempCaCert` but GetMeta returns: +**Symptom:** The KMS health endpoint responds but GetMeta returns: ```json {"error": "error sending request for url (http://127.0.0.1:9200/): ...Connection refused (os error 111)"} ``` diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock index 550291383..7f86192db 100644 --- a/dstack/Cargo.lock +++ b/dstack/Cargo.lock @@ -832,6 +832,7 @@ dependencies = [ "dstack-types", "hex", "http-client", + "mock-attestation", "ra-rpc", "ra-tls", "rocket", @@ -1904,6 +1905,7 @@ dependencies = [ "rocket", "serde", "serde_json", + "tempfile", ] [[package]] diff --git a/dstack/cert-client/Cargo.toml b/dstack/cert-client/Cargo.toml index 092824e28..8a715e379 100644 --- a/dstack/cert-client/Cargo.toml +++ b/dstack/cert-client/Cargo.toml @@ -20,10 +20,12 @@ tdx-attest.workspace = true dstack-guest-agent-rpc.workspace = true http-client = { workspace = true, features = ["prpc"] } tokio.workspace = true +x509-parser = { workspace = true, features = ["verify"] } [dev-dependencies] +ra-rpc = { workspace = true, features = ["rocket", "client"] } +mock-attestation = { path = "../crates/mock-attestation" } hex.workspace = true rocket.workspace = true tempfile.workspace = true -x509-parser.workspace = true tokio = { workspace = true, features = ["macros", "rt", "rt-multi-thread", "sync"] } diff --git a/dstack/cert-client/src/lib.rs b/dstack/cert-client/src/lib.rs index b7fdda86a..65498f275 100644 --- a/dstack/cert-client/src/lib.rs +++ b/dstack/cert-client/src/lib.rs @@ -2,15 +2,15 @@ // // SPDX-License-Identifier: Apache-2.0 -use std::sync::Arc; +use std::sync::{Arc, Mutex}; use anyhow::{bail, Context, Result}; -use dstack_kms_rpc::{kms_client::KmsClient, SignCertRequest}; +use dstack_kms_rpc::{kms_client::KmsClient, AppKeyResponse, GetAppKeyRequest, SignCertRequest}; use dstack_types::{AppKeys, KeyProvider}; use ra_rpc::client::{CertInfo, RaClient, RaClientConfig}; use ra_tls::{ attestation::{AppInfo, AttestationVerifier, VersionedAttestation}, - cert::{generate_self_signed_ra_cert, CaCert, CertSigningRequestV2}, + cert::{generate_self_signed_ra_cert, CaCert, CertPair, CertSigningRequestV2}, }; pub enum CertRequestClient { @@ -49,6 +49,73 @@ pub fn validate_kms_rpc_cert(cert: Option) -> Result<()> { Ok(()) } +/// Fetch keys directly over the attested connection, without a metadata/CA preflight. +/// A supplied CA always pins the connection, independently of the epoch selector. +/// Callers retain the startup KMS CA for the CVM lifetime, including later key +/// requests. Without a supplied CA, bind the response CA to this connection and +/// establish its identity through the provider pin or launch-measurement policy. +pub async fn request_app_keys( + url: String, + request: GetAppKeyRequest, + client_cert: CertPair, + attestation_verifier: Arc, + trusted_ca: Option, +) -> Result { + let server_cert = Arc::new(Mutex::new(None)); + let captured = server_cert.clone(); + let client = RaClientConfig::builder() + .remote_uri(url) + .tls_client_cert(client_cert.cert_pem) + .tls_client_key(client_cert.key_pem) + .tls_skip_ca_verification(trusted_ca.is_none()) + .tls_built_in_root_certs(false) + .maybe_tls_ca_cert(trusted_ca.clone()) + .attestation_verifier(attestation_verifier) + .cert_validator(Box::new(move |info| { + let info = info.context("missing KMS server certificate")?; + if info.attestation.is_none() { + bail!("KMS server certificate has no verified attestation"); + } + let cert_der = info.cert_der.clone(); + validate_kms_rpc_cert(Some(info))?; + *captured + .lock() + .map_err(|_| anyhow::anyhow!("KMS certificate lock poisoned"))? = Some(cert_der); + Ok(()) + })) + .build() + .into_client()?; + let requested_root_epoch = request.root_epoch; + let response = KmsClient::new(client).get_app_key(request).await?; + if requested_root_epoch.is_some_and(|epoch| epoch != response.root_epoch) { + bail!("KMS returned a different root epoch than requested"); + } + let cert = server_cert + .lock() + .map_err(|_| anyhow::anyhow!("KMS certificate lock poisoned"))?; + validate_kms_root( + cert.as_deref().context("missing KMS server certificate")?, + trusted_ca.as_deref().unwrap_or(&response.ca_cert), + )?; + Ok(response) +} + +fn validate_kms_root(server_cert: &[u8], root_ca: &str) -> Result<()> { + let (_, pem) = + x509_parser::pem::parse_x509_pem(root_ca.as_bytes()).context("invalid KMS root CA")?; + let root = pem.parse_x509().context("invalid KMS root CA")?; + let (_, leaf) = + x509_parser::parse_x509_certificate(server_cert).context("invalid KMS RPC certificate")?; + if !root.is_ca() || !root.validity().is_valid() || !leaf.validity().is_valid() { + bail!("invalid or expired KMS certificate"); + } + if leaf.issuer() != root.subject() { + bail!("KMS RPC certificate issuer does not match the root CA"); + } + leaf.verify_signature(Some(root.public_key())) + .context("KMS root CA did not sign the attested RPC certificate") +} + fn kms_client( url: String, ca_cert: String, @@ -374,6 +441,248 @@ mod tests { ) } + #[rocket::post("/GetAppKey", data = "<_request>")] + fn direct_app_keys( + _request: String, + _cert: rocket::mtls::Certificate<'_>, + state: &rocket::State<(Arc>, std::sync::atomic::AtomicUsize)>, + ) -> rocket::response::content::RawJson { + state.1.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + rocket::response::content::RawJson(state.0.lock().unwrap().clone()) + } + + #[tokio::test(flavor = "multi_thread")] + async fn direct_key_fetch_verifies_the_response_connection_without_preflight() { + use ra_tls::attestation::{ + AttestationVerifierConfig, DstackNitroQuote, QuoteContentType, RootCaPaths, + }; + use rocket::tls::Resolver as _; + let dir = tempfile::tempdir().unwrap(); + let attester = mock_attestation::nsm::NsmGenerator::new().unwrap(); + let attestation_root = dir.path().join("attestation-root.pem"); + std::fs::write(&attestation_root, attester.root_ca_pem()).unwrap(); + let verifier = Arc::new( + AttestationVerifier::load(&AttestationVerifierConfig { + insecure_allow_external_trust_anchors: true, + root_ca: RootCaPaths { + aws_nitro_enclave: Some(attestation_root), + ..Default::default() + }, + ..Default::default() + }) + .unwrap(), + ); + let root = kms_root(); + let rpc_key = p256_key(); + let report_data = QuoteContentType::RaTlsCert.to_report_data(&rpc_key.public_key_der()); + let pcrs = [0, 1, 2, 4] + .into_iter() + .map(|i| (i, vec![i as u8; 48])) + .collect(); + let evidence = attester.attest_with_pcrs(&report_data, pcrs).unwrap(); + let attestation = Attestation { + quote: AttestationQuote::DstackNitroEnclave(DstackNitroQuote { + nsm_quote: evidence, + }), + runtime_events: Vec::new(), + report_data, + config: String::new(), + report: (), + } + .into_versioned(); + let rpc_cert = root + .sign( + CertRequest::builder() + .subject("localhost") + .alt_names(&["127.0.0.1".into()]) + .key(&rpc_key) + .special_usage("kms:rpc") + .usage_server_auth(true) + .attestation(&attestation) + .build(), + ) + .unwrap(); + let cert_path = dir.path().join("rpc.crt"); + let key_path = dir.path().join("rpc.key"); + std::fs::write(&cert_path, rpc_cert.pem()).unwrap(); + std::fs::write(&key_path, rpc_key.serialize_pem()).unwrap(); + let response = AppKeyResponse { + ca_cert: root.pem_cert.clone(), + root_epoch: 1, + ..Default::default() + }; + let figment = rocket::Config::figment() + .merge(("port", 0)) + .merge(("address", "127.0.0.1")) + .merge(("log_level", "off")) + .merge(("shutdown.ctrlc", false)) + .merge(("tls.certs", cert_path)) + .merge(("tls.key", key_path)); + let (tx, rx) = tokio::sync::oneshot::channel(); + let tx = Mutex::new(Some(tx)); + let response_body = Arc::new(Mutex::new(serde_json::to_string(&response).unwrap())); + let server = rocket::custom(figment) + .attach(ra_rpc::ratls_client_verifier::RaTlsClientAuth::fairing()) + .manage(( + response_body.clone(), + std::sync::atomic::AtomicUsize::new(0), + )) + .mount("/", rocket::routes![direct_app_keys]) + .attach(rocket::fairing::AdHoc::on_liftoff("port", move |rocket| { + if let Some(tx) = tx.lock().unwrap().take() { + let _ = tx.send(rocket.endpoints().find_map(|e| e.port()).unwrap()); + } + Box::pin(async {}) + })) + .ignite() + .await + .unwrap(); + let shutdown = server.shutdown(); + let task = tokio::spawn(server.launch()); + let url = format!("https://127.0.0.1:{}", rx.await.unwrap()); + let client_cert = || { + let key = p256_key(); + let report_data = QuoteContentType::RaTlsCert.to_report_data(&key.public_key_der()); + let pcrs = [0, 1, 2, 4] + .into_iter() + .map(|i| (i, vec![i as u8; 48])) + .collect(); + let evidence = attester.attest_with_pcrs(&report_data, pcrs).unwrap(); + let attestation = Attestation { + quote: AttestationQuote::DstackNitroEnclave(DstackNitroQuote { + nsm_quote: evidence, + }), + runtime_events: Vec::new(), + report_data, + config: String::new(), + report: (), + } + .into_versioned(); + let cert = CertRequest::builder() + .subject("guest") + .key(&key) + .attestation(&attestation) + .usage_client_auth(true) + .build() + .self_signed() + .unwrap(); + CertPair { + cert_pem: cert.pem(), + key_pem: key.serialize_pem(), + } + }; + let request = || GetAppKeyRequest { + api_version: 1, + vm_config: String::new(), + root_epoch: None, + }; + let result = request_app_keys( + url.clone(), + request(), + client_cert(), + verifier.clone(), + None, + ) + .await + .unwrap(); + assert_eq!(result.ca_cert, response.ca_cert); + assert_eq!(result.root_epoch, 1); + + let other_ca = kms_root().pem_cert; + for requested_root_epoch in [None, Some(0), Some(1)] { + let request = || GetAppKeyRequest { + api_version: 1, + vm_config: String::new(), + root_epoch: requested_root_epoch, + }; + let selected = AppKeyResponse { + root_epoch: requested_root_epoch.unwrap_or(1), + ..response.clone() + }; + *response_body.lock().unwrap() = serde_json::to_string(&selected).unwrap(); + // Omission and explicit epoch selectors have the same trust semantics. + assert!(request_app_keys( + url.clone(), + request(), + client_cert(), + verifier.clone(), + None + ) + .await + .is_ok()); + // An unrelated pin must fail in TLS, before the RPC handler runs. + assert!(request_app_keys( + url.clone(), + request(), + client_cert(), + verifier.clone(), + Some(other_ca.clone()) + ) + .await + .is_err()); + + let different_epoch_ca = AppKeyResponse { + ca_cert: other_ca.clone(), + ..selected + }; + *response_body.lock().unwrap() = serde_json::to_string(&different_epoch_ca).unwrap(); + // A key response cannot replace the startup CA pin, even for default requests. + let result = request_app_keys( + url.clone(), + request(), + client_cert(), + verifier.clone(), + Some(root.pem_cert.clone()), + ) + .await + .unwrap(); + assert_eq!(result.ca_cert, other_ca); + // First-contact validation still requires the response CA to sign the RPC cert. + assert!(request_app_keys( + url.clone(), + request(), + client_cert(), + verifier.clone(), + None + ) + .await + .is_err()); + } + *response_body.lock().unwrap() = serde_json::to_string(&response).unwrap(); + let production = Arc::new(AttestationVerifier::new_prod(None).unwrap()); + assert!( + request_app_keys(url, request(), client_cert(), production, None) + .await + .is_err() + ); + shutdown.notify(); + let server = task.await.unwrap().unwrap(); + assert_eq!( + server + .state::<(Arc>, std::sync::atomic::AtomicUsize)>() + .unwrap() + .1 + .load(std::sync::atomic::Ordering::SeqCst), + 11 + ); + } + + #[test] + fn the_returned_root_must_sign_the_actual_rpc_certificate() { + let root = kms_root(); + let (leaf, _) = kms_rpc_cert(&root, "kms.example"); + let (_, leaf) = x509_parser::pem::parse_x509_pem(leaf.as_bytes()).unwrap(); + validate_kms_root(&leaf.contents, &root.pem_cert).unwrap(); + let unrelated = kms_root(); + assert!(validate_kms_root(&leaf.contents, &unrelated.pem_cert).is_err()); + assert!(validate_kms_root(&leaf.contents, "not a certificate").is_err()); + assert!(validate_kms_root(b"not a certificate", &root.pem_cert).is_err()); + + let (app_leaf, _) = app_cert(&root, &[0xa1; 20], "kms.example"); + let (_, app_leaf) = x509_parser::pem::parse_x509_pem(app_leaf.as_bytes()).unwrap(); + assert!(validate_kms_root(&app_leaf.contents, &root.pem_cert).is_err()); + } + #[rocket::post("/GetMeta")] fn get_meta() -> &'static str { r#"{"ca_cert":"served ca cert"}"# diff --git a/dstack/crates/dstack-auth/Cargo.toml b/dstack/crates/dstack-auth/Cargo.toml index 9a374b988..518c712c2 100644 --- a/dstack/crates/dstack-auth/Cargo.toml +++ b/dstack/crates/dstack-auth/Cargo.toml @@ -20,3 +20,6 @@ clap.workspace = true rocket = { workspace = true, features = ["json"] } serde.workspace = true serde_json.workspace = true + +[dev-dependencies] +tempfile.workspace = true diff --git a/dstack/crates/dstack-auth/src/main.rs b/dstack/crates/dstack-auth/src/main.rs index 00e5b72ee..a55f21b13 100644 --- a/dstack/crates/dstack-auth/src/main.rs +++ b/dstack/crates/dstack-auth/src/main.rs @@ -7,8 +7,8 @@ //! //! Runs on the host as `dstack-auth.service`; the KMS-in-CVM reaches it at //! `http://10.0.2.2:` under user-mode networking and POSTs `BootInfo` to -//! `/bootAuth/app` (compose-hash allowlist) and `/bootAuth/kms` (mrAggregated -//! allowlist). The allowlist JSON is re-read on every request, so `dstack run` +//! `/authorizeApp` (compose-hash allowlist) and `/authorizeKms` (mrAggregated +//! allowlist). Legacy `/bootAuth/*` routes remain available for bootstrap. The allowlist JSON is re-read on every request, so `dstack run` //! can add an app without a restart. Fails closed: a missing/invalid allowlist //! denies everything. //! @@ -22,6 +22,7 @@ use anyhow::{Context, Result}; use clap::Parser; +use dstack_types::authorization::AuthorizationOperation as Operation; use rocket::http::Status; use rocket::serde::json::Json; use rocket::{get, post, routes, State}; @@ -73,6 +74,27 @@ struct BootResponse { reason: String, } +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct AuthorizationRequest { + operation: Operation, + #[serde(flatten)] + boot_info: BootInfo, + requested_root_epoch: Option, + requested_app_epoch: Option, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct AuthorizationResult { + is_allowed: bool, + rejection_reason: String, + gateway_app_id: String, + current_root_epoch: usize, + current_root_epoch_hash: String, + current_app_epoch: u32, +} + /// the current KMS root epoch and its chain hash, which commits to the root /// public keys of epochs 0..=number. struct EpochInfo { @@ -84,6 +106,7 @@ struct EpochInfo { #[serde(rename_all = "camelCase", default)] struct Allowlist { os_images: Vec, + kms_contract_addr: String, gateway_app_id: String, kms: KmsRules, apps: HashMap, @@ -101,11 +124,20 @@ struct RootEpoch { } impl Allowlist { + fn app(&self, app_id: &str) -> Option<&AppRules> { + let app_id = norm(app_id); + self.apps + .iter() + .find(|(id, _)| norm(id) == app_id) + .map(|(_, app)| app) + } + /// `None` only when no epoch is configured; invalid configuration is an error. fn current_root_epoch(&self) -> Result> { if self.root_epochs.is_empty() && self.current_root_epoch == 0 { return Ok(None); } + u32::try_from(self.current_root_epoch).context("root epoch exceeds uint32")?; let epochs = self .root_epochs .get(..=self.current_root_epoch) @@ -214,17 +246,11 @@ fn allow(info: &BootInfo, is_kms: bool, al: &Allowlist) -> BootResponse { } } -fn check_app(info: &BootInfo, al: &Allowlist) -> BootResponse { +fn app_policy(info: &BootInfo, al: &Allowlist) -> BootResponse { if !al.os_images.is_empty() && !contains(&al.os_images, &info.os_image_hash) { return deny(al, "os image not allowed"); } - let app_id = norm(&info.app_id); - let Some(app) = al - .apps - .iter() - .find(|(k, _)| norm(k) == app_id) - .map(|(_, v)| v) - else { + let Some(app) = al.app(&info.app_id) else { return deny(al, "app not registered"); }; if !contains(&app.compose_hashes, &info.compose_hash) { @@ -233,17 +259,116 @@ fn check_app(info: &BootInfo, al: &Allowlist) -> BootResponse { if !device_ok(app.allow_any_device, &app.devices, &info.device_id) { return deny(al, "device not allowed"); } - allow(info, false, al) + BootResponse { + is_allowed: true, + gateway_app_id: al.gateway_app_id.clone(), + reason: String::new(), + } } -fn check_kms(info: &BootInfo, al: &Allowlist) -> BootResponse { +fn kms_policy(info: &BootInfo, al: &Allowlist) -> BootResponse { if !contains(&al.kms.mr_aggregated, &info.mr_aggregated) { return deny(al, "kms mrAggregated not allowed"); } if !device_ok(al.kms.allow_any_device, &al.kms.devices, &info.device_id) { return deny(al, "device not allowed"); } - allow(info, true, al) + BootResponse { + is_allowed: true, + gateway_app_id: al.gateway_app_id.clone(), + reason: String::new(), + } +} + +fn check_app(info: &BootInfo, al: &Allowlist) -> BootResponse { + let result = app_policy(info, al); + if result.is_allowed { + allow(info, false, al) + } else { + result + } +} + +fn check_kms(info: &BootInfo, al: &Allowlist) -> BootResponse { + let result = kms_policy(info, al); + if result.is_allowed { + allow(info, true, al) + } else { + result + } +} + +fn authorize( + request: &AuthorizationRequest, + is_kms: bool, + al: &Allowlist, +) -> Result { + let current = al.current_root_epoch()?; + let mut result = AuthorizationResult { + is_allowed: false, + rejection_reason: String::new(), + gateway_app_id: al.gateway_app_id.clone(), + current_root_epoch: current.as_ref().map_or(0, |epoch| epoch.number), + current_root_epoch_hash: current.as_ref().map_or_else( + || format!("0x{}", "00".repeat(32)), + |epoch| epoch.hash.clone(), + ), + current_app_epoch: 0, + }; + let info = &request.boot_info; + result.rejection_reason = if request.operation.is_kms() != is_kms { + "invalid authorization operation".into() + } else if current.is_none() && request.operation != Operation::KmsBootstrap { + "root epoch not registered".into() + } else if request + .requested_root_epoch + .is_some_and(|epoch| epoch as usize > result.current_root_epoch) + { + "requested root epoch is not active".into() + } else if request.requested_app_epoch.unwrap_or(0) != 0 { + "requested app epoch must be zero".into() + } else if request.operation != Operation::KmsBootstrap && norm(&al.kms_contract_addr).is_empty() + { + "kmsContractAddr is not configured".into() + } else if request.operation != Operation::KmsBootstrap + && (norm(&info.app_id) == norm(&al.kms_contract_addr)) != is_kms + { + "caller does not match authorization scope".into() + } else { + let policy = if is_kms { + kms_policy(info, al) + } else { + app_policy(info, al) + }; + result.is_allowed = policy.is_allowed; + policy.reason + }; + Ok(result) +} + +fn unavailable(error: anyhow::Error) -> Status { + rocket::warn!("invalid authorization configuration: {error}"); + Status::ServiceUnavailable +} + +#[post("/authorizeApp", data = "")] +fn authorize_app( + request: Json, + cli: &State, +) -> Result, Status> { + authorize(&request, false, &load(&cli.config)) + .map(Json) + .map_err(unavailable) +} + +#[post("/authorizeKms", data = "")] +fn authorize_kms( + request: Json, + cli: &State, +) -> Result, Status> { + authorize(&request, true, &load(&cli.config)) + .map(Json) + .map_err(unavailable) } /// load the allowlist, failing closed (deny-all) if it's missing or invalid. @@ -297,9 +422,9 @@ fn info(cli: &State) -> Result, Status> { fn info_json(al: &Allowlist) -> Result { let mut info = json!({ "status": "ok", - "kmsContractAddr": "", + "kmsContractAddr": al.kms_contract_addr, "ethRpcUrl": "", - "gatewayAppId": "", + "gatewayAppId": al.gateway_app_id, "chainId": 0, "appImplementation": "", }); @@ -310,15 +435,19 @@ fn info_json(al: &Allowlist) -> Result { Ok(info) } -#[rocket::main] -async fn main() -> Result<()> { - let cli = Cli::parse(); +fn server(cli: Cli) -> rocket::Rocket { let figment = rocket::Config::figment() .merge(("address", cli.address.clone())) .merge(("port", cli.port)); - rocket::custom(figment) - .manage(cli) - .mount("/", routes![info, boot_app, boot_kms]) + rocket::custom(figment).manage(cli).mount( + "/", + routes![info, boot_app, boot_kms, authorize_app, authorize_kms], + ) +} + +#[rocket::main] +async fn main() -> Result<()> { + server(Cli::parse()) .launch() .await .map_err(|e| anyhow::anyhow!("auth webhook failed: {e}"))?; @@ -329,6 +458,109 @@ async fn main() -> Result<()> { mod tests { use super::*; + #[rocket::async_test] + async fn authorization_routes_use_registered_roots_and_separate_caller_scopes() { + use rocket::{http::ContentType, local::asynchronous::Client}; + + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("allowlist.json"); + let mut policy = json!({ + "kmsContractAddr": "0x1111111111111111111111111111111111111111", + "rootEpochs": [{"caPubkey": "aa", "k256Pubkey": "bb"}], + "osImages": ["aa"], + "kms": {"mrAggregated": ["bb"]}, + "apps": {"0x2222222222222222222222222222222222222222": {"composeHashes": ["cc"]}} + }); + std::fs::write(&path, policy.to_string()).unwrap(); + let client = Client::tracked(server(Cli { + config: path.clone(), + address: "127.0.0.1".into(), + port: 0, + })) + .await + .unwrap(); + let kms = json!({"operation": "kms_serve", "appId": "11".repeat(20), "mrAggregated": "bb"}); + let app = json!({"operation": "app_get_key", "appId": "22".repeat(20), "composeHash": "cc", "osImageHash": "aa"}); + for (route, body, allowed) in [ + ("/authorizeKms", kms.clone(), true), + ("/authorizeApp", app.clone(), true), + ("/authorizeKms", app.clone(), false), + ("/authorizeApp", kms.clone(), false), + ] { + let response = client + .post(route) + .header(ContentType::JSON) + .body(body.to_string()) + .dispatch() + .await; + assert_eq!(response.status(), Status::Ok); + let response: serde_json::Value = response.into_json().await.unwrap(); + assert_eq!(response["isAllowed"], allowed); + assert_eq!(response["currentRootEpoch"], 0); + assert_eq!(response["currentAppEpoch"], 0); + assert_ne!( + response["currentRootEpochHash"], + format!("0x{}", "00".repeat(32)) + ); + } + for operation in [serde_json::Value::Null, json!("unknown")] { + let mut body = app.clone(); + body["operation"] = operation; + let response = client + .post("/authorizeApp") + .header(ContentType::JSON) + .body(body.to_string()) + .dispatch() + .await; + assert!(matches!( + response.status(), + Status::BadRequest | Status::UnprocessableEntity + )); + } + for field in ["requestedRootEpoch", "requestedAppEpoch"] { + let mut body = app.clone(); + body[field] = 1.into(); + let response: serde_json::Value = client + .post("/authorizeApp") + .header(ContentType::JSON) + .body(body.to_string()) + .dispatch() + .await + .into_json() + .await + .unwrap(); + assert_eq!(response["isAllowed"], false); + } + policy["rootEpochs"] = json!([]); + std::fs::write(&path, policy.to_string()).unwrap(); + let response: serde_json::Value = client + .post("/authorizeKms") + .header(ContentType::JSON) + .body(kms.to_string()) + .dispatch() + .await + .into_json() + .await + .unwrap(); + assert_eq!(response["isAllowed"], false); + assert_eq!( + response["currentRootEpochHash"], + format!("0x{}", "00".repeat(32)) + ); + policy["currentRootEpoch"] = 1.into(); + std::fs::write(&path, policy.to_string()).unwrap(); + assert_eq!( + client + .post("/authorizeApp") + .header(ContentType::JSON) + .body(app.to_string()) + .dispatch() + .await + .status(), + Status::ServiceUnavailable + ); + } + fn allowlist() -> Allowlist { serde_json::from_str( r#"{ diff --git a/dstack/dstack-types/src/authorization.rs b/dstack/dstack-types/src/authorization.rs new file mode 100644 index 000000000..1fdb48228 --- /dev/null +++ b/dstack/dstack-types/src/authorization.rs @@ -0,0 +1,27 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Operations performed by the attested subject of an authorization request. + +use serde::{Deserialize, Serialize}; + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum AuthorizationOperation { + KmsBootstrap, + KmsServe, + KmsExportKeys, + KmsImportKeys, + AppGetKey, + AppGetCert, +} + +impl AuthorizationOperation { + pub fn is_kms(self) -> bool { + matches!( + self, + Self::KmsBootstrap | Self::KmsServe | Self::KmsExportKeys | Self::KmsImportKeys + ) + } +} diff --git a/dstack/dstack-types/src/lib.rs b/dstack/dstack-types/src/lib.rs index cc1bdf530..c9b276f6e 100644 --- a/dstack/dstack-types/src/lib.rs +++ b/dstack/dstack-types/src/lib.rs @@ -14,6 +14,8 @@ use size_parser::human_size; /// infrastructure-provider initialization stages. pub const MAX_INIT_SCRIPTS: usize = 5; +pub mod authorization; + pub mod init_script_hashes { use serde::{Deserialize, Deserializer, Serialize, Serializer}; use serde_human_bytes::ByteBuf; diff --git a/dstack/dstack-util/src/main.rs b/dstack/dstack-util/src/main.rs index 1046d35d1..f081b3235 100644 --- a/dstack/dstack-util/src/main.rs +++ b/dstack/dstack-util/src/main.rs @@ -639,9 +639,6 @@ fn cmd_attest_strip(args: AttestStripArgs) -> Result<()> { } async fn cmd_get_keys(args: GetKeysArgs) -> Result<()> { - use dstack_kms_rpc::kms_client::KmsClient; - use ra_rpc::client::RaClientConfig; - let kms_url = normalize_prpc_url(&args.kms_url); // Load root CA if provided for TLS pinning @@ -653,57 +650,24 @@ async fn cmd_get_keys(args: GetKeysArgs) -> Result<()> { None }; - // Step 1: Fetch the KMS root CA - eprintln!("Connecting to KMS: {kms_url}"); - let tls_skip_ca_verification = root_ca_pem.is_none(); - if tls_skip_ca_verification { - eprintln!("Warning: no --root-ca provided, TLS certificate verification is disabled for initial connection"); - } - let root_ca = { - let client = RaClientConfig::builder() - .remote_uri(kms_url.clone()) - .tls_skip_ca_verification(tls_skip_ca_verification) - .tls_built_in_root_certs(false) - .maybe_tls_ca_cert(root_ca_pem.clone()) - .build() - .into_client() - .context("failed to create client")?; - KmsClient::new(client) - .get_meta() - .await - .context("failed to get KMS meta")? - .ca_cert - }; - - // Step 2: Generate a self-issued RA-TLS client certificate. The quote inside it - // is the identity the KMS authenticates, so no CA material is needed. let app_id = decode_app_id(args.app_id.as_deref())?; let cert_pair = generate_self_signed_ra_cert(app_id).context("Failed to generate RA cert")?; - - // Step 3: Create authenticated client and request app keys - let ra_client = RaClientConfig::builder() - .tls_skip_ca_verification(false) - .tls_built_in_root_certs(false) - .remote_uri(kms_url.clone()) - .tls_client_cert(cert_pair.cert_pem) - .tls_client_key(cert_pair.key_pem) - .tls_ca_cert(root_ca.clone()) - .build() - .into_client() - .context("Failed to create RA client")?; - - let kms_client = KmsClient::new(ra_client); - let response = kms_client - .get_app_key(dstack_kms_rpc::GetAppKeyRequest { + let response = cert_client::request_app_keys( + kms_url.clone(), + dstack_kms_rpc::GetAppKeyRequest { api_version: 1, - vm_config: "".to_string(), - }) - .await - .context( - "failed to get app key (the KMS must be 0.6.0 or later to accept a self-issued RA-TLS client certificate)", - )?; + vm_config: String::new(), + root_epoch: None, + }, + cert_pair, + std::sync::Arc::new(ra_tls::attestation::AttestationVerifier::new_prod(None)?), + root_ca_pem.clone(), + ) + .await + .context("Failed to get app keys from the attested KMS")?; + let root_ca = root_ca_pem.unwrap_or(response.ca_cert); - // Step 4: Build AppKeys structure + // Build AppKeys from the authenticated response. let (_, ca_pem) = x509_parser::pem::parse_x509_pem(root_ca.as_bytes()).context("Failed to parse CA cert")?; let x509 = ca_pem.parse_x509().context("Failed to parse CA cert")?; @@ -722,7 +686,7 @@ async fn cmd_get_keys(args: GetKeysArgs) -> Result<()> { }, }; - // Step 5: Output result + // Output result. let json = serde_json::to_string_pretty(&keys).context("Failed to serialize app keys")?; if let Some(output_path) = args.output { safe_write_with_mode(&output_path, &json, 0o600).context("Failed to write app keys")?; diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 0d604076b..1beb2b6ad 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -55,7 +55,7 @@ use crate::{ deserialize_json_file, sha256, sha256_file, AppCompose, AppKeys, KeyProviderKind, SysConfig, }, }; -use cert_client::{validate_kms_rpc_cert, CertRequestClient}; +use cert_client::{request_app_keys, CertRequestClient}; use cmd_lib::run_fun as cmd; use dstack_api_auth::jwt; use dstack_gateway_rpc::{ @@ -2417,47 +2417,22 @@ impl<'a> Stage0<'a> { async fn request_app_keys_from_kms_url(&self, kms_url: String) -> Result<(AppKeys, Vec)> { info!("Requesting app keys from KMS: {kms_url}"); - // Learn the KMS root CA over an unauthenticated connection, as with - // `GetTempCaCert` before. Relying parties bind the KMS through the - // `key-provider` launch event; `verify_key_provider_id` also enforces it - // in-guest when the app pins `key_provider_id`. - let root_ca = { - info!("Getting KMS root CA"); - let client = RaClient::new(kms_url.clone(), true)?; - let kms_client = dstack_kms_rpc::kms_client::KmsClient::new(client); - kms_client - .get_meta() - .await - .context("failed to get KMS meta")? - .ca_cert - }; - // Self-issued: the quote inside the certificate is the identity, so there is - // no CA to fetch and no shared private key to hold. let cert_pair = generate_self_signed_ra_cert(None).context("Failed to generate RA-TLS cert")?; - let attestation_verifier = attestation_verifier(&self.shared.sys_config)?; - let ra_client = RaClientConfig::builder() - .tls_skip_ca_verification(false) - .tls_built_in_root_certs(false) - .remote_uri(kms_url.clone()) - .tls_client_cert(cert_pair.cert_pem) - .tls_client_key(cert_pair.key_pem) - .tls_ca_cert(root_ca.clone()) - .attestation_verifier(attestation_verifier) - .cert_validator(Box::new(validate_kms_rpc_cert)) - .build() - .into_client() - .context("Failed to create client")?; - let kms_client = dstack_kms_rpc::kms_client::KmsClient::new(ra_client); - let response = kms_client - .get_app_key(rpc::GetAppKeyRequest { + let response = request_app_keys( + kms_url.clone(), + rpc::GetAppKeyRequest { api_version: 1, vm_config: self.shared.sys_config.vm_config.clone(), - }) - .await - .context( - "failed to get app key (the KMS must be 0.6.0 or later to accept a self-issued RA-TLS client certificate)", - )?; + root_epoch: None, + }, + cert_pair, + attestation_verifier(&self.shared.sys_config)?, + None, + ) + .await + .context("Failed to get app keys from the attested KMS")?; + let root_ca = response.ca_cert; let (_, ca_pem) = x509_parser::pem::parse_x509_pem(root_ca.as_bytes()) .context("Failed to parse ca cert")?; diff --git a/dstack/kms/README.md b/dstack/kms/README.md index 49d15e618..7e5d4b710 100644 --- a/dstack/kms/README.md +++ b/dstack/kms/README.md @@ -242,3 +242,130 @@ The `SignCert` RPC is used by the dstack app to sign a TLS certificate. In this - Verify the CSR signature - Query the smart contract to check if the app is authorized - If authorized, sign the CSR with the CA root key and return the certificate chain to the app + + +## Authorization protocol + +Authorization uses two explicit endpoints: + +- `POST /authorizeKms` for KMS self-authorization and root-key handover. +- `POST /authorizeApp` for application keys and certificates. + +The Ethereum backends call the matching `DstackKms` method. Both accept +`AuthorizationRequest` and return `AuthorizationResult`; no caller-kind flag is +sent. The KMS method checks the KMS policy; the app method applies registered app +policy. +KMS callers do not fetch contract metadata to decide which policy to use. + +`requestedAppEpoch` reserves the application's own key epoch, independently of KMS root +epochs. HTTP requests default it to 0; KMS currently sends 0. Solidity uses the same +`OptionalEpoch` tuple as the root selector, reserving omission for the current +app epoch. Responses include `currentAppEpoch`, currently 0. Nonzero values are +rejected until app-epoch support is implemented, and rejection responses still +carry the current root epoch and chain hash. + +`requestedRootEpoch` is optional in the JSON request. Omit it to use the epoch in +this authorization result; an explicit `0` selects epoch 0. Solidity represents +it as `OptionalEpoch { isSet, value }` alongside `AppInfo` in the request. The +contract rejects a requested epoch newer than the current one; historical +epochs remain available for migration. No serving epoch or serving hash is sent. +KMS-to-KMS handover transfers the complete keyring. The receiver explicitly +authorizes the source to export the highest received epoch before importing it. + +Every request includes a required `operation` describing what the attested +subject is authorized to do. JSON uses `snake_case`; the contract receives +`keccak256(bytes(operation))` as `bytes32`. Rust uses the shared serde-derived +`AuthorizationOperation` enum. Unknown operations and operations on the wrong +endpoint are rejected. The supported operations currently share the existing +material policy, leaving room for independent policies later: + +| Endpoint | Operation | Subject | +| --- | --- | --- | +| `authorizeKms` | `kms_bootstrap` | KMS generating its first roots | +| `authorizeKms` | `kms_serve` | KMS serving app keys, public keys or certificates | +| `authorizeKms` | `kms_export_keys` | Source KMS handing over its keyring | +| `authorizeKms` | `kms_import_keys` | Recipient KMS receiving a keyring | +| `authorizeApp` | `app_get_key` | App receiving derived keys | +| `authorizeApp` | `app_get_cert` | App requesting a certificate | + +The KMS chooses the operation from its code path, not an untrusted RPC field. +Self-authorization caches are keyed by operation. Legacy authorization endpoints +retain their existing semantics for old KMS binaries; fine-grained operation +policies cannot protect calls still served by those old binaries. + +The response is an `AuthorizationResult`: + +```json +{ + "isAllowed": true, + "rejectionReason": "", + "gatewayAppId": "0x...", + "currentRootEpoch": 0, + "currentRootEpochHash": "0x..." +} +``` + +Both successful and rejected decisions include the current epoch and chain hash +from the same contract call or auth-simple configuration snapshot. With no +registered epoch, the result is `isAllowed: false`, +`rejectionReason: "Root epoch not registered"`, epoch 0 and a zero hash. A zero hash +never authorizes key release or requests an epoch switch. + +The KMS holds no serving-epoch snapshot. It verifies the returned current epoch +and chain hash against its keyring, then derives keys from `requestedRootEpoch` when +specified or the returned current epoch otherwise. Missing or mismatched roots +fail closed. Self-authorization retains its one-second cache of complete results. + +The TLS listener has a separate lifetime: when an authorized current CA differs +from the running RPC certificate's issuer, the KMS requests a restart to refresh +that certificate and the caller must retry. The RPC CA does not select key +epochs. Application key responses and signed certificate chains report the +selected epoch's CA. CA certificates +are reloaded after renewal at startup. + +Upgrade the contract and auth backend and register epoch 0 before using this +protocol for key release. Runtime authorization never falls back to the legacy +API if the new endpoint or contract method is unavailable. The old contract +methods and `/bootAuth/app` and `/bootAuth/kms` endpoints remain available for +older clients. Bootstrap uses the `kms_bootstrap` operation, which checks KMS policy at epoch zero +even before roots are registered. Other operations require registered roots. +Onboarding checks its source through `authorizeKms`, without a legacy fallback. For +`auth-simple`, set `kmsContractAddr` to the attested KMS app ID and configure +`rootEpochs`; for `auth-mock`, set `KMS_CONTRACT_ADDR`, `MOCK_ROOT_EPOCH`, and +`MOCK_ROOT_EPOCH_HASH`. + + +## Direct attested key retrieval + +`GetTempCaCert`, its original response fields, and the `temp_ca_key` handover +field remain available for existing OS images and legacy KMS onboarding. The +compatibility CA is stored in `tmp-ca.key`/`tmp-ca.crt`, independent of all root +epochs; existing files are reused and missing files are recreated. Its public +private key authenticates nobody. Client identity still requires attestation. +The returned `ca_cert` is the currently served root CA. + +Upgrade contracts and auth backends, register the original roots as epoch 0, +then upgrade KMS nodes before guests. Keep root and app epochs at zero during +the mixed-version phase; rotate only after guests support key migration. + +Guests and `dstack-util get-keys` generate their self-issued RA-TLS certificate +locally, then call `GetAppKey` directly. They verify the response connection's +attestation and `kms:rpc` usage, and verify that the returned root CA signed that +connection's certificate. Skipping CA-chain validation during this first request +still verifies the TLS handshake signature, so replaying an attested certificate +without its private key cannot impersonate the server. There is no `GetMeta` preflight or first unauthenticated +connection. The optional provider pin and the key-provider launch measurement +still bind the root identity; a valid quote alone is not a KMS policy decision. +`--root-ca` additionally pins the CLI connection before sending the request. + +Onboarding with an admin token uses a self-issued client certificate. Tokenless +onboarding obtains the source temp CA first so pre-0.6 sources remain usable. +Both paths check the source attestation captured from the key-carrying connection +against local KMS policy before importing anything. Retries and reconnects repeat +verification on the response connection; no assumption relies on one TCP session +lasting forever. All subsequent key requests pin the KMS CA established at startup, regardless +of whether the epoch selector is omitted or explicit. A supplied `trusted_ca` +always wins over the response CA. The selected epoch's CA never replaces this +lifetime pin; a TLS CA change requires a fresh boot/trust establishment rather +than silently trusting another KMS. Without a supplied pin, the response CA +must sign the actual RPC certificate, even when an epoch is explicitly selected. diff --git a/dstack/kms/auth-eth-bun/README.md b/dstack/kms/auth-eth-bun/README.md index 8fcafb0bd..bc9c72e19 100644 --- a/dstack/kms/auth-eth-bun/README.md +++ b/dstack/kms/auth-eth-bun/README.md @@ -199,3 +199,24 @@ the test suite (`index.test.ts`) validates: - ✅ error handling behavior - ✅ hex encoding/decoding compatibility - ✅ optional field handling + + +### Authorization endpoints + +`POST /authorizeKms` and `POST /authorizeApp` return `AuthorizationResult`, including +`rejectionReason`, `gatewayAppId`, `currentRootEpoch`, and `currentRootEpochHash`. +The selected endpoint determines the policy; the backend checks that the verified +`appId` matches that scope. `requestedAppEpoch` defaults to 0; nonzero values are currently +rejected. The optional +`requestedRootEpoch` selects historical keys; omission uses the returned current +epoch, and an explicit `0` selects epoch 0. See the +[KMS authorization protocol](../README.md#authorization-protocol) for registration, +bootstrap, and compatibility requirements. The legacy `/bootAuth/*` endpoints +remain available. + +### Authorization operations + +`/authorizeKms` and `/authorizeApp` require an `operation` string. See the +[KMS authorization protocol](../README.md#authorization-protocol) for the names +and subject semantics. Bootstrap alone may authorize before root registration; +unknown operations are rejected. Legacy `/bootAuth/*` routes are unchanged. diff --git a/dstack/kms/auth-eth-bun/index.test.ts b/dstack/kms/auth-eth-bun/index.test.ts index 22af72fc8..dda821bba 100644 --- a/dstack/kms/auth-eth-bun/index.test.ts +++ b/dstack/kms/auth-eth-bun/index.test.ts @@ -10,7 +10,8 @@ const mockReadContract = vi.fn(); const mockGetChainId = vi.fn(); const mockGetBlockNumber = vi.fn(); -vi.mock('viem', () => ({ +vi.mock('viem', async (importOriginal) => ({ + ...await importOriginal(), createPublicClient: vi.fn(() => ({ readContract: mockReadContract, getChainId: mockGetChainId, @@ -832,3 +833,49 @@ describe('request hardening', () => { expect(response.headers.get('x-frame-options')).toBe('SAMEORIGIN'); }); }); + + +describe('POST /authorizeApp', () => { + const appInfo = { + appId: '0x' + '11'.repeat(20), composeHash: '0x' + '22'.repeat(32), + instanceId: '', deviceId: '0x' + '33'.repeat(32), + mrAggregated: '0x' + '44'.repeat(32), mrSystem: '0x' + '55'.repeat(32), + osImageHash: '0x' + '66'.repeat(32), tcbStatus: 'UpToDate', advisoryIds: [], + }; + const authorize = (requestedRootEpoch?: number) => appFetch(new Request('http://localhost/authorizeApp', { + method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ ...appInfo, operation: 'app_get_key', requestedRootEpoch }), + })); + + it('reads the complete result with one contract call at configured finality', async () => { + const result = { + isAllowed: false, rejectionReason: 'App not registered', gatewayAppId: 'gateway', + currentAppEpoch: 0n, currentRootEpoch: 2n, currentRootEpochHash: '0x' + '77'.repeat(32), + }; + mockReadContract.mockResolvedValueOnce(result); + const response = await authorize(); + expect(response.status).toBe(200); + expect(await response.json()).toEqual({ ...result, currentRootEpoch: 2, currentAppEpoch: 0 }); + expect(mockReadContract).toHaveBeenCalledTimes(1); + expect(mockReadContract).toHaveBeenCalledWith(expect.objectContaining({ + functionName: 'authorizeApp', blockNumber: 98n, + args: [expect.objectContaining({ appInfo: expect.objectContaining({ appId: appInfo.appId }), requestedAppEpoch: { isSet: true, value: 0n }, requestedRootEpoch: { isSet: false, value: 0n } })], + })); + }); + + it('encodes an explicit epoch zero as present', async () => { + mockReadContract.mockResolvedValueOnce({ + isAllowed: true, rejectionReason: '', gatewayAppId: 'gateway', + currentAppEpoch: 0n, currentRootEpoch: 1n, currentRootEpochHash: '0x' + '77'.repeat(32), + }); + expect((await authorize(0)).status).toBe(200); + expect(mockReadContract).toHaveBeenCalledWith(expect.objectContaining({ + args: [expect.objectContaining({ requestedRootEpoch: { isSet: true, value: 0n } })], + })); + }); + + it('does not fall back to legacy authorization on a failed contract call', async () => { + mockReadContract.mockRejectedValueOnce(new Error('contract unavailable')); + expect((await authorize()).status).toBe(503); + expect(mockReadContract).toHaveBeenCalledTimes(1); + }); +}); diff --git a/dstack/kms/auth-eth-bun/index.ts b/dstack/kms/auth-eth-bun/index.ts index f7d726377..f4e12ab3a 100644 --- a/dstack/kms/auth-eth-bun/index.ts +++ b/dstack/kms/auth-eth-bun/index.ts @@ -7,7 +7,7 @@ import { bodyLimit } from 'hono/body-limit'; import { secureHeaders } from 'hono/secure-headers'; import { zValidator } from '@hono/zod-validator'; import { z } from 'zod'; -import { createPublicClient, http, type Address, type Hex } from 'viem'; +import { createPublicClient, http, keccak256, toBytes, type Address, type Hex } from 'viem'; // zod schemas for validation - compatible with original fastify implementation const boundedHex = (bytes: number, description: string) => @@ -36,6 +36,13 @@ const BootInfoSchema = z.object({ rootEpochHash: boundedHex(32, 'root epoch chain hash').optional() }); +const AuthorizationRequestSchema = BootInfoSchema.omit({ rootEpoch: true, rootEpochHash: true }).extend({ + operation: z.enum(['kms_bootstrap', 'kms_serve', 'kms_export_keys', 'kms_import_keys', 'app_get_key', 'app_get_cert']), + requestedAppEpoch: z.number().int().min(0).max(0xffffffff).default(0), + requestedRootEpoch: z.number().int().min(0).max(0xffffffff).optional(), +}); +type AuthorizationRequest = z.infer; + const BootResponseSchema = z.object({ isAllowed: z.boolean(), reason: z.string(), @@ -71,8 +78,49 @@ function rootEpochDenial(bootInfo: BootInfo, isKms: boolean, current: RootEpochI return undefined; } +const APP_INFO_COMPONENTS = [ + { name: 'appId', type: 'address' }, + { name: 'composeHash', type: 'bytes32' }, + { name: 'instanceId', type: 'address' }, + { name: 'deviceId', type: 'bytes32' }, + { name: 'mrAggregated', type: 'bytes32' }, + { name: 'mrSystem', type: 'bytes32' }, + { name: 'osImageHash', type: 'bytes32' }, + { name: 'tcbStatus', type: 'string' }, + { name: 'advisoryIds', type: 'string[]' } +] as const; + +const AUTHORIZATION_ABI = { + type: 'function', + stateMutability: 'view', + inputs: [{ name: 'request', type: 'tuple', components: [ + { name: 'appInfo', type: 'tuple', components: APP_INFO_COMPONENTS }, + { name: 'operation', type: 'bytes32' }, + { name: 'requestedAppEpoch', type: 'tuple', components: [ + { name: 'isSet', type: 'bool' }, { name: 'value', type: 'uint256' }, + ] }, + { name: 'requestedRootEpoch', type: 'tuple', components: [ + { name: 'isSet', type: 'bool' }, { name: 'value', type: 'uint256' }, + ] }, + ] }], + outputs: [{ + name: 'result', + type: 'tuple', + components: [ + { name: 'isAllowed', type: 'bool' }, + { name: 'rejectionReason', type: 'string' }, + { name: 'gatewayAppId', type: 'string' }, + { name: 'currentRootEpoch', type: 'uint256' }, + { name: 'currentRootEpochHash', type: 'bytes32' }, + { name: 'currentAppEpoch', type: 'uint256' }, + ], + }], +} as const; + // DstackKms contract ABI (minimal required functions) const DSTACK_KMS_ABI = [ + { ...AUTHORIZATION_ABI, name: 'authorizeKms' }, + { ...AUTHORIZATION_ABI, name: 'authorizeApp' }, { name: 'isKmsAllowed', type: 'function', @@ -81,17 +129,7 @@ const DSTACK_KMS_ABI = [ { name: 'bootInfo', type: 'tuple', - components: [ - { name: 'appId', type: 'address' }, - { name: 'composeHash', type: 'bytes32' }, - { name: 'instanceId', type: 'address' }, - { name: 'deviceId', type: 'bytes32' }, - { name: 'mrAggregated', type: 'bytes32' }, - { name: 'mrSystem', type: 'bytes32' }, - { name: 'osImageHash', type: 'bytes32' }, - { name: 'tcbStatus', type: 'string' }, - { name: 'advisoryIds', type: 'string[]' } - ] + components: APP_INFO_COMPONENTS } ], outputs: [ @@ -107,17 +145,7 @@ const DSTACK_KMS_ABI = [ { name: 'bootInfo', type: 'tuple', - components: [ - { name: 'appId', type: 'address' }, - { name: 'composeHash', type: 'bytes32' }, - { name: 'instanceId', type: 'address' }, - { name: 'deviceId', type: 'bytes32' }, - { name: 'mrAggregated', type: 'bytes32' }, - { name: 'mrSystem', type: 'bytes32' }, - { name: 'osImageHash', type: 'bytes32' }, - { name: 'tcbStatus', type: 'string' }, - { name: 'advisoryIds', type: 'string[]' } - ] + components: APP_INFO_COMPONENTS } ], outputs: [ @@ -198,9 +226,9 @@ class EthereumBackend { return `0x${hex}` as Hex; } - async checkBoot(bootInfo: BootInfo, isKms: boolean): Promise { + private contractAppInfo(bootInfo: BootInfo) { // create boot info struct for contract call - const bootInfoStruct = { + return { appId: this.decodeHex(bootInfo.appId, 20) as Address, composeHash: this.decodeHex(bootInfo.composeHash, 32), instanceId: this.decodeHex(bootInfo.instanceId, 20) as Address, @@ -212,6 +240,33 @@ class EthereumBackend { advisoryIds: bootInfo.advisoryIds || [] }; + } + + async checkAuthorization(appInfo: AuthorizationRequest, method: 'authorizeKms' | 'authorizeApp') { + const blockNumber = await this.finalizedBlockNumber(); + const result = await this.client.readContract({ + address: this.kmsContractAddr, + abi: DSTACK_KMS_ABI, + functionName: method, + args: [{ + appInfo: this.contractAppInfo(appInfo), + operation: keccak256(toBytes(appInfo.operation)), + requestedAppEpoch: { isSet: appInfo.requestedAppEpoch !== undefined, value: BigInt(appInfo.requestedAppEpoch ?? 0) }, + requestedRootEpoch: { isSet: appInfo.requestedRootEpoch !== undefined, value: BigInt(appInfo.requestedRootEpoch ?? 0) }, + }], + blockNumber, + }); + const currentRootEpoch = Number(result.currentRootEpoch); + if (!Number.isSafeInteger(currentRootEpoch)) { + throw new Error('current root epoch exceeds the supported range'); + } + const currentAppEpoch = Number(result.currentAppEpoch); + if (!Number.isSafeInteger(currentAppEpoch)) throw new Error('current app epoch exceeds the supported range'); + return { ...result, currentRootEpoch, currentAppEpoch }; + } + + async checkBoot(bootInfo: BootInfo, isKms: boolean): Promise { + const bootInfoStruct = this.contractAppInfo(bootInfo); const blockNumber = await this.finalizedBlockNumber(); let response; if (isKms) { @@ -370,6 +425,21 @@ app.get('/', async (c) => { } }); +for (const method of ['authorizeKms', 'authorizeApp'] as const) { + app.post(`/${method}`, + bodyLimit({ maxSize: MAX_BOOT_INFO_BYTES }), + zValidator('json', AuthorizationRequestSchema), + async (c) => { + try { + return c.json(await ethereum.checkAuthorization(c.req.valid('json'), method)); + } catch { + console.error('authorization backend failed'); + return c.json({ status: 'error', message: backendUnavailable }, 503); + } + } + ); +} + // app boot authentication app.post('/bootAuth/app', zValidator('json', BootInfoSchema, (result, c) => { diff --git a/dstack/kms/auth-eth-bun/openapi.json b/dstack/kms/auth-eth-bun/openapi.json index a9619eb0a..e431225a9 100644 --- a/dstack/kms/auth-eth-bun/openapi.json +++ b/dstack/kms/auth-eth-bun/openapi.json @@ -98,6 +98,68 @@ } } } + }, + "/authorizeKms": { + "post": { + "summary": "Authorize a Kms identity and return the current root epoch", + "operationId": "authorizeKms", + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/AuthorizationRequest" + } + } + } + }, + "responses": { + "200": { + "description": "Authorization decision, including the epoch on rejection.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/AuthorizationResult" + } + } + } + }, + "503": { + "description": "Authorization backend unavailable; no legacy fallback." + } + } + } + }, + "/authorizeApp": { + "post": { + "summary": "Authorize a App identity and return the current root epoch", + "operationId": "authorizeApp", + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/AuthorizationRequest" + } + } + } + }, + "responses": { + "200": { + "description": "Authorization decision, including the epoch on rejection.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/AuthorizationResult" + } + } + } + }, + "503": { + "description": "Authorization backend unavailable; no legacy fallback." + } + } + } } }, "components": { @@ -198,7 +260,9 @@ "properties": { "status": { "type": "string", - "enum": ["ok"], + "enum": [ + "ok" + ], "description": "System status" }, "kmsContractAddr": { @@ -244,13 +308,142 @@ "properties": { "status": { "type": "string", - "enum": ["error"] + "enum": [ + "error" + ] }, "message": { "type": "string", "description": "Error message" } } + }, + "AppInfo": { + "type": "object", + "required": [ + "mrAggregated", + "osImageHash", + "appId", + "composeHash", + "instanceId", + "deviceId" + ], + "properties": { + "mrAggregated": { + "type": "string", + "description": "Aggregated MR measurement" + }, + "osImageHash": { + "type": "string", + "description": "OS Image hash" + }, + "appId": { + "type": "string", + "description": "Application ID" + }, + "composeHash": { + "type": "string", + "description": "Compose hash" + }, + "instanceId": { + "type": "string", + "description": "Instance ID" + }, + "deviceId": { + "type": "string", + "description": "Device ID" + }, + "tcbStatus": { + "type": "string", + "description": "TCB status (optional for compatibility)" + }, + "advisoryIds": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Advisory IDs (optional for compatibility)" + }, + "mrSystem": { + "type": "string", + "description": "MR System measurement (optional for compatibility)" + } + }, + "description": "Verified attestation identity. The KMS app ID is the KMS contract address." + }, + "AuthorizationResult": { + "type": "object", + "required": [ + "isAllowed", + "rejectionReason", + "gatewayAppId", + "currentRootEpoch", + "currentRootEpochHash", + "currentAppEpoch" + ], + "properties": { + "isAllowed": { + "type": "boolean" + }, + "rejectionReason": { + "type": "string", + "description": "Empty when allowed." + }, + "gatewayAppId": { + "type": "string" + }, + "currentRootEpoch": { + "type": "integer", + "minimum": 0 + }, + "currentRootEpochHash": { + "type": "string", + "pattern": "^0x[0-9a-fA-F]{64}$", + "description": "Zero when no epoch is registered; authorization is denied." + }, + "currentAppEpoch": { + "type": "integer", + "minimum": 0, + "description": "Current application epoch; zero until app rotation is supported." + } + } + }, + "AuthorizationRequest": { + "allOf": [ + { + "$ref": "#/components/schemas/AppInfo" + } + ], + "type": "object", + "properties": { + "requestedRootEpoch": { + "type": "integer", + "minimum": 0, + "maximum": 4294967295, + "description": "Omit to use the current epoch. An explicit zero selects epoch 0; future epochs are denied." + }, + "requestedAppEpoch": { + "type": "integer", + "minimum": 0, + "maximum": 4294967295, + "default": 0, + "description": "Application epoch, independent of KMS root epochs. Only zero is currently supported." + }, + "operation": { + "type": "string", + "enum": [ + "kms_bootstrap", + "kms_serve", + "kms_export_keys", + "kms_import_keys", + "app_get_key", + "app_get_cert" + ] + } + }, + "required": [ + "operation" + ] } } } diff --git a/dstack/kms/auth-eth/README.md b/dstack/kms/auth-eth/README.md index bb543b6dd..c7a8d6e7b 100644 --- a/dstack/kms/auth-eth/README.md +++ b/dstack/kms/auth-eth/README.md @@ -170,3 +170,24 @@ anvil ``` Documentation: https://book.getfoundry.sh/ + + +### Authorization endpoints + +`POST /authorizeKms` and `POST /authorizeApp` return `AuthorizationResult`, including +`rejectionReason`, `gatewayAppId`, `currentRootEpoch`, and `currentRootEpochHash`. +The selected endpoint determines the policy; the backend checks that the verified +`appId` matches that scope. `requestedAppEpoch` defaults to 0; nonzero values are currently +rejected. The optional +`requestedRootEpoch` selects historical keys; omission uses the returned current +epoch, and an explicit `0` selects epoch 0. See the +[KMS authorization protocol](../README.md#authorization-protocol) for registration, +bootstrap, and compatibility requirements. The legacy `/bootAuth/*` endpoints +remain available. + +### Authorization operations + +`/authorizeKms` and `/authorizeApp` require an `operation` string. See the +[KMS authorization protocol](../README.md#authorization-protocol) for the names +and subject semantics. Bootstrap alone may authorize before root registration; +unknown operations are rejected. Legacy `/bootAuth/*` routes are unchanged. diff --git a/dstack/kms/auth-eth/contracts/AuthorizationOperation.sol b/dstack/kms/auth-eth/contracts/AuthorizationOperation.sol new file mode 100644 index 000000000..d5f926616 --- /dev/null +++ b/dstack/kms/auth-eth/contracts/AuthorizationOperation.sol @@ -0,0 +1,24 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +pragma solidity ^0.8.24; + +/// The operation is performed by the attested subject, not the policy caller. +library AuthorizationOperation { + bytes32 internal constant KMS_BOOTSTRAP = keccak256("kms_bootstrap"); + bytes32 internal constant KMS_SERVE = keccak256("kms_serve"); + bytes32 internal constant KMS_EXPORT_KEYS = keccak256("kms_export_keys"); + bytes32 internal constant KMS_IMPORT_KEYS = keccak256("kms_import_keys"); + bytes32 internal constant APP_GET_KEY = keccak256("app_get_key"); + bytes32 internal constant APP_GET_CERT = keccak256("app_get_cert"); + + function isKms(bytes32 operation) internal pure returns (bool) { + return operation == KMS_BOOTSTRAP || operation == KMS_SERVE || operation == KMS_EXPORT_KEYS + || operation == KMS_IMPORT_KEYS; + } + + function isApp(bytes32 operation) internal pure returns (bool) { + return operation == APP_GET_KEY || operation == APP_GET_CERT; + } +} diff --git a/dstack/kms/auth-eth/contracts/DstackApp.sol b/dstack/kms/auth-eth/contracts/DstackApp.sol index 7d90f29ef..d55a6c883 100644 --- a/dstack/kms/auth-eth/contracts/DstackApp.sol +++ b/dstack/kms/auth-eth/contracts/DstackApp.sol @@ -188,7 +188,7 @@ contract DstackApp is } // Check if an app is allowed to boot - function isAppAllowed(IAppAuth.AppBootInfo calldata bootInfo) + function isAppAllowed(IAppAuth.AppInfo calldata bootInfo) external view override diff --git a/dstack/kms/auth-eth/contracts/DstackKms.sol b/dstack/kms/auth-eth/contracts/DstackKms.sol index 7168572e1..a31483e60 100644 --- a/dstack/kms/auth-eth/contracts/DstackKms.sol +++ b/dstack/kms/auth-eth/contracts/DstackKms.sol @@ -7,6 +7,7 @@ pragma solidity ^0.8.24; import "./IAppAuth.sol"; +import "./AuthorizationOperation.sol"; import "@openzeppelin/contracts-upgradeable/proxy/utils/Initializable.sol"; import "@openzeppelin/contracts-upgradeable/access/Ownable2StepUpgradeable.sol"; import "@openzeppelin/contracts-upgradeable/proxy/utils/UUPSUpgradeable.sol"; @@ -14,6 +15,28 @@ import "@openzeppelin/contracts-upgradeable/utils/introspection/ERC165Upgradeabl import "@openzeppelin/contracts/proxy/ERC1967/ERC1967Proxy.sol"; contract DstackKms is Initializable, Ownable2StepUpgradeable, UUPSUpgradeable, ERC165Upgradeable, IAppAuth { + struct OptionalUint256 { + bool isSet; + uint256 value; + } + + struct AuthorizationRequest { + AppInfo appInfo; + bytes32 operation; + // Reserved for application key rotation; only zero is currently supported. + OptionalUint256 requestedAppEpoch; + OptionalUint256 requestedRootEpoch; + } + + struct AuthorizationResult { + bool isAllowed; + string rejectionReason; + string gatewayAppId; + uint256 currentRootEpoch; + bytes32 currentRootEpochHash; + uint256 currentAppEpoch; + } + // Struct for KMS information struct KmsInfo { bytes k256Pubkey; @@ -370,8 +393,73 @@ contract DstackKms is Initializable, Ownable2StepUpgradeable, UUPSUpgradeable, E _emitPolicy("os-image", osImageHash, false); } + /// Authorize an attested identity and return the root epoch from the same state. + /// The caller selects KMS or app policy; KMS authorization requires the proxy app ID. + function authorizeKms(AuthorizationRequest calldata request) external view returns (AuthorizationResult memory) { + return _checkAuthorization(request, true); + } + + function authorizeApp(AuthorizationRequest calldata request) external view returns (AuthorizationResult memory) { + return _checkAuthorization(request, false); + } + + function _checkAuthorization( + AuthorizationRequest calldata request, + bool isKms + ) + private + view + returns (AuthorizationResult memory result) + { + result.gatewayAppId = gatewayAppId; + if (!(isKms + ? AuthorizationOperation.isKms(request.operation) + : AuthorizationOperation.isApp(request.operation))) { + result.rejectionReason = "Invalid authorization operation"; + return result; + } + if (_rootEpochs.length == 0) { + if (request.operation != AuthorizationOperation.KMS_BOOTSTRAP) { + result.rejectionReason = "Root epoch not registered"; + return result; + } + } else { + result.currentRootEpoch = _currentRootEpoch; + result.currentRootEpochHash = _rootEpochs[_currentRootEpoch].chainHash; + } + if (request.requestedRootEpoch.isSet && request.requestedRootEpoch.value > _currentRootEpoch) { + result.rejectionReason = "Requested root epoch is not active"; + return result; + } + if (request.requestedAppEpoch.isSet && request.requestedAppEpoch.value != 0) { + result.rejectionReason = "Requested app epoch must be zero"; + return result; + } + AppInfo calldata appInfo = request.appInfo; + AppInfo memory bootInfo = AppInfo({ + appId: appInfo.appId, + composeHash: appInfo.composeHash, + instanceId: appInfo.instanceId, + deviceId: appInfo.deviceId, + mrAggregated: appInfo.mrAggregated, + mrSystem: appInfo.mrSystem, + osImageHash: appInfo.osImageHash, + tcbStatus: appInfo.tcbStatus, + advisoryIds: appInfo.advisoryIds + }); + if (isKms) { + (result.isAllowed, result.rejectionReason) = _isKmsAllowed(bootInfo); + } else { + (result.isAllowed, result.rejectionReason) = _isAppAllowed(bootInfo); + } + } + // Function to check if KMS is allowed to boot - function isKmsAllowed(AppBootInfo calldata bootInfo) external view returns (bool isAllowed, string memory reason) { + function isKmsAllowed(AppInfo calldata bootInfo) external view returns (bool isAllowed, string memory reason) { + return _isKmsAllowed(bootInfo); + } + + function _isKmsAllowed(AppInfo memory bootInfo) private view returns (bool, string memory) { // Check if the TCB status is up to date if (keccak256(abi.encodePacked(bootInfo.tcbStatus)) != keccak256(abi.encodePacked("UpToDate"))) { return (false, "TCB status is not up to date"); @@ -396,12 +484,16 @@ contract DstackKms is Initializable, Ownable2StepUpgradeable, UUPSUpgradeable, E } // Function to check if an app is allowed to boot - function isAppAllowed(AppBootInfo calldata bootInfo) + function isAppAllowed(AppInfo calldata bootInfo) external view override returns (bool isAllowed, string memory reason) { + return _isAppAllowed(bootInfo); + } + + function _isAppAllowed(AppInfo memory bootInfo) private view returns (bool, string memory) { // Check if app is registered if (!registeredApps[bootInfo.appId]) { return (false, "App not registered"); diff --git a/dstack/kms/auth-eth/contracts/IAppAuth.sol b/dstack/kms/auth-eth/contracts/IAppAuth.sol index 56b768971..896c60954 100644 --- a/dstack/kms/auth-eth/contracts/IAppAuth.sol +++ b/dstack/kms/auth-eth/contracts/IAppAuth.sol @@ -33,7 +33,7 @@ interface IAppAuth is IERC165 { * @param tcbStatus Trusted Computing Base status * @param advisoryIds Array of security advisory identifiers */ - struct AppBootInfo { + struct AppInfo { address appId; bytes32 composeHash; address instanceId; @@ -52,5 +52,5 @@ interface IAppAuth is IERC165 { * @return isAllowed True if the app is authorized to boot, false otherwise * @return reason Human-readable reason for the decision (empty if allowed) */ - function isAppAllowed(AppBootInfo calldata bootInfo) external view returns (bool isAllowed, string memory reason); + function isAppAllowed(AppInfo calldata bootInfo) external view returns (bool isAllowed, string memory reason); } diff --git a/dstack/kms/auth-eth/docs/formal-verification.md b/dstack/kms/auth-eth/docs/formal-verification.md index 67d776c4a..e9987e409 100644 --- a/dstack/kms/auth-eth/docs/formal-verification.md +++ b/dstack/kms/auth-eth/docs/formal-verification.md @@ -124,7 +124,7 @@ Single-call (`test/DstackKms.symbolic.t.sol`): - [x] `check_RegisterApp_AnyCallerCanRegisterNonZeroAddress` + `_RejectsZeroAddress` — codifies the permissionless-by-design behavior; see "Findings" - [x] `check_IsAppAllowed_RejectsUnregisteredApp` / `_RejectsUnknownOsImage` - — fully symbolic `AppBootInfo`; the failing gate produces the + — fully symbolic `AppInfo`; the failing gate produces the right rejection reason without delegating - [x] `check_IsAppAllowed_DelegatesFaithfully` — when both KMS gates pass, the outer return equals the registered `IAppAuth`'s return. diff --git a/dstack/kms/auth-eth/docs/specification.md b/dstack/kms/auth-eth/docs/specification.md index 41cd2b885..01f12bf22 100644 --- a/dstack/kms/auth-eth/docs/specification.md +++ b/dstack/kms/auth-eth/docs/specification.md @@ -125,7 +125,7 @@ Signature: `deployAndRegisterApp(address initialOwner, bool disableUpgrades, boo - Semantics: equivalent to the 6-arg overload with `requireTcbUpToDate = false`. - (verified: `DstackKmsSymbolicTest.check_DeployAndRegisterApp5Arg_DefaultsTcbToFalse`) -### 3.5 `DstackKms.isAppAllowed(AppBootInfo b) external view returns (bool, string)` +### 3.5 `DstackKms.isAppAllowed(AppInfo b) external view returns (bool, string)` Decision (in order): @@ -147,7 +147,7 @@ Decision (in order): behavior of an adversarial registered app; those propagate to the outer call by EVM semantics (§5.1). -### 3.6 `DstackKms.isKmsAllowed(AppBootInfo b) external view returns (bool, string)` +### 3.6 `DstackKms.isKmsAllowed(AppInfo b) external view returns (bool, string)` Decision (in order): @@ -200,7 +200,7 @@ For each of `setGatewayAppId`, - (verified: `DstackAppSymbolicTest.check_Initialize5Arg_DefaultsTcbToFalse`, `check_Initialize6Arg_HonorsTcbFlag`) -### 3.9 `DstackApp.isAppAllowed(AppBootInfo b) external view returns (bool, string)` +### 3.9 `DstackApp.isAppAllowed(AppInfo b) external view returns (bool, string)` Decision (in order, all evaluated under `b.appId == address(this)` by convention): diff --git a/dstack/kms/auth-eth/script/Query.s.sol b/dstack/kms/auth-eth/script/Query.s.sol index 32faedec9..933d13578 100644 --- a/dstack/kms/auth-eth/script/Query.s.sol +++ b/dstack/kms/auth-eth/script/Query.s.sol @@ -124,7 +124,7 @@ contract CheckKmsAllowed is BaseQueryScript { require(address(kms) != address(0), "KMS_CONTRACT_ADDR not set"); // Read boot info from environment - IAppAuth.AppBootInfo memory bootInfo = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory bootInfo = IAppAuth.AppInfo({ appId: vm.envAddress("APP_ID"), composeHash: vm.envBytes32("COMPOSE_HASH"), instanceId: vm.envOr("INSTANCE_ID", address(0)), @@ -150,7 +150,7 @@ contract CheckAppAllowed is BaseQueryScript { require(address(kms) != address(0), "KMS_CONTRACT_ADDR not set"); // Read boot info from environment - IAppAuth.AppBootInfo memory bootInfo = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory bootInfo = IAppAuth.AppInfo({ appId: vm.envAddress("APP_ID"), composeHash: vm.envBytes32("COMPOSE_HASH"), instanceId: vm.envOr("INSTANCE_ID", address(0)), diff --git a/dstack/kms/auth-eth/src/ethereum.ts b/dstack/kms/auth-eth/src/ethereum.ts index 6eab5d874..2aedc8b54 100644 --- a/dstack/kms/auth-eth/src/ethereum.ts +++ b/dstack/kms/auth-eth/src/ethereum.ts @@ -3,10 +3,12 @@ // SPDX-License-Identifier: Apache-2.0 import { ethers } from 'ethers'; -import { BootInfo, BootResponse, RootEpochInfo } from './types'; +import { BootInfo, BootResponse, RootEpochInfo, AuthorizationResult, AuthorizationRequest } from './types'; // Minimal ABI for DstackKms contract const DSTACK_KMS_ABI = [ + "function authorizeKms(((address appId,bytes32 composeHash,address instanceId,bytes32 deviceId,bytes32 mrAggregated,bytes32 mrSystem,bytes32 osImageHash,string tcbStatus,string[] advisoryIds) appInfo,bytes32 operation,(bool isSet,uint256 value) requestedAppEpoch,(bool isSet,uint256 value) requestedRootEpoch) request) view returns ((bool isAllowed,string rejectionReason,string gatewayAppId,uint256 currentRootEpoch,bytes32 currentRootEpochHash,uint256 currentAppEpoch))", + "function authorizeApp(((address appId,bytes32 composeHash,address instanceId,bytes32 deviceId,bytes32 mrAggregated,bytes32 mrSystem,bytes32 osImageHash,string tcbStatus,string[] advisoryIds) appInfo,bytes32 operation,(bool isSet,uint256 value) requestedAppEpoch,(bool isSet,uint256 value) requestedRootEpoch) request) view returns ((bool isAllowed,string rejectionReason,string gatewayAppId,uint256 currentRootEpoch,bytes32 currentRootEpochHash,uint256 currentAppEpoch))", "function isAppAllowed((address appId,bytes32 composeHash,address instanceId,bytes32 deviceId,bytes32 mrAggregated,bytes32 mrSystem,bytes32 osImageHash,string tcbStatus,string[] advisoryIds) bootInfo) view returns (bool, string)", "function isKmsAllowed((address appId,bytes32 composeHash,address instanceId,bytes32 deviceId,bytes32 mrAggregated,bytes32 mrSystem,bytes32 osImageHash,string tcbStatus,string[] advisoryIds) bootInfo) view returns (bool, string)", "function gatewayAppId() view returns (string)", @@ -61,9 +63,9 @@ export class EthereumBackend { return '0x' + hex; } - async checkBoot(bootInfo: BootInfo, isKms: boolean): Promise { + private contractAppInfo(bootInfo: BootInfo) { // Create boot info struct for contract call - const bootInfoStruct = { + return { appId: this.decodeHex(bootInfo.appId, 20), instanceId: this.decodeHex(bootInfo.instanceId, 20), composeHash: this.decodeHex(bootInfo.composeHash, 32), @@ -74,6 +76,27 @@ export class EthereumBackend { tcbStatus: bootInfo.tcbStatus, advisoryIds: bootInfo.advisoryIds }; + } + + async checkAuthorization(appInfo: AuthorizationRequest, method: 'authorizeKms' | 'authorizeApp'): Promise { + const result = await this.kmsContract[method]({ + appInfo: this.contractAppInfo(appInfo), + operation: ethers.id(appInfo.operation), + requestedAppEpoch: { isSet: appInfo.requestedAppEpoch !== undefined, value: appInfo.requestedAppEpoch ?? 0 }, + requestedRootEpoch: { isSet: appInfo.requestedRootEpoch !== undefined, value: appInfo.requestedRootEpoch ?? 0 }, + }); + return { + isAllowed: result.isAllowed, + rejectionReason: result.rejectionReason, + gatewayAppId: result.gatewayAppId, + currentRootEpoch: ethers.toNumber(result.currentRootEpoch), + currentRootEpochHash: result.currentRootEpochHash, + currentAppEpoch: ethers.toNumber(result.currentAppEpoch), + }; + } + + async checkBoot(bootInfo: BootInfo, isKms: boolean): Promise { + const bootInfoStruct = this.contractAppInfo(bootInfo); // Read the whole decision from one block. const blockTag = await this.provider.getBlockNumber(); let response; diff --git a/dstack/kms/auth-eth/src/main.test.ts b/dstack/kms/auth-eth/src/main.test.ts index e7ad49739..06884976e 100644 --- a/dstack/kms/auth-eth/src/main.test.ts +++ b/dstack/kms/auth-eth/src/main.test.ts @@ -91,6 +91,33 @@ describe('Server', () => { }); }); + describe('POST /authorizeApp', () => { + const appRequest = { ...mockBootInfo, operation: 'app_get_key', requestedAppEpoch: 0 }; + it('preserves the complete decision, including the epoch on rejection', async () => { + const decision = { + isAllowed: false, rejectionReason: 'App not registered', gatewayAppId: 'gateway', + currentAppEpoch: 0, currentRootEpoch: 1, currentRootEpochHash: '0x' + '11'.repeat(32), + }; + app.ethereum.checkAuthorization = jest.fn().mockResolvedValue(decision); + const response = await app.inject({ method: 'POST', url: '/authorizeApp', payload: appRequest }); + expect(response.statusCode).toBe(200); + expect(JSON.parse(response.payload)).toEqual(decision); + expect(app.ethereum.checkAuthorization).toHaveBeenCalledWith(appRequest, 'authorizeApp'); + await app.inject({ method: 'POST', url: '/authorizeKms', payload: appRequest }); + expect(app.ethereum.checkAuthorization).toHaveBeenLastCalledWith(appRequest, 'authorizeKms'); + await app.inject({ method: 'POST', url: '/authorizeApp', payload: { ...appRequest, requestedRootEpoch: 0 } }); + expect(app.ethereum.checkAuthorization).toHaveBeenLastCalledWith({ ...appRequest, requestedRootEpoch: 0 }, 'authorizeApp'); + }); + + it('fails closed when the new contract method is unavailable', async () => { + jest.clearAllMocks(); + app.ethereum.checkAuthorization = jest.fn().mockRejectedValue(new Error('missing contract method')); + const response = await app.inject({ method: 'POST', url: '/authorizeApp', payload: appRequest }); + expect(response.statusCode).toBe(500); + expect(app.ethereum.checkBoot).not.toHaveBeenCalled(); + }); + }); + describe('key epochs', () => { it('should pass the KMS root epoch through to the backend', async () => { const mockCheckBoot = jest.fn().mockResolvedValue({ isAllowed: true, reason: '', gatewayAppId: '' }); diff --git a/dstack/kms/auth-eth/src/server.ts b/dstack/kms/auth-eth/src/server.ts index 33c80d1d5..e3d81424a 100644 --- a/dstack/kms/auth-eth/src/server.ts +++ b/dstack/kms/auth-eth/src/server.ts @@ -4,7 +4,7 @@ import fastify, { FastifyInstance } from 'fastify'; import { EthereumBackend } from './ethereum'; -import { BootInfo, BootResponse } from './types'; +import { BootInfo, BootResponse, AuthorizationResult, AuthorizationRequest } from './types'; import { ethers } from 'ethers'; declare module 'fastify' { @@ -25,8 +25,7 @@ export async function build(): Promise { description, }); - server.addSchema({ - $id: 'bootInfo', + const appInfoSchema = { type: 'object', required: ['mrAggregated', 'osImageHash', 'appId', 'composeHash', 'instanceId', 'deviceId'], properties: { @@ -39,9 +38,28 @@ export async function build(): Promise { tcbStatus: { type: 'string', maxLength: 128, default: '' }, advisoryIds: { type: 'array', maxItems: 128, items: { type: 'string', maxLength: 256 }, default: [] }, mrSystem: { ...hex(32, 'System MR measurement'), default: '' }, + } + }; + + server.addSchema({ + ...appInfoSchema, + $id: 'bootInfo', + properties: { + ...appInfoSchema.properties, rootEpoch: { type: 'integer', minimum: 0 }, rootEpochHash: hex(32, 'Chain hash of the root epochs up to rootEpoch'), - } + }, + }); + server.addSchema({ + ...appInfoSchema, + $id: 'authorizationRequest', + required: [...appInfoSchema.required, 'operation'], + properties: { + ...appInfoSchema.properties, + operation: { type: 'string', enum: ['kms_bootstrap', 'kms_serve', 'kms_export_keys', 'kms_import_keys', 'app_get_key', 'app_get_cert'] }, + requestedAppEpoch: { type: 'integer', minimum: 0, maximum: 4294967295, default: 0 }, + requestedRootEpoch: { type: 'integer', minimum: 0, maximum: 4294967295 }, + }, }); server.addSchema({ @@ -104,6 +122,28 @@ export async function build(): Promise { } }); + for (const method of ['authorizeKms', 'authorizeApp'] as const) { + server.post<{ Body: AuthorizationRequest; Reply: AuthorizationResult }>(`/${method}`, { + schema: { + body: { $ref: 'authorizationRequest#' }, + response: { + 200: { + type: 'object', + required: ['isAllowed', 'rejectionReason', 'gatewayAppId', 'currentRootEpoch', 'currentRootEpochHash', 'currentAppEpoch'], + properties: { + isAllowed: { type: 'boolean' }, + rejectionReason: { type: 'string' }, + gatewayAppId: { type: 'string' }, + currentRootEpoch: { type: 'integer', minimum: 0 }, + currentAppEpoch: { type: 'integer', minimum: 0 }, + currentRootEpochHash: hex(32, 'Current root epoch chain hash'), + }, + }, + }, + }, + }, async (request) => server.ethereum.checkAuthorization(request.body, method)); + } + // Define routes server.post<{ Body: BootInfo; diff --git a/dstack/kms/auth-eth/src/types.ts b/dstack/kms/auth-eth/src/types.ts index c92edcfe0..8f7223c6a 100644 --- a/dstack/kms/auth-eth/src/types.ts +++ b/dstack/kms/auth-eth/src/types.ts @@ -32,3 +32,18 @@ export interface RootEpochInfo { } // Removed KMS_CONTRACT_ABI and APP_CONTRACT_ABI since we're using typechain types now + +export interface AuthorizationResult { + isAllowed: boolean; + rejectionReason: string; + gatewayAppId: string; + currentRootEpoch: number; + currentRootEpochHash: string; + currentAppEpoch: number; +} + +export interface AuthorizationRequest extends Omit { + operation: 'kms_bootstrap' | 'kms_serve' | 'kms_export_keys' | 'kms_import_keys' | 'app_get_key' | 'app_get_cert'; + requestedAppEpoch?: number; + requestedRootEpoch?: number; +} diff --git a/dstack/kms/auth-eth/test/DstackApp.t.sol b/dstack/kms/auth-eth/test/DstackApp.t.sol index 9f590d2e3..6776e0b2d 100644 --- a/dstack/kms/auth-eth/test/DstackApp.t.sol +++ b/dstack/kms/auth-eth/test/DstackApp.t.sol @@ -170,7 +170,7 @@ contract DstackAppTest is Test { app.addComposeHash(composeHash); vm.stopPrank(); - IAppAuth.AppBootInfo memory bootInfo = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory bootInfo = IAppAuth.AppInfo({ appId: address(app), composeHash: composeHash, instanceId: address(0), @@ -212,7 +212,7 @@ contract DstackAppTest is Test { app.addComposeHash(allowedHash); vm.stopPrank(); - IAppAuth.AppBootInfo memory bootInfo = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory bootInfo = IAppAuth.AppInfo({ appId: address(app), composeHash: unallowedHash, instanceId: address(0), @@ -270,7 +270,7 @@ contract DstackAppTest is Test { string[] memory advisoryIds = new string[](1); advisoryIds[0] = "INTEL-SA-00614"; - IAppAuth.AppBootInfo memory bootInfo = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory bootInfo = IAppAuth.AppInfo({ appId: address(app), composeHash: composeHash, instanceId: address(0), @@ -318,7 +318,7 @@ contract DstackAppTest is Test { assertTrue(tcbApp.allowedDeviceIds(deviceId)); assertTrue(tcbApp.allowedComposeHashes(composeHash)); - IAppAuth.AppBootInfo memory bootInfo = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory bootInfo = IAppAuth.AppInfo({ appId: address(tcbApp), composeHash: composeHash, instanceId: address(0), diff --git a/dstack/kms/auth-eth/test/DstackKms.symbolic.t.sol b/dstack/kms/auth-eth/test/DstackKms.symbolic.t.sol index 47b79af2c..ad9f7ac63 100644 --- a/dstack/kms/auth-eth/test/DstackKms.symbolic.t.sol +++ b/dstack/kms/auth-eth/test/DstackKms.symbolic.t.sol @@ -32,7 +32,7 @@ contract MockConfigurableApp { _ret = ret; } - function isAppAllowed(IAppAuth.AppBootInfo calldata) external view returns (bool, string memory) { + function isAppAllowed(IAppAuth.AppInfo calldata) external view returns (bool, string memory) { return (_ret, ""); } } @@ -41,7 +41,7 @@ contract MockConfigurableApp { /// to verify that a malicious registered contract's revert /// propagates out of `kms.isAppAllowed` rather than being swallowed. contract MockRevertingApp { - function isAppAllowed(IAppAuth.AppBootInfo calldata) external pure returns (bool, string memory) { + function isAppAllowed(IAppAuth.AppInfo calldata) external pure returns (bool, string memory) { revert("malicious app revert"); } } @@ -87,14 +87,14 @@ contract DstackKmsSymbolicTest is Test { // the (false, reason) return without delegating. // --------------------------------------------------------------- - function check_IsAppAllowed_RejectsUnregisteredApp(IAppAuth.AppBootInfo calldata bootInfo) external view { + function check_IsAppAllowed_RejectsUnregisteredApp(IAppAuth.AppInfo calldata bootInfo) external view { // No apps registered in setUp(), so any bootInfo.appId is rejected. (bool allowed, string memory reason) = kms.isAppAllowed(bootInfo); assert(!allowed); assert(keccak256(bytes(reason)) == keccak256(bytes("App not registered"))); } - function check_IsAppAllowed_RejectsUnknownOsImage(address appId, IAppAuth.AppBootInfo calldata bootInfo) external { + function check_IsAppAllowed_RejectsUnknownOsImage(address appId, IAppAuth.AppInfo calldata bootInfo) external { vm.assume(appId != address(0)); // Register the appId so the registration check passes. @@ -128,7 +128,7 @@ contract DstackKmsSymbolicTest is Test { kms.addOsImageHash(osImageHash); vm.stopPrank(); - IAppAuth.AppBootInfo memory b = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory b = IAppAuth.AppInfo({ appId: address(mock), composeHash: bytes32(0), instanceId: address(0), @@ -158,7 +158,7 @@ contract DstackKmsSymbolicTest is Test { kms.addOsImageHash(osImageHash); vm.stopPrank(); - IAppAuth.AppBootInfo memory b = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory b = IAppAuth.AppInfo({ appId: address(mock), composeHash: bytes32(0), instanceId: address(0), @@ -192,7 +192,7 @@ contract DstackKmsSymbolicTest is Test { vm.prank(OWNER); kms.addOsImageHash(osImageHash); - IAppAuth.AppBootInfo memory b = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory b = IAppAuth.AppInfo({ appId: address(0), composeHash: bytes32(0), instanceId: address(0), @@ -222,7 +222,7 @@ contract DstackKmsSymbolicTest is Test { vm.stopPrank(); // kmsAllowedDeviceIds is left empty. - IAppAuth.AppBootInfo memory b = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory b = IAppAuth.AppInfo({ appId: address(0), composeHash: bytes32(0), instanceId: address(0), diff --git a/dstack/kms/auth-eth/test/DstackKms.t.sol b/dstack/kms/auth-eth/test/DstackKms.t.sol index 037374aec..138759983 100644 --- a/dstack/kms/auth-eth/test/DstackKms.t.sol +++ b/dstack/kms/auth-eth/test/DstackKms.t.sol @@ -442,6 +442,164 @@ contract DstackKmsTest is Test { vm.stopPrank(); } + function authorizationInfo(address appId) internal pure returns (IAppAuth.AppInfo memory) { + return IAppAuth.AppInfo({ + appId: appId, + composeHash: bytes32(uint256(789)), + instanceId: address(0), + deviceId: bytes32(uint256(123)), + mrAggregated: bytes32(uint256(456)), + mrSystem: bytes32(0), + osImageHash: bytes32(uint256(111)), + tcbStatus: "UpToDate", + advisoryIds: new string[](0) + }); + } + + function authorize( + IAppAuth.AppInfo memory info, + bool isKms + ) + internal + view + returns (DstackKms.AuthorizationResult memory) + { + DstackKms.AuthorizationRequest memory request = DstackKms.AuthorizationRequest( + info, + isKms ? keccak256("kms_serve") : keccak256("app_get_key"), + DstackKms.OptionalUint256(false, 0), + DstackKms.OptionalUint256(false, 0) + ); + return isKms ? kms.authorizeKms(request) : kms.authorizeApp(request); + } + + function test_AuthorizationOperationsFailClosedAndBootstrapBeforeRegistration() public { + IAppAuth.AppInfo memory info = authorizationInfo(address(kms)); + vm.startPrank(owner); + kms.addOsImageHash(info.osImageHash); + kms.addKmsDevice(info.deviceId); + kms.addKmsAggregatedMr(info.mrAggregated); + vm.stopPrank(); + DstackKms.AuthorizationRequest memory request = DstackKms.AuthorizationRequest( + info, keccak256("kms_bootstrap"), DstackKms.OptionalUint256(false, 0), DstackKms.OptionalUint256(false, 0) + ); + assertTrue(kms.authorizeKms(request).isAllowed); + assertEq(kms.authorizeKms(request).currentRootEpochHash, bytes32(0)); + request.operation = keccak256("kms_export_keys"); + assertEq(kms.authorizeKms(request).rejectionReason, "Root epoch not registered"); + vm.prank(owner); + kms.registerRootEpoch(ca(0xa0), k256(0xb0), "https://example.com/epoch-0"); + assertTrue(kms.authorizeKms(request).isAllowed); + request.operation = keccak256("kms_import_keys"); + assertTrue(kms.authorizeKms(request).isAllowed); + assertEq(kms.authorizeApp(request).rejectionReason, "Invalid authorization operation"); + request.operation = keccak256("app_get_cert"); + assertEq(kms.authorizeKms(request).rejectionReason, "Invalid authorization operation"); + request.operation = bytes32(0); + assertEq(kms.authorizeKms(request).rejectionReason, "Invalid authorization operation"); + assertEq(kms.authorizeApp(request).rejectionReason, "Invalid authorization operation"); + } + + function test_CheckAuthorization_OptionalUint256DistinguishesZeroFromDefault() public { + IAppAuth.AppInfo memory info = authorizationInfo(address(kms)); + vm.startPrank(owner); + kms.registerRootEpoch(ca(0xa0), k256(0xb0), "https://example.com/epoch-0"); + kms.registerRootEpoch(ca(0xa1), k256(0xb1), "https://example.com/epoch-1"); + kms.addOsImageHash(info.osImageHash); + kms.addKmsDevice(info.deviceId); + kms.addKmsAggregatedMr(info.mrAggregated); + vm.stopPrank(); + + DstackKms.AuthorizationRequest memory request = DstackKms.AuthorizationRequest( + info, keccak256("kms_serve"), DstackKms.OptionalUint256(false, 0), DstackKms.OptionalUint256(true, 1) + ); + DstackKms.AuthorizationResult memory result = kms.authorizeKms(request); + assertFalse(result.isAllowed); + assertEq(result.rejectionReason, "Requested root epoch is not active"); + bytes32 h1 = chainHash(chainHash(bytes32(0), ca(0xa0), k256(0xb0)), ca(0xa1), k256(0xb1)); + vm.prank(owner); + kms.activateRootEpoch(1, h1); + assertTrue(kms.authorizeKms(request).isAllowed); + request.requestedRootEpoch.value = 0; + assertTrue(kms.authorizeKms(request).isAllowed); + request.requestedRootEpoch.value = type(uint256).max; + assertFalse(kms.authorizeKms(request).isAllowed); + request.requestedRootEpoch.isSet = false; + result = kms.authorizeKms(request); + assertTrue(result.isAllowed); + assertEq(result.currentRootEpoch, 1); + assertEq(result.currentRootEpochHash, h1); + } + + function test_CheckAuthorization_UnregisteredEpochReturnsRejection() public view { + DstackKms.AuthorizationResult memory result = authorize(authorizationInfo(address(kms)), true); + assertFalse(result.isAllowed); + assertEq(result.rejectionReason, "Root epoch not registered"); + assertEq(result.currentRootEpoch, 0); + assertEq(result.currentRootEpochHash, bytes32(0)); + } + + function test_CheckAuthorization_ValidatesRequestedPolicy() public { + IAppAuth.AppInfo memory info = authorizationInfo(address(kms)); + vm.startPrank(owner); + kms.registerRootEpoch(ca(0xa0), k256(0xb0), "https://example.com/epoch-0"); + kms.setGatewayAppId("gateway"); + kms.addOsImageHash(info.osImageHash); + kms.addKmsDevice(info.deviceId); + kms.addKmsAggregatedMr(info.mrAggregated); + address appId = kms.deployAndRegisterApp(user, false, false, info.deviceId, info.composeHash); + vm.stopPrank(); + + DstackKms.AuthorizationResult memory result = authorize(info, true); + assertTrue(result.isAllowed); + assertEq(result.rejectionReason, ""); + assertEq(result.gatewayAppId, "gateway"); + assertEq(result.currentRootEpochHash, chainHash(bytes32(0), ca(0xa0), k256(0xb0))); + + info.appId = user; + result = authorize(info, false); + assertFalse(result.isAllowed); + assertEq(result.rejectionReason, "App not registered"); + + info.appId = appId; + info.mrAggregated = bytes32(0); + result = authorize(info, false); + assertTrue(result.isAllowed); + result = authorize(info, true); + assertFalse(result.isAllowed); + assertEq(result.rejectionReason, "Aggregated MR not allowed"); + DstackKms.AuthorizationRequest memory appRequest = DstackKms.AuthorizationRequest( + info, keccak256("app_get_key"), DstackKms.OptionalUint256(true, 1), DstackKms.OptionalUint256(false, 0) + ); + result = kms.authorizeApp(appRequest); + assertFalse(result.isAllowed); + assertEq(result.rejectionReason, "Requested app epoch must be zero"); + assertEq(result.currentRootEpochHash, chainHash(bytes32(0), ca(0xa0), k256(0xb0))); + info.appId = address(kms); + result = authorize(info, false); + assertFalse(result.isAllowed); + assertEq(result.rejectionReason, "App not registered"); + result = authorize(info, true); + assertFalse(result.isAllowed); + assertEq(result.rejectionReason, "Aggregated MR not allowed"); + } + + function test_CheckAuthorization_RejectionIncludesActivatedEpoch() public { + vm.startPrank(owner); + kms.registerRootEpoch(ca(0xa0), k256(0xb0), "https://example.com/epoch-0"); + kms.registerRootEpoch(ca(0xa1), k256(0xb1), "https://example.com/epoch-1"); + kms.activateRootEpoch(1, chainHash(chainHash(bytes32(0), ca(0xa0), k256(0xb0)), ca(0xa1), k256(0xb1))); + vm.stopPrank(); + + DstackKms.AuthorizationResult memory result = authorize(authorizationInfo(user), false); + assertFalse(result.isAllowed); + assertEq(result.rejectionReason, "App not registered"); + assertEq(result.currentRootEpoch, 1); + assertEq( + result.currentRootEpochHash, chainHash(chainHash(bytes32(0), ca(0xa0), k256(0xb0)), ca(0xa1), k256(0xb1)) + ); + } + function test_IsKmsAllowed() public { bytes32 deviceId = bytes32(uint256(123)); bytes32 mrAggregated = bytes32(uint256(456)); @@ -454,7 +612,7 @@ contract DstackKmsTest is Test { kms.addOsImageHash(osImageHash); vm.stopPrank(); - IAppAuth.AppBootInfo memory bootInfo = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory bootInfo = IAppAuth.AppInfo({ appId: address(0), composeHash: bytes32(0), instanceId: address(0), @@ -483,7 +641,7 @@ contract DstackKmsTest is Test { kms.addOsImageHash(osImageHash); vm.stopPrank(); - IAppAuth.AppBootInfo memory bootInfo = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory bootInfo = IAppAuth.AppInfo({ appId: address(0), composeHash: bytes32(0), instanceId: address(0), @@ -511,7 +669,7 @@ contract DstackKmsTest is Test { address appId = kms.deployAndRegisterApp(user, false, false, deviceId, composeHash); vm.stopPrank(); - IAppAuth.AppBootInfo memory bootInfo = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory bootInfo = IAppAuth.AppInfo({ appId: appId, composeHash: composeHash, instanceId: address(0), diff --git a/dstack/kms/auth-eth/test/UpgradesWithPlugin.t.sol b/dstack/kms/auth-eth/test/UpgradesWithPlugin.t.sol index 0cb05013d..770482fba 100644 --- a/dstack/kms/auth-eth/test/UpgradesWithPlugin.t.sol +++ b/dstack/kms/auth-eth/test/UpgradesWithPlugin.t.sol @@ -284,7 +284,7 @@ contract UpgradesWithPluginTest is Test { assertTrue(app.allowAnyDevice()); assertTrue(app.allowedComposeHashes(composeHash)); - IAppAuth.AppBootInfo memory bootInfo = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory bootInfo = IAppAuth.AppInfo({ appId: appId, composeHash: composeHash, instanceId: address(0), @@ -335,7 +335,7 @@ contract UpgradesWithPluginTest is Test { assertFalse(app.requireTcbUpToDate()); // Outdated TCB still allowed when flag is off - IAppAuth.AppBootInfo memory bootInfo = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory bootInfo = IAppAuth.AppInfo({ appId: appId, composeHash: composeHash, instanceId: address(0), @@ -399,7 +399,7 @@ contract UpgradesWithPluginTest is Test { assertEq(app.owner(), owner); // Test the full flow still works - IAppAuth.AppBootInfo memory bootInfo = IAppAuth.AppBootInfo({ + IAppAuth.AppInfo memory bootInfo = IAppAuth.AppInfo({ appId: appId, composeHash: composeHash, instanceId: address(0), diff --git a/dstack/kms/auth-mock/README.md b/dstack/kms/auth-mock/README.md index c44c207e8..4ddb11646 100644 --- a/dstack/kms/auth-mock/README.md +++ b/dstack/kms/auth-mock/README.md @@ -167,3 +167,24 @@ to switch from mock to real backend: 4. update environment variables the API remains identical, so no client code changes are needed. + + +### Authorization endpoints + +`POST /authorizeKms` and `POST /authorizeApp` return `AuthorizationResult`, including +`rejectionReason`, `gatewayAppId`, `currentRootEpoch`, and `currentRootEpochHash`. +The selected endpoint determines the policy; the backend checks that the verified +`appId` matches that scope. `requestedAppEpoch` defaults to 0; nonzero values are currently +rejected. The optional +`requestedRootEpoch` selects historical keys; omission uses the returned current +epoch, and an explicit `0` selects epoch 0. See the +[KMS authorization protocol](../README.md#authorization-protocol) for registration, +bootstrap, and compatibility requirements. The legacy `/bootAuth/*` endpoints +remain available. + +### Authorization operations + +`/authorizeKms` and `/authorizeApp` require an `operation` string. See the +[KMS authorization protocol](../README.md#authorization-protocol) for the names +and subject semantics. Bootstrap alone may authorize before root registration; +unknown operations are rejected. Legacy `/bootAuth/*` routes are unchanged. diff --git a/dstack/kms/auth-mock/index.test.ts b/dstack/kms/auth-mock/index.test.ts index 68a3cfbf4..d244b938f 100644 --- a/dstack/kms/auth-mock/index.test.ts +++ b/dstack/kms/auth-mock/index.test.ts @@ -382,3 +382,32 @@ describe('Mock Behavior Verification', () => { } }); }); + + +describe('authorization endpoints', () => { + const authorize = (appId: string, isKms = false, requestedAppEpoch = 0) => appFetch(new Request(`http://localhost/${isKms ? 'authorizeKms' : 'authorizeApp'}`, { + method: 'POST', headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ appId, operation: isKms ? 'kms_serve' : 'app_get_key', requestedAppEpoch, mrAggregated: '', osImageHash: '', composeHash: '', instanceId: '', deviceId: '' }), + })); + + it('requires registered roots and selects policy by app ID', async () => { + vi.stubEnv('MOCK_ROOT_EPOCH', ''); + vi.stubEnv('MOCK_ROOT_EPOCH_HASH', ''); + vi.stubEnv('MOCK_POLICY', 'deny-kms'); + try { + expect(await (await authorize('app')).json()).toMatchObject({ + isAllowed: false, rejectionReason: 'Root epoch not registered', + currentRootEpoch: 0, currentRootEpochHash: '0x' + '00'.repeat(32), + }); + vi.stubEnv('MOCK_ROOT_EPOCH', '1'); + vi.stubEnv('MOCK_ROOT_EPOCH_HASH', '0x' + '11'.repeat(32)); + expect(await (await authorize('app')).json()).toMatchObject({ isAllowed: true, rejectionReason: '', currentRootEpoch: 1 }); + expect(await (await authorize(process.env.KMS_CONTRACT_ADDR!, true)).json()).toMatchObject({ + isAllowed: false, rejectionReason: 'mock policy: deny-kms', currentRootEpoch: 1, + currentRootEpochHash: '0x' + '11'.repeat(32), + }); + } finally { + vi.unstubAllEnvs(); + } + }); +}); diff --git a/dstack/kms/auth-mock/index.ts b/dstack/kms/auth-mock/index.ts index 6d01a947f..518b3e334 100644 --- a/dstack/kms/auth-mock/index.ts +++ b/dstack/kms/auth-mock/index.ts @@ -29,6 +29,13 @@ const BootInfoSchema = z.object({ rootEpochHash: z.string().optional() }); +const AuthorizationRequestSchema = BootInfoSchema.omit({ rootEpoch: true, rootEpochHash: true }).extend({ + operation: z.enum(['kms_bootstrap', 'kms_serve', 'kms_export_keys', 'kms_import_keys', 'app_get_key', 'app_get_cert']), + requestedAppEpoch: z.number().int().min(0).max(0xffffffff).default(0), + requestedRootEpoch: z.number().int().min(0).max(0xffffffff).optional(), +}); +type AuthorizationRequest = z.infer; + const BootResponseSchema = z.object({ isAllowed: z.boolean(), reason: z.string(), @@ -106,6 +113,31 @@ class MockBackend { this.mockAppImplementation = process.env.MOCK_APP_IMPLEMENTATION || '0xmockapp9876543210987654321098765432109'; } + async checkAuthorization(appInfo: AuthorizationRequest, isKms: boolean) { + const current = getRootEpoch(); + const epoch = { + currentAppEpoch: 0, + currentRootEpoch: current.rootEpoch ?? 0, + currentRootEpochHash: current.rootEpochHash ?? '0x' + '00'.repeat(32), + }; + const kmsOperation = ['kms_bootstrap', 'kms_serve', 'kms_export_keys', 'kms_import_keys'].includes(appInfo.operation); + if (kmsOperation !== isKms) { + return { isAllowed: false, rejectionReason: 'Invalid authorization operation', gatewayAppId: this.mockGatewayAppId, ...epoch }; + } + if (current.rootEpoch === undefined && appInfo.operation !== 'kms_bootstrap') { + return { isAllowed: false, rejectionReason: 'Root epoch not registered', gatewayAppId: this.mockGatewayAppId, ...epoch }; + } + if (appInfo.requestedRootEpoch !== undefined && appInfo.requestedRootEpoch > (current.rootEpoch ?? 0)) { + return { isAllowed: false, rejectionReason: 'Requested root epoch is not active', gatewayAppId: this.mockGatewayAppId, ...epoch }; + } + if (appInfo.requestedAppEpoch !== 0) { + return { isAllowed: false, rejectionReason: 'Requested app epoch must be zero', gatewayAppId: this.mockGatewayAppId, ...epoch }; + } + const result = this.evaluateBoot(appInfo, isKms); + return { isAllowed: result.isAllowed, rejectionReason: result.isAllowed ? '' : result.reason, + gatewayAppId: this.mockGatewayAppId, ...epoch }; + } + async checkBoot(bootInfo: BootInfo, isKms: boolean): Promise { const response = this.evaluateBoot(bootInfo, isKms); const epochDenial = response.isAllowed ? rootEpochDenial(bootInfo, isKms) : undefined; @@ -202,6 +234,16 @@ app.get('/', async (c) => { } }); +for (const method of ['authorizeKms', 'authorizeApp'] as const) { + app.post(`/${method}`, zValidator('json', AuthorizationRequestSchema), async (c) => { + try { + return c.json(await mockBackend.checkAuthorization(c.req.valid('json'), method === 'authorizeKms')); + } catch { + return c.json({ status: 'error', message: 'authorization backend unavailable' }, 503); + } + }); +} + // app boot authentication app.post('/bootAuth/app', zValidator('json', BootInfoSchema), diff --git a/dstack/kms/auth-mock/openapi.json b/dstack/kms/auth-mock/openapi.json index 6a35cbbc3..73b1b16c5 100644 --- a/dstack/kms/auth-mock/openapi.json +++ b/dstack/kms/auth-mock/openapi.json @@ -98,6 +98,68 @@ } } } + }, + "/authorizeKms": { + "post": { + "summary": "Authorize a Kms identity and return the current root epoch", + "operationId": "authorizeKms", + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/AuthorizationRequest" + } + } + } + }, + "responses": { + "200": { + "description": "Authorization decision, including the epoch on rejection.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/AuthorizationResult" + } + } + } + }, + "503": { + "description": "Authorization backend unavailable; no legacy fallback." + } + } + } + }, + "/authorizeApp": { + "post": { + "summary": "Authorize a App identity and return the current root epoch", + "operationId": "authorizeApp", + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/AuthorizationRequest" + } + } + } + }, + "responses": { + "200": { + "description": "Authorization decision, including the epoch on rejection.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/AuthorizationResult" + } + } + } + }, + "503": { + "description": "Authorization backend unavailable; no legacy fallback." + } + } + } } }, "components": { @@ -201,7 +263,9 @@ "properties": { "status": { "type": "string", - "enum": ["ok"], + "enum": [ + "ok" + ], "description": "System status" }, "kmsContractAddr": { @@ -229,8 +293,14 @@ "description": "Mock app implementation address", "example": "0xmockapp9876543210987654321098765432109" }, - "rootEpoch": { "type": "integer", "description": "MOCK_ROOT_EPOCH" }, - "rootEpochHash": { "type": "string", "description": "MOCK_ROOT_EPOCH_HASH" }, + "rootEpoch": { + "type": "integer", + "description": "MOCK_ROOT_EPOCH" + }, + "rootEpochHash": { + "type": "string", + "description": "MOCK_ROOT_EPOCH_HASH" + }, "note": { "type": "string", "description": "Note about mock backend behavior", @@ -247,13 +317,142 @@ "properties": { "status": { "type": "string", - "enum": ["error"] + "enum": [ + "error" + ] }, "message": { "type": "string", "description": "Error message" } } + }, + "AppInfo": { + "type": "object", + "required": [ + "mrAggregated", + "osImageHash", + "appId", + "composeHash", + "instanceId", + "deviceId" + ], + "properties": { + "mrAggregated": { + "type": "string", + "description": "Aggregated MR measurement" + }, + "osImageHash": { + "type": "string", + "description": "OS Image hash" + }, + "appId": { + "type": "string", + "description": "Application ID" + }, + "composeHash": { + "type": "string", + "description": "Compose hash" + }, + "instanceId": { + "type": "string", + "description": "Instance ID" + }, + "deviceId": { + "type": "string", + "description": "Device ID" + }, + "tcbStatus": { + "type": "string", + "description": "TCB status (optional for compatibility)" + }, + "advisoryIds": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Advisory IDs (optional for compatibility)" + }, + "mrSystem": { + "type": "string", + "description": "MR System measurement (optional for compatibility)" + } + }, + "description": "Verified attestation identity. The KMS app ID is the KMS contract address." + }, + "AuthorizationResult": { + "type": "object", + "required": [ + "isAllowed", + "rejectionReason", + "gatewayAppId", + "currentRootEpoch", + "currentRootEpochHash", + "currentAppEpoch" + ], + "properties": { + "isAllowed": { + "type": "boolean" + }, + "rejectionReason": { + "type": "string", + "description": "Empty when allowed." + }, + "gatewayAppId": { + "type": "string" + }, + "currentRootEpoch": { + "type": "integer", + "minimum": 0 + }, + "currentRootEpochHash": { + "type": "string", + "pattern": "^0x[0-9a-fA-F]{64}$", + "description": "Zero when no epoch is registered; authorization is denied." + }, + "currentAppEpoch": { + "type": "integer", + "minimum": 0, + "description": "Current application epoch; zero until app rotation is supported." + } + } + }, + "AuthorizationRequest": { + "allOf": [ + { + "$ref": "#/components/schemas/AppInfo" + } + ], + "type": "object", + "properties": { + "requestedRootEpoch": { + "type": "integer", + "minimum": 0, + "maximum": 4294967295, + "description": "Omit to use the current epoch. An explicit zero selects epoch 0; future epochs are denied." + }, + "requestedAppEpoch": { + "type": "integer", + "minimum": 0, + "maximum": 4294967295, + "default": 0, + "description": "Application epoch, independent of KMS root epochs. Only zero is currently supported." + }, + "operation": { + "type": "string", + "enum": [ + "kms_bootstrap", + "kms_serve", + "kms_export_keys", + "kms_import_keys", + "app_get_key", + "app_get_cert" + ] + } + }, + "required": [ + "operation" + ] } } } diff --git a/dstack/kms/auth-simple/README.md b/dstack/kms/auth-simple/README.md index dc48e7c50..bfa0951b1 100644 --- a/dstack/kms/auth-simple/README.md +++ b/dstack/kms/auth-simple/README.md @@ -252,3 +252,24 @@ bun run test:run - [auth-eth](../auth-eth/) - On-chain governance auth server - [auth-mock](../auth-mock/) - Development/testing auth server (always allows) - [Deployment Guide](../../../docs/deployment.md) - Full deployment instructions + + +### Authorization endpoints + +`POST /authorizeKms` and `POST /authorizeApp` return `AuthorizationResult`, including +`rejectionReason`, `gatewayAppId`, `currentRootEpoch`, and `currentRootEpochHash`. +The selected endpoint determines the policy; the backend checks that the verified +`appId` matches that scope. `requestedAppEpoch` defaults to 0; nonzero values are currently +rejected. The optional +`requestedRootEpoch` selects historical keys; omission uses the returned current +epoch, and an explicit `0` selects epoch 0. See the +[KMS authorization protocol](../README.md#authorization-protocol) for registration, +bootstrap, and compatibility requirements. The legacy `/bootAuth/*` endpoints +remain available. + +### Authorization operations + +`/authorizeKms` and `/authorizeApp` require an `operation` string. See the +[KMS authorization protocol](../README.md#authorization-protocol) for the names +and subject semantics. Bootstrap alone may authorize before root registration; +unknown operations are rejected. Legacy `/bootAuth/*` routes are unchanged. diff --git a/dstack/kms/auth-simple/index.test.ts b/dstack/kms/auth-simple/index.test.ts index 5e7e37238..2a08a5877 100644 --- a/dstack/kms/auth-simple/index.test.ts +++ b/dstack/kms/auth-simple/index.test.ts @@ -472,6 +472,50 @@ describe('auth-simple', () => { const info = async () => (await app.fetch(new Request('http://localhost/', { method: 'GET' }))).json(); + const authorize = async (fields: object = {}, isKms = false) => + (await app.fetch(new Request(`http://localhost/${isKms ? 'authorizeKms' : 'authorizeApp'}`, { + method: 'POST', headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ ...baseBootInfo, operation: isKms ? 'kms_serve' : 'app_get_key', ...fields }), + }))).json(); + + it('checks the requested policy and returns the epoch on rejection', async () => { + const kmsContractAddr = '0x' + '11'.repeat(20); + writeTestConfig({ ...config, kmsContractAddr, currentRootEpoch: 1 }); + expect(await authorize()).toMatchObject({ + isAllowed: true, rejectionReason: '', currentRootEpoch: 1, currentRootEpochHash: h1, + }); + expect(await authorize({ requestedRootEpoch: 0 })).toMatchObject({ isAllowed: true, currentRootEpoch: 1 }); + expect(await authorize({ requestedRootEpoch: 1 })).toMatchObject({ isAllowed: true }); + expect(await authorize({ requestedAppEpoch: 1 })).toMatchObject({ isAllowed: false, rejectionReason: 'Requested app epoch must be zero', currentRootEpoch: 1 }); + expect(await authorize({ requestedAppEpoch: 0 })).toMatchObject({ isAllowed: true }); + expect(await authorize({ requestedRootEpoch: 2 })).toMatchObject({ + isAllowed: false, rejectionReason: 'Requested root epoch is not active', currentRootEpoch: 1, currentRootEpochHash: h1, + }); + expect(await authorize({ appId: kmsContractAddr, composeHash: 'not-an-app-compose' }, true)).toMatchObject({ isAllowed: true }); + expect(await authorize({ appId: kmsContractAddr, mrAggregated: '0xbad' }, true)).toMatchObject({ + isAllowed: false, currentRootEpoch: 1, currentRootEpochHash: h1, + }); + expect(await authorize({ appId: '0x' + '22'.repeat(20) })).toMatchObject({ isAllowed: false }); + }); + + it('denies authorization without registered roots, while bootstrap remains available', async () => { + writeTestConfig({ ...config, rootEpochs: [] }); + expect(await authorize()).toMatchObject({ + isAllowed: false, rejectionReason: 'Root epoch not registered', + currentRootEpoch: 0, currentRootEpochHash: '0x' + '00'.repeat(32), + }); + expect((await boot('kms', {})).isAllowed).toBe(true); + expect(await authorize({ operation: 'kms_bootstrap' }, true)).toMatchObject({ isAllowed: true }); + expect(await authorize({ operation: 'kms_export_keys' }, true)).toMatchObject({ isAllowed: false }); + }); + + it('rejects operations belonging to the other authorization scope', async () => { + writeTestConfig(config); + expect(await authorize({ operation: 'kms_serve' })).toMatchObject({ isAllowed: false, rejectionReason: 'Invalid authorization operation' }); + expect(await authorize({ operation: 'app_get_cert' }, true)).toMatchObject({ isAllowed: false, rejectionReason: 'Invalid authorization operation' }); + expect(await authorize({ operation: 'app_get_cert' })).toMatchObject({ isAllowed: true }); + }); + it('allows only a KMS serving the current root epoch', async () => { writeTestConfig(config); expect(await info()).toMatchObject({ rootEpoch: 0, rootEpochHash: h0 }); diff --git a/dstack/kms/auth-simple/index.ts b/dstack/kms/auth-simple/index.ts index 4a242ac7e..7444a38ab 100644 --- a/dstack/kms/auth-simple/index.ts +++ b/dstack/kms/auth-simple/index.ts @@ -27,6 +27,13 @@ const BootInfoSchema = z.object({ rootEpochHash: z.string().optional() }); +const AuthorizationRequestSchema = BootInfoSchema.omit({ rootEpoch: true, rootEpochHash: true }).extend({ + operation: z.enum(['kms_bootstrap', 'kms_serve', 'kms_export_keys', 'kms_import_keys', 'app_get_key', 'app_get_cert']), + requestedAppEpoch: z.number().int().min(0).max(0xffffffff).default(0), + requestedRootEpoch: z.number().int().min(0).max(0xffffffff).optional(), +}); +type AuthorizationRequest = z.infer; + const BootResponseSchema = z.object({ isAllowed: z.boolean(), reason: z.string(), @@ -152,6 +159,31 @@ class ConfigBackend { } } + async checkAuthorization(appInfo: AuthorizationRequest, isKms: boolean) { + const config = this.loadConfig(); + const current = currentRootEpoch(config); + const epoch = { + currentAppEpoch: 0, + currentRootEpoch: current.rootEpoch ?? 0, + currentRootEpochHash: current.rootEpochHash ?? '0x' + '00'.repeat(32), + }; + const kmsOperation = ['kms_bootstrap', 'kms_serve', 'kms_export_keys', 'kms_import_keys'].includes(appInfo.operation); + if (kmsOperation !== isKms) { + return { isAllowed: false, rejectionReason: 'Invalid authorization operation', gatewayAppId: config.gatewayAppId, ...epoch }; + } + if (current.rootEpoch === undefined && appInfo.operation !== 'kms_bootstrap') { + return { isAllowed: false, rejectionReason: 'Root epoch not registered', gatewayAppId: config.gatewayAppId, ...epoch }; + } + if (appInfo.requestedRootEpoch !== undefined && appInfo.requestedRootEpoch > (current.rootEpoch ?? 0)) { + return { isAllowed: false, rejectionReason: 'Requested root epoch is not active', gatewayAppId: config.gatewayAppId, ...epoch }; + } + if (appInfo.requestedAppEpoch !== 0) { + return { isAllowed: false, rejectionReason: 'Requested app epoch must be zero', gatewayAppId: config.gatewayAppId, ...epoch }; + } + const result = this.evaluateBoot(appInfo, isKms, config); + return { isAllowed: result.isAllowed, rejectionReason: result.reason, gatewayAppId: result.gatewayAppId, ...epoch }; + } + async checkBoot(bootInfo: BootInfo, isKms: boolean): Promise { const config = this.loadConfig(); const response = this.evaluateBoot(bootInfo, isKms, config); @@ -332,6 +364,17 @@ app.get('/', async (c) => { } }); +for (const method of ['authorizeKms', 'authorizeApp'] as const) { + app.post(`/${method}`, zValidator('json', AuthorizationRequestSchema), async (c) => { + try { + return c.json(await backend.checkAuthorization(c.req.valid('json'), method === 'authorizeKms')); + } catch { + console.error('authorization backend failed'); + return c.json({ status: 'error', message: 'authorization backend unavailable' }, 503); + } + }); +} + // app boot authentication app.post('/bootAuth/app', zValidator('json', BootInfoSchema), diff --git a/dstack/kms/rpc/proto/kms_rpc.proto b/dstack/kms/rpc/proto/kms_rpc.proto index 6853df2be..13083647b 100644 --- a/dstack/kms/rpc/proto/kms_rpc.proto +++ b/dstack/kms/rpc/proto/kms_rpc.proto @@ -11,6 +11,9 @@ package kms; message GetAppKeyRequest { uint32 api_version = 1; string vm_config = 2; + // Root key epoch to derive from; unset means the current epoch. Epoch 0 is + // the original root keys. Older epochs remain available so CVMs can migrate. + optional uint32 root_epoch = 3; } message AppId { @@ -46,6 +49,8 @@ message AppKeyResponse { string gateway_app_id = 7; // OS Image hash bytes os_image_hash = 8; + // Root key epoch all epoch-dependent fields above were derived from. + uint32 root_epoch = 9; } message GetMetaResponse { @@ -73,15 +78,12 @@ message KmsKeys { } message KmsKeyResponse { + // Retained for legacy KMS onboarding. This public CA authenticates no caller. string temp_ca_key = 1; + // Root keys of every epoch the KMS holds; the index is the epoch. repeated KmsKeys keys = 2; } -// See the note on `GetTempCaCert`. -// -// `temp_ca_key` is a CA private key handed to any caller, so it authenticates -// nobody. It exists because the KMS pinned this CA for mutual TLS, which left a -// self-issued RA-TLS certificate with nothing to chain to. message GetTempCaCertResponse { string temp_ca_cert = 1; string temp_ca_key = 2; @@ -109,19 +111,7 @@ service KMS { rpc GetAppEnvEncryptPubKey(AppId) returns (PublicKeyResponse); // Request the KMS instance metadata for use as a probe and health check. rpc GetMeta(google.protobuf.Empty) returns (GetMetaResponse); - // Request the temporary CA certificate and key. - // - // Both current callers - guests at boot, and KMS-to-KMS onboarding - fetch this CA - // and mint their client certificate from it, because the KMS used to pin the CA for - // mutual TLS and a self-issued certificate had nothing to chain to. - // - // That pin is gone: the KMS now verifies client certificates by the attestation - // they carry and ignores the issuer, so a self-issued certificate would be accepted - // on the same terms. Neither caller has been migrated yet, so this RPC still has to - // work; it is scheduled for removal once both are. - // - // Do not build new callers on it. A caller that only needs the KMS root CA can read - // `ca_cert` from `GetMeta` instead. + // Compatibility for existing OS images and tokenless legacy onboarding. rpc GetTempCaCert(google.protobuf.Empty) returns (GetTempCaCertResponse); // Sign a certificate rpc SignCert(SignCertRequest) returns (SignCertResponse); @@ -158,11 +148,18 @@ message OnboardRequest { string domain = 2; // Optional bearer token for the source KMS admin listener. string source_token = 3; + // Import the source's epochs 0..N-1 and generate epoch N locally. The source + // must be at the current epoch N-1. + bool rotate = 4; } message OnboardResponse { - // k256 public key (secp256k1) inherited from source KMS + // k256 public key (secp256k1) of the current epoch bytes k256_pubkey = 1; + // Set only by a rotation: the generated CA public key and the attestation + // binding the new epoch's public keys. + bytes ca_pubkey = 2; + bytes attestation = 3; } // Attestation info needed for on-chain KMS authorization. diff --git a/dstack/kms/src/config.rs b/dstack/kms/src/config.rs index fd2200fb8..507e4f136 100644 --- a/dstack/kms/src/config.rs +++ b/dstack/kms/src/config.rs @@ -21,6 +21,8 @@ const RPC_CERT: &str = "rpc.crt"; const RPC_KEY: &str = "rpc.key"; const RPC_DOMAIN: &str = "rpc-domain"; const K256_KEY: &str = "root-k256.key"; +const ROOT_EPOCH: &str = "root-epoch"; +const EPOCHS_DIR: &str = "epochs"; const BOOTSTRAP_INFO: &str = "bootstrap-info.json"; #[derive(Debug, Clone, Deserialize)] @@ -130,9 +132,7 @@ impl KmsConfig { } pub fn keys_exists(&self) -> bool { - self.tmp_ca_cert().exists() - && self.tmp_ca_key().exists() - && self.root_ca_cert().exists() + self.root_ca_cert().exists() && self.root_ca_key().exists() && self.rpc_cert().exists() && self.rpc_key().exists() @@ -171,6 +171,28 @@ impl KmsConfig { self.cert_dir.join(K256_KEY) } + /// Records the newest epoch this KMS holds, whose roots are the top-level + /// root files. Absent means 0. + pub fn root_epoch(&self) -> PathBuf { + self.cert_dir.join(ROOT_EPOCH) + } + + /// Root key paths of `epoch` in a keyring whose newest epoch is `latest`: + /// the newest roots are the top-level root files, so paths that predate + /// epochs are unchanged. Older epochs live in `epochs//`. + pub fn root_key_paths(&self, epoch: u32, latest: u32) -> RootKeyPaths { + let dir = if epoch == latest { + self.cert_dir.clone() + } else { + self.cert_dir.join(EPOCHS_DIR).join(epoch.to_string()) + }; + RootKeyPaths { + ca_cert: dir.join(ROOT_CA_CERT), + ca_key: dir.join(ROOT_CA_KEY), + k256_key: dir.join(K256_KEY), + } + } + pub fn bootstrap_info(&self) -> PathBuf { self.cert_dir.join(BOOTSTRAP_INFO) } @@ -209,6 +231,12 @@ pub(crate) struct Dev { pub gateway_app_id: String, } +pub(crate) struct RootKeyPaths { + pub ca_cert: PathBuf, + pub ca_key: PathBuf, + pub k256_key: PathBuf, +} + #[derive(Debug, Clone, Deserialize)] pub(crate) struct OnboardConfig { pub enabled: bool, diff --git a/dstack/kms/src/main.rs b/dstack/kms/src/main.rs index f4bf428a8..205790d79 100644 --- a/dstack/kms/src/main.rs +++ b/dstack/kms/src/main.rs @@ -3,7 +3,7 @@ // SPDX-License-Identifier: Apache-2.0 use anyhow::{anyhow, bail, Context, Result}; -use clap::Parser; +use clap::{Parser, Subcommand}; use config::KmsConfig; use main_service::{KmsState, RpcHandler}; use ra_rpc::ratls_client_verifier::RaTlsClientAuth; @@ -35,6 +35,32 @@ struct Args { /// Path to the configuration file #[arg(short, long)] config: Option, + #[command(subcommand)] + command: Option, +} + +#[derive(Subcommand)] +enum Command { + /// Print, as hex, the attestation `registerRootEpoch` takes for epoch 0, + /// converted from the legacy `kmsInfo` quote and event log + LegacyAttestation { + /// TDX quote, hex + #[arg(long)] + quote: String, + /// Event log JSON, hex-encoded as `kmsInfo()` returns it + #[arg(long)] + eventlog: String, + }, +} + +fn legacy_attestation(quote: &str, eventlog: &str) -> Result { + let decode = |hex_str: &str| hex::decode(hex_str.trim().trim_start_matches("0x")); + let quote = decode(quote).context("invalid quote hex")?; + let eventlog = decode(eventlog).context("invalid event log hex")?; + let attestation = dstack_attest::attestation::Attestation::from_tdx_quote(quote, &eventlog)? + .into_versioned() + .to_bytes()?; + Ok(format!("0x{}", hex::encode(attestation))) } async fn run_onboard_service(kms_config: KmsConfig, figment: Figment) -> Result<()> { @@ -123,6 +149,10 @@ async fn main() -> Result<()> { fmt().with_env_filter(filter).with_ansi(false).init(); } let args = Args::parse(); + if let Some(Command::LegacyAttestation { quote, eventlog }) = args.command { + println!("{}", legacy_attestation("e, &eventlog)?); + return Ok(()); + } let figment = config::load_config_figment(args.config.as_deref()); let config: KmsConfig = figment.focus("core").extract()?; @@ -143,8 +173,12 @@ async fn main() -> Result<()> { } } + let keyring = main_service::Keyring::load(&config).context("Failed to load the root keys")?; + let rpc_epoch = main_service::wait_for_servable_epoch(&config, &keyring).await; + info!("Issuing RPC certificate under root epoch {rpc_epoch}"); + info!("Updating certs"); - if let Err(err) = onboard_service::update_certs(&config).await { + if let Err(err) = onboard_service::update_certs(&config, rpc_epoch, keyring.latest()).await { if config.attest_rpc_cert { return Err(err).context( "Failed to reissue the attested KMS RPC certificate; refusing to start with a \ @@ -176,7 +210,12 @@ async fn main() -> Result<()> { } else { None }; - let state = main_service::KmsState::new(config).context("Failed to initialize KMS state")?; + // Reload certificates after renewal before exposing them through RPCs. + let rpc_ca_cert = + fs_err::read_to_string(config.root_key_paths(rpc_epoch, keyring.latest()).ca_cert)?; + let keyring = main_service::Keyring::load(&config).context("Failed to reload the root keys")?; + let state = main_service::KmsState::new(config, keyring, rpc_ca_cert) + .context("Failed to initialize KMS state")?; let quote_verifier = QuoteVerifier::new(state.attestation_verifier()); let admin_quote_verifier = QuoteVerifier::new(state.attestation_verifier()); let figment = figment @@ -213,6 +252,11 @@ async fn main() -> Result<()> { rocket = rocket.manage(quote_verifier); let main_srv = rocket.launch(); + // Refresh the TLS listener when the authorized root CA changes. + let restart = { + let state = state.clone(); + async move { state.restart_requested().await } + }; match admin_setup { Some((admin_figment, admin_fairing)) => { if admin_config.insecure_no_auth { @@ -233,14 +277,46 @@ async fn main() -> Result<()> { .manage(state) .manage(admin_quote_verifier) .launch(); - tokio::try_join!( - async { main_srv.await.map_err(|err| anyhow!(err.to_string())) }, - async { admin_srv.await.map_err(|err| anyhow!(err.to_string())) }, - )?; - } - None => { - main_srv.await.map_err(|err| anyhow!(err.to_string()))?; + let servers = async { + tokio::try_join!( + async { main_srv.await.map_err(|err| anyhow!(err.to_string())) }, + async { admin_srv.await.map_err(|err| anyhow!(err.to_string())) }, + ) + .map(drop) + }; + tokio::select! { + result = servers => result?, + () = restart => bail!("the RPC root CA changed; restarting to refresh the TLS certificate"), + } } + None => tokio::select! { + result = main_srv => result.map(drop).map_err(|err| anyhow!(err.to_string()))?, + () = restart => bail!("the RPC root CA changed; restarting to refresh the TLS certificate"), + }, } Ok(()) } + +#[cfg(test)] +mod tests { + use dstack_attest::attestation::VersionedAttestation; + + #[test] + fn legacy_attestation_converts_a_get_quote_response() { + let fixture: serde_json::Value = serde_json::from_str(include_str!( + "../../verifier/fixtures/tdx-lite-getquote.json" + )) + .unwrap(); + let quote = fixture["quote"].as_str().unwrap(); + let eventlog = hex::encode(fixture["event_log"].as_str().unwrap()); + + let attestation = super::legacy_attestation(quote, &format!("0x{eventlog}")).unwrap(); + let attestation = + VersionedAttestation::from_bytes(&hex::decode(&attestation[2..]).unwrap()).unwrap(); + let attestation = attestation.into_v1(); + assert_eq!( + hex::encode(attestation.report_data().unwrap()), + fixture["report_data"].as_str().unwrap() + ); + } +} diff --git a/dstack/kms/src/main_service.rs b/dstack/kms/src/main_service.rs index d845a82c6..605092339 100644 --- a/dstack/kms/src/main_service.rs +++ b/dstack/kms/src/main_service.rs @@ -26,15 +26,17 @@ use ra_tls::{ attestation::{resolve_vm_config, AttestationVerifier, TeeVariant, VerifiedAttestation}, cert::{CaCert, CertRequest, CertSigningRequestV1, CertSigningRequestV2, Csr}, kdf, + rcgen::KeyPair, }; use scale::Decode; use tracing::warn; use upgrade_authority::{ - build_boot_info, ensure_app_id_len, local_kms_boot_info, BootInfo, GetInfoResponse, + build_boot_info, ensure_app_id_len, local_kms_boot_info, AuthorizationResult, BootInfo, + GetInfoResponse, Operation, RootEpochRegistry, }; use crate::{ - config::KmsConfig, + config::{KmsConfig, RootKeyPaths}, crypto::{derive_k256_key, sign_message, sign_message_with_timestamp}, }; @@ -43,7 +45,7 @@ pub(crate) mod upgrade_authority; /// How long the auth API's info and this KMS's own authorization are reused. /// -/// `GetMeta`, `GetAppEnvEncryptPubKey` and `GetTempCaCert` need no client certificate, and each +/// `GetMeta` and `GetAppEnvEncryptPubKey` need no client certificate, and each /// would otherwise cost the auth API a round of chain RPC calls. Concurrent callers share one /// in-flight request. App authorization is never cached. const AUTH_API_CACHE_TTL: Duration = Duration::from_secs(1); @@ -67,16 +69,109 @@ impl std::ops::Deref for KmsState { } } +/// The root key pair of one epoch. +struct RootKeys { + ca: CaCert, + k256: SigningKey, +} + +impl RootKeys { + fn load(paths: &RootKeyPaths) -> Result { + let ca = CaCert::load(&paths.ca_cert, &paths.ca_key) + .context("failed to load root CA certificate")?; + let key_bytes = fs::read(&paths.k256_key).context("failed to read ECDSA root key")?; + let k256 = SigningKey::from_slice(&key_bytes).context("failed to load ECDSA root key")?; + Ok(Self { ca, k256 }) + } +} + +/// The root keys of epochs 0..=latest this KMS holds, and their chain hashes. +pub(crate) struct Keyring { + roots: Vec, + chain_hashes: Vec<[u8; 32]>, +} + +impl Keyring { + pub(crate) fn load(config: &KmsConfig) -> Result { + let latest = read_root_epoch(config)?; + let roots = (0..=latest) + .map(|epoch| { + RootKeys::load(&config.root_key_paths(epoch, latest)) + .with_context(|| format!("failed to load root keys of epoch {epoch}")) + }) + .collect::>>()?; + let chain_hashes = root_chain_hashes(roots.iter().map(|root| (&root.ca.key, &root.k256))); + Ok(Self { + roots, + chain_hashes, + }) + } + + pub(crate) fn latest(&self) -> u32 { + self.roots.len() as u32 - 1 + } + + pub(crate) fn servable_epoch(&self, registry: RootEpochRegistry) -> Result { + servable_epoch(&self.chain_hashes, registry) + } +} + +/// Chain hashes of the root keys of epochs 0, 1, ... +pub(crate) fn root_chain_hashes<'a>( + roots: impl Iterator, +) -> Vec<[u8; 32]> { + let pubkeys: Vec<_> = roots + .map(|(ca, k256)| { + ( + ca.public_key_der(), + k256.verifying_key().to_sec1_bytes().to_vec(), + ) + }) + .collect(); + dstack_types::root_epoch::chain_hashes(pubkeys.iter().map(|(ca, k256)| (&ca[..], &k256[..]))) +} + +/// The epoch a keyring with `chain_hashes` serves under `registry`: the +/// current epoch, which the keyring must hold with a matching chain hash. A +/// keyring nobody registered is servable only at epoch 0, as before root +/// epochs existed. +pub(crate) fn servable_epoch( + chain_hashes: &[[u8; 32]], + registry: RootEpochRegistry, +) -> Result { + let latest = chain_hashes.len() as u32 - 1; + let current = match registry { + RootEpochRegistry::Unchecked => return Ok(latest), + RootEpochRegistry::Empty if latest == 0 => return Ok(0), + RootEpochRegistry::Empty => { + bail!("the keyring holds root epoch {latest}, but no root epoch is registered") + } + RootEpochRegistry::Current(current) => current, + }; + match chain_hashes.get(current.epoch as usize) { + Some(hash) if *hash == current.chain_hash => Ok(current.epoch), + Some(_) => bail!( + "the root keys up to epoch {} are not the registered ones", + current.epoch + ), + None => bail!( + "root epoch {} is current, but the keyring holds only up to {latest}", + current.epoch + ), + } +} + pub struct KmsStateInner { config: KmsConfig, - root_ca: CaCert, - k256_key: SigningKey, - temp_ca_cert: String, - temp_ca_key: String, + keyring: Keyring, + temp_ca: CaCert, + /// The CA of the running TLS listener, independent of key selection. + rpc_ca_pubkey: Vec, + restart: tokio::sync::Notify, verifier: CvmVerifier, attestation_verifier: Arc, self_boot_info: Cache<(), BootInfo>, - self_allowed: Cache<(), ()>, + self_allowed: Cache, auth_api_info: Cache<(), GetInfoResponse>, metrics: KmsMetrics, } @@ -112,9 +207,13 @@ impl KmsMetrics { } } -fn short_lived_cache() -> Cache<(), V> { +fn short_lived_cache(capacity: u64) -> Cache +where + K: Eq + std::hash::Hash + Send + Sync + 'static, + V: Clone + Send + Sync + 'static, +{ Cache::builder() - .max_capacity(1) + .max_capacity(capacity) .time_to_live(AUTH_API_CACHE_TTL) .build() } @@ -127,16 +226,15 @@ impl KmsState { self.verifier.clear_cache(image_hash, config_hash) } - pub fn new(config: KmsConfig) -> Result { - let root_ca = CaCert::load(config.root_ca_cert(), config.root_ca_key()) - .context("Failed to load root CA certificate")?; - let key_bytes = fs::read(config.k256_key()).context("Failed to read ECDSA root key")?; - let k256_key = - SigningKey::from_slice(&key_bytes).context("Failed to load ECDSA root key")?; - let temp_ca_key = - fs::read_to_string(config.tmp_ca_key()).context("Faeild to read temp ca key")?; - let temp_ca_cert = - fs::read_to_string(config.tmp_ca_cert()).context("Faeild to read temp ca cert")?; + pub fn new(config: KmsConfig, keyring: Keyring, rpc_ca_cert: String) -> Result { + let (_, pem) = x509_parser::pem::parse_x509_pem(rpc_ca_cert.as_bytes()) + .context("invalid RPC CA certificate")?; + let rpc_ca_pubkey = pem + .parse_x509() + .context("invalid RPC CA certificate")? + .public_key() + .raw + .to_vec(); let attestation_verifier = Arc::new( AttestationVerifier::load(&config.attestation) .context("failed to load attestation verifier")?, @@ -154,19 +252,19 @@ impl KmsState { } Ok(Self { inner: Arc::new(KmsStateInner { + temp_ca: crate::onboard_service::ensure_temp_ca(&config)?, config, - root_ca, - k256_key, - temp_ca_cert, - temp_ca_key, + keyring, + rpc_ca_pubkey, + restart: tokio::sync::Notify::new(), verifier, attestation_verifier, self_boot_info: Cache::builder() .max_capacity(1) .time_to_live(SELF_BOOT_INFO_TTL) .build(), - self_allowed: short_lived_cache(), - auth_api_info: short_lived_cache(), + self_allowed: short_lived_cache(4), + auth_api_info: short_lived_cache(1), metrics: KmsMetrics::default(), }), }) @@ -179,6 +277,120 @@ impl KmsState { pub(crate) fn attestation_verifier(&self) -> Arc { self.inner.attestation_verifier.clone() } + + fn epoch_root(&self, epoch: u32) -> Result<&RootKeys> { + self.keyring + .roots + .get(epoch as usize) + .with_context(|| format!("root epoch {epoch} is not held by this KMS")) + } + + pub async fn restart_requested(&self) { + self.restart.notified().await + } + + async fn current_epoch_from_info(&self) -> Result { + let info = self + .auth_api_info + .try_get_with((), self.config.auth_api.get_info()) + .await + .map_err(|err| anyhow!("{err:#}"))?; + self.keyring.servable_epoch(info.root_epochs) + } + + /// Resolve this request from its authorization result; no serving epoch is cached. + fn authorized_epoch( + &self, + result: &AuthorizationResult, + requested: Option, + ) -> Result { + if result.current_app_epoch != 0 { + bail!("this KMS does not support nonzero app epochs"); + } + let current = if self.config.auth_api.is_dev() { + self.keyring.latest() + } else { + let current = result + .current_root()? + .context("Root epoch not registered")?; + self.keyring + .servable_epoch(RootEpochRegistry::Current(current))? + }; + if !result.is_allowed { + bail!("Authorization denied: {}", result.rejection_reason); + } + let epoch = requested.unwrap_or(current); + if epoch > current { + bail!("root epoch {epoch} is not active; the current one is {current}"); + } + self.epoch_root(epoch)?; + Ok(epoch) + } + + /// Keep the running TLS issuer consistent with default key responses until reload. + fn ensure_rpc_ca(&self, current_epoch: u32) -> Result<()> { + if self.epoch_root(current_epoch)?.ca.key.public_key_der() != self.rpc_ca_pubkey { + self.restart.notify_one(); + bail!("the RPC root CA changed; retry after the TLS certificate is refreshed"); + } + Ok(()) + } +} + +/// Wait for the registered current roots to issue the RPC certificate at startup. +/// An epoch-0 keyring may start while the info endpoint is unreachable, but +/// key release still requires an authorization result with registered roots. +pub(crate) async fn wait_for_servable_epoch(config: &KmsConfig, keyring: &Keyring) -> u32 { + const RETRY: Duration = Duration::from_secs(10); + loop { + let result = match config.auth_api.get_info().await { + Ok(info) => keyring.servable_epoch(info.root_epochs), + Err(err) if keyring.latest() == 0 => { + warn!("skipping the root epoch check: {err:#}"); + return 0; + } + Err(err) => Err(err.context("failed to read the current root epoch")), + }; + match result { + Ok(epoch) => return epoch, + Err(err) => { + warn!("not serving yet: {err:#}; retrying in {}s", RETRY.as_secs()); + tokio::time::sleep(RETRY).await; + } + } + } +} + +struct AppKeys { + disk_crypt_key: Vec, + env_crypt_key: Vec, + k256_key: Vec, + k256_signature: Vec, +} + +fn derive_app_keys(root: &RootKeys, app_id: &[u8], instance_id: &[u8]) -> Result { + let disk_crypt_key = + kdf::derive_dh_secret(&root.ca.key, &[app_id, instance_id, b"app-disk-crypt-key"]) + .context("Failed to derive app disk key")?; + let env_crypt_key = kdf::derive_dh_secret(&root.ca.key, &[app_id, b"env-encrypt-key"]) + .context("Failed to derive env encrypt key")?; + let env_crypt_key = x25519_dalek::StaticSecret::from(env_crypt_key).to_bytes(); + let (k256_key, k256_signature) = + derive_k256_key(&root.k256, app_id).context("Failed to derive app ecdsa key")?; + Ok(AppKeys { + disk_crypt_key: disk_crypt_key.to_vec(), + env_crypt_key: env_crypt_key.to_vec(), + k256_key: k256_key.to_bytes().to_vec(), + k256_signature, + }) +} + +pub(crate) fn read_root_epoch(config: &KmsConfig) -> Result { + match fs::read_to_string(config.root_epoch()) { + Ok(epoch) => epoch.trim().parse().context("invalid root epoch file"), + Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(0), + Err(err) => Err(err).context("failed to read root epoch file"), + } } pub struct RpcHandler { @@ -195,17 +407,23 @@ impl RpcHandler { } pub(crate) async fn handover_keys(self, request: GetKmsKeyRequest) -> Result { - self.ensure_self_allowed() + self.ensure_self_allowed(Operation::KmsExportKeys) .await .context("KMS self authorization failed")?; let info = self.ensure_kms_allowed(&request.vm_config).await?; ensure_key_release_allowed(&info, (&self.state.config).into())?; Ok(KmsKeyResponse { - temp_ca_key: self.state.inner.temp_ca_key.clone(), - keys: vec![KmsKeys { - ca_key: self.state.inner.root_ca.key.serialize_pem(), - k256_key: self.state.inner.k256_key.to_bytes().to_vec(), - }], + temp_ca_key: self.state.temp_ca.key.serialize_pem(), + keys: self + .state + .keyring + .roots + .iter() + .map(|root| KmsKeys { + ca_key: root.ca.key.serialize_pem(), + k256_key: root.k256.to_bytes().to_vec(), + }) + .collect(), }) } } @@ -213,6 +431,7 @@ impl RpcHandler { struct BootConfig { boot_info: BootInfo, gateway_app_id: String, + root_epoch: u32, } pub(crate) fn build_boot_info_for_attestation( @@ -268,20 +487,12 @@ fn ensure_key_release_allowed(boot_info: &BootInfo, policy: KeyReleasePolicy) -> } } -fn ensure_self_key_release_allowed( - self_boot_info: Option<&BootInfo>, - policy: KeyReleasePolicy, -) -> Result<()> { - if let Some(boot_info) = self_boot_info { - ensure_key_release_allowed(boot_info, policy)?; - } - Ok(()) -} - impl RpcHandler { - async fn ensure_self_allowed(&self) -> Result> { + async fn ensure_self_allowed(&self, operation: Operation) -> Result { if !self.state.config.enforce_self_authorization { - return Ok(None); + let epoch = self.state.current_epoch_from_info().await?; + self.state.ensure_rpc_ca(epoch)?; + return Ok(epoch); } let boot_info = self .state @@ -290,24 +501,25 @@ impl RpcHandler { .await .map_err(|err| anyhow!("{err:#}")) .context("Failed to load cached self boot info")?; - self.state + let response = self + .state .self_allowed - .try_get_with((), async { + .try_get_with(operation, async { let response = self .state .config .auth_api - .is_app_allowed(&boot_info, true) + .authorize_kms(&boot_info, operation, None) .await .context("Failed to call self KMS auth check")?; - if !response.is_allowed { - bail!("KMS is not allowed: {}", response.reason); - } - Ok(()) + let epoch = self.state.authorized_epoch(&response, None)?; + self.state.ensure_rpc_ca(epoch)?; + Ok::<_, anyhow::Error>(response) }) .await .map_err(|err| anyhow!("{err:#}"))?; - Ok(Some(boot_info)) + let epoch = self.state.authorized_epoch(&response, None)?; + Ok(epoch) } fn ensure_attested(&self) -> Result<&VerifiedAttestation> { @@ -319,57 +531,91 @@ impl RpcHandler { async fn ensure_kms_allowed(&self, vm_config: &str) -> Result { let att = self.ensure_attested()?; - self.ensure_app_attestation_allowed(att, true, false, vm_config) - .await - .map(|c| c.boot_info) + let boot_info = build_boot_info_for_attestation(att, false, vm_config)?; + let response = self + .state + .config + .auth_api + .authorize_kms(&boot_info, Operation::KmsImportKeys, None) + .await?; + self.state.authorized_epoch(&response, None)?; + self.ensure_attestation_policy(att, vm_config, &boot_info, &response) + .await?; + Ok(boot_info) } - async fn ensure_app_boot_allowed(&self, vm_config: &str) -> Result { + async fn ensure_app_boot_allowed( + &self, + vm_config: &str, + requested_root_epoch: Option, + ) -> Result { let att = self.ensure_attested()?; - self.ensure_app_attestation_allowed(att, false, false, vm_config) - .await + let boot_info = build_boot_info_for_attestation(att, false, vm_config)?; + self.ensure_app_attestation_allowed( + att, + boot_info, + vm_config, + Operation::AppGetKey, + requested_root_epoch, + ) + .await } async fn ensure_app_attestation_allowed( &self, att: &VerifiedAttestation, - is_kms: bool, - use_boottime_mr: bool, - vm_config_str: &str, + boot_info: BootInfo, + vm_config: &str, + operation: Operation, + requested_root_epoch: Option, ) -> Result { - let boot_info = build_boot_info_for_attestation(att, use_boottime_mr, vm_config_str)?; let response = self .state .config .auth_api - .is_app_allowed(&boot_info, is_kms) + .authorize_app(&boot_info, operation, requested_root_epoch) .await?; - if !response.is_allowed { - bail!("Boot denied: {}", response.reason); - } + let root_epoch = self + .state + .authorized_epoch(&response, requested_root_epoch)?; + self.ensure_attestation_policy(att, vm_config, &boot_info, &response) + .await?; + Ok(BootConfig { + boot_info, + gateway_app_id: response.gateway_app_id, + root_epoch, + }) + } + + async fn ensure_attestation_policy( + &self, + att: &VerifiedAttestation, + vm_config: &str, + boot_info: &BootInfo, + response: &AuthorizationResult, + ) -> Result<()> { + let current_epoch = if self.state.config.auth_api.is_dev() { + self.state.keyring.latest() + } else { + response.current_root_epoch + }; + self.state.ensure_rpc_ca(current_epoch)?; // SNP rootfs/app/config binding is handled by the SNP launch-measurement // helper above. The legacy OS-image verifier is TDX-oriented and still // rejects SNP quotes; keep SNP on the explicit fail-closed helper path. if boot_info.tee_variant != TeeVariant::DstackAmdSevSnp { self.state .verifier - .verify_os_image_hash( - vm_config_str.into(), - att, - &mut VerificationDetails::default(), - ) + .verify_os_image_hash(vm_config.into(), att, &mut VerificationDetails::default()) .await .context("Failed to verify os image hash")?; } - Ok(BootConfig { - boot_info, - gateway_app_id: response.gateway_app_id, - }) + Ok(()) } - fn derive_app_ca(&self, app_id: &[u8]) -> Result { + fn derive_app_ca(&self, app_id: &[u8], root: &RootKeys) -> Result { let context_data = vec![app_id, b"app-ca"]; - let app_key = kdf::derive_p256_key_pair(&self.state.root_ca.key, &context_data) + let app_key = kdf::derive_p256_key_pair(&root.ca.key, &context_data) .context("Failed to derive app disk key")?; let req = CertRequest::builder() .key(&app_key) @@ -379,11 +625,7 @@ impl RpcHandler { .app_id(app_id) .special_usage("app:ca") .build(); - let app_ca = self - .state - .root_ca - .sign(req) - .context("Failed to sign App CA")?; + let app_ca = root.ca.sign(req).context("Failed to sign App CA")?; Ok(CaCert::from_parts(app_key, app_ca)) } } @@ -393,57 +635,43 @@ impl KmsRpc for RpcHandler { if request.api_version > 1 { bail!("Unsupported API version: {}", request.api_version); } - self.ensure_self_allowed() + self.ensure_self_allowed(Operation::KmsServe) .await .context("KMS self authorization failed")?; let BootConfig { boot_info, gateway_app_id, + root_epoch, } = self - .ensure_app_boot_allowed(&request.vm_config) + .ensure_app_boot_allowed(&request.vm_config, request.root_epoch) .await .context("App not allowed")?; ensure_key_release_allowed(&boot_info, (&self.state.config).into())?; - let app_id = boot_info.app_id; - let instance_id = boot_info.instance_id; - let os_image_hash = boot_info.os_image_hash; - - let context_data = vec![&app_id[..], &instance_id[..], b"app-disk-crypt-key"]; - let app_disk_key = kdf::derive_dh_secret(&self.state.root_ca.key, &context_data) - .context("Failed to derive app disk key")?; - let env_crypt_key = { - let secret = - kdf::derive_dh_secret(&self.state.root_ca.key, &[&app_id[..], b"env-encrypt-key"]) - .context("Failed to derive env encrypt key")?; - let secret = x25519_dalek::StaticSecret::from(secret); - secret.to_bytes() - }; - - let (k256_key, k256_signature) = { - let (k256_app_key, signature) = derive_k256_key(&self.state.k256_key, &app_id) - .context("Failed to derive app ecdsa key")?; - (k256_app_key.to_bytes().to_vec(), signature) - }; + let root = self.state.epoch_root(root_epoch)?; + let keys = derive_app_keys(root, &boot_info.app_id, &boot_info.instance_id)?; Ok(AppKeyResponse { - ca_cert: self.state.root_ca.pem_cert.clone(), - disk_crypt_key: app_disk_key.to_vec(), - env_crypt_key: env_crypt_key.to_vec(), - k256_key, - k256_signature, + ca_cert: root.ca.pem_cert.clone(), + disk_crypt_key: keys.disk_crypt_key, + env_crypt_key: keys.env_crypt_key, + k256_key: keys.k256_key, + k256_signature: keys.k256_signature, tproxy_app_id: gateway_app_id.clone(), gateway_app_id, - os_image_hash, + os_image_hash: boot_info.os_image_hash, + root_epoch, }) } async fn get_app_env_encrypt_pub_key(self, request: AppId) -> Result { - self.ensure_self_allowed() + let epoch = self + .ensure_self_allowed(Operation::KmsServe) .await .context("KMS self authorization failed")?; + let root = self.state.epoch_root(epoch)?; ensure_app_id_len(&request.app_id)?; let secret = kdf::derive_dh_secret( - &self.state.root_ca.key, + &root.ca.key, &[&request.app_id[..], "env-encrypt-key".as_bytes()], ) .context("Failed to derive env encrypt key")?; @@ -458,7 +686,7 @@ impl KmsRpc for RpcHandler { // Legacy signature (without timestamp) for backward compatibility let signature = sign_message( - &self.state.k256_key, + &root.k256, b"dstack-env-encrypt-pubkey", &request.app_id, &public_key, @@ -467,7 +695,7 @@ impl KmsRpc for RpcHandler { // New signature with timestamp to prevent replay attacks let signature_v1 = sign_message_with_timestamp( - &self.state.k256_key, + &root.k256, b"dstack-env-encrypt-pubkey", &request.app_id, timestamp, @@ -493,16 +721,12 @@ impl KmsRpc for RpcHandler { .try_get_with((), self.state.config.auth_api.get_info()) .await .map_err(|err| anyhow!("{err:#}"))?; + let epoch = self.state.keyring.servable_epoch(info.root_epochs)?; + let root = self.state.epoch_root(epoch)?; Ok(GetMetaResponse { - ca_cert: self.state.inner.root_ca.pem_cert.clone(), + ca_cert: root.ca.pem_cert.clone(), allow_any_upgrade: self.state.inner.config.auth_api.is_dev(), - k256_pubkey: self - .state - .inner - .k256_key - .verifying_key() - .to_sec1_bytes() - .to_vec(), + k256_pubkey: root.k256.verifying_key().to_sec1_bytes().to_vec(), bootstrap_info, is_dev: self.state.config.auth_api.is_dev(), kms_contract_address: info.kms_contract_address, @@ -520,26 +744,26 @@ impl KmsRpc for RpcHandler { self.handover_keys(request).await } - /// Serve the temp CA certificate and key. - /// - /// Retained for older guest images and tokenless KMS onboarding. Current - /// guests use self-issued RA-TLS certificates and fetch the root CA via GetMeta. - /// The returned key authenticates nobody; client identity comes from attestation. async fn get_temp_ca_cert(self) -> Result { - let self_boot_info = self - .ensure_self_allowed() - .await - .context("KMS self authorization failed")?; - ensure_self_key_release_allowed(self_boot_info.as_ref(), (&self.state.config).into())?; + let epoch = self.ensure_self_allowed(Operation::KmsServe).await?; + if self.state.config.enforce_self_authorization { + let boot_info = self + .state + .self_boot_info + .get(&()) + .await + .context("missing authorized KMS boot info")?; + ensure_key_release_allowed(&boot_info, (&self.state.config).into())?; + } Ok(GetTempCaCertResponse { - temp_ca_cert: self.state.inner.temp_ca_cert.clone(), - temp_ca_key: self.state.inner.temp_ca_key.clone(), - ca_cert: self.state.inner.root_ca.pem_cert.clone(), + temp_ca_cert: self.state.temp_ca.pem_cert.clone(), + temp_ca_key: self.state.temp_ca.key.serialize_pem(), + ca_cert: self.state.epoch_root(epoch)?.ca.pem_cert.clone(), }) } async fn sign_cert(self, request: SignCertRequest) -> Result { - self.ensure_self_allowed() + self.ensure_self_allowed(Operation::KmsServe) .await .context("KMS self authorization failed")?; let csr = match request.api_version { @@ -566,11 +790,19 @@ impl KmsRpc for RpcHandler { .verify_with_ra_pubkey(&csr.pubkey, &self.state.attestation_verifier) .await .context("Quote verification failed")?; + let boot_info = build_boot_info_for_attestation(&attestation, true, &request.vm_config)?; let app_info = self - .ensure_app_attestation_allowed(&attestation, false, true, &request.vm_config) + .ensure_app_attestation_allowed( + &attestation, + boot_info, + &request.vm_config, + Operation::AppGetCert, + None, + ) .await?; ensure_key_release_allowed(&app_info.boot_info, (&self.state.config).into())?; - let app_ca = self.derive_app_ca(&app_info.boot_info.app_id)?; + let root = self.state.epoch_root(app_info.root_epoch)?; + let app_ca = self.derive_app_ca(&app_info.boot_info.app_id, root)?; // Stamp the identity verified above, not the requester's own claims. // Runtime measurements (`false`) are what certificates have always carried. let cert_app_info = attestation @@ -583,7 +815,7 @@ impl KmsRpc for RpcHandler { certificate_chain: vec![ cert.pem(), app_ca.pem_cert.clone(), - self.state.root_ca.pem_cert.clone(), + root.ca.pem_cert.clone(), ], }) } @@ -1171,16 +1403,6 @@ mod tests { .expect("TCB/advisory policy should be decided by the auth API, not this local gate"); } - #[test] - fn snp_self_boot_info_uses_same_release_policy_for_temp_ca() { - let boot_info = snp_boot_info(); - - ensure_self_key_release_allowed(Some(&boot_info), KeyReleasePolicy::default()) - .expect_err("disabled SNP self boot info must not receive temp CA key material"); - ensure_self_key_release_allowed(Some(&boot_info), SNP_ENABLED) - .expect("enabled clean SNP self boot info should pass the temp CA release gate"); - } - #[test] fn nitro_enclave_key_release_requires_explicit_enablement() { let mut boot_info = snp_boot_info(); @@ -1198,13 +1420,6 @@ mod tests { // explicitly enabled permits it ensure_key_release_allowed(&boot_info, NITRO_ENCLAVE_ENABLED) .expect("explicitly enabled nitro enclave key release should allow the boot info"); - - // the temp CA path uses the same gate - ensure_self_key_release_allowed(Some(&boot_info), KeyReleasePolicy::default()).expect_err( - "disabled nitro enclave self boot info must not receive temp CA key material", - ); - ensure_self_key_release_allowed(Some(&boot_info), NITRO_ENCLAVE_ENABLED) - .expect("enabled nitro enclave self boot info should pass the temp CA release gate"); } #[test] @@ -1280,19 +1495,8 @@ mod tests { .build() .self_signed() .unwrap(); - let tmp_ca_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256).unwrap(); - let tmp_ca_cert = CertRequest::builder() - .org_name("Dstack") - .subject("Dstack Client Temp CA") - .ca_level(0) - .key(&tmp_ca_key) - .build() - .self_signed() - .unwrap(); fs::write(cert_dir.join("root-ca.key"), ca_key.serialize_pem()).unwrap(); fs::write(cert_dir.join("root-ca.crt"), ca_cert.pem()).unwrap(); - fs::write(cert_dir.join("tmp-ca.key"), tmp_ca_key.serialize_pem()).unwrap(); - fs::write(cert_dir.join("tmp-ca.crt"), tmp_ca_cert.pem()).unwrap(); fs::write( cert_dir.join("root-k256.key"), SigningKey::random(&mut rand::rngs::OsRng).to_bytes(), @@ -1324,11 +1528,149 @@ mod tests { .focus("core") .extract() .unwrap(); - KmsState::new(config).unwrap() + let keyring = Keyring::load(&config).unwrap(); + KmsState::new(config, keyring, ca_cert.pem()).unwrap() + } + + #[rocket::async_test] + async fn authorization_selects_epochs_without_a_serving_epoch_snapshot() { + let dir = tempfile::tempdir().unwrap(); + let initial = kms_state(dir.path(), "http://127.0.0.1:1"); + let mut keyring = Keyring::load(&initial.config).unwrap(); + let key = KeyPair::generate_for(&ra_tls::rcgen::PKCS_ECDSA_P256_SHA256).unwrap(); + let cert = CertRequest::builder() + .key(&key) + .subject("Root epoch 1") + .ca_level(1) + .build() + .self_signed() + .unwrap(); + keyring.roots.push(RootKeys { + ca: CaCert::from_parts(key, cert), + k256: SigningKey::random(&mut rand::rngs::OsRng), + }); + keyring.chain_hashes = + root_chain_hashes(keyring.roots.iter().map(|root| (&root.ca.key, &root.k256))); + let state = KmsState::new( + initial.config.clone(), + keyring, + initial.keyring.roots[0].ca.pem_cert.clone(), + ) + .unwrap(); + let result = |epoch: u32| AuthorizationResult { + is_allowed: true, + rejection_reason: String::new(), + gateway_app_id: String::new(), + current_root_epoch: epoch, + current_app_epoch: 0, + current_root_epoch_hash: hex::encode(state.keyring.chain_hashes[epoch as usize]), + }; + assert_eq!(state.authorized_epoch(&result(0), None).unwrap(), 0); + assert!(state.authorized_epoch(&result(0), Some(1)).is_err()); + assert_eq!(state.authorized_epoch(&result(1), None).unwrap(), 1); + assert_eq!(state.authorized_epoch(&result(1), Some(0)).unwrap(), 0); + assert_eq!(state.authorized_epoch(&result(1), Some(1)).unwrap(), 1); + assert!(state.authorized_epoch(&result(1), Some(u32::MAX)).is_err()); + // Key selection already follows epoch 1; only the TLS listener needs refreshing. + assert!(state.ensure_rpc_ca(1).is_err()); + assert!( + tokio::time::timeout(Duration::from_millis(10), state.restart_requested()) + .await + .is_ok() + ); + assert_eq!(state.rpc_ca_pubkey, initial.rpc_ca_pubkey); + + let mut rejected = result(1); + rejected.is_allowed = false; + rejected.rejection_reason = "Policy denied".into(); + assert!(state.authorized_epoch(&rejected, Some(0)).is_err()); + rejected.is_allowed = true; + for hash in [ + "invalid".into(), + hex::encode([0; 32]), + hex::encode([0x55; 32]), + ] { + rejected.current_root_epoch_hash = hash; + assert!(state.authorized_epoch(&rejected, None).is_err()); + } + let mut missing = result(1); + missing.current_root_epoch = u32::MAX; + assert!(state.authorized_epoch(&missing, Some(0)).is_err()); + assert!(state.epoch_root(u32::MAX).is_err()); } const AUTH_API_INFO: &str = r#"{"status":"ok","kmsContractAddr":"0xkms","ethRpcUrl":"https://rpc.example","gatewayAppId":"0xgateway","chainId":1,"appImplementation":"0ximpl"}"#; + #[rocket::async_test] + async fn self_authorization_cache_cannot_authorize_a_different_operation() { + use upgrade_authority::tests::{boot_info, serve}; + let dir = tempfile::tempdir().unwrap(); + let initial = kms_state(dir.path(), "http://127.0.0.1:1"); + let response = |allowed| { + serde_json::json!({ + "isAllowed": allowed, "rejectionReason": "export denied", "gatewayAppId": "", + "currentRootEpoch": 0, "currentAppEpoch": 0, + "currentRootEpochHash": hex::encode(initial.keyring.chain_hashes[0]), + }) + .to_string() + }; + let (url, server) = serve(vec![response(true), response(false)]); + let mut config = initial.config.clone(); + config.enforce_self_authorization = true; + config.auth_api = upgrade_authority::tests::webhook(url); + let keyring = Keyring::load(&config).unwrap(); + let state = KmsState::new( + config, + keyring, + initial.epoch_root(0).unwrap().ca.pem_cert.clone(), + ) + .unwrap(); + state.self_boot_info.insert((), boot_info(1)).await; + let handler = RpcHandler { + state, + attestation: None, + }; + assert_eq!( + handler + .ensure_self_allowed(Operation::KmsServe) + .await + .unwrap(), + 0 + ); + assert!(handler + .ensure_self_allowed(Operation::KmsExportKeys) + .await + .is_err()); + assert_eq!( + handler + .ensure_self_allowed(Operation::KmsServe) + .await + .unwrap(), + 0 + ); + let requests = server.join().unwrap(); + assert_eq!(requests[0].1["operation"], "kms_serve"); + assert_eq!(requests[1].1["operation"], "kms_export_keys"); + } + + #[rocket::async_test] + async fn legacy_temp_ca_rpc_returns_a_separate_ca_and_the_served_root() { + assert!(dstack_kms_rpc::kms_server::KMSMethod::from_str("GetTempCaCert").is_some()); + let dir = tempfile::tempdir().unwrap(); + let (url, _) = serve_concurrent_auth_api(AUTH_API_INFO); + let state = kms_state(dir.path(), &url); + let response = RpcHandler { + state: state.clone(), + attestation: None, + } + .get_temp_ca_cert() + .await + .unwrap(); + let temp_ca = CaCert::new(response.temp_ca_cert, response.temp_ca_key).unwrap(); + assert_ne!(temp_ca.key.public_key_der(), state.rpc_ca_pubkey); + assert_eq!(response.ca_cert, state.epoch_root(0).unwrap().ca.pem_cert); + } + #[rocket::async_test] async fn concurrent_get_meta_calls_share_one_auth_api_request() { let dir = tempfile::tempdir().unwrap(); diff --git a/dstack/kms/src/main_service/upgrade_authority.rs b/dstack/kms/src/main_service/upgrade_authority.rs index 79f4a7c23..c265a555b 100644 --- a/dstack/kms/src/main_service/upgrade_authority.rs +++ b/dstack/kms/src/main_service/upgrade_authority.rs @@ -8,6 +8,7 @@ use anyhow::{bail, Context, Result}; use dstack_guest_agent_rpc::v0::{ dstack_guest_client::DstackGuestClient, AttestResponse, RawQuoteArgs, }; +pub(crate) use dstack_types::authorization::AuthorizationOperation as Operation; use http_client::prpc::PrpcClient; use ra_tls::attestation::{AttestationVerifier, VerifiedAttestation, VersionedAttestation}; use serde::de::DeserializeOwned; @@ -78,6 +79,7 @@ pub(crate) async fn local_kms_boot_info(verifier: &AttestationVerifier) -> Resul build_boot_info_for_attestation(&verified, false, "") } +#[cfg(test)] #[derive(Debug, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] pub(crate) struct BootResponse { @@ -86,6 +88,91 @@ pub(crate) struct BootResponse { pub reason: String, } +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(crate) struct AuthorizationResult { + pub is_allowed: bool, + pub rejection_reason: String, + pub gateway_app_id: String, + pub current_root_epoch: u32, + pub current_app_epoch: u32, + pub current_root_epoch_hash: String, +} + +impl AuthorizationResult { + pub fn current_root(&self) -> Result> { + let chain_hash: [u8; 32] = + hex::decode(self.current_root_epoch_hash.trim_start_matches("0x")) + .ok() + .and_then(|hash| hash.try_into().ok()) + .context("invalid authorization root epoch chain hash")?; + if chain_hash == [0; 32] { + return Ok(None); + } + Ok(Some(CurrentRootEpoch { + epoch: self.current_root_epoch, + chain_hash, + })) + } +} + +/// The root epoch fields: the KMS sends the epoch it serves in boot checks, +/// and auth backends report the current one in info responses. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub(crate) struct RootEpochFields { + #[serde(default, skip_serializing_if = "Option::is_none")] + pub root_epoch: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub root_epoch_hash: Option, +} + +impl From for RootEpochFields { + fn from(current: CurrentRootEpoch) -> Self { + Self { + root_epoch: Some(current.epoch), + root_epoch_hash: Some(format!("0x{}", hex::encode(current.chain_hash))), + } + } +} + +impl RootEpochFields { + pub fn registry(&self) -> Result { + match (self.root_epoch, &self.root_epoch_hash) { + (None, None) => Ok(RootEpochRegistry::Empty), + (Some(epoch), Some(hash)) => { + let chain_hash = hex::decode(hash.trim_start_matches("0x")) + .ok() + .and_then(|hash| hash.try_into().ok()) + .context("invalid root epoch chain hash")?; + Ok(RootEpochRegistry::Current(CurrentRootEpoch { + epoch, + chain_hash, + })) + } + _ => bail!("the auth API reports a root epoch without its chain hash"), + } + } +} + +/// The root epoch governance made current, with the chain hash committing to +/// the root public keys of epochs 0..=epoch. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct CurrentRootEpoch { + pub epoch: u32, + pub chain_hash: [u8; 32], +} + +/// What governance says about root epochs. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) enum RootEpochRegistry { + /// The dev backend: there is nothing to check against. + Unchecked, + /// No epoch is registered, or the backend predates root epochs. + Empty, + Current(CurrentRootEpoch), +} + #[derive(Debug, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] pub(crate) struct AuthApiInfoResponse { @@ -96,6 +183,8 @@ pub(crate) struct AuthApiInfoResponse { pub gateway_app_id: String, pub chain_id: u64, pub app_implementation: String, + #[serde(flatten)] + pub root_epoch: RootEpochFields, } #[derive(Debug, Clone, Serialize, Deserialize)] @@ -107,6 +196,7 @@ pub(crate) struct GetInfoResponse { pub eth_rpc_url: Option, pub chain_id: Option, pub app_implementation: Option, + pub root_epochs: RootEpochRegistry, } const AUTH_API_CONNECT_TIMEOUT: Duration = Duration::from_secs(5); @@ -167,7 +257,91 @@ async fn send_request(req: reqwest::RequestBuilder, url: &s } impl AuthApi { - pub async fn is_app_allowed(&self, boot_info: &BootInfo, is_kms: bool) -> Result { + pub async fn authorize_kms( + &self, + app_info: &BootInfo, + operation: Operation, + requested_root_epoch: Option, + ) -> Result { + if !operation.is_kms() { + bail!("invalid KMS authorization operation"); + } + self.check_authorization(app_info, "authorizeKms", operation, requested_root_epoch) + .await + } + + pub async fn authorize_app( + &self, + app_info: &BootInfo, + operation: Operation, + requested_root_epoch: Option, + ) -> Result { + if operation.is_kms() { + bail!("invalid app authorization operation"); + } + self.check_authorization(app_info, "authorizeApp", operation, requested_root_epoch) + .await + } + + async fn check_authorization( + &self, + app_info: &BootInfo, + method: &str, + operation: Operation, + requested_root_epoch: Option, + ) -> Result { + #[derive(Serialize)] + #[serde(rename_all = "camelCase")] + struct AuthorizationRequest<'a> { + #[serde(flatten)] + app_info: &'a BootInfo, + operation: Operation, + requested_app_epoch: u32, + #[serde(skip_serializing_if = "Option::is_none")] + requested_root_epoch: Option, + } + ensure_identity_widths(app_info)?; + match self { + AuthApi::Dev { dev } => Ok(AuthorizationResult { + is_allowed: true, + rejection_reason: String::new(), + gateway_app_id: dev.gateway_app_id.clone(), + current_root_epoch: 0, + current_app_epoch: 0, + current_root_epoch_hash: hex::encode([0; 32]), + }), + AuthApi::Webhook { webhook } => { + let url = url_join(&webhook.url, method); + http_post( + &url, + &AuthorizationRequest { + app_info, + operation, + requested_app_epoch: 0, + requested_root_epoch, + }, + webhook.timeout, + ) + .await + } + } + } + + #[cfg(test)] + pub async fn is_app_allowed( + &self, + boot_info: &BootInfo, + is_kms: bool, + root_epoch: Option, + ) -> Result { + #[derive(Serialize)] + struct BootAuthRequest<'a> { + #[serde(flatten)] + boot_info: &'a BootInfo, + #[serde(flatten)] + root_epoch: RootEpochFields, + } + ensure_identity_widths(boot_info)?; match self { AuthApi::Dev { dev } => Ok(BootResponse { @@ -182,7 +356,11 @@ impl AuthApi { "bootAuth/app" }; let url = url_join(&webhook.url, path); - http_post(&url, &boot_info, webhook.timeout).await + let request = BootAuthRequest { + boot_info, + root_epoch: root_epoch.map(Into::into).unwrap_or_default(), + }; + http_post(&url, &request, webhook.timeout).await } } } @@ -196,6 +374,7 @@ impl AuthApi { gateway_app_id: Some(dev.gateway_app_id.clone()), chain_id: None, app_implementation: None, + root_epochs: RootEpochRegistry::Unchecked, }), AuthApi::Webhook { webhook } => { let info: AuthApiInfoResponse = http_get(&webhook.url, webhook.timeout).await?; @@ -211,6 +390,7 @@ impl AuthApi { chain_id: Some(info.chain_id), gateway_app_id: Some(info.gateway_app_id.clone()), app_implementation: Some(info.app_implementation.clone()), + root_epochs: info.root_epoch.registry()?, }) } } @@ -255,20 +435,24 @@ pub(crate) async fn ensure_self_kms_allowed( .context("failed to build local KMS boot info")?; let response = cfg .auth_api - .is_app_allowed(&boot_info, true) + .authorize_kms(&boot_info, Operation::KmsBootstrap, None) .await - .context("failed to call KMS auth check")?; + .context("failed to call KMS bootstrap authorization")?; if !response.is_allowed { - bail!("boot denied: {}", response.reason); + bail!("boot denied: {}", response.rejection_reason); } Ok(()) } +/// `allow_legacy_cert` accepts a source whose certificate predates the unified +/// attestation format, judging its image by mrAggregated alone. pub(crate) async fn ensure_kms_allowed( cfg: &KmsConfig, attestation: &VerifiedAttestation, verifier: &AttestationVerifier, -) -> Result<()> { + allow_legacy_cert: bool, + root_epoch: u32, +) -> Result { let mut boot_info = build_boot_info_for_attestation(attestation, false, "") .context("failed to build KMS boot info from attestation")?; // Workaround: old source KMS instances use the legacy cert format (separate TDX_QUOTE + @@ -277,6 +461,9 @@ pub(crate) async fn ensure_kms_allowed( // A source that presents a config but no os_image_hash is not legacy and gets no fallback. // TODO: remove once all source KMS instances use the unified PHALA_RATLS_ATTESTATION format. if boot_info.os_image_hash.is_empty() && attestation.config.is_empty() { + if !allow_legacy_cert { + bail!("the source KMS certificate does not carry its OS image hash"); + } warn!("source KMS uses the legacy cert format, falling back to the local os_image_hash"); let local_info = local_kms_boot_info(verifier) .await @@ -285,17 +472,23 @@ pub(crate) async fn ensure_kms_allowed( } let response = cfg .auth_api - .is_app_allowed(&boot_info, true) + .authorize_kms(&boot_info, Operation::KmsExportKeys, Some(root_epoch)) .await - .context("failed to call KMS auth check")?; + .context("failed to authorize source KMS key export")?; if !response.is_allowed { - bail!("boot denied: {}", response.reason); + bail!("boot denied: {}", response.rejection_reason); } - Ok(()) + if cfg.auth_api.is_dev() { + return Ok(RootEpochRegistry::Unchecked); + } + let current = response + .current_root()? + .context("root epoch not registered")?; + Ok(RootEpochRegistry::Current(current)) } #[cfg(test)] -mod tests { +pub(crate) mod tests { use super::*; use crate::config::Webhook; use ra_tls::attestation::TeeVariant; @@ -304,7 +497,7 @@ mod tests { use std::net::TcpListener; use std::thread; - fn boot_info(identity: u8) -> BootInfo { + pub(crate) fn boot_info(identity: u8) -> BootInfo { BootInfo { tee_variant: TeeVariant::DstackTdx, mr_aggregated: vec![identity; 32], @@ -320,7 +513,7 @@ mod tests { } } - fn serve>( + pub(crate) fn serve>( responses: Vec, ) -> (String, thread::JoinHandle>) { let responses: Vec = responses.into_iter().map(Into::into).collect(); @@ -364,9 +557,12 @@ mod tests { .nth(1) .unwrap() .to_string(); - let body = - serde_json::from_slice(&request[header_end..header_end + content_length]) - .unwrap(); + let body = &request[header_end..header_end + content_length]; + let body = if body.is_empty() { + Value::Null + } else { + serde_json::from_slice(body).unwrap() + }; requests.push((path, body)); write!( stream, @@ -381,7 +577,7 @@ mod tests { (format!("http://{address}"), handle) } - fn webhook(url: String) -> AuthApi { + pub(crate) fn webhook(url: String) -> AuthApi { AuthApi::Webhook { webhook: Webhook { url, @@ -436,7 +632,7 @@ mod tests { let mut aliasing_compose_hash = boot_info(1); aliasing_compose_hash.compose_hash.truncate(31); let err = auth - .is_app_allowed(&aliasing_compose_hash, false) + .is_app_allowed(&aliasing_compose_hash, false, None) .await .unwrap_err(); assert!(err.to_string().contains("compose_hash must be 32 bytes")); @@ -444,7 +640,7 @@ mod tests { let mut aliasing_instance_id = boot_info(1); aliasing_instance_id.instance_id.truncate(19); let err = auth - .is_app_allowed(&aliasing_instance_id, false) + .is_app_allowed(&aliasing_instance_id, false, None) .await .unwrap_err(); assert!(err @@ -455,7 +651,7 @@ mod tests { let mut no_instance_id = boot_info(1); no_instance_id.instance_id.clear(); assert!( - auth.is_app_allowed(&no_instance_id, false) + auth.is_app_allowed(&no_instance_id, false, None) .await .unwrap() .is_allowed @@ -472,8 +668,8 @@ mod tests { let auth = webhook(url); let info = boot_info(1); - let first = auth.is_app_allowed(&info, false).await.unwrap(); - let repeated = auth.is_app_allowed(&info, false).await.unwrap(); + let first = auth.is_app_allowed(&info, false, None).await.unwrap(); + let repeated = auth.is_app_allowed(&info, false, None).await.unwrap(); assert!(first.is_allowed); assert!(!repeated.is_allowed); assert_eq!(repeated.reason, "revoked"); @@ -495,8 +691,18 @@ mod tests { let app = boot_info(10); let kms = boot_info(20); - assert!(auth.is_app_allowed(&app, false).await.unwrap().is_allowed); - assert!(auth.is_app_allowed(&kms, true).await.unwrap().is_allowed); + assert!( + auth.is_app_allowed(&app, false, None) + .await + .unwrap() + .is_allowed + ); + assert!( + auth.is_app_allowed(&kms, true, None) + .await + .unwrap() + .is_allowed + ); let requests = server.join().unwrap(); assert_eq!(requests[0].0, "/bootAuth/app"); @@ -509,6 +715,146 @@ mod tests { assert_ne!(requests[0].1["mrAggregated"], requests[1].1["mrAggregated"]); } + #[rocket::async_test] + async fn authorization_operations_are_typed_and_sent_without_legacy_fallback() { + let operations = [ + Operation::KmsBootstrap, + Operation::KmsServe, + Operation::KmsExportKeys, + Operation::KmsImportKeys, + Operation::AppGetKey, + Operation::AppGetCert, + ]; + let response = r#"{"isAllowed":true,"rejectionReason":"","gatewayAppId":"","currentRootEpoch":0,"currentAppEpoch":0,"currentRootEpochHash":"0000000000000000000000000000000000000000000000000000000000000000"}"#; + let (url, server) = serve(vec![response; operations.len()]); + let auth = webhook(url); + for operation in operations { + if operation.is_kms() { + auth.authorize_kms(&boot_info(1), operation, Some(0)) + .await + .unwrap(); + } else { + auth.authorize_app(&boot_info(1), operation, None) + .await + .unwrap(); + } + } + for ((route, body), operation) in server.join().unwrap().into_iter().zip(operations) { + assert_eq!( + route, + if operation.is_kms() { + "/authorizeKms" + } else { + "/authorizeApp" + } + ); + assert_eq!( + serde_json::from_value::(body["operation"].clone()).unwrap(), + operation + ); + } + assert!(serde_json::from_str::(r#""unknown""#).is_err()); + } + + #[rocket::async_test] + async fn authorization_returns_the_epoch_with_each_decision() { + let response = |allowed: bool, epoch: u32| { + format!( + r#"{{"isAllowed":{allowed},"rejectionReason":"","gatewayAppId":"gw", "currentRootEpoch":{epoch},"currentAppEpoch":0,"currentRootEpochHash":"0x{}"}}"#, + "11".repeat(32) + ) + }; + let (url, server) = serve(vec![ + response(true, 0), + response(false, 1), + response(true, 1), + response(true, 1), + ]); + let auth = webhook(url); + let info = boot_info(50); + let first = auth + .authorize_app(&info, Operation::AppGetKey, None) + .await + .unwrap(); + let second = auth + .authorize_app(&info, Operation::AppGetKey, None) + .await + .unwrap(); + assert!(first.is_allowed); + assert!(!second.is_allowed); + assert_eq!(first.current_root().unwrap().unwrap().epoch, 0); + assert_eq!(second.current_root().unwrap().unwrap().epoch, 1); + auth.authorize_app(&info, Operation::AppGetKey, Some(0)) + .await + .unwrap(); + auth.authorize_kms(&info, Operation::KmsServe, None) + .await + .unwrap(); + let requests = server.join().unwrap(); + assert_eq!(requests.len(), 4); + assert!(requests[..3] + .iter() + .all(|(path, _)| path == "/authorizeApp")); + assert_eq!(requests[3].0, "/authorizeKms"); + let mut expected = serde_json::to_value(info).unwrap(); + expected["requestedAppEpoch"] = 0.into(); + expected["operation"] = "app_get_key".into(); + assert_eq!(requests[0].1, expected); + assert_eq!(requests[2].1["requestedAppEpoch"], 0); + assert!(requests[0].1.get("requestedRootEpoch").is_none()); + assert_eq!(requests[2].1["requestedRootEpoch"], 0); + } + + #[rocket::async_test] + async fn authorization_without_epoch_fields_is_not_accepted_as_legacy() { + let (url, server) = serve(vec![ + r#"{"isAllowed":true,"reason":"","gatewayAppId":"gw"}"#, + ]); + assert!(webhook(url) + .authorize_app(&boot_info(50), Operation::AppGetKey, None) + .await + .is_err()); + assert_eq!(server.join().unwrap().len(), 1); + } + + #[rocket::async_test] + async fn root_epochs_are_read_from_info_and_sent_in_boot_checks() { + let hash = "0x".to_string() + &"11".repeat(32); + let (url, server) = serve(vec![ + format!( + r#"{{"status":"ok","kmsContractAddr":"0x00","gatewayAppId":"gw","chainId":1, + "appImplementation":"0x00","rootEpoch":3,"rootEpochHash":"{hash}"}}"# + ), + r#"{"status":"ok","kmsContractAddr":"0x00","gatewayAppId":"gw","chainId":1, + "appImplementation":"0x00","rootEpoch":3}"# + .into(), + r#"{"isAllowed":true,"gatewayAppId":"gw","reason":""}"#.into(), + r#"{"isAllowed":true,"gatewayAppId":"gw","reason":""}"#.into(), + ]); + let auth = webhook(url); + let current = CurrentRootEpoch { + epoch: 3, + chain_hash: [0x11; 32], + }; + + assert_eq!( + auth.get_info().await.unwrap().root_epochs, + RootEpochRegistry::Current(current) + ); + assert!(auth.get_info().await.is_err()); + let info = boot_info(50); + auth.is_app_allowed(&info, false, Some(current)) + .await + .unwrap(); + auth.is_app_allowed(&info, true, None).await.unwrap(); + + let requests = server.join().unwrap(); + assert_eq!(requests[2].1["rootEpoch"], 3); + assert_eq!(requests[2].1["rootEpochHash"], hash); + assert_eq!(requests[2].1["appId"], requests[3].1["appId"]); + assert!(requests[3].1.get("rootEpoch").is_none()); + } + #[rocket::async_test] async fn malformed_backend_response_fails_closed_and_next_request_recovers() { let (url, server) = serve(vec![ @@ -518,9 +864,9 @@ mod tests { let auth = webhook(url); let info = boot_info(30); - let error = auth.is_app_allowed(&info, false).await.unwrap_err(); + let error = auth.is_app_allowed(&info, false, None).await.unwrap_err(); assert!(error.to_string().contains("failed to decode response")); - let recovered = auth.is_app_allowed(&info, false).await.unwrap(); + let recovered = auth.is_app_allowed(&info, false, None).await.unwrap(); assert!(recovered.is_allowed); assert_eq!(recovered.reason, "recovered"); assert_eq!(server.join().unwrap().len(), 2); @@ -537,7 +883,7 @@ mod tests { let (url, server) = serve(vec![body]); let error = webhook(url) - .is_app_allowed(&boot_info(40), false) + .is_app_allowed(&boot_info(40), false, None) .await .unwrap_err(); diff --git a/dstack/kms/src/onboard_service.rs b/dstack/kms/src/onboard_service.rs index 480f9b62a..e7b944433 100644 --- a/dstack/kms/src/onboard_service.rs +++ b/dstack/kms/src/onboard_service.rs @@ -36,9 +36,10 @@ use tracing::info; use crate::{ config::KmsConfig, main_service::{ - build_boot_info_for_attestation, + build_boot_info_for_attestation, root_chain_hashes, servable_epoch, upgrade_authority::{ app_attest, dstack_client, ensure_kms_allowed, ensure_self_kms_allowed, pad64, + RootEpochRegistry, }, }, }; @@ -146,16 +147,7 @@ impl OnboardRpc for OnboardHandler { let keys = Keys::generate(&request.domain, self.state.config.attest_rpc_cert) .await .context("Failed to generate keys")?; - - let k256_pubkey = keys.k256_key.verifying_key().to_sec1_bytes().to_vec(); - let ca_pubkey = keys.ca_key.public_key_der(); - let attestation = attest_keys(&ca_pubkey, &k256_pubkey).await?; - - let response = BootstrapResponse { - ca_pubkey, - k256_pubkey, - attestation, - }; + let response = keys.attest().await?; // Store the bootstrap info safe_write(cfg.bootstrap_info(), serde_json::to_vec(&response)?)?; keys.store(cfg)?; @@ -180,13 +172,29 @@ impl OnboardRpc for OnboardHandler { &source_url, &request.source_token, &request.domain, + request.rotate, self.state.attestation_verifier.clone(), ) .await .context("Failed to onboard")?; - let k256_pubkey = keys.k256_key.verifying_key().to_sec1_bytes().to_vec(); + if !request.rotate { + let k256_pubkey = keys.k256_key.verifying_key().to_sec1_bytes().to_vec(); + keys.store(cfg).context("Failed to store keys")?; + return Ok(OnboardResponse { + k256_pubkey, + ..Default::default() + }); + } + // The attestation over the new epoch is what governance registers, so + // it is persisted with the keys and stays readable via GetSetupStatus. + let info = keys.attest().await?; + safe_write(cfg.bootstrap_info(), serde_json::to_vec(&info)?)?; keys.store(cfg).context("Failed to store keys")?; - Ok(OnboardResponse { k256_pubkey }) + Ok(OnboardResponse { + k256_pubkey: info.k256_pubkey, + ca_pubkey: info.ca_pubkey, + attestation: info.attestation, + }) } async fn get_attestation_info(self) -> Result { @@ -287,6 +295,8 @@ mod tests { use crate::main_service::amd_attest::{ compute_expected_measurement, MeasurementInput, OvmfSectionParam, }; + use crate::main_service::upgrade_authority::CurrentRootEpoch; + use dstack_kms_rpc::KmsKeys; use sha2::Digest; fn hex_of(byte: u8, len: usize) -> String { @@ -536,6 +546,210 @@ mod tests { assert!(status.attestation.is_empty()); } + fn handover(epochs: u32) -> KmsKeyResponse { + KmsKeyResponse { + temp_ca_key: String::new(), + keys: (0..epochs) + .map(|_| KmsKeys { + ca_key: KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256) + .unwrap() + .serialize_pem(), + k256_key: SigningKey::random(&mut rand::rngs::OsRng) + .to_bytes() + .to_vec(), + }) + .collect(), + } + } + + /// The registry a correct handover of `res` would be checked against. + fn current_of(res: &KmsKeyResponse, epoch: u32) -> RootEpochRegistry { + let roots: Vec<_> = res + .keys + .iter() + .map(|k| { + ( + KeyPair::from_pem(&k.ca_key).unwrap().public_key_der(), + SigningKey::from_slice(&k.k256_key) + .unwrap() + .verifying_key() + .to_sec1_bytes() + .to_vec(), + ) + }) + .collect(); + let hashes = dstack_types::root_epoch::chain_hashes( + roots.iter().map(|(ca, k256)| (&ca[..], &k256[..])), + ); + RootEpochRegistry::Current(CurrentRootEpoch { + epoch, + chain_hash: hashes[epoch as usize], + }) + } + + async fn onboard( + cfg: &KmsConfig, + res: KmsKeyResponse, + rotate: bool, + registry: RootEpochRegistry, + ) -> Result { + Keys::from_key_response(cfg, res, "kms.example.com", rotate, registry).await + } + + #[test] + fn compatibility_ca_survives_restart_and_recovers_a_missing_certificate() { + let dir = tempfile::tempdir().unwrap(); + let cfg = auto_bootstrap_config(dir.path()); + let first = ensure_temp_ca(&cfg).unwrap(); + let second = ensure_temp_ca(&cfg).unwrap(); + assert_eq!(first.key.public_key_der(), second.key.public_key_der()); + assert_eq!(first.pem_cert, second.pem_cert); + fs::remove_file(cfg.tmp_ca_cert()).unwrap(); + let recovered = ensure_temp_ca(&cfg).unwrap(); + assert_eq!(first.key.public_key_der(), recovered.key.public_key_der()); + assert!(!cfg.root_keys_exist()); + } + + #[rocket::async_test] + async fn join_imports_and_stores_every_epoch() { + let cert_dir = tempfile::tempdir().unwrap(); + let cfg = auto_bootstrap_config(cert_dir.path()); + let res = handover(3); + let sent: Vec<_> = res.keys.iter().map(|k| k.k256_key.clone()).collect(); + + let keys = onboard(&cfg, res, false, RootEpochRegistry::Unchecked) + .await + .unwrap(); + keys.store(&cfg).unwrap(); + assert!(cfg.tmp_ca_key().exists()); + assert!(cfg.tmp_ca_cert().exists()); + + assert_eq!(crate::main_service::read_root_epoch(&cfg).unwrap(), 2); + for (epoch, k256) in (0..).zip(&sent) { + assert_eq!( + &fs::read(cfg.root_key_paths(epoch, 2).k256_key).unwrap(), + k256 + ); + } + assert_eq!(&fs::read(cfg.k256_key()).unwrap(), &sent[2]); + } + + #[rocket::async_test] + async fn rotation_imports_the_keyring_and_generates_the_next_epoch() { + let cert_dir = tempfile::tempdir().unwrap(); + let cfg = auto_bootstrap_config(cert_dir.path()); + let res = handover(2); + let registry = current_of(&res, 1); + let sent: Vec<_> = res.keys.iter().map(|k| k.k256_key.clone()).collect(); + + let keys = onboard(&cfg, res, true, registry).await.unwrap(); + assert_eq!(keys.epoch, 2); + keys.store(&cfg).unwrap(); + + assert_eq!( + fs::read(cfg.root_key_paths(1, 2).k256_key).unwrap(), + sent[1] + ); + let current = fs::read(cfg.k256_key()).unwrap(); + assert!(!sent.contains(¤t)); + } + + #[rocket::async_test] + async fn onboarding_checks_the_handover_against_the_current_epoch() { + let cert_dir = tempfile::tempdir().unwrap(); + let cfg = auto_bootstrap_config(cert_dir.path()); + let res = handover(2); + let other = handover(2); + + // A join may hold an epoch pending activation beyond the current one. + assert!(onboard(&cfg, res.clone(), false, current_of(&res, 1)) + .await + .is_ok()); + assert!(onboard(&cfg, res.clone(), false, current_of(&res, 0)) + .await + .is_ok()); + // A rotation must start at the current epoch. + assert!(onboard(&cfg, res.clone(), true, current_of(&res, 0)) + .await + .is_err()); + // A keyring other than the registered one. + assert!(onboard(&cfg, res.clone(), false, current_of(&other, 1)) + .await + .is_err()); + assert!(onboard(&cfg, res.clone(), true, current_of(&other, 1)) + .await + .is_err()); + // A source behind the current epoch. + assert!(onboard(&cfg, handover(1), false, current_of(&res, 1)) + .await + .is_err()); + // A missing epoch shifts the rest, so the chain hash no longer matches. + let mut gap = res.clone(); + gap.keys.remove(0); + assert!(onboard(&cfg, gap, false, current_of(&res, 0)) + .await + .is_err()); + assert!( + onboard(&cfg, handover(0), false, RootEpochRegistry::Unchecked) + .await + .is_err() + ); + } + + #[rocket::async_test] + async fn an_unregistered_deployment_can_only_join_at_epoch_0() { + let cert_dir = tempfile::tempdir().unwrap(); + let cfg = auto_bootstrap_config(cert_dir.path()); + assert!(onboard(&cfg, handover(1), false, RootEpochRegistry::Empty) + .await + .is_ok()); + assert!(onboard(&cfg, handover(2), false, RootEpochRegistry::Empty) + .await + .is_err()); + assert!(onboard(&cfg, handover(1), true, RootEpochRegistry::Empty) + .await + .is_err()); + } + + /// A stored keyring at `latest`, loaded back. + async fn stored_keyring(epochs: u32) -> (tempfile::TempDir, KmsConfig, KmsKeyResponse) { + let cert_dir = tempfile::tempdir().unwrap(); + let cfg = auto_bootstrap_config(cert_dir.path()); + let res = handover(epochs); + onboard(&cfg, res.clone(), false, RootEpochRegistry::Unchecked) + .await + .unwrap() + .store(&cfg) + .unwrap(); + (cert_dir, cfg, res) + } + + #[rocket::async_test] + async fn a_keyring_serves_the_current_epoch_it_holds() { + use crate::main_service::Keyring; + + let (_dir, cfg, res) = stored_keyring(3).await; + let keyring = Keyring::load(&cfg).unwrap(); + assert_eq!(keyring.latest(), 2); + assert_eq!(keyring.servable_epoch(current_of(&res, 1)).unwrap(), 1); + assert_eq!(keyring.servable_epoch(current_of(&res, 2)).unwrap(), 2); + assert_eq!( + keyring + .servable_epoch(RootEpochRegistry::Unchecked) + .unwrap(), + 2 + ); + assert!(keyring.servable_epoch(RootEpochRegistry::Empty).is_err()); + assert!(keyring.servable_epoch(current_of(&handover(3), 1)).is_err()); + let mut ahead = handover(4); + ahead.keys[..3].clone_from_slice(&res.keys); + assert!(keyring.servable_epoch(current_of(&ahead, 3)).is_err()); + + let (_dir, cfg, _) = stored_keyring(1).await; + let keyring = Keyring::load(&cfg).unwrap(); + assert_eq!(keyring.servable_epoch(RootEpochRegistry::Empty).unwrap(), 0); + } + #[rocket::async_test] async fn setup_status_includes_bootstrap_info_when_present() { let cert_dir = tempfile::tempdir().unwrap(); @@ -607,10 +821,29 @@ mod tests { } } +/// Root keys of an epoch older than the current one. +struct HistoricalRoot { + ca_key: KeyPair, + ca_cert: Certificate, + k256_key: SigningKey, +} + +impl HistoricalRoot { + fn new(ca_key: KeyPair, k256_key: SigningKey) -> Result { + let ca_cert = self_signed_root_ca(&ca_key)?; + Ok(Self { + ca_key, + ca_cert, + k256_key, + }) + } +} + struct Keys { + /// Epoch of the root keys below; `history` holds epochs `0..epoch`. + epoch: u32, + history: Vec, k256_key: SigningKey, - tmp_ca_key: KeyPair, - tmp_ca_cert: Certificate, ca_key: KeyPair, ca_cert: Certificate, rpc_key: KeyPair, @@ -620,45 +853,33 @@ struct Keys { impl Keys { async fn generate(domain: &str, attest_rpc_cert: bool) -> Result { - let tmp_ca_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; let ca_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; let rpc_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; let k256_key = SigningKey::random(&mut rand::rngs::OsRng); - Self::from_keys( - tmp_ca_key, - ca_key, - rpc_key, - k256_key, - domain, - attest_rpc_cert, - ) - .await + Self::from_keys(ca_key, rpc_key, k256_key, domain, attest_rpc_cert).await + } + + /// Attest the current root public keys. Epoch 0 keeps the original + /// bootstrap encoding; later epochs also bind the epoch number. + async fn attest(&self) -> Result { + let ca_pubkey = self.ca_key.public_key_der(); + let k256_pubkey = self.k256_key.verifying_key().to_sec1_bytes().to_vec(); + let attestation = attest_keys(self.epoch, &ca_pubkey, &k256_pubkey).await?; + Ok(BootstrapResponse { + ca_pubkey, + k256_pubkey, + attestation, + }) } async fn from_keys( - tmp_ca_key: KeyPair, ca_key: KeyPair, rpc_key: KeyPair, k256_key: SigningKey, domain: &str, attest_rpc_cert: bool, ) -> Result { - let tmp_ca_cert = CertRequest::builder() - .org_name("Dstack") - .subject("Dstack Client Temp CA") - .ca_level(0) - .key(&tmp_ca_key) - .build() - .self_signed()?; - - // Create self-signed KMS cert - let ca_cert = CertRequest::builder() - .org_name("Dstack") - .subject("Dstack KMS CA") - .ca_level(1) - .key(&ca_key) - .build() - .self_signed()?; + let ca_cert = self_signed_root_ca(&ca_key)?; // The only place the KMS embeds its own attestation. Skipping it lets // the KMS run outside a TEE for development; it does not affect the // verification of quotes presented *to* the KMS, which is a separate @@ -689,9 +910,9 @@ impl Keys { .build() .signed_by(&ca_cert, &ca_key)?; Ok(Keys { + epoch: 0, + history: Vec::new(), k256_key, - tmp_ca_key, - tmp_ca_cert, ca_key, ca_cert, rpc_key, @@ -705,6 +926,7 @@ impl Keys { other_kms_url: &str, source_token: &str, domain: &str, + rotate: bool, attestation_verifier: Arc, ) -> Result { let attestation_slot = Arc::new(Mutex::new(None::)); @@ -730,17 +952,14 @@ impl Keys { .context("Missing source KMS attestation") }; - // With a token, `other_kms_url` is the source's admin listener, which accepts - // a self-issued RA-TLS client certificate. Without one, the source predates - // 0.6.0: it serves `KMS.GetKmsKey` publicly and pins its temp CA. - let tmp_ca = if source_token.is_empty() { + let temp_ca = if source_token.is_empty() { let client = RaClient::new(other_kms_url.to_string(), true)?; - let tmp_ca = KmsClient::new(client).get_temp_ca_cert().await?; - Some((tmp_ca.temp_ca_cert, tmp_ca.temp_ca_key)) + let ca = KmsClient::new(client).get_temp_ca_cert().await?; + Some((ca.temp_ca_cert, ca.temp_ca_key)) } else { None }; - let (ra_cert, ra_key) = gen_ra_cert(tmp_ca).await?; + let (ra_cert, ra_key) = gen_ra_cert(temp_ca).await?; let client = RaClientConfig::builder() .tls_skip_ca_verification(true) .tls_built_in_root_certs(false) @@ -763,52 +982,92 @@ impl Keys { }; // The attestation is captured only on the connection that returned the keys; // nothing received is used before the source passes this check. - ensure_kms_allowed(cfg, &source_attestation()?, &attestation_verifier) - .await - .context("Source KMS is not allowed for onboarding")?; - Self::from_key_response(cfg, keys_res, domain).await + let highest = keys_res + .keys + .len() + .checked_sub(1) + .context("source KMS handed over no keys")?; + let highest = u32::try_from(highest).context("too many root epochs")?; + let registry = ensure_kms_allowed( + cfg, + &source_attestation()?, + &attestation_verifier, + !rotate, + highest, + ) + .await + .context("source KMS is not allowed for onboarding")?; + Self::from_key_response(cfg, keys_res, domain, rotate, registry).await } + /// Import the source's whole keyring, epochs 0..=latest; a rotation then + /// generates epoch latest+1. async fn from_key_response( cfg: &KmsConfig, keys_res: KmsKeyResponse, domain: &str, + rotate: bool, + registry: RootEpochRegistry, ) -> Result { - if keys_res.keys.len() != 1 { - return Err(anyhow::anyhow!("Invalid keys")); - } - let keys = keys_res.keys[0].clone(); - let tmp_ca_key_pem = keys_res.temp_ca_key; - let root_ca_key_pem = keys.ca_key; - let root_k256_key = keys.k256_key; + let latest = keys_res + .keys + .len() + .checked_sub(1) + .context("source KMS handed over no keys")? as u32; + let mut roots = keys_res + .keys + .into_iter() + .map(|keys| { + let ca_key = KeyPair::from_pem(&keys.ca_key).context("Failed to parse CA key")?; + let k256_key = + SigningKey::from_slice(&keys.k256_key).context("Failed to parse ECDSA key")?; + Ok((ca_key, k256_key)) + }) + .collect::>>()?; + ensure_handover_registered(&roots, registry, rotate)?; + let epoch = if rotate { latest + 1 } else { latest }; + let (ca_key, k256_key) = if rotate { + ( + KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?, + SigningKey::random(&mut rand::rngs::OsRng), + ) + } else { + roots.pop().context("source KMS handed over no keys")? + }; + let history = roots + .into_iter() + .map(|(ca_key, k256_key)| HistoricalRoot::new(ca_key, k256_key)) + .collect::>>()?; let rpc_key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; - let ca_key = KeyPair::from_pem(&root_ca_key_pem).context("Failed to parse CA key")?; - let tmp_ca_key = - KeyPair::from_pem(&tmp_ca_key_pem).context("Failed to parse tmp CA key")?; - let ecdsa_key = - SigningKey::from_slice(&root_k256_key).context("Failed to parse ECDSA key")?; - Self::from_keys( - tmp_ca_key, - ca_key, - rpc_key, - ecdsa_key, - domain, - cfg.attest_rpc_cert, - ) - .await + let keys = Self::from_keys(ca_key, rpc_key, k256_key, domain, cfg.attest_rpc_cert).await?; + Ok(Self { + epoch, + history, + ..keys + }) } /// The root keys go last: see [`KmsConfig::root_keys_exist`]. fn store(&self, cfg: &KmsConfig) -> Result<()> { + ensure_temp_ca(cfg)?; + for (epoch, root) in (0..).zip(&self.history) { + let paths = cfg.root_key_paths(epoch, self.epoch); + if let Some(dir) = paths.ca_key.parent() { + fs::create_dir_all(dir)?; + } + safe_write(&paths.ca_cert, root.ca_cert.pem())?; + safe_write_with_mode(&paths.ca_key, root.ca_key.serialize_pem(), 0o600)?; + safe_write_with_mode(&paths.k256_key, root.k256_key.to_bytes(), 0o600)?; + } self.store_certs(cfg)?; safe_write(cfg.rpc_domain(), self.rpc_domain.as_bytes())?; + safe_write(cfg.root_epoch(), self.epoch.to_string())?; self.store_keys(cfg)?; Ok(()) } fn store_keys(&self, cfg: &KmsConfig) -> Result<()> { - safe_write_with_mode(cfg.tmp_ca_key(), self.tmp_ca_key.serialize_pem(), 0o600)?; safe_write_with_mode(cfg.rpc_key(), self.rpc_key.serialize_pem(), 0o600)?; safe_write_with_mode(cfg.root_ca_key(), self.ca_key.serialize_pem(), 0o600)?; safe_write_with_mode(cfg.k256_key(), self.k256_key.to_bytes(), 0o600)?; @@ -816,21 +1075,52 @@ impl Keys { } fn store_certs(&self, cfg: &KmsConfig) -> Result<()> { - safe_write(cfg.tmp_ca_cert(), self.tmp_ca_cert.pem())?; safe_write(cfg.root_ca_cert(), self.ca_cert.pem())?; safe_write(cfg.rpc_cert(), self.rpc_cert.pem())?; Ok(()) } } -pub(crate) async fn update_certs(cfg: &KmsConfig) -> Result<()> { +/// Keep the public compatibility CA independent of all epoch root keys. +pub(crate) fn ensure_temp_ca(cfg: &KmsConfig) -> Result { + let had_key = cfg.tmp_ca_key().exists(); + let key = if had_key { + KeyPair::from_pem(&fs::read_to_string(cfg.tmp_ca_key())?)? + } else { + let key = KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256)?; + safe_write_with_mode(cfg.tmp_ca_key(), key.serialize_pem(), 0o600)?; + key + }; + let cert = CertRequest::builder() + .org_name("Dstack") + .subject("Dstack Client Temp CA") + .ca_level(0) + .key(&key) + .build() + .self_signed()?; + if had_key && cfg.tmp_ca_cert().exists() { + renew_ca_cert_if_expiring( + cfg.tmp_ca_cert(), + cert.pem(), + "temporary client CA certificate", + )?; + } else { + safe_write(cfg.tmp_ca_cert(), cert.pem())?; + } + CaCert::load(cfg.tmp_ca_cert(), cfg.tmp_ca_key()) +} + +/// Reissue the RPC certificate under the root CA of the served `epoch`, so +/// guests reach this KMS through the CA of the keys it serves. +pub(crate) async fn update_certs(cfg: &KmsConfig, epoch: u32, latest: u32) -> Result<()> { + ensure_temp_ca(cfg)?; + let roots = cfg.root_key_paths(epoch, latest); // Read existing keys - let tmp_ca_key = KeyPair::from_pem(&fs::read_to_string(cfg.tmp_ca_key())?)?; - let ca_key = KeyPair::from_pem(&fs::read_to_string(cfg.root_ca_key())?)?; + let ca_key = KeyPair::from_pem(&fs::read_to_string(&roots.ca_key)?)?; let rpc_key = KeyPair::from_pem(&fs::read_to_string(cfg.rpc_key())?)?; // Read k256 key - let k256_key_bytes = fs::read(cfg.k256_key())?; + let k256_key_bytes = fs::read(&roots.k256_key)?; let k256_key = SigningKey::from_slice(&k256_key_bytes)?; let domain = if cfg.onboard.auto_bootstrap_domain.is_empty() { @@ -841,28 +1131,15 @@ pub(crate) async fn update_certs(cfg: &KmsConfig) -> Result<()> { let domain = domain.trim(); // Regenerate certificates using existing keys - let keys = Keys::from_keys( - tmp_ca_key, - ca_key, - rpc_key, - k256_key, - domain, - cfg.attest_rpc_cert, - ) - .await - .context("Failed to regenerate certificates")?; + let keys = Keys::from_keys(ca_key, rpc_key, k256_key, domain, cfg.attest_rpc_cert) + .await + .context("Failed to regenerate certificates")?; renew_ca_cert_if_expiring( - cfg.root_ca_cert(), + &roots.ca_cert, keys.ca_cert.pem(), "KMS root CA certificate", )?; - renew_ca_cert_if_expiring( - cfg.tmp_ca_cert(), - keys.tmp_ca_cert.pem(), - "temporary client CA certificate", - )?; - // The RPC leaf depends on the refreshed domain and platform attestation, so // it is reissued on every startup. safe_write(cfg.rpc_cert(), keys.rpc_cert.pem())?; @@ -922,10 +1199,48 @@ pub(crate) async fn bootstrap_keys(cfg: &KmsConfig, verifier: &AttestationVerifi Ok(()) } -async fn attest_keys(p256_pubkey: &[u8], k256_pubkey: &[u8]) -> Result> { +/// Check the handed-over roots against governance: they must include the +/// current epoch with a matching chain hash. A rotation must start from the +/// current epoch with nothing pending activation, or it would collide with the +/// next registered epoch. +fn ensure_handover_registered( + roots: &[(KeyPair, SigningKey)], + registry: RootEpochRegistry, + rotate: bool, +) -> Result<()> { + if rotate && registry == RootEpochRegistry::Empty { + bail!("register root epoch 0 before rotating"); + } + let hashes = root_chain_hashes(roots.iter().map(|(ca, k256)| (ca, k256))); + let current = servable_epoch(&hashes, registry)?; + let latest = roots.len() as u32 - 1; + if rotate && current != latest { + bail!( + "the source holds root epochs up to {latest}, but {current} is current; rotate from \ + a source at the current epoch with no epoch pending activation" + ); + } + Ok(()) +} + +fn self_signed_root_ca(ca_key: &KeyPair) -> Result { + CertRequest::builder() + .org_name("Dstack") + .subject("Dstack KMS CA") + .ca_level(1) + .key(ca_key) + .build() + .self_signed() +} + +async fn attest_keys(epoch: u32, p256_pubkey: &[u8], k256_pubkey: &[u8]) -> Result> { let p256_hex = hex::encode(p256_pubkey); let k256_hex = hex::encode(k256_pubkey); - let content_to_quote = format!("dstack-kms-genereted-keys-v1:{p256_hex};{k256_hex};"); + let content_to_quote = if epoch == 0 { + format!("dstack-kms-genereted-keys-v1:{p256_hex};{k256_hex};") + } else { + format!("dstack-kms-generated-keys-v2:{epoch};{p256_hex};{k256_hex};") + }; let hash = keccak256(content_to_quote.as_bytes()); let report_data = pad64(hash); let res = app_attest(report_data).await?; @@ -939,7 +1254,7 @@ fn keccak256(msg: &[u8]) -> [u8; 32] { hasher.finalize().into() } -/// Mint an RA-TLS client certificate, signed by `ca` (cert and key PEM) or self-issued. +/// Legacy sources require a temp-CA-issued client certificate. async fn gen_ra_cert(ca: Option<(String, String)>) -> Result<(String, String)> { use ra_tls::cert::CertRequest; use ra_tls::rcgen::{KeyPair, PKCS_ECDSA_P256_SHA256}; @@ -953,14 +1268,15 @@ async fn gen_ra_cert(ca: Option<(String, String)>) -> Result<(String, String)> { let attestation = VersionedAttestation::from_bytes(&response.attestation).context("Invalid attestation")?; let req = CertRequest::builder() - .subject("RA-TLS TEMP Cert") + .subject("RA-TLS Client Cert") .attestation(&attestation) .key(&key) + .usage_client_auth(true) .build(); let cert = match ca { - Some((cert_pem, key_pem)) => CaCert::new(cert_pem, key_pem)?.sign(req), + Some((cert, key)) => CaCert::new(cert, key)?.sign(req), None => req.self_signed(), } - .context("Failed to sign certificate")?; + .context("failed to sign client certificate")?; Ok((cert.pem(), key.serialize_pem())) } diff --git a/dstack/kms/src/www/onboard.html b/dstack/kms/src/www/onboard.html index 4794cf724..d476e64e8 100644 --- a/dstack/kms/src/www/onboard.html +++ b/dstack/kms/src/www/onboard.html @@ -261,6 +261,12 @@

Onboard from an Existing KMS Instance

+
+ +
Onboard from an Existing KMS Instance sourceUrl: '', sourceToken: '', onboardDomain: '', + rotate: false, error: '', success: '', result: '', @@ -350,7 +357,7 @@

Onboard from an Existing KMS Instance

return; } if (data.attestation) { - this.success = 'Bootstrap successful!'; + this.success = 'New root keys generated!'; this.result = JSON.stringify({ caPubkey: '0x' + data.ca_pubkey, k256Pubkey: '0x' + data.k256_pubkey, @@ -391,7 +398,8 @@

Onboard from an Existing KMS Instance

const data = await rpcCall('Onboard', { source_url: this.sourceUrl + "/prpc", source_token: this.sourceToken, - domain: this.onboardDomain + domain: this.onboardDomain, + rotate: this.rotate }); if (data.error) throw new Error(data.error); diff --git a/dstack/ra-rpc/src/ratls_client_verifier.rs b/dstack/ra-rpc/src/ratls_client_verifier.rs index d3c467808..f886aac5b 100644 --- a/dstack/ra-rpc/src/ratls_client_verifier.rs +++ b/dstack/ra-rpc/src/ratls_client_verifier.rs @@ -61,7 +61,7 @@ impl ClientCertVerifier for RaTlsClientVerifier { /// Anonymous connections stay allowed, matching the previous /// `[rpc.tls.mutual] mandatory = false`. Handlers that need an attested caller - /// enforce it themselves; the unauthenticated RPCs (`GetMeta`, `GetTempCaCert`) + /// enforce it themselves; the unauthenticated RPCs (such as `GetMeta`) /// are reachable without a certificate by design. fn client_auth_mandatory(&self) -> bool { false diff --git a/dstack/tests/docs/kms-bootstrap-onboard.md b/dstack/tests/docs/kms-bootstrap-onboard.md index f9f4d3234..ac8b32164 100644 --- a/dstack/tests/docs/kms-bootstrap-onboard.md +++ b/dstack/tests/docs/kms-bootstrap-onboard.md @@ -76,7 +76,7 @@ Operational notes: 2. `Boot Progress: done` does **not** guarantee the onboard endpoint is ready. 3. The onboarding completion endpoint is **GET `/finish`**. 4. On teepod with gateway, onboard mode usually uses the `-8000` URL, while runtime TLS KMS RPC usually uses the `-8000s` URL. **Port forwarding** (`--port tcp:0.0.0.0::8000`) is simpler than gateway for testing, because gateway requires the auth API to return a `gatewayAppId` at boot time. -5. If you use a very small custom webhook instead of the real auth service, `KMS.GetMeta` may fail because `auth_api.get_info()` expects extra chain / contract metadata fields. In that case, use `GetTempCaCert` as the runtime readiness probe. +5. If you use a very small custom webhook instead of the real auth service, `KMS.GetMeta` may fail because `auth_api.get_info()` expects extra chain / contract metadata fields. In that case, use `GetAppEnvEncryptPubKey` as the runtime readiness probe. 6. dstack CVMs use QEMU user-mode networking — the host is reachable at **`10.0.2.2`** from inside the CVM. The `source_url` in `Onboard.Onboard` must use a CVM-reachable address (e.g., `https://10.0.2.2:/prpc`), not `127.0.0.1`. 7. **~~Remote KMS attestation has an empty `osImageHash`.~~** Fixed: RA-TLS certs now use the unified `PHALA_RATLS_ATTESTATION` format which preserves `vm_config`. For old source KMS instances, the receiver-side check fills `osImageHash` from the local KMS's own value automatically. No special `"0x"` entry in `osImages` is needed anymore. @@ -205,7 +205,7 @@ curl -sk "${KMS_SRC_RUNTIME%/}/prpc/KMS.GetMeta?json" \ - `k256_pubkey` - `bootstrap_info` -If `KMS.GetMeta` fails because your minimal webhook does not return chain / contract info, use `GetTempCaCert` below as the runtime readiness probe instead. +If `KMS.GetMeta` fails because your minimal webhook does not return chain / contract info, use `GetAppEnvEncryptPubKey` below as the runtime readiness probe instead. --- @@ -273,24 +273,25 @@ This section folds the runtime trusted-RPC verification into the same flow. | bootstrap deny | source-side auth leaves `kms.mrAggregated` empty or omits the current `kms-src` MR | `Onboard.Bootstrap` on `kms-src` | `KMS is not allowed to bootstrap`, `MR aggregated not allowed` | | onboard deny (receiver-side) | destination-side auth leaves `kms.mrAggregated` empty or omits the current `kms-src` MR | `Onboard.Onboard` on `kms-dst` | source KMS not allowed / onboarding failed | | onboard deny (source-side) | source-side auth leaves `kms.mrAggregated` empty or omits the current `kms-dst` MR | `Onboard.Onboard` on `kms-dst` | source rejected destination caller / `GetKmsKey` authorization failed | -| runtime deny | auth removes the running KMS from `kms.mrAggregated` | `GetTempCaCert` or another trusted RPC | `KMS self authorization failed`, `KMS is not allowed` | +| runtime deny | auth removes the running KMS from `kms.mrAggregated` | `GetAppEnvEncryptPubKey` or another trusted RPC | `KMS self authorization failed`, `KMS is not allowed` | Use the happy-path steps below first, then flip policies one by one and rerun the indicated probe. -### 7.1 Minimum canary: `GetTempCaCert` +### 7.1 Minimum canary: `GetAppEnvEncryptPubKey` ```bash -curl -sk "${KMS_SRC_RUNTIME%/}/prpc/KMS.GetTempCaCert?json" \ - | tee /tmp/kms-bootstrap-onboard/kms-src-get-temp-ca.json | jq . +curl -sk "${KMS_SRC_RUNTIME%/}/prpc/KMS.GetAppEnvEncryptPubKey?json" \ + -H 'Content-Type: application/json' -d '{"app_id":"1111111111111111111111111111111111111111"}' \ + | tee /tmp/kms-bootstrap-onboard/kms-src-get-env-pubkey.json | jq . ``` Expected result: - success - response contains: - - `temp_ca_cert` - - `temp_ca_key` - - `ca_cert` + - `public_key` + - `signature_v1` + - `timestamp` ### 7.2 `GetKmsKey` @@ -332,8 +333,9 @@ Expected result: After a normal happy-path run, flip source-side auth policy to deny `kms-src` itself and retry: ```bash -curl -sk "${KMS_SRC_RUNTIME%/}/prpc/KMS.GetTempCaCert?json" \ - | tee /tmp/kms-bootstrap-onboard/kms-src-get-temp-ca-after-deny.json | jq . +curl -sk "${KMS_SRC_RUNTIME%/}/prpc/KMS.GetAppEnvEncryptPubKey?json" \ + -H 'Content-Type: application/json' -d '{"app_id":"1111111111111111111111111111111111111111"}' \ + | tee /tmp/kms-bootstrap-onboard/kms-src-get-env-pubkey-after-deny.json | jq . ``` Expected result: @@ -405,8 +407,9 @@ Then restore both source-side and destination-side allowlists and rerun the norm After a successful bootstrap or onboard, remove the running KMS's own MR from `kms.mrAggregated` and retry: ```bash -curl -sk "${KMS_SRC_RUNTIME%/}/prpc/KMS.GetTempCaCert?json" \ - | tee /tmp/kms-bootstrap-onboard/kms-src-get-temp-ca-denied.json | jq . +curl -sk "${KMS_SRC_RUNTIME%/}/prpc/KMS.GetAppEnvEncryptPubKey?json" \ + -H 'Content-Type: application/json' -d '{"app_id":"1111111111111111111111111111111111111111"}' \ + | tee /tmp/kms-bootstrap-onboard/kms-src-get-env-pubkey-denied.json | jq . ``` Expected result: @@ -414,7 +417,7 @@ Expected result: - response contains `.error` - error indicates KMS self authorization failed or the KMS is not allowed -You can repeat the same check on `kms-dst` after onboard by removing `kms-dst` from destination-side policy and retrying `KMS.GetTempCaCert`. +You can repeat the same check on `kms-dst` after onboard by removing `kms-dst` from destination-side policy and retrying `KMS.GetAppEnvEncryptPubKey`. --- @@ -427,8 +430,8 @@ For each run, save: 3. onboard response 4. `/finish` responses 5. runtime `KMS.GetMeta` responses -6. trusted RPC responses such as `GetTempCaCert` -7. deny-case responses such as `kms-src-bootstrap-denied.json`, `kms-dst-onboard-denied.json`, `kms-dst-onboard-denied-by-src.json`, and `kms-src-get-temp-ca-denied.json` +6. trusted RPC responses such as `GetAppEnvEncryptPubKey` +7. deny-case responses such as `kms-src-bootstrap-denied.json`, `kms-dst-onboard-denied.json`, `kms-dst-onboard-denied-by-src.json`, and `kms-src-get-env-pubkey-denied.json` 8. auth policy snapshots used during the run Recommended archive: @@ -447,8 +450,8 @@ The flow is considered validated if all of the following are true: 2. `kms-src` transitions to runtime mode successfully 3. `kms-dst` onboard succeeds against `kms-src` 4. `kms-dst` transitions to runtime mode successfully -5. runtime metadata probes succeed on both KMS instances, or `GetTempCaCert` succeeds when `GetMeta` is unavailable with a minimal webhook -6. at least one trusted runtime RPC such as `GetTempCaCert` succeeds +5. runtime metadata probes succeed on both KMS instances, or `GetAppEnvEncryptPubKey` succeeds when `GetMeta` is unavailable with a minimal webhook +6. at least one trusted runtime RPC such as `GetAppEnvEncryptPubKey` succeeds 7. the selected deny cases fail at the expected RPC with an authorization error --- diff --git a/dstack/tests/docs/kms-self-authorization.md b/dstack/tests/docs/kms-self-authorization.md index d9e365fd9..88b33c787 100644 --- a/dstack/tests/docs/kms-self-authorization.md +++ b/dstack/tests/docs/kms-self-authorization.md @@ -6,7 +6,7 @@ The goal is to validate the following behaviors without depending on `kms/e2e/` 1. **Bootstrap self-check**: a KMS must call the auth API and verify that **itself** is allowed before bootstrap succeeds. 2. **Onboard receiver-side source check**: a new KMS must reject onboarding if the **source KMS** is not allowed by the receiver's auth policy. -3. **Trusted RPC self-check**: trusted KMS RPCs such as `GetTempCaCert`, `GetKmsKey`, `GetAppKey`, and `SignCert` must fail when the running KMS is no longer allowed by its auth policy. +3. **Trusted RPC self-check**: trusted KMS RPCs such as `GetAppEnvEncryptPubKey`, `GetKmsKey`, `GetAppKey`, and `SignCert` must fail when the running KMS is no longer allowed by its auth policy. 4. **Attestation requirement**: KMS always requires attestation; for local development without TDX hardware, use `sdk/simulator`. This guide is written as a deployment-and-test runbook so an AI agent can follow it end-to-end. @@ -589,15 +589,16 @@ Verify that a running KMS re-checks its own authorization on trusted RPCs. ### Recommended canary RPC -Use `GetTempCaCert` first. It is simpler than `GetAppKey` because it does not require preparing an attested app client, but it still exercises the new runtime self-check. +Use `GetAppEnvEncryptPubKey` first. It is simpler than `GetAppKey` because it does not require preparing an attested app client, but it still exercises the new runtime self-check. ### Steps 1. While `kms-src` is healthy, confirm the canary RPC works: ```bash -curl -sk "${KMS_SRC_RUNTIME%/}/prpc/KMS.GetTempCaCert?json" \ - | tee /tmp/kms-self-auth/get-temp-ca-allowed.json | jq . +curl -sk "${KMS_SRC_RUNTIME%/}/prpc/KMS.GetAppEnvEncryptPubKey?json" \ + -H 'Content-Type: application/json' -d '{"app_id":"1111111111111111111111111111111111111111"}' \ + | tee /tmp/kms-self-auth/get-env-pubkey-allowed.json | jq . ``` 2. Flip `auth-simple-src` to deny `kms-src` itself: @@ -609,13 +610,14 @@ cp /tmp/kms-self-auth/auth-src-deny-self.json /tmp/kms-self-auth/auth-src.json 3. Retry the same RPC: ```bash -curl -sk "${KMS_SRC_RUNTIME%/}/prpc/KMS.GetTempCaCert?json" \ - | tee /tmp/kms-self-auth/get-temp-ca-denied.json | jq . +curl -sk "${KMS_SRC_RUNTIME%/}/prpc/KMS.GetAppEnvEncryptPubKey?json" \ + -H 'Content-Type: application/json' -d '{"app_id":"1111111111111111111111111111111111111111"}' \ + | tee /tmp/kms-self-auth/get-env-pubkey-denied.json | jq . ``` ### Expected result -- before the policy flip: `GetTempCaCert` succeeds +- before the policy flip: `GetAppEnvEncryptPubKey` succeeds - after the policy flip: the response contains `.error` - the error should indicate KMS self-authorization failed, or that the KMS is not allowed @@ -667,7 +669,7 @@ For each run, save: 2. auth config snapshots used for each step 3. bootstrap/onboard RPC responses 4. `KMS.GetMeta` output after successful boot -5. `GetTempCaCert` allow/deny responses +5. `GetAppEnvEncryptPubKey` allow/deny responses 6. relevant CVM logs if a step fails unexpectedly Recommended archive: