feat(kms)!: refuse to serve without a key handover path

CHANGELOG.md

@@ -33,6 +33,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- KMS: a KMS requires a key handover path and otherwise refuses to start,
+  before onboarding or bootstrap, naming the missing setting: unless
+  `core.onboard.public_key_handover = true`, set `[core.admin] enabled = true`
+  and `[core.admin.tls]` with `key` and `certs`. There is no opt-out, dev auth
+  included. The built-in config now enables the admin listener with TLS on the
+  RPC key and certificate, so a KMS on it needs an admin token to start:
+  `core.admin.auth_token`, `DSTACK_KMS_ADMIN_TOKEN`, or the new
+  `core.admin.auth_token_hash` (hex SHA-256), which keeps the token out of a
+  public app compose; the hash is then public, so use a high-entropy random
+  token such as `openssl rand -hex 32`. When upgrading, add the admin TLS and a token, or enable
+  the public handover. The `dstack-app` composes and `dstackup` generate the
+  token and configure its hash; the deploy scripts publish the admin listener
+  when `KMS_ADMIN_ADDR` is set.
 - KMS: `core.onboard.public_key_handover` defaults to `false`, so
   `KMS.GetKmsKey` is no longer served on the public port. Onboard through the
   source's admin listener with `source_token`; a source that still onboards

docs/deployment.md

@@ -510,6 +510,8 @@ Additional KMS instances can onboard from an existing KMS to share the same root
 
 **Configure the source KMS for key handover:**
 
+Every KMS needs this, or `core.onboard.public_key_handover = true`: without a handover path it refuses to start, before onboarding or bootstrap, naming the missing setting. In a public config such as an app compose, set `auth_token_hash` (the token's SHA-256) instead of `auth_token`, with a high-entropy random token (`openssl rand -hex 32`), since the hash can be brute-forced offline.
+
 `Admin.GetKmsKey` only works on an admin listener with `[core.admin.tls]`: without TLS the listener speaks plain HTTP, so the new KMS can present no RA-TLS certificate and the call fails with `No attestation provided`. The admin listener binds to `127.0.0.1` by default; bind it to an address the new KMS can reach and restrict access to the KMS nodes at the network layer.
 
 ```toml

docs/rc-testing-runbook.md

@@ -154,7 +154,10 @@ KMS first, then gateway, then the app. Each needs the previous one's identity.
    simplest authorization that releases keys without a chain is
    `[core.auth_api] type = "dev"` with `gateway_app_id = "any"` — no extra
    containers, unlike the on-chain compose, which runs auth-eth next to the KMS
-   and needs an Ethereum RPC endpoint.
+   and needs an Ethereum RPC endpoint. The KMS refuses to start without a
+   key handover path, dev auth included: give it `[core.admin]` with
+   `[core.admin.tls]` (`key`, `certs`) and an `auth_token_hash`, or set
+   `core.onboard.public_key_handover = true`.
 
 3. **Gateway as a CVM** with `--kms`. Internal ports are RPC 8000, admin 8001,
    WireGuard 51820/udp, proxy 443.

docs/tutorials/kms-build-configuration.md

@@ -204,6 +204,19 @@ type = "webhook"
 [core.auth_api.webhook]
 url = "http://127.0.0.1:9200"
 
+# Admin listener, which serves the key handover to other KMS nodes; a serving
+# KMS requires it. ADMIN_TOKEN_HASH is the token's SHA-256, filled in by the
+# KMS CVM deployment tutorial.
+[core.admin]
+enabled = true
+address = "0.0.0.0"
+port = 9101
+auth_token_hash = "ADMIN_TOKEN_HASH"
+
+[core.admin.tls]
+key = "/etc/kms/certs/rpc.key"
+certs = "/etc/kms/certs/rpc.crt"
+
 # Onboarding Configuration
 [core.onboard]
 enabled = true
@@ -224,6 +237,7 @@ EOF
 | `[core]` | `cert_dir` | Directory for certificates |
 | `[core]` | `pccs_url` | PCCS endpoint for quote verification |
 | `[core.auth_api]` | `url` | auth-eth webhook service URL |
+| `[core.admin]` | `auth_token_hash` | SHA-256 of the admin token; the admin listener with `[core.admin.tls]` is required to serve |
 | `[core.onboard]` | `enabled` | Enable bootstrap/onboard mode |
 
 ## Step 5: Prepare the auth-eth Service

docs/tutorials/kms-cvm-deployment.md

@@ -135,6 +135,7 @@ services:
     image: registry.yourdomain.com/dstack-kms:fixed
     ports:
       - "9100:9100"
+      - "9101:9101"
     volumes:
       - /var/run/dstack.sock:/var/run/dstack.sock
       - kms-certs:/etc/kms/certs
@@ -184,6 +185,18 @@ configs:
       [core.auth_api.webhook]
       url = "http://127.0.0.1:9200"
 
+      # Admin listener: serves the key handover to other KMS nodes, so a
+      # KMS requires it. Only the token's SHA-256 goes into the compose.
+      [core.admin]
+      enabled = true
+      address = "0.0.0.0"
+      port = 9101
+      auth_token_hash = "ADMIN_TOKEN_HASH"
+
+      [core.admin.tls]
+      key = "/etc/kms/certs/rpc.key"
+      certs = "/etc/kms/certs/rpc.crt"
+
       [core.onboard]
       enabled = true
       auto_bootstrap_domain = ""
@@ -203,6 +216,10 @@ sed -i 's|kms.yourdomain.com|kms.your-actual-domain.com|g' docker-compose.yaml
 
 # KMS contract address (from contract deployment tutorial)
 sed -i "s|YOUR_CONTRACT_ADDRESS|$(cat ~/.dstack/secrets/kms-contract-address)|g" docker-compose.yaml
+
+# KMS admin token: keep the token, put only its SHA-256 in the compose
+(umask 077 && openssl rand -hex 32 > ~/.dstack/secrets/kms-admin-token)
+sed -i "s|ADMIN_TOKEN_HASH|$(tr -d '\n' < ~/.dstack/secrets/kms-admin-token | sha256sum | cut -d' ' -f1)|g" docker-compose.yaml
 ```
 
 This docker-compose uses a Docker `configs` section to inject a complete `kms.toml` into the container at `/etc/kms/kms.toml`, overriding the config baked into the image. This approach lets you change KMS configuration without rebuilding the Docker image.
@@ -215,6 +232,7 @@ This docker-compose uses a Docker `configs` section to inject a complete `kms.to
 | `[rpc.tls]` | TLS certificate paths for HTTPS |
 | `[core.image]` | OS image verification — downloads images from `download.dstack.org` to compute expected TDX measurements |
 | `[core.auth_api]` | Authentication via auth-eth webhook on localhost:9200 |
+| `[core.admin]` | Admin listener on port 9101 with TLS; other KMS nodes onboard through it with the admin token. Without it (or `core.onboard.public_key_handover = true`) the KMS does not start |
 | `[core.onboard]` | Bootstrap settings — `auto_bootstrap_domain` is empty so KMS enters onboard mode for manual bootstrap |
 
 > **Why manual bootstrap?** With `auto_bootstrap_domain` left empty, KMS starts in "onboard mode" — a plain HTTP server on port 9100 that waits for you to trigger bootstrap via an RPC call. This ensures `bootstrap-info.json` (containing the TDX attestation quote and public keys) is written to disk. You'll need this file later to register the KMS on-chain.

docs/upgrading-to-0.6.md

@@ -236,6 +236,13 @@ Configuration changes:
   `KMS.GetKmsKey` from the public port. While a 0.5.x KMS still onboards from
   this cluster, it can only use that public entry, so set the option to `true`
   on the source node, and remove it once every node is upgraded.
+- A KMS without a key handover path does not start. Configure the admin
+  listener with `[core.admin.tls]` (`key` and `certs`) and an admin token, or
+  set `core.onboard.public_key_handover = true`. The built-in default enables
+  the admin listener with TLS, so it needs `core.admin.auth_token`,
+  `DSTACK_KMS_ADMIN_TOKEN` or `core.admin.auth_token_hash` to start. In a KMS
+  app compose, which is public, use `auth_token_hash` with a high-entropy random
+  token (`openssl rand -hex 32`), since its hash can be brute-forced offline.
 - Auto-bootstrap refuses to overwrite existing root keys.
 
 If you use `auth-simple`, replace it with `dstack-auth`, which reads the same

dstack/crates/dstack-cli-core/src/config.rs

@@ -100,6 +100,9 @@ pub struct HostConfig {
     pub image_download_url: String,
     /// confidential-computing platform (selects SNP-specific KMS settings).
     pub platform: Platform,
+    /// hex SHA-256 of the KMS admin token; the KMS refuses to start without
+    /// an admin credential.
+    pub kms_admin_token_hash: String,
 }
 
 impl Default for HostConfig {
@@ -110,6 +113,7 @@ impl Default for HostConfig {
             os_image_hash: String::new(),
             image_download_url: DEFAULT_IMAGE_DOWNLOAD_URL.to_string(),
             platform: Platform::Tdx,
+            kms_admin_token_hash: String::new(),
         }
     }
 }
@@ -148,6 +152,17 @@ type = "webhook"
 [core.auth_api.webhook]
 url = "{webhook_url}"
 
+# the key handover path a serving KMS requires; the token stays on the host.
+[core.admin]
+enabled = true
+address = "127.0.0.1"
+port = 8001
+auth_token_hash = "{admin_token_hash}"
+
+[core.admin.tls]
+key = "/kms/certs/rpc.key"
+certs = "/kms/certs/rpc.crt"
+
 [core.onboard]
 enabled = true
 auto_bootstrap_domain = "{bootstrap_domain}"
@@ -164,6 +179,7 @@ port = 8000
         download_url = cfg.image_download_url,
         webhook_url = cfg.auth_webhook_url,
         bootstrap_domain = cfg.kms_bootstrap_domain,
+        admin_token_hash = cfg.kms_admin_token_hash,
     )
 }
 
@@ -478,10 +494,20 @@ mod tests {
         let cfg = HostConfig {
             auth_webhook_url: "http://10.0.2.2:8001".into(),
             kms_bootstrap_domain: "10.0.2.2".into(),
+            kms_admin_token_hash: "ab".repeat(32),
             ..Default::default()
         };
         let toml = kms_toml(&cfg);
         assert!(toml.contains("enforce_self_authorization = false"));
+        let v: toml::Value = toml::from_str(&toml).unwrap();
+        let admin = &v["core"]["admin"];
+        assert_eq!(admin["enabled"].as_bool(), Some(true));
+        assert_eq!(
+            admin["auth_token_hash"].as_str(),
+            Some("ab".repeat(32).as_str())
+        );
+        assert_eq!(admin["tls"]["key"].as_str(), Some("/kms/certs/rpc.key"));
+        assert_eq!(admin["tls"]["certs"].as_str(), Some("/kms/certs/rpc.crt"));
         assert!(toml.contains(r#"auto_bootstrap_domain = "10.0.2.2""#));
         assert!(toml.contains(r#"type = "webhook""#));
         assert!(toml.contains(r#"url = "http://10.0.2.2:8001""#));

dstack/crates/dstackup/src/install.rs

@@ -14,6 +14,7 @@ use dstack_cli_core::host::Platform;
 use dstack_cli_core::layout::{path_string, InstallLayout};
 use dstack_cli_core::vmm::Vmm;
 use dstack_cli_core::{host, ports, rpc};
+use sha2::{Digest, Sha256};
 use std::env;
 use std::fs;
 use std::path::{Path, PathBuf};
@@ -137,9 +138,16 @@ pub(crate) async fn cmd_install(mut o: InstallOpts, release_api_base_url: &str)
     // install state, so it authenticates automatically.
     let vmm_token = match existing_token {
         Some(t) => t,
-        None => generate_vmm_token()?,
+        None => generate_token()?,
     };
     write_token_file(&token_path, &vmm_token)?;
+    // the KMS admin token, which the KMS config holds only as a hash.
+    let kms_admin_token_path = layout.config_dir.join("kms-admin-token");
+    let kms_admin_token = match read_token_file(&kms_admin_token_path) {
+        Some(t) => t,
+        None => generate_token()?,
+    };
+    write_token_file(&kms_admin_token_path, &kms_admin_token)?;
 
     // 8. resolve the key provider - run our own unless told to use an existing
     //    one (TDX only; SNP has no SGX local provider).
@@ -183,6 +191,7 @@ pub(crate) async fn cmd_install(mut o: InstallOpts, release_api_base_url: &str)
         auth_webhook_url: format!("http://10.0.2.2:{}", o.auth_port),
         os_image_hash: os_image_hash.unwrap_or_default(),
         platform,
+        kms_admin_token_hash: hex::encode(Sha256::digest(kms_admin_token.as_bytes())),
         ..Default::default()
     };
     let kms = config::kms_toml(&host_cfg);
@@ -1216,8 +1225,8 @@ async fn wait_ready(client_url: &str, token: Option<&str>, timeout: Duration) ->
     }
 }
 
-/// mint a 256-bit management-API token, hex-encoded, from the OS RNG.
-fn generate_vmm_token() -> Result<String> {
+/// mint a 256-bit API token, hex-encoded, from the OS RNG.
+fn generate_token() -> Result<String> {
     let mut buf = [0u8; 32];
     let mut f = fs::File::open("/dev/urandom").context("opening /dev/urandom")?;
     std::io::Read::read_exact(&mut f, &mut buf).context("reading /dev/urandom")?;

dstack/kms/README.md

@@ -55,7 +55,11 @@ certs = "/etc/kms/certs/rpc.crt"
 ```
 
 The token can also be supplied via the `DSTACK_KMS_ADMIN_TOKEN` or
-`ADMIN_API_TOKEN` environment variables instead of the config file. The admin
+`ADMIN_API_TOKEN` environment variables instead of the config file, or as its
+hex SHA-256 in `auth_token_hash` (`printf '%s' "$TOKEN" | sha256sum`) for a
+config that is not secret, such as a KMS app compose. The hash is then public,
+so the token must be high-entropy random, such as `openssl rand -hex 32`: a
+weak token can be brute-forced offline from its hash. The admin
 listener fails closed: enabled with neither `auth_token` nor `htpasswd_file`
 (and `insecure_no_auth = false`) refuses to start. With `[core.admin.tls]` the
 listener serves HTTPS (use `http://` without it). Call it with, for example:
@@ -77,6 +81,17 @@ need no client certificate. `core.onboard.public_key_handover` is off by default
 `true` only to also serve `KMS.GetKmsKey` on the public listener for older
 clients.
 
+A KMS refuses to start serving without a handover path, so that other nodes can
+always onboard from it: unless `public_key_handover = true`, it requires
+`[core.admin] enabled = true` and `[core.admin.tls]` with `key` and `certs`, and
+the error names the missing setting. The check runs at startup, before
+onboarding or bootstrap, so a misconfigured node fails before it gets or
+generates root keys. There is no opt-out, `auth_api = "dev"` included. The default `kms.toml`
+enables the admin listener with TLS on the RPC key and certificate, so it
+starts only once an admin token is set (`auth_token`,
+`DSTACK_KMS_ADMIN_TOKEN` or `auth_token_hash`). The `dstack-app` composes and
+`dstackup` set `auth_token_hash` from a token they generate.
+
 Three settings relax authorization, each over a different scope:
 
 | Setting | Effect |

dstack/kms/dstack-app/compose-dev.yaml

@@ -39,6 +39,7 @@ services:
       - /var/run/dstack.sock:/var/run/dstack.sock
     ports:
       - 8000:8000
+      - 8002:8001
     depends_on:
       - auth-api
     restart: unless-stopped
@@ -75,6 +76,18 @@ configs:
       [core.auth_api.webhook]
       url = "http://auth-api:8000"
 
+      [core.admin]
+      enabled = true
+      # The onboarding KMS reaches it through the CVM port 8002.
+      address = "0.0.0.0"
+      port = 8001
+      # SHA-256 of the admin token, which stays with the operator.
+      auth_token_hash = "${ADMIN_TOKEN_HASH}"
+
+      [core.admin.tls]
+      key = "/kms/certs/rpc.key"
+      certs = "/kms/certs/rpc.crt"
+
       [core.onboard]
       enabled = true
       auto_bootstrap_domain = ""

dstack/kms/dstack-app/compose-simple.yaml

@@ -14,6 +14,7 @@ services:
       - /var/run/dstack.sock:/var/run/dstack.sock
     ports:
       - 8000:8000
+      - 8002:8001
     restart: unless-stopped
     configs:
       - source: kms_config
@@ -48,6 +49,18 @@ configs:
       [core.auth_api.webhook]
       url = "${AUTH_WEBHOOK_URL}"
 
+      [core.admin]
+      enabled = true
+      # The onboarding KMS reaches it through the CVM port 8002.
+      address = "0.0.0.0"
+      port = 8001
+      # SHA-256 of the admin token, which stays with the operator.
+      auth_token_hash = "${ADMIN_TOKEN_HASH}"
+
+      [core.admin.tls]
+      key = "/kms/certs/rpc.key"
+      certs = "/kms/certs/rpc.crt"
+
       [core.onboard]
       enabled = true
       auto_bootstrap_domain = ""

dstack/kms/dstack-app/deploy-simple.sh

@@ -37,6 +37,10 @@ else
 # Required: The address of the KMS service listening on Host machine
 # KMS_RPC_ADDR=0.0.0.0:9201
 
+# Optional: the address of the KMS admin listener on the Host machine, which
+# other KMS nodes onboard from with ADMIN_TOKEN; firewall it to KMS nodes only
+# KMS_ADMIN_ADDR=127.0.0.1:9203
+
 # Required: The address of the guest agent service listening on Host machine
 # GUEST_AGENT_ADDR=127.0.0.1:9205
 
@@ -130,6 +134,7 @@ echo "Deploying KMS to dstack-vmm..."
   --compose .app-compose.json \
   --image "$OS_IMAGE" \
   --port "tcp:$KMS_RPC_ADDR:8000" \
+  ${KMS_ADMIN_ADDR:+--port "tcp:$KMS_ADMIN_ADDR:8002"} \
   --port "tcp:$GUEST_AGENT_ADDR:8090" \
   --vcpu 8 \
   --memory 8G \

dstack/kms/dstack-app/deploy-to-vmm.sh

@@ -32,6 +32,10 @@ else
 # The address of the KMS service listening on Host machine
 # KMS_RPC_ADDR=0.0.0.0:9201
 
+# Optional: the address of the KMS admin listener on the Host machine, which
+# other KMS nodes onboard from with ADMIN_TOKEN; firewall it to KMS nodes only
+# KMS_ADMIN_ADDR=127.0.0.1:9203
+
 # The address of the guest agent service listening on Host machine
 # GUEST_AGENT_ADDR=127.0.0.1:9205
 
@@ -152,6 +156,7 @@ $CLI deploy \
   --compose .app-compose.json \
   --image "$OS_IMAGE" \
   --port tcp:"$KMS_RPC_ADDR":8000 \
+  ${KMS_ADMIN_ADDR:+--port "tcp:$KMS_ADMIN_ADDR:8002"} \
   --port tcp:"$AUTH_API_RPC_ADDR":8001 \
   --port tcp:"$GUEST_AGENT_ADDR":8090 \
   --vcpu 8 \

dstack/kms/dstack-app/entrypoint.sh

@@ -12,6 +12,9 @@ cat <<EOF > ./kms.toml
 cache_dir = "./images"
 download_url = "${IMAGE_DOWNLOAD_URL}"
 download_timeout = "2m"
+
+[core.admin]
+auth_token_hash = "${ADMIN_TOKEN_HASH}"
 EOF
 
 exec dstack-kms -c ./kms.toml