deployment.md

Deploying dstack

This guide is for self-hosted deployments on your own TDX or AMD SEV-SNP hardware. For cloud deployments, see Quickstart.

This guide covers the full manual deployment path for self-hosted dstack. Before deploying, prepare the host with Hardware enablement. If you want the shortest path to a first app on one host, start with Self-hosted quick onboarding.

Overview

dstack can be deployed in two ways:

• Single-node deployment: dstackup downloads a verified guest image, renders host config, starts the VMM and auth webhook, bootstraps a KMS CVM, and deploys apps through direct port mappings. Use Self-hosted quick onboarding for the first-app path.
• Production deployment: KMS and Gateway run as CVMs with hardware-rooted security. Uses an auth server for authorization and OS image allowlisting. Required for multi-node deployments, Gateway routing, custom domains, or on-chain governance.

For local development and contribution workflows, see Contributing.

Prerequisites

Hardware:

• Bare metal TDX or AMD SEV-SNP server. See Hardware enablement.
• At least 16GB RAM, 100GB free disk space
• Public IPv4 address
• Optional: NVIDIA H100 or Blackwell GPU for Confidential Computing workloads

Network:

• Domain with DNS access (for Gateway TLS)

Note: See Hardware Requirements for server recommendations.

Production Deployment

For production, deploy KMS and Gateway as CVMs with hardware-rooted security. Production deployments require:

• KMS running in a CVM (not on the host)
• Auth server for authorization (webhook mode)
• KMS measurements allowlisted before bootstrap / onboarding / trusted RPCs can succeed

If you skip the KMS allowlist step, the VM may boot and the onboard UI may still appear, but the KMS will reject bootstrap, onboarding, or later trusted RPCs with authorization errors.

Production Checklist

Required:

1. Set up TDX host with dstack-vmm
2. Deploy KMS as CVM (with auth server, capture its attestation info, and allowlist the KMS mrAggregated before bootstrap)
3. Deploy Gateway as CVM
4. Zero Trust HTTPS - the gateway cannot serve an app over TLS until it holds a certificate for the domain

Optional Add-ons:

5. Certificate Transparency monitoring
6. Multi-node deployment
7. On-chain governance - Smart contract-based authorization

---

1. Set Up TDX Host

Clone and build dstack-vmm:

git clone https://github.com/Dstack-TEE/dstack
cd dstack
cargo build --manifest-path dstack/Cargo.toml --release -p dstack-vmm -p supervisor
mkdir -p vmm-data
cp dstack/target/release/dstack-vmm vmm-data/
cp dstack/target/release/supervisor vmm-data/
cd vmm-data/

Create vmm.toml:

address = "tcp:0.0.0.0:9080"
reuse = true
image_path = "./images"
run_path = "./run/vm"

[auth]
enabled = true
# generate with: openssl rand -hex 32
tokens = ["<paste output of: openssl rand -hex 32>"]

[cvm]
kms_urls = []
gateway_urls = []
cid_start = 30000
cid_pool_size = 1000

[cvm.port_mapping]
enabled = true
address = "127.0.0.1"
range = [
    { protocol = "tcp", from = 1, to = 20000 },
    { protocol = "udp", from = 1, to = 20000 },
]

[host_api]
address = "vsock:2"
port = 10000

VMM API authentication (required for non-localhost binds). Since PR #796, the [auth] token guards the entire VMM surface — creating and stopping CVMs, the web UI, and all pRPC calls — not just /logs. Because the example above binds tcp:0.0.0.0:9080 (reachable off the host), you MUST enable [auth]; otherwise the whole control API is exposed unauthenticated. Auth is fail-closed: with enabled = true and no usable credential, requests are rejected.

Clients authenticate with Authorization: Bearer <token> (or the X-Admin-Token: <token> header). Instead of inline tokens, you can point to an Apache bcrypt htpasswd file with htpasswd_file = "/etc/dstack/admin.htpasswd" (create it with htpasswd -B -c /etc/dstack/admin.htpasswd admin). Only bcrypt (-B) entries are accepted.

Download guest images from dstack guest-OS releases and extract to ./images/.

For reproducible builds and verification, see the Security Model.

Start VMM:

./dstack-vmm -c vmm.toml

---

2. Deploy KMS as CVM

Production KMS requires:

• KMS: The key management service inside a CVM
• Auth server: Webhook server that validates boot requests and returns authorization decisions

Auth Server Options

Note: the boot-authorization webhook below (dstack-auth / auth-eth) is a different mechanism from the KMS admin-API authentication. The webhook allowlists which CVMs may boot and receive keys; the admin API ([core.admin] in kms.toml) guards operator RPCs. See dstack-kms admin authentication.

Server	Use Case	Configuration
dstack-auth	Config-file-based whitelisting	JSON config file
auth-eth	On-chain governance via the DstackKms/DstackApp contracts in the same directory	Ethereum RPC + contract
Custom	Your own authorization logic	Implement webhook interface

All auth servers implement the same webhook interface:

• GET / - Health check and current root epoch
• POST /authorizeApp - Application key and certificate authorization
• POST /authorizeKms - KMS self-authorization and key handover

The legacy /bootAuth/app and /bootAuth/kms routes remain available for older clients and KMS bootstrap/onboarding checks.

Using dstack-auth (Config-Based)

dstack-auth validates requests against a JSON config file. Build it:

cd dstack
cargo build --release -p dstack-auth

Create auth-config.json for initial KMS deployment:

{
  "osImages": ["0x<os-image-hash>"],
  "allowedTcbStatuses": ["UpToDate"],
  "allowedAdvisoryIds": [],
  "kms": {
    "mrAggregated": ["0x<kms-mr-aggregated>"],
    "allowAnyDevice": true
  },
  "apps": {}
}

Important: an empty kms.mrAggregated allowlist denies every KMS. Capture the current KMS measurement with Onboard.GetAttestationInfo and add it before bootstrap.

allowedTcbStatuses and allowedAdvisoryIds are opt-in: without them the TCB status and advisories are not checked. Keep them unless you operate and trust the host yourself; see TCB and advisory policy.

Bootstrap needs only the fields above. Before the KMS serves keys, set kmsContractAddr to the KMS CVM's app ID and register its root keys as rootEpochs (see Rotating KMS Root Keys); every other operation is denied until both are set.

Run dstack-auth on the operator host:

./target/release/dstack-auth --config /path/to/auth-config.json --port 3001

It binds to 127.0.0.1, which a KMS CVM on QEMU user-mode networking reaches as http://10.0.2.2:3001. Add --address 0.0.0.0 if the KMS reaches the host over a real network.

For adding Gateway, apps, and other config fields, see dstack-auth Operations Guide.

Using auth-eth (On-Chain)

For decentralized governance via smart contracts, see On-Chain Governance.

auth-eth reads policy at the block ETH_BLOCK_TAG names, finalized by default. Only finalized protects policy from reorgs. With latest (required for an OP-stack chain read through Helios) or safe, a reorg can revert both activations and revocations: a removed image or compose hash is authorized again, and a rolled-back root epoch activation makes every KMS restart onto the previous epoch. ETH_FINALITY_CONFIRMATIONS=N hides reorgs shallower than N blocks but delays every policy change, emergency revocations included, by N blocks. See reorgs and the block tag.

Getting OS Image Hash

The OS image hash is in the digest.txt file inside the guest image tarball:

# Extract hash from release tarball
tar -xzf dstack-0.5.5.tar.gz
cat dstack-0.5.5/digest.txt
# Output: 0b327bcd642788b0517de3ff46d31ebd3847b6c64ea40bacde268bb9f1c8ec83

Add 0x prefix for the dstack-auth config: 0x0b327bcd...

Deploy KMS CVM

Choose the deployment script based on your auth server:

For dstack-auth (external webhook):

dstack-auth runs on your infrastructure, outside the CVM.

cd dstack/kms/dstack-app/

Edit .env.simple:

VMM_RPC=http://127.0.0.1:9080
AUTH_WEBHOOK_URL=http://10.0.2.2:3001
KMS_RPC_ADDR=0.0.0.0:9201
GUEST_AGENT_ADDR=127.0.0.1:9205
OS_IMAGE=dstack-0.5.5
IMAGE_DOWNLOAD_URL=https://github.com/Dstack-TEE/meta-dstack/releases/download/v0.5.5/dstack-0.5.5.tar.gz

Then run:

./deploy-simple.sh

For auth-eth (on-chain governance):

See On-Chain Governance Guide for deploying KMS with smart contract-based authorization.

Monitor startup:

tail -f ../../../vmm-data/run/vm/<vm-id>/serial.log

Wait for [ OK ] Finished App Compose Service.

Bootstrap KMS

Open http://127.0.0.1:9201/ in your browser.

1. Click Bootstrap
2. Enter the domain for your KMS (e.g., kms.example.com)
3. Click Finish setup

The KMS will display its public key and TDX quote:

---

3. Deploy Gateway as CVM

Prerequisites

Before deploying Gateway:

1. Register the Gateway app in your auth server config (add to apps section in auth-config.json)
2. Note the App ID you assign - you'll need it for the .env file

For on-chain governance, see On-Chain Governance for registration steps.

Deploy Gateway CVM

cd dstack/gateway/dstack-app/
./deploy-to-vmm.sh

Edit .env with required variables:

# VMM connection (use TCP if VMM is on same host, or remote URL)
VMM_RPC=http://127.0.0.1:9080

# Cloudflare (for DNS-01 ACME challenge)
CF_API_TOKEN=your_cloudflare_api_token

# Domain configuration
SRV_DOMAIN=example.com
PUBLIC_IP=$(curl -s ifconfig.me)

# Gateway app ID (from registration above)
GATEWAY_APP_ID=32467b43BFa67273FC7dDda0999Ee9A12F2AaA08

# Gateway URLs
MY_URL=https://gateway.example.com:9202
BOOTNODE_URL=https://gateway.example.com:9202

# WireGuard (uses same port as RPC)
WG_ADDR=0.0.0.0:9202

# Network settings
SUBNET_INDEX=0
ACME_STAGING=no  # Set to 'yes' for testing
OS_IMAGE=dstack-0.5.5

Note on hex formats:

• Gateway .env file: Use raw hex without 0x prefix (e.g., GATEWAY_APP_ID=32467b43...)
• dstack-auth config: Use 0x prefix (e.g., "0x32467b43..."). The server normalizes both formats.

Run the script again:

./deploy-to-vmm.sh

The script will display the compose file and compose hash, then prompt for confirmation:

Docker compose file:
...
Compose hash: 0x700a50336df7c07c82457b116e144f526c29f6d8...
Configuration:
...
Continue? [y/N]

Before pressing 'y', add the compose hash to your auth server whitelist:

• For dstack-auth: Add to composeHashes array in auth-config.json
• For auth-eth: Use Foundry scripts (see On-Chain Governance)

Then return to the first terminal and press 'y' to deploy.

Update VMM Configuration

After Gateway is running, update vmm.toml with KMS and Gateway URLs:

[cvm]
kms_urls = ["https://kms.example.com:9201"]
gateway_urls = ["https://gateway.example.com:9202"]

gateway_urls is a failover list for one gateway cluster. To register a CVM with independently operated clusters, configure explicit groups instead. Each cluster gets a separate WireGuard interface and key pair; their WireGuard address ranges must not overlap. All clusters must run the gateway app identity authorized by the CVM's KMS-issued app keys.

Clusters refresh independently. If one cluster is unavailable, its last working WireGuard configuration remains active while other clusters continue to register and update normally.

gateway_urls and gateway_clusters are mutually exclusive in the VMM configuration. The VMM refuses to start if both are non-empty. For compatibility with sys-config files produced elsewhere, the guest prefers gateway_clusters and logs a warning when both forms are present.

[cvm]
kms_urls = ["https://kms.example.com:9201"]

[[cvm.gateway_clusters]]
name = "primary"
urls = [
  "https://gateway-a.example.com:9202",
  "https://gateway-b.example.com:9202",
]
[[cvm.gateway_clusters]]
name = "secondary"
urls = ["https://gateway-c.example.com:9202"]

Restart dstack-vmm to apply changes.

---

4. Zero Trust HTTPS

The gateway issues its own certificates from inside the CVM. dstack-gateway links the certbot crate directly: it answers dns-01 challenges with the Cloudflare credential you give it, and keeps the ACME account key and every certificate in the CVM's WaveKV store. Neither key is ever written to the host, which is what makes the monitoring in step 5 worth running — every certificate the CT logs show for your domain should carry a public key the gateway published. Nothing on the host issues or holds these certificates; the certbot CLI under dstack/certbot/cli is a testing tool for the same crate and has no part in this path.

bootstrap-cluster.sh configures all of it, reading CF_API_TOKEN, SRV_DOMAIN and ACME_STAGING from the .env you filled in during step 3 and calling the gateway's admin API:

cd dstack/gateway/dstack-app/
bash bootstrap-cluster.sh

RPC	What it sets
SetCertbotConfig	ACME directory URL and the renewal schedule
CreateDnsCredential	the Cloudflare token used for dns-01, as the default credential
AddZtDomain	a domain to keep a wildcard certificate for

Run it once per cluster. Additional nodes receive all three through cluster sync, so do not repeat it per node.

Every name the gateway terminates TLS on needs its own ZT domain. The script adds SRV_DOMAIN; if app URLs sit one level deeper — <id>-<port>.gateway.example.com — then gateway.example.com needs an entry of its own, because a wildcard certificate for *.example.com does not cover subdomains of subdomains.

Certificates are requested on the next renewal round rather than the moment a domain is added. Watch for them to arrive:

ADMIN_ADDR=127.0.0.1:9203  # GATEWAY_ADMIN_RPC_ADDR in .env
curl -sf -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  "http://$ADMIN_ADDR/prpc/ListZtDomains" | jq '.domains[] | {domain: .config.domain, cert: .cert_status}'

has_cert: true with a not_after roughly 90 days out means the domain is served. POST /prpc/RenewCert forces a round immediately instead of waiting for renew_interval_secs.

Pin issuance to your own ACME account with POST /prpc/SetCaa, which writes CAA records naming Let's Encrypt and the gateway's account URI for every configured domain. Any other account is then refused by the CA rather than merely noticed after the fact by step 5.

Start on Let's Encrypt staging (ACME_STAGING=yes), whose certificates are not browser-trusted but whose rate limits leave room for mistakes. Switching to production takes two calls: SetCertbotConfig with the production directory URL, then RotateAcmeCredentials to register an account there and re-pin every domain's CAA to it. Renewals refuse to run while the stored account and the configured ACME URL disagree, so do not skip the rotation.

For the same flow driven by hand, one curl at a time, see the Gateway service setup tutorial. For a multi-node cluster, see Cluster deployment.

---

5. Certificate Transparency Monitoring (Optional)

Monitor for unauthorized certificates issued to your domain.

cd /path/to/dstack
cargo build --manifest-path dstack/Cargo.toml --release -p ct_monitor
./dstack/target/release/ct_monitor \
  --gateway-uri https://<gateway-domain> \
  --domain <your-domain>

How it works:

1. Fetches known public keys from Gateway (/acme-info endpoint)
2. Queries crt.sh for certificates issued to your domain
3. Verifies each certificate's public key matches the known keys
4. Logs errors (❌) when certificates are issued to unknown public keys

The monitor runs in a loop, checking every 60 seconds. Integrate with your alerting system by monitoring stderr for error messages.

---

6. Multi-Node Deployment (Optional)

Scale by adding VMM nodes and KMS replicas for high availability.

Adding VMM Nodes

On each additional TDX host:

1. Set up dstack-vmm (see step 1)
2. Configure vmm.toml with existing KMS/Gateway URLs
3. Start VMM

[cvm]
kms_urls = ["https://kms.example.com:9201"]
gateway_urls = ["https://gateway.example.com:9202"]

Adding KMS Replicas (Onboarding)

Additional KMS instances can onboard from an existing KMS to share the same root keys. This enables:

• High availability (multiple KMS nodes)
• Geographic distribution
• Load balancing

How it works:

1. New KMS starts in onboard mode (empty auto_bootstrap_domain)
2. New KMS generates an RA-TLS client certificate with its attestation: self-issued with an admin token, or signed by the source temporary CA for tokenless legacy onboarding
3. New KMS calls GetKmsKey over mTLS: Admin.GetKmsKey with the token, or the explicitly enabled public KMS.GetKmsKey without one
4. Source KMS verifies the attestation and authorizes handover via /authorizeKms
5. If approved, source KMS returns root keys
6. New KMS verifies the source's attestation on that same connection and checks it against its own policy before using the keys
7. Both KMS instances now derive identical keys

Configure the source KMS for key handover:

Every KMS needs this, or core.onboard.public_key_handover = true: without a handover path it refuses to start, before onboarding or bootstrap, naming the missing setting. In a public config such as an app compose, set auth_token_hash (the token's SHA-256) instead of auth_token, with a high-entropy random token (openssl rand -hex 32), since the hash can be brute-forced offline.

Admin.GetKmsKey only works on an admin listener with [core.admin.tls]: without TLS the listener speaks plain HTTP, so the new KMS can present no RA-TLS certificate and the call fails with No attestation provided. The admin listener binds to 127.0.0.1 by default; bind it to an address the new KMS can reach and restrict access to the KMS nodes at the network layer.

[core.admin]
enabled = true
address = "0.0.0.0"          # reachable by the new KMS; firewall it to KMS nodes only
port = 8001
auth_token = "<source-admin-token>"

[core.admin.tls]
key = "/etc/kms/certs/rpc.key"
certs = "/etc/kms/certs/rpc.crt"
# No [core.admin.tls.mutual]: client certificates are verified by their attestation.

Configure new KMS for onboarding:

[core.onboard]
enabled = true
auto_bootstrap_domain = ""   # Empty = onboard mode
address = "0.0.0.0"
port = 9203                  # HTTP port for onboard UI

Trigger onboard via API:

curl -X POST http://<new-kms>:9203/prpc/Onboard.Onboard?json \
  -H "Content-Type: application/json" \
  -d '{"source_url": "https://<source-kms>:8001/prpc", "source_token": "<source-admin-token>", "domain": "kms2.example.com"}'

Finish and restart:

curl http://<new-kms>:9203/finish
# Restart KMS - it will now serve as a full KMS with shared keys

Note: KMS onboarding requires attested KMS instances, and both sides must already be authorized. Add the relevant KMS mrAggregated hashes to your auth backend first:

• the destination KMS must allow the source KMS
• the source KMS must allow the destination KMS

If you skip this, Onboard.Onboard or later trusted RPCs will fail with KMS authorization errors.

Admin authentication. The KMS admin RPCs (for example ClearImageCache) are served on a dedicated [core.admin] listener behind the shared HTTP authenticator, just like the VMM and gateway: set [core.admin] enabled = true with an auth_token (or the DSTACK_KMS_ADMIN_TOKEN / ADMIN_API_TOKEN env vars), and clients send Authorization: Bearer <token> or X-Admin-Token. Enabled with neither auth_token nor htpasswd_file (and insecure_no_auth = false) fails closed — the KMS refuses to start. See dstack-kms admin authentication.

Key handover. It is served as Admin.GetKmsKey on the [core.admin] listener, which needs [core.admin.tls] so it can verify the onboarding KMS's RA-TLS certificate. Pass the source admin URL (https://) and its token (source_token) in the onboarding request; the token is required unless the source enables the public handover. The legacy public KMS.GetKmsKey is off by default; set core.onboard.public_key_handover = true on the source only when an onboarding KMS cannot use the admin listener, such as a 0.5.x KMS or an onboarding request without source_token.

Rotating KMS Root Keys

KMS root keys are versioned by epoch; epoch 0 is the original roots. Every KMS keeps all epochs so CVMs can migrate from any of them, and serves the epoch governance made current. See the rotation design for the full procedure; the KMS steps are:

1. If epochs were never registered, publish the attestation the KMS produced at bootstrap (kept in bootstrap-info.json on the bootstrapped node) and register the current root public keys as epoch 0: DstackKms.registerRootEpoch(caPubkey, k256Pubkey, evidenceUrl), via script/Manage.s.sol:RegisterRootEpoch, or a rootEpochs entry in the dstack-auth config. Epoch 0 is current once registered, and must be the kmsInfo keys when those are set. A deployment whose kmsInfo holds a legacy quote and eventlog instead of an attestation converts them with dstack-kms legacy-attestation --quote <hex> --eventlog <hex> and publishes the result first: from then on kmsInfo follows the current epoch's keys and its quote and event log are cleared.
2. Onboard a fresh KMS with "rotate": true (the Rotate checkbox in the onboard UI) from a KMS at the current epoch N-1. It imports epochs 0..N-1, generates epoch N, and returns the new public keys with an attestation whose report data is keccak256("dstack-kms-genereted-keys-v1:{ca_pubkey_hex};{k256_pubkey_hex};") zero-padded to 64 bytes, the same encoding as the original bootstrap (the spelling is part of the format). It binds both public keys; the epoch number comes from the on-chain registration order.
3. Before registering, verify the attestation with dstack-verifier (POST /verify): check the quote, the approved KMS image and measurements, and that its report data equals the value recomputed from the returned public keys. Then publish it and register epoch N with its URL. The cluster keeps serving N-1.
4. Onboard the other KMS nodes from the rotated one; they import 0..N and keep serving N-1. A join checks the source's keyring against the latest registered epoch as well as the current one, so a source holding an unregistered or different epoch N is refused.
5. Activate epoch N (ActivateRootEpoch, or currentRootEpoch for dstack-auth). Each KMS restarts onto N on its next uncached authorization check; a KMS that does not hold N waits until it is onboarded again. If the nodes holding N are lost before N is activated, drop it with PopRootEpoch (or remove the rootEpochs entry) and rotate again; nodes that imported the dropped N must be onboarded again. On chain, the policy rows prepared for the dropped N do not apply to the new one; approve them again before activating it.
6. Boot CVMs on an OS that supports root epochs. At boot they re-key their data disk from the epoch recorded in its LUKS header to the current one. If an encrypted environment still uses an older epoch, the CVM re-encrypts it during boot and the VMM replaces the ciphertext only if it was not changed concurrently. See SealedEnv for the new format, file precedence, and rollback requirements.

Each authorizeKms or authorizeApp result carries the current epoch and its chain hash, and the latest registered epoch (latestRootEpoch, latestRootEpochHash), which may be pending activation, including when authorization is denied. KMS verifies these against its keyring; there is no cached serving epoch. An optional requestedRootEpoch selects historical keys, and omission selects the returned current epoch. Requests for future epochs are denied. No registered epoch returns a rejection with rejectionReason: "Root epoch not registered" and a zero hash.

RPC certificates still require a listener restart when their root CA changes; callers retry after the certificate is refreshed. Register epoch 0 before normal key release. Bootstrap keeps using scoped KMS authorization so it can generate the first roots before registration. See the authorization protocol.

---

Deploying Apps

After setup, deploy apps via the VMM dashboard or CLI.

Register App

Before deploying, register your app in your auth server:

• For dstack-auth: See dstack-auth Operations Guide
• For auth-eth: See On-Chain Governance

Deploy via UI

Open http://localhost:9080:

• Select the OS image
• Enter the App ID (from registration above)
• Upload your docker-compose.yaml

After startup, click Dashboard to view:

---

Troubleshooting

Error: vhost-vsock: unable to set guest cid: Address already in use

The CID range conflicts with existing VMs.

1. Find used CIDs: ps aux | grep 'guest-cid='
2. Update vmm.toml:

   [cvm]
   cid_start = 33000
   cid_pool_size = 1000

High-concurrency deployments: conntrack table full

When running Gateway with many concurrent connections (>100K), the host's conntrack table may fill up, causing silent packet drops:

dmesg: nf_conntrack: table full, dropping packet

Each proxied connection creates multiple conntrack entries (client→gateway, gateway→WireGuard→backend). The default nf_conntrack_max (typically 262,144) is insufficient for high-concurrency gateways.

Fix:

# Check current limit
sysctl net.netfilter.nf_conntrack_max

# Increase for production (persistent)
echo "net.netfilter.nf_conntrack_max = 1048576" >> /etc/sysctl.d/99-dstack.conf
echo "net.netfilter.nf_conntrack_buckets = 262144" >> /etc/sysctl.d/99-dstack.conf
sysctl -p /etc/sysctl.d/99-dstack.conf

Also increase inside bridge-mode CVMs if they handle many connections:

sysctl -w net.netfilter.nf_conntrack_max=524288

Sizing rule of thumb: Set nf_conntrack_max to at least 4× your target concurrent connection count (each connection may use 2-3 conntrack entries across NAT/bridge layers).

Error: Operation not permitted when building guest image

Ubuntu 23.10+ restricts unprivileged user namespaces:

sudo sysctl kernel.apparmor_restrict_unprivileged_userns=0