Add Daytona sandbox provider

AGENTS.md

@@ -13,7 +13,7 @@ It owns:
 - Template records, async template builds, and their provider images
 - Inline provisioning in `POST /hosts`
 - Provider VM creation and deletion (exe.dev, AWS, Hetzner, Exoscale, Boat,
-  local Docker, Docker Sandboxes)
+  Daytona, local Docker, Docker Sandboxes)
 - Tailscale auth key creation, device discovery, and cleanup
 - SSH host key scanning and `known_hosts` material
 - An SSH gateway for hosts of gateway providers (`python -m gateway.server`)

docs/daytona.md

@@ -0,0 +1,99 @@
+# Daytona sandboxes
+
+Create a Daytona host through the existing API:
+
+```json
+{"provider": "daytona"}
+```
+
+The `image` field selects a prepared Daytona snapshot name or ID. The
+snapshot can be a Linux VM or a container; the provider rejects Windows
+snapshots. CPU, memory, and disk come from the snapshot. Per-host
+`instance_type` and `disk_gb` are not supported.
+
+A Linux VM has its own kernel. A container shares the host kernel, and
+Daytona cannot pause it. Linux VMs can be unavailable to an organization: a
+VM create then fails with "Snapshot ... is not available in region ...". Use
+a container snapshot, or ask Daytona to enable VMs.
+
+The Daytona organization must be Tier 3 or higher. Tier 1 and Tier 2 limit
+outbound traffic to an allowlist without Tailscale, and a sandbox setting
+cannot change that.
+
+## Prepare the snapshot
+
+The snapshot needs the tools in `images/daytona/Dockerfile`: Tailscale with
+iptables and iproute2, Bash, jq, Git, GitHub CLI, sudo, and CA tools. For a
+custom image, keep these tools and passwordless root access. Do not install
+an SSH server: Tailscale SSH serves the host, and an image with `/etc/ssh`
+host keys gives every sandbox the same keys. Never snapshot a sandbox that
+has joined Tailscale into a reusable image.
+
+Daytona's Dockerfile builder can build a container snapshot from
+`images/daytona/Dockerfile`. A Linux VM snapshot needs the image in a
+registry that Daytona can read:
+
+```bash
+docker build --platform linux/amd64 -t registry.example.com/team/drukbox-daytona:1 images/daytona
+docker push registry.example.com/team/drukbox-daytona:1
+```
+
+Replace the example registry and image name. Set the CPU, memory, and disk
+on the snapshot: every host from it gets that size. If you do not set them,
+Daytona applies its default sandbox size. For more than one size, create one
+snapshot per size and select it with `image`. Wait for the snapshot to become
+active in the target region. Use its name or ID as `DAYTONA_DEFAULT_IMAGE`.
+Image builds, stored snapshots, and sandboxes use provider resources.
+
+## Configure Drukbox
+
+| Variable | Value |
+| --- | --- |
+| `DAYTONA_API_KEY` | Required API key; can use `/run/secrets/DAYTONA_API_KEY` |
+| `DAYTONA_DEFAULT_IMAGE` | Required active snapshot name or ID |
+| `DAYTONA_TARGET` | Required target region where the snapshot is available |
+| `DAYTONA_API_TIMEOUT` | API request timeout in seconds; default `150` |
+| `DAYTONA_LIFECYCLE_TIMEOUT_SECONDS` | Start or deletion polling budget; default `180` |
+| `DAYTONA_BOOTSTRAP_SSH_TIMEOUT_SECONDS` | SSH readiness budget; default `120` |
+| `TAILSCALE_ENABLED` | Required: `true` |
+
+Configure the tailnet and Tailscale credentials as described in
+[Networking](networking.md). Hosts use Tailscale SSH as `root`; they have no
+public SSH address. The adapter does not use Daytona's expiring SSH tokens.
+
+Drukbox disables Daytona auto-stop, auto-pause, auto-delete, and wall-clock
+TTL. Deleting the host, or the janitor at lease expiry, removes the sandbox.
+Drukbox owns the lease; keep the janitor running. Pause, resume, and template
+builds through Drukbox are not supported by this adapter.
+
+Bootstrap persists the environment and installs the secrets proxy CA. Without
+systemd, as in a Daytona container, bootstrap starts `tailscaled` itself and
+keeps its state in `/var/lib/tailscale`. If the sandbox has a TUN device,
+programs in it get tailnet routes and MagicDNS. Without one, `tailscaled`
+falls back to userspace networking: inbound SSH works, but programs in the
+sandbox cannot reach tailnet addresses, so the secrets proxy must be
+reachable off the tailnet. HTTP previews stay private.
+
+A paused VM keeps `tailscaled` running. Nothing restarts `tailscaled` if it
+exits. A stop clears the sandbox's processes, so the host leaves the
+tailnet. Do not stop Drukbox sandboxes in Daytona; delete the host instead.
+
+Drukbox tags the sandbox with `SERVICE_LABEL`, checks ownership before
+deletion, and waits for destruction. If provider deletion fails, Drukbox
+keeps the host record for a retry. Do not rename sandboxes, change their
+ownership label, or change `SERVICE_LABEL` while hosts exist.
+
+`/doctor` checks API authentication and the default snapshot class and
+state. It does not check that the snapshot is available in `DAYTONA_TARGET`.
+If it is not, a create fails with Daytona's reason in the host error.
+`/doctor` does not provision a sandbox or prove Tailscale connectivity. Live
+provisioning and SSH require a separate test with a Daytona account and
+configured tailnet.
+
+## Sources
+
+- [Daytona sandboxes](https://www.daytona.io/docs/en/sandboxes/)
+- [Daytona snapshots](https://www.daytona.io/docs/en/snapshots/)
+- [Daytona Tailscale support](https://www.daytona.io/docs/en/vpn-connections/)
+- [Daytona REST API](https://www.daytona.io/docs/openapi.json)
+- [Daytona command API](https://www.daytona.io/docs/toolbox-openapi.json)

docs/deploy.md

@@ -96,6 +96,7 @@ account token returns `503`. See [API](api.md#service-accounts).
 
 | Provider | Sandboxes | Where |
 | --- | --- | --- |
+| `daytona` | Linux VMs or containers ([Daytona setup](daytona.md)) | Remote |
 | `exe` | exe.dev VMs | Remote |
 | `aws` | EC2 instances | Remote |
 | `hetzner` | Hetzner Cloud VMs | Remote |
@@ -111,8 +112,8 @@ all provider extras.
 ## Local sandboxes with Docker
 
 The `docker` provider runs each sandbox as a local container with sshd,
-so you can try drukbox with no cloud account or API token. The published
-image includes the Docker CLI. Set `DEFAULT_HOST_PROVIDER=docker`,
+so you can try drukbox with no cloud account or API token.
+Set `DEFAULT_HOST_PROVIDER=docker`,
 `TAILSCALE_ENABLED=false`, and `UVICORN_HOST=127.0.0.1` in `drukbox.env`,
 then run:
 
@@ -688,7 +689,7 @@ Docker provider:
 | `DOCKER_SSH_USERNAME` | `root` | In-container user callers SSH as. The entrypoint seeds its `authorized_keys`; a derived image adds the user. |
 | `DOCKER_BOOTSTRAP_SSH_TIMEOUT_SECONDS` | `30.0` | ssh-keyscan retry budget for a fresh container. |
 
-The published image includes the Docker CLI. Mount the local daemon socket
+Mount the local daemon socket
 with its supplemental group on Linux, or use `DOCKER_HOST` for a remote or
 rootless daemon. Drukbox mints a per-VM ed25519 key and publishes sshd on a
 random port at `DOCKER_SSH_HOST`. See
@@ -710,3 +711,8 @@ The published image does not contain the `sbx` CLI. Mount the binary and
 the sbx directories of the host, as
 [Local microVMs with Docker Sandboxes](#local-microvms-with-docker-sandboxes)
 shows.
+
+## Daytona
+
+See [Daytona setup](daytona.md) for image preparation, all provider settings,
+Tailscale SSH, and lifecycle limits.

images/daytona/Dockerfile

@@ -0,0 +1,8 @@
+FROM tailscale/tailscale:v1.102.5 AS tailscale
+FROM ubuntu:24.04
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends bash ca-certificates sudo jq git gh openssh-client iptables iproute2 \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY --from=tailscale /usr/local/bin/tailscale /usr/local/bin/tailscaled /usr/local/bin/

src/hosts/scripts/sandbox_bootstrap.sh

@@ -1,19 +1,4 @@
 #!/usr/bin/env bash
-# Sandbox host first-boot bootstrap. Delivered to the VM at create time via the
-# VM provider's setup-script mechanism (exe.dev's --setup-script today).
-#
-# The VM brings itself onto the tailnet and exits. Drukbox observes the new
-# device by polling Tailscale's API from outside; no callback into drukbox
-# is made from this script, and drukbox is never reachable from the box.
-#
-# Required env:
-#   TAILSCALE_AUTHKEY
-#   TAILSCALE_HOSTNAME
-#
-# Optional env:
-#   TAILSCALE_ADVERTISE_TAGS              (default: tag:sandbox)
-#   TAILSCALE_LOGIN_SERVER                (default: unset)
-
 set -euo pipefail
 
 state_dir=/var/lib/sandbox
@@ -35,9 +20,6 @@ run_privileged() {
 
 run_privileged install -d -m 755 -o "$(id -u)" -g "$(id -g)" "$state_dir"
 
-# Defensive against re-runs: --setup-script is documented as run-once, and the
-# legacy in-image systemd unit (if present on transitional images) also gates on
-# this flag. Either path converges on the same end state.
 if [[ -f "$done_path" ]]; then
   exit 0
 fi
@@ -52,26 +34,31 @@ require_var() {
 require_var TAILSCALE_AUTHKEY
 require_var TAILSCALE_HOSTNAME
 
-advertise_tags="${TAILSCALE_ADVERTISE_TAGS:-tag:sandbox}"
-
-run_privileged systemctl enable --now tailscaled.service
-
-tailscale_running() {
-  tailscale status --json 2>/dev/null | jq -e '.BackendState == "Running"' >/dev/null
-}
+if [[ -d /run/systemd/system ]]; then
+  run_privileged systemctl enable --now tailscaled.service
+else
+  tailscaled_tun=userspace-networking
+  if [[ -c /dev/net/tun ]]; then
+    tailscaled_tun=tailscale0
+  fi
+  # Tailscale SSH keeps its host keys in the state directory.
+  run_privileged sh -c "nohup tailscaled --tun=$tailscaled_tun --statedir=/var/lib/tailscale </dev/null >/var/log/tailscaled.log 2>&1 &"
+  for _ in {1..60}; do
+    [[ -S /var/run/tailscale/tailscaled.sock ]] && break
+    sleep 0.5
+  done
+fi
 
-if ! tailscale_running; then
+if ! tailscale status --json 2>/dev/null | jq -e '.BackendState == "Running"' >/dev/null; then
   tailscale_args=(
     up
     "--authkey=$TAILSCALE_AUTHKEY"
     --accept-dns=true
     --ssh
     "--hostname=$TAILSCALE_HOSTNAME"
+    "--advertise-tags=${TAILSCALE_ADVERTISE_TAGS:-tag:sandbox}"
     --reset
   )
-  if [[ -n "$advertise_tags" ]]; then
-    tailscale_args+=("--advertise-tags=$advertise_tags")
-  fi
   if [[ -n "${TAILSCALE_LOGIN_SERVER:-}" ]]; then
     tailscale_args+=("--login-server=$TAILSCALE_LOGIN_SERVER")
   fi

src/providers/__init__.py

@@ -1,5 +1,6 @@
 import providers.aws
 import providers.boat
+import providers.daytona
 import providers.docker
 import providers.docker_sbx
 import providers.exe

src/providers/daytona/__init__.py

@@ -0,0 +1,4 @@
+from providers.daytona.provider import DaytonaProvider
+from providers.registry import register_vm_provider
+
+register_vm_provider(DaytonaProvider)

src/providers/daytona/api.py

@@ -0,0 +1,140 @@
+import asyncio
+import contextlib
+import shlex
+from typing import Any
+from urllib.parse import quote
+
+import httpx
+
+from providers.exceptions import (
+    ProviderAuthError,
+    ProviderCommandError,
+    ProviderNotFoundError,
+    ProviderTransportError,
+)
+
+from .settings import DaytonaSettings
+
+LINUX_SANDBOX_CLASSES = {"linux-vm", "container"}
+STARTING_STATES = {"creating", "starting", "pending_build", "pulling_snapshot"}
+
+
+class DaytonaAPI:
+    def __init__(self, settings: DaytonaSettings) -> None:
+        self.settings = settings
+        self.client = httpx.AsyncClient(
+            base_url="https://app.daytona.io/api/",
+            headers={"Authorization": f"Bearer {settings.api_key}"},
+            timeout=httpx.Timeout(settings.api_timeout, connect=5),
+        )
+
+    async def check_snapshot(self, image: str) -> None:
+        snapshot = await self.request("GET", f"snapshots/{quote(image, safe='')}")
+
+        if snapshot["sandboxClass"] not in LINUX_SANDBOX_CLASSES:
+            raise ProviderCommandError("Daytona requires a Linux VM or container snapshot")
+        if snapshot["state"] != "active":
+            raise ProviderCommandError("Daytona snapshot is not active")
+
+    async def create_sandbox(self, name: str, *, image: str, label: str) -> None:
+        await self.request(
+            "POST",
+            "sandbox",
+            json={
+                "name": name,
+                "snapshot": image,
+                "target": self.settings.target,
+                "user": "root",
+                "labels": {"managed-by": label},
+                "public": False,
+                "autoStopInterval": 0,
+                "autoPauseInterval": 0,
+                "autoDeleteInterval": -1,
+                "ttlMinutes": 0,
+            },
+        )
+
+    async def wait_for_started(self, name: str) -> dict[str, Any]:
+        try:
+            async with asyncio.timeout(self.settings.lifecycle_timeout_seconds):
+                sandbox = await self.request("GET", f"sandbox/{name}")
+                while sandbox["state"] in STARTING_STATES:
+                    await asyncio.sleep(1)
+                    sandbox = await self.request("GET", f"sandbox/{name}")
+        except TimeoutError as exc:
+            raise ProviderTransportError("Daytona sandbox start timed out") from exc
+
+        if sandbox["state"] != "started":
+            raise ProviderTransportError("Daytona sandbox did not start")
+        return sandbox
+
+    async def bootstrap(self, sandbox: dict[str, Any], script: str) -> None:
+        # Daytona's rejection can quote the command, and the command carries
+        # the Tailscale auth key, so the reason stays out of the error.
+        try:
+            result = await self.request(
+                "POST",
+                f"{sandbox['toolboxProxyUrl']}/{sandbox['id']}/process/execute",
+                json={"command": f"sudo -n bash -e -c {shlex.quote(script)}", "timeout": 120},
+            )
+        except ProviderCommandError:
+            raise ProviderCommandError("Daytona rejected the bootstrap command") from None
+
+        if result["exitCode"]:
+            raise ProviderCommandError("Daytona sandbox bootstrap failed")
+
+    async def delete_sandbox(self, name: str, *, label: str) -> None:
+        sandbox = await self.request("GET", f"sandbox/{name}")
+
+        if sandbox["labels"].get("managed-by") != label:
+            raise ProviderCommandError("Daytona sandbox does not belong to this deployment")
+        if sandbox["state"] == "destroyed":
+            return
+        path = f"sandbox/{sandbox['id']}"
+        await self.request("DELETE", path)
+        try:
+            async with asyncio.timeout(self.settings.lifecycle_timeout_seconds):
+                while True:
+                    try:
+                        sandbox = await self.request("GET", path)
+                    except ProviderNotFoundError:
+                        return
+                    if sandbox["state"] == "destroyed":
+                        return
+                    if sandbox["state"] == "error":
+                        raise ProviderTransportError("Daytona sandbox deletion failed")
+                    await asyncio.sleep(1)
+        except TimeoutError as exc:
+            raise ProviderTransportError("Daytona sandbox deletion timed out") from exc
+
+    async def diagnose(self) -> str:
+        await self.check_snapshot(self.settings.default_image)
+        return "Daytona API authentication succeeded; snapshot is active"
+
+    async def aclose(self) -> None:
+        await self.client.aclose()
+
+    async def request(
+        self, method: str, path: str, *, json: dict[str, Any] | None = None
+    ) -> dict[str, Any]:
+        try:
+            response = await self.client.request(method, path, json=json)
+        except httpx.RequestError as exc:
+            raise ProviderTransportError("Daytona API transport failed") from exc
+        if response.status_code in {401, 403}:
+            raise ProviderAuthError("Daytona API authentication failed")
+        if response.status_code == 404:
+            raise ProviderNotFoundError("Daytona resource was not found")
+        if response.status_code >= 500 or response.status_code == 429:
+            raise ProviderTransportError(f"Daytona API returned HTTP {response.status_code}")
+        if not 200 <= response.status_code < 300:
+            reason = f"Daytona API rejected request (HTTP {response.status_code})"
+            with contextlib.suppress(ValueError, KeyError, TypeError):
+                reason = f"{reason}: {response.json()['message']}"
+            raise ProviderCommandError(reason)
+        if not response.content:
+            return {}
+        try:
+            return response.json()
+        except ValueError as exc:
+            raise ProviderTransportError("Invalid Daytona API response") from exc