diff --git a/crates/openshell-core/src/driver_mounts.rs b/crates/openshell-core/src/driver_mounts.rs index f786f76ce4..bb64fff1d6 100644 --- a/crates/openshell-core/src/driver_mounts.rs +++ b/crates/openshell-core/src/driver_mounts.rs @@ -31,6 +31,21 @@ pub enum SelinuxLabel { /// directory and by drivers whose workspace remains fixed. pub const DEFAULT_WORKSPACE_ROOT: &str = "/sandbox"; +/// Paths reserved by VM guest initialization. +pub const VM_RESERVED_GUEST_PATHS: &[&str] = &[ + "/.openshell", + "/.openshell-bootstrap", + "/overlay", + "/lower", + "/newroot", + "/image-cache", + "/srv", + "/proc", + "/sys", + "/dev", + "/tmp", +]; + /// Validate a non-empty driver mount source. pub fn validate_mount_source(source: &str, field: &str) -> Result<(), String> { if source.is_empty() { @@ -120,6 +135,38 @@ pub fn resolve_oci_workspace_root(working_dir: &str) -> Result { Ok(workspace_root) } +/// Resolve a VM workload's image-declared working directory. +pub fn resolve_vm_image_workspace_root(working_dir: &str) -> Result { + let root = resolve_oci_workspace_root(working_dir)?; + for reserved in VM_RESERVED_GUEST_PATHS.iter().copied().chain(["/run"]) { + if paths_overlap(Path::new(&root), Path::new(reserved)) { + return Err(format!( + "VM workspace '{root}' conflicts with guest path '{reserved}'" + )); + } + } + Ok(root) +} + +/// Resolve a VM workspace that guest init will assign to the sandbox user. +pub fn resolve_vm_managed_workspace_root(root: &str) -> Result { + if root.is_empty() || root == "/" || root.contains(':') { + return Err("VM workspace_root must be a non-root absolute path without ':'".to_string()); + } + let root = resolve_vm_image_workspace_root(root)?; + if matches!(root.as_str(), "/home" | "/var") + || [ + "/etc", "/root", "/boot", "/bin", "/sbin", "/lib", "/lib32", "/lib64", "/libx32", + "/usr", + ] + .iter() + .any(|reserved| Path::new(&root).starts_with(reserved)) + { + return Err(format!("VM workspace '{root}' is a protected system path")); + } + Ok(root) +} + fn normalize_absolute_container_path(value: &str, field: &str) -> Result { if value.is_empty() { return Err(format!("{field} must not be empty")); diff --git a/crates/openshell-driver-vm/README.md b/crates/openshell-driver-vm/README.md index cb9433d08e..7f6bf14e89 100644 --- a/crates/openshell-driver-vm/README.md +++ b/crates/openshell-driver-vm/README.md @@ -173,6 +173,8 @@ Select the VM driver with `--compute-driver vm`, `OPENSHELL_COMPUTE_DRIVER=vm`, | `vcpus` | `2` | vCPUs per sandbox. | | `mem_mib` | `2048` | Memory per sandbox, in MiB. | | `overlay_disk_mib` | `4096` | Sparse writable overlay disk size per sandbox, in MiB. | +| `workspace_mode` | `"managed"` | `managed` prepares the configured guest directory for the sandbox user. `image` uses the image's OCI `WORKDIR`, falling back to the managed `/sandbox` for missing, empty, `/`, or `/sandbox` values. Rootfs tar archives have no OCI `WORKDIR`. | +| `workspace_root` | `"/sandbox"` | Absolute path in the guest used by `managed` mode. | | `krun_log_level` | `1` | libkrun verbosity (0-5). | | `guest_tls_ca` | unset | Historical key name for the host supervisor's gateway CA certificate. Required when `grpc_endpoint` uses `https://`; never copied into the guest. | | `https_proxy` | unset | Corporate forward proxy (`http://host:port` or `https://host:port`) that host control chains policy-approved TLS CONNECT egress through. Host-loopback proxy URLs work because control runs on the gateway host. | @@ -182,10 +184,12 @@ Select the VM driver with `--compute-driver vm`, `OPENSHELL_COMPUTE_DRIVER=vm`, | `proxy_connect_by_hostname` | unset | Send hostnames rather than validated IPs in CONNECT. Last resort for proxies whose ACLs reject IP CONNECT targets. | | `proxy_ca_bundle` | unset | Gateway-host PEM CA bundle trusted for the corporate proxy and TLS-intercepted server certificates. The driver validates it and stages it at a fixed non-secret guest path in the protected overlay. Requires `https_proxy`. | | `provider_spiffe_workload_api_tcp_endpoint` | unset | Explicit guest-reachable `tcp:IP:port` SPIFFE Workload API listener for provider token exchange. It requires `provider_spiffe_allow_guest_tcp = true`; a host UNIX socket is never silently exposed to a VM guest. | -| `enable_bind_mounts` | `false` | Allow per-sandbox `driver_config.vm.mounts` host directory shares (libkrun virtiofs only). The driver writes a `tag\ttarget\tmode` manifest to `/.openshell/mounts.manifest` in the overlay, and guest init mounts each share after `/sandbox` ownership fixups. Rejected while resource admission is enabled. | +| `enable_bind_mounts` | `false` | Allow per-sandbox `driver_config.vm.mounts` host directory shares (libkrun virtiofs only). The driver writes a `tag\ttarget\tmode` manifest to `/.openshell/mounts.manifest` in the overlay, and guest init mounts each share after workspace setup. Rejected while resource admission is enabled. | The proxy settings are operator-owned and deployment-level: they are not accepted through `template.driver_config.vm`, and the driver passes them only to native host control. Every present-but-invalid value is fatal at gateway or sandbox startup rather than degrading to a direct dial. +In `managed` mode, OpenShell assigns ownership of a custom workspace directory to the sandbox user and preserves existing files and their ownership. In `image` mode, the directory must already exist without symlink components and be writable by the resolved sandbox user. Existing VM image preparation still normalizes the compatibility `/sandbox` tree. The driver records the selected workspace at creation and reuses it with the sandbox's writable overlay across restarts. It also retains the prepared workload image for a custom workspace or `image` mode. + See [`openshell-gateway --help`](../openshell-server/src/cli.rs) for the gateway process flag surface. ## Verifying the gateway @@ -242,7 +246,7 @@ At startup, the driver reclaims inactive attempts in this directory. It preserve Each sandbox gets its own sparse writable `/sandboxes//overlay.ext4`. Guest init mounts overlayfs as `/` with the prepared image rootfs as lowerdir when present, otherwise the bootstrap -rootfs is used directly. Writes to `/sandbox` and other mutable paths land in +rootfs is used directly. Writes to the workspace and other mutable paths land in the overlay while cached image disks remain unchanged. The overlay disk must be large enough to hold the compressed payload, unpacked rootfs, and sandbox writes during the first prepare. diff --git a/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh b/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh index 37950675b0..0590c4055b 100644 --- a/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh +++ b/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh @@ -48,6 +48,42 @@ root_path() { printf '%s%s\n' "${ROOT_PREFIX:-}" "$path" } +load_sandbox_workspace() { + # Read driver-owned overlay metadata directly. The workload image cannot + # supply workspace configuration through the merged root filesystem. + local config=/overlay/config/openshell-workspace extra + WORKSPACE_MODE=managed + WORKSPACE_ROOT=/sandbox + [ -e "$config" ] || return 0 + + if ! { + IFS= read -r WORKSPACE_MODE && + IFS= read -r WORKSPACE_ROOT && + ! IFS= read -r extra && + [ -z "$extra" ] + } < "$config"; then + ts "FATAL: invalid sandbox workspace configuration" + exit 1 + fi + case "$WORKSPACE_MODE" in + managed|image) ;; + *) + ts "FATAL: invalid sandbox workspace mode" + exit 1 + ;; + esac + case "$WORKSPACE_ROOT/" in + /|*//*|*/./*|*/../*|[!/]*) + ts "FATAL: invalid sandbox workspace path" + exit 1 + ;; + esac + if [ "$WORKSPACE_MODE" = managed ] && [[ "$WORKSPACE_ROOT" == *:* ]]; then + ts "FATAL: managed workspace path contains a passwd field separator" + exit 1 + fi +} + sandbox_owner() { sandbox_owner_from_passwd "$(root_path /etc/passwd)" } @@ -377,7 +413,7 @@ setup_gpu() { reconcile_sandbox_account() { local sandbox_uid="${OPENSHELL_VM_SANDBOX_UID:-}" local sandbox_gid="${OPENSHELL_VM_SANDBOX_GID:-}" - local etc + local etc home="" [ -n "$sandbox_uid" ] && [ -n "$sandbox_gid" ] || return 0 [[ "$sandbox_uid" =~ ^[0-9]+$ ]] && [[ "$sandbox_gid" =~ ^[0-9]+$ ]] || { @@ -399,11 +435,19 @@ reconcile_sandbox_account() { else printf 'sandbox:x:%s:\n' "$sandbox_gid" >> "$etc/group" fi + if [ "$WORKSPACE_MODE" = managed ] && [ "$WORKSPACE_ROOT" != /sandbox ]; then + home="$WORKSPACE_ROOT" + fi if grep -q '^sandbox:' "$etc/passwd"; then - # Preserve image-owned account metadata (home, shell, and description) - # while restoring the UID/GID contract recorded for this overlay. - if ! awk -F: -v OFS=: -v uid="$sandbox_uid" -v gid="$sandbox_gid" \ - '$1 == "sandbox" { $3 = uid; $4 = gid } { print }' \ + # Update the account home only for an explicitly managed custom path. + if ! OPENSHELL_VM_ACCOUNT_HOME="$home" \ + awk -F: -v OFS=: -v uid="$sandbox_uid" -v gid="$sandbox_gid" \ + '$1 == "sandbox" { + $3 = uid + $4 = gid + home = ENVIRON["OPENSHELL_VM_ACCOUNT_HOME"] + if (home != "") $6 = home + } { print }' \ "$etc/passwd" >"$etc/passwd.openshell"; then rm -f "$etc/passwd.openshell" ts "FATAL: failed to reconcile sandbox account" @@ -411,7 +455,8 @@ reconcile_sandbox_account() { fi mv "$etc/passwd.openshell" "$etc/passwd" else - printf 'sandbox:x:%s:%s:OpenShell Sandbox:/sandbox:/bin/sh\n' "$sandbox_uid" "$sandbox_gid" >> "$etc/passwd" + printf 'sandbox:x:%s:%s:OpenShell Sandbox:%s:/bin/sh\n' \ + "$sandbox_uid" "$sandbox_gid" "${home:-/sandbox}" >> "$etc/passwd" fi grep -q '^sandbox:' "$etc/gshadow" || printf 'sandbox:!::\n' >> "$etc/gshadow" grep -q '^sandbox:' "$etc/shadow" || printf 'sandbox:!:20123:0:99999:7:::\n' >> "$etc/shadow" @@ -422,8 +467,24 @@ setup_sandbox_workdir() { local sandbox_dir local owner local current_owner - sandbox_dir="$(root_path /sandbox)" + if [ "$WORKSPACE_ROOT" != /sandbox ] && [ "$WORKSPACE_MODE" = image ]; then + # The boundary checks the image directory as the actual workload user. + return + fi owner="$(sandbox_owner)" + if [ "$WORKSPACE_ROOT" != /sandbox ]; then + sandbox_dir="$( + umask 022 + prepare_guest_directory "$WORKSPACE_ROOT" "$owner" "$WORKSPACE_ROOT" + )" || exit 1 + if ! chown "$owner" "$sandbox_dir" || ! chmod u+rwx "$sandbox_dir"; then + ts "FATAL: failed to prepare workspace ${WORKSPACE_ROOT} (${owner})" + exit 1 + fi + return + fi + + sandbox_dir="$(root_path /sandbox)" mkdir -p "$sandbox_dir" current_owner="$(stat -c '%u:%g' "$sandbox_dir" 2>/dev/null || true)" if [ "$owner" = "10001:10001" ]; then @@ -561,11 +622,10 @@ run_openshell_init_dropins() { done < <(LC_ALL=C sort -u "$manifest") } -# Create a virtiofs mount point under ROOT_PREFIX without following symlinks, -# so an image cannot redirect a validated target onto another guest path. -# Directories created under /sandbox are handed to the sandbox user. -prepare_virtiofs_target() { - local target="$1" owner="$2" +# Walk a guest directory path without following image symlinks. New +# directories below the workspace belong to the sandbox account. +prepare_guest_directory() { + local target="$1" owner="$2" workspace_root="$3" local path="${ROOT_PREFIX:-}" component local -a components IFS=/ read -r -a components <<<"${target#/}" @@ -573,31 +633,31 @@ prepare_virtiofs_target() { [ -n "$component" ] || continue path="${path}/${component}" if [ -L "$path" ]; then - ts >&2 "FATAL: virtiofs mount target ${target} traverses symlink ${path#"${ROOT_PREFIX:-}"}" + ts >&2 "FATAL: guest path ${target} traverses symlink ${path#"${ROOT_PREFIX:-}"}" exit 1 fi if [ ! -e "$path" ]; then if ! mkdir "$path"; then - ts >&2 "FATAL: failed to create virtiofs mount point ${target}" + ts >&2 "FATAL: failed to create guest directory ${target}" exit 1 fi case "${path#"${ROOT_PREFIX:-}"}" in - /sandbox/*) + "$workspace_root"/*) if ! chown "$owner" "$path"; then - ts >&2 "FATAL: failed to hand virtiofs mount point parent ${path#"${ROOT_PREFIX:-}"} to ${owner}" + ts >&2 "FATAL: failed to assign guest directory ${path#"${ROOT_PREFIX:-}"} to ${owner}" exit 1 fi ;; esac elif [ ! -d "$path" ]; then - ts >&2 "FATAL: virtiofs mount target ${target} is not a directory" + ts >&2 "FATAL: guest path ${target} is not a directory" exit 1 fi done printf '%s\n' "$path" } -# Runs after every /sandbox ownership fixup and the root-run init drop-ins so +# Runs after workspace ownership setup and the root-run init drop-ins so # nothing in init walks into host-backed shares. mount_virtiofs_shares() { local manifest @@ -618,7 +678,7 @@ mount_virtiofs_shares() { exit 1 ;; esac - guest_target="$(prepare_virtiofs_target "$target" "$owner")" || exit 1 + guest_target="$(prepare_guest_directory "$target" "$owner" "$WORKSPACE_ROOT")" || exit 1 # shellcheck disable=SC2086 if mount -t virtiofs $mount_opts "$tag" "$guest_target"; then ts " mounted virtiofs ${tag} -> ${target} (${mode})" @@ -654,6 +714,7 @@ run_post_overlay_setup() { mount -t cgroup2 cgroup2 "$(root_path /sys/fs/cgroup)" 2>/dev/null & wait + load_sandbox_workspace reconcile_sandbox_account setup_sandbox_workdir @@ -678,7 +739,7 @@ EOF # The boundary transport mediates network and DNS requests. Only loopback is # configured in the guest; no public resolver or guest NIC is needed. -export HOME=/sandbox +export HOME="$WORKSPACE_ROOT" export USER=sandbox # Fix /sandbox ownership. The host-side CLI extracts OCI layers as a non-root @@ -688,7 +749,7 @@ export USER=sandbox # chown succeeds. On Linux non-root hosts with virtiofs, guest root maps to # the host user, so chown is denied — this is non-fatal because the # supervisor's own filesystem preparation handles the paths that matter. -if [ -d /sandbox ]; then +if [ "$WORKSPACE_ROOT" = /sandbox ] && [ -d /sandbox ]; then _sb_uid=$(id -u sandbox 2>/dev/null || true) _sb_gid=$(id -g sandbox 2>/dev/null || true) if [ -n "$_sb_uid" ] && [ -n "$_sb_gid" ]; then diff --git a/crates/openshell-driver-vm/src/driver.rs b/crates/openshell-driver-vm/src/driver.rs index 528c786b60..ee09b1c05a 100644 --- a/crates/openshell-driver-vm/src/driver.rs +++ b/crates/openshell-driver-vm/src/driver.rs @@ -46,7 +46,7 @@ use oci_client::secrets::RegistryAuth; use oci_client::{Reference, RegistryOperation}; use openshell_core::UpstreamProxyConfig; use openshell_core::config::DEFAULT_STOP_TIMEOUT_SECS; -use openshell_core::driver_mounts; +use openshell_core::driver_mounts::{self, VM_RESERVED_GUEST_PATHS}; use openshell_core::gpu::{ driver_gpu_requirements, effective_driver_gpu_count, validate_specific_gpu_device_request, }; @@ -101,7 +101,7 @@ use std::process::{ExitStatus, Stdio}; use std::sync::Arc; use std::sync::atomic::{AtomicU64, Ordering}; use std::time::Duration; -use tokio::io::{AsyncBufReadExt, AsyncWriteExt}; +use tokio::io::{AsyncBufReadExt, AsyncReadExt, AsyncWriteExt}; use tokio::process::{Child, Command}; use tokio::sync::{Mutex, broadcast, mpsc}; use tokio::task::JoinHandle; @@ -121,6 +121,7 @@ const MAX_REGISTRY_LAYER_DOWNLOAD_CONCURRENCY: usize = 16; const REGISTRY_REQUEST_MAX_ATTEMPTS: usize = 4; const REGISTRY_RETRY_INITIAL_DELAY: Duration = Duration::from_millis(250); const REGISTRY_RETRY_MAX_DELAY: Duration = Duration::from_secs(1); +const MAX_REGISTRY_IMAGE_CONFIG_BYTES: usize = 16 * 1024 * 1024; /// 10 GiB — configurable via `rootfs_tar_max_bytes`. const DEFAULT_ROOTFS_TAR_MAX_BYTES: u64 = 10 * 1024 * 1024 * 1024; const ROOTFS_TAR_STAGING_DIR: &str = "rootfs-tar-staging"; @@ -248,6 +249,8 @@ const SANDBOX_OWNER_STATE_FILE: &str = "sandbox-owner-state"; const SANDBOX_OWNER_STATE_V1: &str = "sandbox-owner-v1"; const SANDBOX_OWNER_STATE_V2: &str = "sandbox-owner-v2"; const SANDBOX_REQUEST_FILE: &str = "sandbox.pb"; +const SANDBOX_WORKSPACE_FILE: &str = "workspace.json"; +const OVERLAY_WORKSPACE_FILE: &str = "/config/openshell-workspace"; const SANDBOX_STOPPED_FILE: &str = "stopped"; /// Durable tombstone preventing driver restart from relaunching a sandbox /// whose canonical main process already terminated. @@ -281,12 +284,15 @@ struct RuntimeImagePlan { image_disk: Option, image_identity: String, bootstrap_image_identity: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + working_dir: Option, } #[derive(Debug, Clone)] struct PreparedImageDisk { image_identity: String, disk_path: PathBuf, + working_dir: Option, } #[derive(Debug, Clone)] @@ -337,6 +343,47 @@ impl std::fmt::Debug for VmNetworkSupervisorTrustSnapshot { } } +/// How the VM driver selects a workspace for a new sandbox. +#[derive( + Clone, + Copy, + Debug, + Default, + PartialEq, + Eq, + serde::Serialize, + serde::Deserialize, + clap::ValueEnum, +)] +#[serde(rename_all = "snake_case")] +pub enum VmWorkspaceMode { + #[default] + Managed, + Image, +} + +#[derive(Debug, serde::Serialize, serde::Deserialize)] +struct VmWorkspace { + mode: VmWorkspaceMode, + root: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + image_identity: Option, +} + +impl Default for VmWorkspace { + fn default() -> Self { + Self { + mode: VmWorkspaceMode::Managed, + root: default_vm_workspace_root(), + image_identity: None, + } + } +} + +fn default_vm_workspace_root() -> String { + driver_mounts::DEFAULT_WORKSPACE_ROOT.to_string() +} + #[derive(Clone, serde::Serialize, serde::Deserialize)] #[serde(deny_unknown_fields)] #[allow(clippy::struct_excessive_bools)] @@ -363,6 +410,10 @@ pub struct VmDriverConfig { pub vcpus: u8, pub mem_mib: u32, pub overlay_disk_mib: u64, + #[serde(default)] + pub workspace_mode: VmWorkspaceMode, + #[serde(default = "default_vm_workspace_root")] + pub workspace_root: String, pub guest_tls_ca: Option, pub guest_tls_cert: Option, pub guest_tls_key: Option, @@ -509,6 +560,8 @@ impl Default for VmDriverConfig { vcpus: DEFAULT_VCPUS, mem_mib: DEFAULT_MEM_MIB, overlay_disk_mib: DEFAULT_OVERLAY_DISK_MIB, + workspace_mode: VmWorkspaceMode::Managed, + workspace_root: default_vm_workspace_root(), guest_tls_ca: None, guest_tls_cert: None, guest_tls_key: None, @@ -533,6 +586,25 @@ impl Default for VmDriverConfig { } impl VmDriverConfig { + fn configured_workspace_root(&self) -> Option<&str> { + (self.workspace_mode == VmWorkspaceMode::Managed).then_some(self.workspace_root.as_str()) + } + + fn resolve_workspace(&self, working_dir: Option<&str>) -> Result { + let root = self + .configured_workspace_root() + .unwrap_or_else(|| working_dir.unwrap_or_default()); + let root = validate_vm_workspace_root(root, self.workspace_mode)?; + if root == driver_mounts::DEFAULT_WORKSPACE_ROOT { + return Ok(VmWorkspace::default()); + } + Ok(VmWorkspace { + mode: self.workspace_mode, + root, + image_identity: None, + }) + } + /// Resolve a fallback sandbox UID for an image that has no sandbox account. pub fn resolve_sandbox_uid(&self) -> u32 { self.sandbox_uid.unwrap_or(DEFAULT_SANDBOX_UID) @@ -1169,6 +1241,9 @@ impl VmDriver { .validate() .map_err(|err| err.message().to_string())?; config.validate_sandbox_identity()?; + if config.workspace_mode == VmWorkspaceMode::Managed { + config.workspace_root = config.resolve_workspace(None)?.root; + } config.validate_runtime_security_config()?; config.validate_bootstrap_image_config()?; config.validate_rootfs_tar_config()?; @@ -1365,6 +1440,7 @@ impl VmDriver { .map_err(|error| Status::internal(format!("stage host additional CA: {error}"))) } + #[allow(clippy::too_many_arguments)] async fn spawn_host_supervisor( &self, sandbox: &Sandbox, @@ -1373,6 +1449,7 @@ impl VmDriver { runtime_descriptor: &SandboxRuntimeDescriptor, auth_bundle: &openshell_core::jwt::SupervisorAuthBundle, sandbox_owner: SandboxOwnerIdentity, + workspace_root: &str, ) -> Result<(Child, Option), Status> { // The destination CA is staged on the host before either resolving // the supervisor binary or spawning it. It never crosses into a VM @@ -1441,7 +1518,7 @@ impl VmDriver { .arg("--auth-bundle-file") .arg(&auth_bundle_path) .arg("--workdir") - .arg("/sandbox") + .arg(workspace_root) .args(upstream_proxy_args) .args(network_trust_args) .env( @@ -1582,6 +1659,15 @@ impl VmDriver { // gRPC API surface; boxing here would diverge from every other handler. #[allow(clippy::result_large_err)] pub fn validate_sandbox(&self, sandbox: &Sandbox) -> Result<(), Status> { + self.validate_sandbox_for_workspace(sandbox, self.config.configured_workspace_root()) + } + + #[allow(clippy::result_large_err)] + fn validate_sandbox_for_workspace( + &self, + sandbox: &Sandbox, + workspace_root: Option<&str>, + ) -> Result<(), Status> { openshell_core::resource_admission::check_sandbox_driver_config( self.config.allow_driver_config, sandbox, @@ -1594,6 +1680,9 @@ impl VmDriver { &self.config, sandbox_requests_gpu(sandbox), )?; + if let Some(workspace_root) = workspace_root { + validate_vm_workspace_mounts(&driver_config.mounts, workspace_root)?; + } let has_rootfs_tar = driver_config.rootfs_tar_path.is_some(); if self.resolved_sandbox_image(sandbox).is_none() && !has_rootfs_tar { return Err(Status::failed_precondition( @@ -1603,6 +1692,19 @@ impl VmDriver { Ok(()) } + async fn validate_persisted_sandbox( + &self, + sandbox: &Sandbox, + state_dir: &Path, + ) -> Result<(), Status> { + let workspace = read_sandbox_workspace(state_dir).await?; + let workspace_root = workspace + .as_ref() + .map(|workspace| workspace.root.as_str()) + .or_else(|| self.config.configured_workspace_root()); + self.validate_sandbox_for_workspace(sandbox, workspace_root) + } + // `tonic::Status` is large but is the standard error type across the // gRPC API surface; boxing here would diverge from every other handler. #[allow(clippy::result_large_err)] @@ -1825,6 +1927,15 @@ impl VmDriver { } Some(_) | None => None, }; + let persisted_workspace = if overlay_preparation == OverlayPreparation::PreserveExisting { + read_sandbox_workspace(&state_dir).await? + } else { + None + }; + let prepared_workspace_mode = (persisted_workspace.is_none() + && self.config.configured_workspace_root() + != Some(driver_mounts::DEFAULT_WORKSPACE_ROOT)) + .then_some(self.config.workspace_mode); self.publish_platform_event( sandbox.id.clone(), @@ -1836,28 +1947,37 @@ impl VmDriver { ), ); - let image_plan = if overlay_preparation == OverlayPreparation::PreserveExisting + let mut retained_image_identity = persisted_workspace + .as_ref() + .and_then(|workspace| workspace.image_identity.clone()); + if retained_image_identity.is_none() + && overlay_preparation == OverlayPreparation::PreserveExisting && driver_config_had_rootfs_tar { - let persisted_identity = + retained_image_identity = Some( read_persisted_image_identity(&state_dir).await.map_err(|err| { Status::internal(format!( "cannot restore rootfs-tar sandbox: persisted image identity not found: {err}" )) - })?; + })?, + ); + } + let image_plan = if let Some(retained_image_identity) = retained_image_identity { let bootstrap_image_ref = self.bootstrap_image_ref()?; let bootstrap_image_identity = self - .prepare_images_in_worker(&sandbox.id, &bootstrap_image_ref, None, true) + .prepare_images_in_worker(&sandbox.id, &bootstrap_image_ref, None, true, None) .await? .bootstrap_image_identity; let root_disk = image_cache_rootfs_image(&self.config.state_dir, &bootstrap_image_identity); - let image_disk = image_cache_rootfs_image(&self.config.state_dir, &persisted_identity); + let image_disk = + image_cache_rootfs_image(&self.config.state_dir, &retained_image_identity); RuntimeImagePlan { root_disk, image_disk: Some(image_disk), - image_identity: persisted_identity, + image_identity: retained_image_identity, bootstrap_image_identity, + working_dir: None, } } else { self.prepare_images_in_worker( @@ -1865,9 +1985,26 @@ impl VmDriver { &image_ref, rootfs_tar_path.as_deref(), false, + prepared_workspace_mode, ) .await? }; + let workspace = if let Some(workspace) = persisted_workspace { + workspace + } else { + let mut workspace = self + .config + .resolve_workspace(image_plan.working_dir.as_deref()) + .map_err(Status::invalid_argument)?; + validate_vm_workspace_mounts(&driver_config.mounts, &workspace.root)?; + // Keep original image contents across restarts even for + // workspace files that have not been copied up. + if prepared_workspace_mode.is_some() { + workspace.image_identity = Some(image_plan.image_identity.clone()); + } + write_sandbox_workspace(&state_dir, &workspace).await?; + workspace + }; let image_identity = image_plan.image_identity.clone(); self.ensure_provisioning_active(&sandbox.id).await?; info!( @@ -2045,6 +2182,7 @@ impl VmDriver { inject_guest_mount_manifest(&overlay_disk, &driver_config.mounts) .map_err(|error| Status::internal(format!("inject VM mount manifest: {error}"))) }) + .and_then(|()| inject_guest_workspace(&overlay_disk, &workspace)) { self.lifecycle_extensions .after_launch_failed(&sandbox, &state_dir, LaunchAbortReason::GuestPrepareFailed) @@ -2251,6 +2389,7 @@ impl VmDriver { &runtime_descriptor, &launch_authentication.supervisor, sandbox_owner_state, + &workspace.root, ) .await { @@ -2479,7 +2618,8 @@ impl VmDriver { .map_err(|error| { Status::failed_precondition(format!("read VM admission provenance: {error}")) })?; - self.validate_sandbox(&sandbox)?; + self.validate_persisted_sandbox(&sandbox, &state_dir) + .await?; if already_running { // The gateway keeps launch sessions in memory. A non-empty bundle // during startup recovery represents a new gateway session, so @@ -2812,7 +2952,7 @@ impl VmDriver { warn!(sandbox_id = %sandbox.id, reason = %error.message(), "VM recovery denied by launch authentication"); return false; } - if let Err(error) = self.validate_sandbox(&sandbox) { + if let Err(error) = self.validate_persisted_sandbox(&sandbox, &state_dir).await { warn!(sandbox_id = %sandbox.id, reason = %error.message(), "VM recovery denied by admission"); return false; } @@ -3287,9 +3427,17 @@ impl VmDriver { image_ref: &str, rootfs_tar: Option<&Path>, bootstrap_only: bool, + prepared_workspace_mode: Option, ) -> Result { match self - .run_preparation_in_worker(sandbox_id, image_ref, rootfs_tar, bootstrap_only, None) + .run_preparation_in_worker( + sandbox_id, + image_ref, + rootfs_tar, + bootstrap_only, + prepared_workspace_mode, + None, + ) .await? { preparation::Output::Images(plan) => Ok(plan), @@ -3318,7 +3466,7 @@ impl VmDriver { .map(preparation::WorkloadIdentitySelectors::from), }; match self - .run_preparation_in_worker(sandbox_id, "", None, false, Some(overlay)) + .run_preparation_in_worker(sandbox_id, "", None, false, None, Some(overlay)) .await? { preparation::Output::Overlay(owner) => Ok(owner), @@ -3366,6 +3514,7 @@ impl VmDriver { image_ref: &str, rootfs_tar: Option<&Path>, bootstrap_only: bool, + prepared_workspace_mode: Option, overlay: Option, ) -> Result { let span_status = openshell_otel::ErrorStatusGuard::current(); @@ -3408,6 +3557,7 @@ impl VmDriver { image_ref: image_ref.to_string(), rootfs_tar: rootfs_tar.map(Path::to_path_buf), bootstrap_only, + prepared_workspace_mode, overlay, lease_fd: -1, parent_pid: 0, @@ -3517,6 +3667,7 @@ impl VmDriver { sandbox_id: &str, image_ref: &str, rootfs_tar_path: Option<&Path>, + prepared_workspace_mode: Option, ) -> Result { let span_status = openshell_otel::ErrorStatusGuard::current(); let bootstrap_image_ref = self.bootstrap_image_ref()?; @@ -3534,26 +3685,32 @@ impl VmDriver { image_disk: Some(prepared.disk_path), image_identity: prepared.image_identity, bootstrap_image_identity, + working_dir: None, }); } - if image_ref.trim() == bootstrap_image_ref.trim() { + // Prepare custom workspaces from a distinct guest image disk so the + // host bootstrap builder cannot change image file ownership there. + if prepared_workspace_mode.is_none() && image_ref.trim() == bootstrap_image_ref.trim() { return span_status.finish(Ok(RuntimeImagePlan { root_disk, image_disk: None, image_identity: bootstrap_image_identity.clone(), bootstrap_image_identity, + working_dir: None, })); } + let inspect_workdir = prepared_workspace_mode == Some(VmWorkspaceMode::Image); let prepared = self - .ensure_prepared_image_disk(sandbox_id, image_ref, &root_disk) + .ensure_prepared_image_disk(sandbox_id, image_ref, &root_disk, inspect_workdir) .await?; span_status.finish(Ok(RuntimeImagePlan { root_disk, image_disk: Some(prepared.disk_path), image_identity: prepared.image_identity, bootstrap_image_identity, + working_dir: prepared.working_dir, })) } @@ -3716,7 +3873,7 @@ impl VmDriver { image_ref: &str, ) -> Result { let span_status = openshell_otel::ErrorStatusGuard::current(); - if let Some((engine, image_identity)) = + if let Some((engine, image_identity, _)) = self.resolve_local_container_image(image_ref).await? { let result = self @@ -3938,7 +4095,7 @@ impl VmDriver { async fn resolve_local_container_image( &self, image_ref: &str, - ) -> Result, Status> { + ) -> Result)>, Status> { let required_local_image = is_openshell_local_build_image_ref(image_ref); let engine = match connect_local_container_engine().await { Some(engine) => engine, @@ -3986,7 +4143,8 @@ impl VmDriver { image_identity = %image_identity, "vm driver: resolved image from local container engine" ); - Ok(Some((engine, image_identity))) + let working_dir = inspect.config.and_then(|config| config.working_dir); + Ok(Some((engine, image_identity, working_dir))) } Err(err) if is_docker_not_found_error(&err) && required_local_image => { Err(Status::failed_precondition(format!( @@ -4095,11 +4253,12 @@ impl VmDriver { sandbox_id: &str, image_ref: &str, bootstrap_root_disk: &Path, + inspect_workdir: bool, ) -> Result { - if let Some((docker, image_identity)) = + if let Some((docker, image_identity, working_dir)) = self.resolve_local_container_image(image_ref).await? { - return self + let mut prepared = self .ensure_prepared_local_image_disk( sandbox_id, image_ref, @@ -4107,7 +4266,9 @@ impl VmDriver { &image_identity, bootstrap_root_disk, ) - .await; + .await?; + prepared.working_dir = working_dir.filter(|_| inspect_workdir); + return Ok(prepared); } self.ensure_prepared_registry_image_disk( @@ -4116,6 +4277,7 @@ impl VmDriver { bootstrap_root_disk, ®istry_client(), ®istry_auth(image_ref)?, + inspect_workdir, ) .await } @@ -4128,7 +4290,12 @@ impl VmDriver { image_identity: &str, bootstrap_root_disk: &Path, ) -> Result { - let cache_identity = prepared_image_cache_identity(image_identity, &self.config); + // Old local cache entries were exported by tag. Keep those entries + // separate from exports bound to the inspected image and its config. + let cache_identity = prepared_image_cache_identity( + &format!("local-image-id:{image_identity}"), + &self.config, + ); let image_path = image_cache_rootfs_image(&self.config.state_dir, &cache_identity); if tokio::fs::metadata(&image_path).await.is_ok() { @@ -4136,6 +4303,7 @@ impl VmDriver { return Ok(PreparedImageDisk { image_identity: cache_identity, disk_path: image_path, + working_dir: None, }); } @@ -4146,6 +4314,7 @@ impl VmDriver { return Ok(PreparedImageDisk { image_identity: cache_identity, disk_path: image_path, + working_dir: None, }); } @@ -4164,7 +4333,7 @@ impl VmDriver { ]), ); if let Err(err) = - export_local_image_rootfs_to_path(docker, image_ref, &rootfs_archive).await + export_local_image_rootfs_to_path(docker, image_identity, &rootfs_archive).await { let _ = tokio::fs::remove_dir_all(&staging_dir).await; return Err(err); @@ -4189,6 +4358,7 @@ impl VmDriver { Ok(PreparedImageDisk { image_identity: cache_identity, disk_path: image_path, + working_dir: None, }) } @@ -4243,6 +4413,7 @@ impl VmDriver { return Ok(PreparedImageDisk { image_identity: cache_identity, disk_path: image_path, + working_dir: None, }); } @@ -4259,6 +4430,7 @@ impl VmDriver { return Ok(PreparedImageDisk { image_identity: cache_identity, disk_path: image_path, + working_dir: None, }); } @@ -4332,6 +4504,7 @@ impl VmDriver { Ok(PreparedImageDisk { image_identity: cache_identity, disk_path: image_path, + working_dir: None, }) } @@ -4342,6 +4515,7 @@ impl VmDriver { bootstrap_root_disk: &Path, client: &OciClient, auth: &RegistryAuth, + inspect_workdir: bool, ) -> Result { let reference = parse_registry_reference(image_ref)?; @@ -4383,21 +4557,25 @@ impl VmDriver { let cache_identity = prepared_image_cache_identity(&source_image_identity, &self.config); let image_path = image_cache_rootfs_image(&self.config.state_dir, &cache_identity); - if tokio::fs::metadata(&image_path).await.is_ok() { - self.publish_prepared_cache_hit(sandbox_id, image_ref, "registry", &cache_identity); - return Ok(PreparedImageDisk { - image_identity: cache_identity, - disk_path: image_path, - }); - } - - self.publish_prepared_cache_miss(sandbox_id, image_ref, "registry", &cache_identity); - let _cache_guard = self.image_cache_lock.lock().await; - if tokio::fs::metadata(&image_path).await.is_ok() { + let cache_hit = tokio::fs::metadata(&image_path).await.is_ok(); + let cache_guard = if cache_hit { + None + } else { + self.publish_prepared_cache_miss(sandbox_id, image_ref, "registry", &cache_identity); + Some(self.image_cache_lock.lock().await) + }; + if cache_hit || tokio::fs::metadata(&image_path).await.is_ok() { + drop(cache_guard); self.publish_prepared_cache_hit(sandbox_id, image_ref, "registry", &cache_identity); + let working_dir = if inspect_workdir { + pull_registry_image_workdir(client, &reference, auth, image_ref).await? + } else { + None + }; return Ok(PreparedImageDisk { image_identity: cache_identity, disk_path: image_path, + working_dir, }); } @@ -4427,6 +4605,15 @@ impl VmDriver { "config", ) .await?; + let working_dir = if inspect_workdir { + let path = + oci_layout_blob_path(&layout_dir, &manifest.config.digest).map_err(|err| { + Status::failed_precondition(format!("invalid config digest: {err}")) + })?; + read_registry_image_workdir(&path, image_ref).await? + } else { + None + }; let total_layers = manifest.layers.len(); let total_bytes: i64 = manifest.layers.iter().map(|layer| layer.size.max(0)).sum(); @@ -4481,6 +4668,7 @@ impl VmDriver { Ok(PreparedImageDisk { image_identity: cache_identity, disk_path: image_path, + working_dir, }) } @@ -5242,6 +5430,7 @@ pub async fn run_image_preparation_worker(request_path: &Path) -> Result<(), Str image_disk: None, image_identity: identity.clone(), bootstrap_image_identity: identity, + working_dir: None, })) } else { driver @@ -5249,6 +5438,7 @@ pub async fn run_image_preparation_worker(request_path: &Path) -> Result<(), Str &request.sandbox_id, &request.image_ref, request.rootfs_tar.as_deref(), + request.prepared_workspace_mode, ) .await .map(preparation::Output::Images) @@ -5580,19 +5770,12 @@ fn validate_vm_sandbox_template(template: &SandboxTemplate) -> Result<(), Status /// from GSI 32), so 32 leaves headroom on both. const VM_MAX_BIND_MOUNTS: usize = if cfg!(target_arch = "aarch64") { 32 } else { 3 }; -const VM_RESERVED_GUEST_PATHS: &[&str] = &[ - "/.openshell", - "/.openshell-bootstrap", - "/overlay", - "/lower", - "/newroot", - "/image-cache", - "/srv", - "/proc", - "/sys", - "/dev", - "/tmp", -]; +fn validate_vm_workspace_root(root: &str, mode: VmWorkspaceMode) -> Result { + match mode { + VmWorkspaceMode::Managed => driver_mounts::resolve_vm_managed_workspace_root(root), + VmWorkspaceMode::Image => driver_mounts::resolve_vm_image_workspace_root(root), + } +} const VIRTIOFS_TAG_PREFIX: &str = "osfs"; @@ -5673,11 +5856,6 @@ fn validate_vm_driver_mounts( driver_mounts::validate_container_mount_target(&mount.target) .map_err(Status::invalid_argument)?; let normalized = driver_mounts::normalize_mount_target(&mount.target); - driver_mounts::validate_workspace_mount_target( - &normalized, - driver_mounts::DEFAULT_WORKSPACE_ROOT, - ) - .map_err(Status::invalid_argument)?; for reserved in VM_RESERVED_GUEST_PATHS { if driver_mounts::paths_overlap(Path::new(&normalized), Path::new(reserved)) { return Err(Status::invalid_argument(format!( @@ -5702,6 +5880,15 @@ fn validate_vm_driver_mounts( Ok(()) } +#[allow(clippy::result_large_err)] +fn validate_vm_workspace_mounts(mounts: &[VmMountConfig], root: &str) -> Result<(), Status> { + for mount in mounts { + driver_mounts::validate_workspace_mount_target(&mount.target, root) + .map_err(Status::invalid_argument)?; + } + Ok(()) +} + /// Host filesystem checks, run at provisioning rather than in /// `validate_sandbox` so a briefly missing source (for example an NFS share /// not yet mounted after a reboot) fails provisioning visibly instead of @@ -5921,6 +6108,7 @@ async fn export_local_image_rootfs_to_path( Some(create_options), ContainerCreateBody { image: Some(image_ref.to_string()), + working_dir: Some("/".to_string()), ..Default::default() }, ) @@ -6006,6 +6194,77 @@ async fn pin_registry_reference( Ok((reference.clone_with_digest(digest.clone()), digest)) } +async fn pull_registry_image_workdir( + client: &OciClient, + pinned_reference: &Reference, + auth: &RegistryAuth, + image_ref: &str, +) -> Result, Status> { + let config = retry_registry_request("pull image working directory", || async { + let (manifest, _) = client.pull_image_manifest(pinned_reference, auth).await?; + let mut stream = client + .pull_blob_stream(pinned_reference, &manifest.config) + .await?; + let mut config = Vec::new(); + while let Some(chunk) = stream.try_next().await? { + if config.len().saturating_add(chunk.len()) > MAX_REGISTRY_IMAGE_CONFIG_BYTES { + return Err(OciDistributionError::GenericError(Some( + "image config exceeds the size limit".to_string(), + ))); + } + config.extend_from_slice(&chunk); + } + Ok(config) + }) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "failed to read config for vm sandbox image '{image_ref}': {err}" + )) + })?; + parse_registry_image_workdir(&config, image_ref) +} + +async fn read_registry_image_workdir( + config_path: &Path, + image_ref: &str, +) -> Result, Status> { + let mut config = Vec::new(); + tokio::fs::File::open(config_path) + .await + .map_err(|err| Status::internal(format!("open staged OCI image config: {err}")))? + .take(MAX_REGISTRY_IMAGE_CONFIG_BYTES as u64 + 1) + .read_to_end(&mut config) + .await + .map_err(|err| Status::internal(format!("read staged OCI image config: {err}")))?; + if config.len() > MAX_REGISTRY_IMAGE_CONFIG_BYTES { + return Err(Status::failed_precondition(format!( + "config for vm sandbox image '{image_ref}' exceeds the size limit" + ))); + } + parse_registry_image_workdir(&config, image_ref) +} + +fn parse_registry_image_workdir(config: &[u8], image_ref: &str) -> Result, Status> { + #[derive(serde::Deserialize)] + struct ImageConfig { + config: Option, + } + + #[derive(serde::Deserialize)] + struct ExecutionConfig { + #[serde(rename = "WorkingDir")] + working_dir: Option, + } + + let image: ImageConfig = serde_json::from_slice(config).map_err(|err| { + Status::failed_precondition(format!( + "invalid config for vm sandbox image '{image_ref}': {err}" + )) + })?; + Ok(image.config.and_then(|config| config.working_dir)) +} + async fn retry_registry_request( operation: &str, request: F, @@ -7358,6 +7617,45 @@ async fn read_persisted_image_identity(state_dir: &Path) -> Result Result, Status> { + let bytes = match tokio::fs::read(state_dir.join(SANDBOX_WORKSPACE_FILE)).await { + Ok(bytes) => bytes, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + // Older overlays have no workspace metadata. A create interrupted + // before resolving its workspace has neither a marker nor a disk. + return tokio::fs::try_exists(sandbox_runtime_disk_paths(state_dir).overlay_disk) + .await + .map(|exists| exists.then(VmWorkspace::default)) + .map_err(|error| Status::internal(format!("stat VM overlay disk: {error}"))); + } + Err(error) => return Err(Status::internal(format!("read VM workspace: {error}"))), + }; + let workspace: VmWorkspace = serde_json::from_slice(&bytes) + .map_err(|error| Status::internal(format!("decode VM workspace: {error}")))?; + let root = + validate_vm_workspace_root(&workspace.root, workspace.mode).map_err(Status::internal)?; + if workspace.root != root { + return Err(Status::internal( + "persisted VM workspace path is not normalized", + )); + } + Ok(Some(workspace)) +} + +#[allow(clippy::result_large_err)] +async fn write_sandbox_workspace(state_dir: &Path, workspace: &VmWorkspace) -> Result<(), Status> { + let bytes = serde_json::to_vec(workspace) + .map_err(|error| Status::internal(format!("encode VM workspace: {error}")))?; + let staging = state_dir.join("workspace.json.tmp"); + write_private_file(&staging, bytes) + .await + .map_err(|error| Status::internal(format!("stage VM workspace: {error}")))?; + tokio::fs::rename(staging, state_dir.join(SANDBOX_WORKSPACE_FILE)) + .await + .map_err(|error| Status::internal(format!("persist VM workspace: {error}"))) +} + async fn write_sandbox_request(state_dir: &Path, sandbox: &Sandbox) -> Result<(), std::io::Error> { restrict_owner_only_dir(state_dir).await?; let destination = state_dir.join(SANDBOX_REQUEST_FILE); @@ -7646,6 +7944,21 @@ fn inject_guest_mount_manifest( Ok(()) } +#[allow(clippy::result_large_err)] +fn inject_guest_workspace(overlay_disk: &Path, workspace: &VmWorkspace) -> Result<(), Status> { + if workspace.root == driver_mounts::DEFAULT_WORKSPACE_ROOT { + return Ok(()); + } + let mode = match workspace.mode { + VmWorkspaceMode::Managed => "managed", + VmWorkspaceMode::Image => "image", + }; + let contents = format!("{mode}\n{}\n", workspace.root); + write_rootfs_image_file(overlay_disk, OVERLAY_WORKSPACE_FILE, contents.as_bytes()) + .and_then(|()| set_rootfs_image_file_mode(overlay_disk, OVERLAY_WORKSPACE_FILE, 0o600)) + .map_err(|error| Status::internal(format!("inject VM workspace: {error}"))) +} + fn guest_boundary_config_path(generation: &str) -> String { format!("{GUEST_BOUNDARY_CONFIG_DIR}/bootstrap-{generation}.json") } @@ -10912,6 +11225,7 @@ mod tests { image_disk: None, image_identity: "warm".to_string(), bootstrap_image_identity: "warm".to_string(), + working_dir: None, }))); let overlay = preparation::Message::Complete(Ok(preparation::Output::Overlay( SandboxOwnerIdentity { @@ -10970,7 +11284,7 @@ mod tests { let slow_driver = driver.clone(); let slow = tokio::spawn(async move { slow_driver - .prepare_images_in_worker("slow-a", "test-bootstrap", None, true) + .prepare_images_in_worker("slow-a", "test-bootstrap", None, true, None) .await }); let warm_driver = driver.clone(); @@ -10980,7 +11294,7 @@ mod tests { tokio::time::sleep(Duration::from_millis(10)).await; } let plan = warm_driver - .prepare_images_in_worker("warm-b", "test-bootstrap", None, true) + .prepare_images_in_worker("warm-b", "test-bootstrap", None, true, None) .await .map_err(|error| error.to_string())?; if plan.root_disk != cached_root { @@ -11050,6 +11364,7 @@ mod tests { image_ref: "test-image".to_string(), rootfs_tar: None, bootstrap_only: false, + prepared_workspace_mode: None, overlay: None, lease_fd: -1, parent_pid: 0, @@ -14352,6 +14667,62 @@ while :; do sleep 0.05; done"#, config } + #[tokio::test] + async fn vm_workspace_selection_persists_and_protects_the_selected_path() { + let state_dir = tempfile::tempdir().unwrap(); + let state_dir = state_dir.path(); + let mut config = mount_test_config(true); + let default = driver_mounts::DEFAULT_WORKSPACE_ROOT; + assert_eq!( + config.resolve_workspace(Some("/home/app")).unwrap().root, + default + ); + + config.workspace_mode = VmWorkspaceMode::Image; + assert_eq!(config.resolve_workspace(None).unwrap().root, default); + assert!(config.resolve_workspace(Some("/overlay/data")).is_err()); + let mut workspace = config.resolve_workspace(Some("/home/app")).unwrap(); + assert_eq!(workspace.root, "/home/app"); + assert_eq!(workspace.mode, VmWorkspaceMode::Image); + workspace.image_identity = Some("pinned-image".to_string()); + + let overlay_disk = sandbox_runtime_disk_paths(state_dir).overlay_disk; + tokio::fs::write(&overlay_disk, b"pre-existing overlay") + .await + .unwrap(); + assert_eq!( + read_sandbox_workspace(state_dir) + .await + .unwrap() + .unwrap() + .root, + default + ); + write_sandbox_workspace(state_dir, &workspace) + .await + .unwrap(); + + config.workspace_mode = VmWorkspaceMode::Managed; + config.workspace_root = "/work".to_string(); + assert_eq!(config.resolve_workspace(None).unwrap().root, "/work"); + let restored = read_sandbox_workspace(state_dir).await.unwrap().unwrap(); + assert_eq!(restored.root, workspace.root); + assert_eq!(restored.mode, workspace.mode); + assert_eq!(restored.image_identity, workspace.image_identity); + + let mut mount = VmMountConfig { + mount_type: None, + source: "/host/data".to_string(), + target: "/home".to_string(), + read_only: true, + }; + let error = + validate_vm_workspace_mounts(std::slice::from_ref(&mount), &restored.root).unwrap_err(); + assert!(error.message().contains("workspace")); + mount.target = "/home/app/data".to_string(); + assert!(validate_vm_workspace_mounts(&[mount], &restored.root).is_ok()); + } + #[test] fn vm_mount_validation_rejects_when_resource_admission_enabled() { let dir = std::env::temp_dir(); @@ -14764,17 +15135,17 @@ while :; do sleep 0.05; done"#, )); } - /// Runs `prepare_virtiofs_target` from the guest init script against a + /// Runs `prepare_guest_directory` from the guest init script against a /// host directory standing in for the guest root. #[cfg(unix)] fn run_prepare_virtiofs_target(root: &Path, target: &str) -> std::process::Output { let script = include_str!("../scripts/openshell-vm-sandbox-init.sh"); let start = script - .find("prepare_virtiofs_target() {") - .expect("init script defines prepare_virtiofs_target"); + .find("prepare_guest_directory() {") + .expect("init script defines prepare_guest_directory"); let end = start + script[start..].find("\n}\n").expect("function end") + 3; let program = format!( - "set -euo pipefail\nts() {{ echo \"$@\"; }}\n{}\nprepare_virtiofs_target \"$1\" \"$(id -u):$(id -g)\"\n", + "set -euo pipefail\nts() {{ echo \"$@\"; }}\n{}\nprepare_guest_directory \"$1\" \"$(id -u):$(id -g)\" /sandbox\n", &script[start..end] ); std::process::Command::new("bash") diff --git a/crates/openshell-driver-vm/src/main.rs b/crates/openshell-driver-vm/src/main.rs index 56f7c9a5d8..8b2c377e8d 100644 --- a/crates/openshell-driver-vm/src/main.rs +++ b/crates/openshell-driver-vm/src/main.rs @@ -7,6 +7,7 @@ use miette::{IntoDiagnostic, Result}; use openshell_core::VERSION; use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer; use openshell_core::sandbox_env::SupervisorOtlpArgs; +use openshell_driver_vm::driver::VmWorkspaceMode; #[cfg(target_os = "macos")] use openshell_driver_vm::{VM_RUNTIME_DIR_ENV, configured_runtime_dir}; use openshell_driver_vm::{ @@ -220,6 +221,14 @@ struct Args { #[arg(long, env = "OPENSHELL_VM_SANDBOX_GID")] sandbox_gid: Option, + /// Use a managed workspace or the image's OCI working directory. + #[arg(long, value_enum, default_value = "managed")] + workspace_mode: VmWorkspaceMode, + + /// Absolute guest workspace path used in `managed` mode. + #[arg(long, default_value = openshell_core::driver_mounts::DEFAULT_WORKSPACE_ROOT)] + workspace_root: String, + // Corporate forward proxy for sandbox egress. Operator-owned: these reach // the host supervisor on its argv, which the sandbox image and the // user-supplied environment cannot influence. @@ -365,6 +374,8 @@ async fn main() -> Result<()> { gpu_vcpus: args.gpu_vcpus, sandbox_uid: args.sandbox_uid, sandbox_gid: args.sandbox_gid, + workspace_mode: args.workspace_mode, + workspace_root: args.workspace_root, rootfs_tar_staging_dir: args.rootfs_tar_staging_dir.clone(), rootfs_tar_max_bytes: args.rootfs_tar_max_bytes, enable_bind_mounts: args.enable_bind_mounts, diff --git a/crates/openshell-driver-vm/src/preparation.rs b/crates/openshell-driver-vm/src/preparation.rs index 23c623f9ee..e76b19b9c4 100644 --- a/crates/openshell-driver-vm/src/preparation.rs +++ b/crates/openshell-driver-vm/src/preparation.rs @@ -26,7 +26,7 @@ use tokio::sync::Mutex; use tonic::Status; use super::{ - OverlayPreparation, RuntimeImagePlan, SandboxOwnerIdentity, VmDriverConfig, + OverlayPreparation, RuntimeImagePlan, SandboxOwnerIdentity, VmDriverConfig, VmWorkspaceMode, WorkloadIdentityRequest, }; @@ -43,6 +43,7 @@ pub(super) struct Request { pub image_ref: String, pub rootfs_tar: Option, pub bootstrap_only: bool, + pub prepared_workspace_mode: Option, pub overlay: Option, pub lease_fd: i32, pub parent_pid: u32, @@ -576,6 +577,7 @@ mod tests { image_ref: "test-image".to_string(), rootfs_tar: None, bootstrap_only: false, + prepared_workspace_mode: None, overlay: None, lease_fd: -1, parent_pid: 0, diff --git a/crates/openshell-driver-vm/src/registry_cache_tests.rs b/crates/openshell-driver-vm/src/registry_cache_tests.rs index 85a5f2084c..14fd58cb71 100644 --- a/crates/openshell-driver-vm/src/registry_cache_tests.rs +++ b/crates/openshell-driver-vm/src/registry_cache_tests.rs @@ -6,8 +6,8 @@ use oci_client::client::ClientProtocol; use oci_client::manifest::OCI_IMAGE_INDEX_MEDIA_TYPE; use tokio::io::AsyncReadExt as _; -/// A loopback registry with independent HEAD identity and GET contents. Blob -/// requests deliberately fail, so a rejected manifest needs no VM or formatter. +/// A loopback registry with independent HEAD identity and GET contents. Only +/// explicitly registered blobs are served. struct Registry { address: String, requests: Arc>>, @@ -16,10 +16,27 @@ struct Registry { impl Registry { async fn start( + head_digest: String, + manifests: HashMap>, + get_digest: Option, + retry_once: bool, + ) -> Self { + Self::start_with_blobs( + head_digest, + manifests, + get_digest, + retry_once, + HashMap::new(), + ) + .await + } + + async fn start_with_blobs( head_digest: String, manifests: HashMap>, get_digest: Option, mut retry_once: bool, + blobs: HashMap>, ) -> Self { let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); let address = listener.local_addr().unwrap().to_string(); @@ -54,6 +71,12 @@ impl Registry { digest = Some(get_digest.clone().unwrap_or_else(|| sha256(body))); ("200 OK", body.as_slice()) } + } else if let Some(body) = path + .strip_prefix("/v2/test/image/blobs/") + .and_then(|key| blobs.get(key)) + { + digest = Some(sha256(body)); + ("200 OK", body.as_slice()) } else { ("404 Not Found", b"fixture has no blob".as_slice()) }; @@ -186,6 +209,7 @@ async fn ensure_image( Path::new("unused"), client, &RegistryAuth::Anonymous, + false, ) .await .map(|prepared| prepared.image_identity) @@ -267,6 +291,54 @@ async fn prepared_registry_cache_rejects_manifest_identity_mismatch() { rejects_unbound_manifest(false).await; } +#[tokio::test] +async fn registry_workspace_workdir_is_bound_to_the_cached_image() { + let config = r#"{"config":{"WorkingDir":"/home/app"}}"#; + let trusted = manifest(config); + let image_digest = sha256(&trusted); + let config_digest = sha256(config.as_bytes()); + let registry = Registry::start_with_blobs( + image_digest.clone(), + HashMap::from([ + ( + "latest".to_string(), + manifest(r#"{"config":{"WorkingDir":"/other"}}"#), + ), + (image_digest.clone(), trusted), + ]), + None, + false, + HashMap::from([(config_digest.clone(), config.as_bytes().to_vec())]), + ) + .await; + let root = tempfile::tempdir().unwrap(); + let driver = driver(root.path()); + let identity = prepared_image_cache_identity(&image_digest, &driver.config); + let path = image_cache_rootfs_image(root.path(), &identity); + fs::create_dir_all(path.parent().unwrap()).unwrap(); + fs::write(path, b"cached disk").unwrap(); + + let reference = registry.reference(None); + let client = client(); + let prepared = driver + .ensure_prepared_registry_image_disk( + "test", + &reference, + Path::new("unused"), + &client, + &RegistryAuth::Anonymous, + true, + ) + .await + .unwrap(); + assert_eq!(prepared.working_dir.as_deref(), Some("/home/app")); + + let requests = registry.requests(); + assert!(requests.contains(&format!("GET /v2/test/image/manifests/{image_digest}"))); + assert!(requests.contains(&format!("GET /v2/test/image/blobs/{config_digest}"))); + assert!(!requests.contains(&"GET /v2/test/image/manifests/latest".to_string())); +} + #[tokio::test] async fn bootstrap_registry_cache_rejects_manifest_identity_mismatch() { rejects_unbound_manifest(true).await; diff --git a/crates/openshell-gateway/src/vm.rs b/crates/openshell-gateway/src/vm.rs index 1f64a329a2..e55bdda9f6 100644 --- a/crates/openshell-gateway/src/vm.rs +++ b/crates/openshell-gateway/src/vm.rs @@ -60,6 +60,14 @@ const DRIVER_BIN_NAME: &str = "openshell-driver-vm"; const COMPUTE_DRIVER_SOCKET_RUN_DIR: &str = "run"; const COMPUTE_DRIVER_SOCKET_NAME: &str = "compute-driver.sock"; +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum VmWorkspaceMode { + #[default] + Managed, + Image, +} + /// Configuration for launching and talking to the VM compute driver. #[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] #[serde(default, deny_unknown_fields)] @@ -100,6 +108,12 @@ pub struct VmComputeConfig { /// Optional GID override for the VM guest sandbox account. pub sandbox_gid: Option, + /// Select an operator-managed workspace or the image's OCI working directory. + pub workspace_mode: VmWorkspaceMode, + + /// Absolute guest workspace path used in `managed` mode. + pub workspace_root: String, + /// Directory shared with the gateway for request-scoped rootfs tar staging. pub rootfs_tar_staging_dir: Option, @@ -178,6 +192,10 @@ impl VmComputeConfig { )); } validate_vm_sandbox_identity(self)?; + if self.workspace_mode == VmWorkspaceMode::Managed { + openshell_core::driver_mounts::resolve_vm_managed_workspace_root(&self.workspace_root) + .map_err(|error| Error::config(format!("invalid workspace_root: {error}")))?; + } if self .rootfs_tar_staging_dir .as_ref() @@ -252,6 +270,8 @@ impl Default for VmComputeConfig { overlay_disk_mib: Self::default_overlay_disk_mib(), sandbox_uid: None, sandbox_gid: None, + workspace_mode: VmWorkspaceMode::default(), + workspace_root: openshell_core::driver_mounts::DEFAULT_WORKSPACE_ROOT.to_string(), rootfs_tar_staging_dir: None, rootfs_tar_max_bytes: None, guest_tls_ca: None, @@ -577,6 +597,13 @@ pub async fn spawn( .arg("--overlay-disk-mib") .arg(vm_config.overlay_disk_mib.to_string()); append_vm_identity_args(&mut command, vm_config); + if vm_config.workspace_mode == VmWorkspaceMode::Image { + command.arg("--workspace-mode").arg("image"); + } else if vm_config.workspace_root != openshell_core::driver_mounts::DEFAULT_WORKSPACE_ROOT { + command + .arg("--workspace-root") + .arg(&vm_config.workspace_root); + } append_vm_rootfs_tar_args(&mut command, vm_config); if let Some(tls) = guest_tls_paths { command.arg("--guest-tls-ca").arg(tls.ca); @@ -1351,6 +1378,17 @@ mod tests { assert!(error.to_string().contains("sandbox_uid 0")); } + #[test] + fn vm_gateway_config_rejects_reserved_workspace_root() { + let config = VmComputeConfig { + grpc_endpoint: "http://127.0.0.1:50051".to_string(), + workspace_root: "/overlay".to_string(), + ..Default::default() + }; + let error = config.validate_configuration().unwrap_err().to_string(); + assert!(error.contains("workspace_root") && error.contains("/overlay")); + } + #[test] fn resolve_driver_bin_uses_driver_dir_when_binary_present() { let dir = tempdir().unwrap(); diff --git a/docs/how-it-works/gateways/configuration.mdx b/docs/how-it-works/gateways/configuration.mdx index 52bb44113d..d23438e2bb 100644 --- a/docs/how-it-works/gateways/configuration.mdx +++ b/docs/how-it-works/gateways/configuration.mdx @@ -1918,6 +1918,11 @@ overlay_disk_mib = 4096 # Values must fall within OpenShell's allowed non-root sandbox identity range. # sandbox_uid = 20001 # sandbox_gid = 20001 +# Prepare this guest directory for the sandbox user in "managed" mode. +# In "image" mode, use the image's OCI WORKDIR, or the managed /sandbox +# when there is no custom WORKDIR. The workspace is fixed at sandbox creation. +# workspace_mode = "managed" +# workspace_root = "/sandbox" # Corporate forward proxy for sandbox egress. The keys, their semantics, and # the fail-closed contract are identical to the Podman driver above: only TLS # (CONNECT) egress is chained, plain-HTTP destination requests always dial diff --git a/docs/how-it-works/sandboxes/runtimes.mdx b/docs/how-it-works/sandboxes/runtimes.mdx index d16938da01..21b4e3f029 100644 --- a/docs/how-it-works/sandboxes/runtimes.mdx +++ b/docs/how-it-works/sandboxes/runtimes.mdx @@ -245,6 +245,22 @@ VM sandboxes have no network interface. All traffic flows through the OpenShell If the gateway is briefly unavailable while a VM starts, its supervisor retries the session connection. The sandbox stays in provisioning until the gateway accepts the session. +### MicroVM Workspace + +By default, the VM driver prepares `/sandbox` as the workspace for the sandbox user. To select another managed directory, set an absolute guest path in the gateway configuration: + +```toml +[openshell.drivers.vm] +workspace_mode = "managed" +workspace_root = "/home/agent" +``` + +OpenShell prepares the selected directory for the sandbox user and preserves ownership of existing contents in custom managed directories. + +To use the image's OCI `WORKDIR`, set `workspace_mode = "image"`. The directory must already exist in the guest image without symlink components and be writable by the resolved sandbox user. VM image preparation continues to normalize the compatibility `/sandbox` tree. Images with an unset `WORKDIR`, `/`, or `/sandbox`, and rootfs tar archives without OCI metadata, use the managed `/sandbox` workspace. + +Each sandbox keeps its selected workspace across stop and start. The workspace lives on the sandbox's writable guest overlay. + ### MicroVM Stop Durability When you run `openshell sandbox stop` while the main process is running, OpenShell terminates the workload and flushes the guest root filesystem before acknowledging termination. The guest waits up to three seconds for the flush. The host allows up to 22 seconds for supervisor shutdown, then up to five seconds to stop the VM. @@ -375,6 +391,6 @@ On Docker and Podman, set `process.run_as_user` and `process.run_as_group` in th MicroVM persists the resolved UID:GID with the writable overlay and retains it across restarts, even if driver defaults change. Explicit policy selectors must match that identity. For example, `run_as_user: "10000"` fails when the overlay owner is `1000:1000`; OpenShell reports both values without changing file ownership. The symbolic selector `sandbox` refers to the guest account for that owner, and each omitted selector accepts its driver default. The `WorkloadIdentity` condition in `openshell sandbox get ` shows the resolved pair separately from workload readiness. Canonical and exec processes use this same pair; live policy updates cannot change it. -On Docker and Podman, the image's `WORKDIR` becomes the workspace. Images with no `WORKDIR`, `/`, or `/sandbox` use `/sandbox`. Any other `WORKDIR` must exist in the image and be writable by the sandbox user. Kubernetes and MicroVM always use `/sandbox`. +On Docker and Podman, the image's `WORKDIR` becomes the workspace. Images with no `WORKDIR`, `/`, or `/sandbox` use `/sandbox`. Any other `WORKDIR` must exist in the image and be writable by the sandbox user. Kubernetes uses `/sandbox`. MicroVM defaults to `/sandbox` and supports [operator configuration](#microvm-workspace). Custom Docker and Podman workspaces use the workload container's writable layer, not a managed named volume. Files survive stop/start of the same container, but removing or replacing that container removes its workspace changes. Export files you need to retain before deleting the sandbox.