Repository navigation
Description
User Story
I use OpenShell through NemoClaw, which runs OpenClaw and Hermes agents in OpenShell sandboxes on the Docker driver and on the Kubernetes driver (kind with the default local-path storage in NemoClaw's CI). On Kubernetes, the sandbox user's home, /sandbox, is a root-owned, world-writable directory without the sticky bit. Software that requires a private home refuses to start there, and the same software works on the Docker driver.
Problem Statement
On the Kubernetes driver with a provisioner that creates the workspace PVC root as root:root 0777 (rancher local-path, the default StorageClass in kind and k3s), the sandbox sees:
$ stat -c '%n %a %u:%g' /sandbox
/sandbox 777 0:0
The Docker and Podman drivers create /sandbox owned by the sandbox user with mode 0700 (Docker: docker_sandbox_bundle_archive in crates/openshell-driver-docker/src/lib.rs; Podman: workspace.directory(".", 0o700, true) in crates/openshell-driver-podman/src/isolation.rs).
On Kubernetes the agent container mounts the workspace PVC root at /sandbox (apply_workspace_persistence in crates/openshell-driver-kubernetes/src/driver.rs), so the root's owner and mode come from the provisioner. The workspace-init container (openshell-sandbox seed-workspace) runs as the sandbox UID with no capabilities. It copies the image's /sandbox contents and leaves the root alone, and it couldn't chown or chmod a root-owned directory anyway. The pod's fsGroup doesn't help: kubelet doesn't apply fsGroup to local-path's hostPath volumes, and on volume types where it does apply, the root becomes root:<gid> with the setgid bit, which is still not owned by the sandbox user.
This is the same at v0.1.9 (ce58cf8e, the version NemoClaw pins) and on main (78ba3477). I found no existing issue for the PVC root. #2761 covered the ownership of seeded contents under the former root seeder, and #5047 asks for private subpaths for caller-provided mounts.
Impact / Why This Matters
- Tools that check that their home or state directory is private refuse a world-writable home. NeMo-Fabric started refusing it in fix: persist invocation results for SDK and CLI recovery NeMo-Fabric#629, and NemoClaw's kind E2E jobs fail on chore(fabric): pin Fabric release/0.7 with the OpenClaw anthropic and web_fetch fixes NemoClaw#14615 (amd64, arm64) while the Docker jobs pass.
- A world-writable directory without the sticky bit lets any UID that can reach it delete or replace the files in it, including
~/.bashrc,~/.profile, and tool configuration that the sandbox user later runs. Today the workload's processes share one UID, so I don't know of a way to exploit this, which is why this is a public issue. It is still weaker than what Docker and Podman give the same workload, and it removes a layer if another identity ever runs in the pod. - The workaround is to replace the default workspace with a caller-provided PVC mount, which gives up the driver-managed workspace and needs the same ownership preparation outside OpenShell (feat(kubernetes): support private PVC subpaths for non-root workloads #5047).
Acceptance Criteria
- On the Kubernetes driver with local-path storage,
/sandboxin a new sandbox is owned by the sandbox UID with mode0700, as on Docker and Podman. - The sandbox pod stays non-root: no init container gains root or capabilities to do this.
- Files in
/sandboxsurvive stop and start, and existing sandboxes keep their workspace contents after a gateway upgrade. - A conformance test checks that
/sandboxbelongs to the sandbox user and isn't group- or world-writable.
Reproduction Steps
- Create a kind cluster (
kind create cluster); its default StorageClass is local-path. - Install the OpenShell v0.1.9 Helm chart with the Kubernetes driver as in the Kubernetes setup guide.
openshell sandbox create --name perms -- sleep infinityopenshell sandbox exec perms -- sh -c "id; stat -c '%n %a %U %u:%g' /sandbox"
Observed: /sandbox 777 root 0:0. Expected: /sandbox 700 sandbox, with the sandbox user's UID and GID, as with the Docker driver.
The same result without a gateway: in a kind cluster, a Pod with runAsUser: 10001, fsGroup: 10001, fsGroupChangePolicy: OnRootMismatch that mounts a fresh local-path PVC sees the mount root as 777 0:0. On the node, the provisioner created /var/local-path-provisioner/pvc-… as 777 0:0.
Environment
- OpenShell: v0.1.9 (
ce58cf8e9b953281352d98a637e41616295b1318), also currentmain - Kubernetes: kind v0.33.0 (
kindest/node:v1.37.0), StorageClassstandard(rancher.io/local-path) - Host: Ubuntu 24.04 (arm64) and GitHub-hosted runners (amd64 and arm64)
- Docker driver, same image:
/sandboxis700and owned by the sandbox user
Activity