Repository navigation
Description
User Story
I run agent workloads on OpenShell on Kubernetes. I directly need sandbox workloads that get full virtual-machine semantics — a dedicated guest OS (Linux today; other guest operating systems are follow-up work), and a class of agent that needs low-level access: writing the base disk, loading a kernel, replacing a bootloader. This affects which workloads I can safely hand to a sandbox on a shared cluster.
Problem Statement
Today OpenShell's Kubernetes compute driver runs every sandbox as a pod: a container sharing the host kernel. That is fast, but the workload gets a Linux container and nothing else. Two capabilities are missing:
- Full VM semantics: a dedicated per-sandbox machine and a guest OS selected per workload — Linux guests today. Non-Linux guests are a goal, not a current capability: each guest OS would need its own bootstrap and traffic-steering story (read the config disk, run the boundary, steer traffic to the supervisor), and that is listed as follow-up work.
- A root-workload class: agents that need low-level access (base disk, kernel, bootloader). You cannot safely give a container that power, because a compromised root container shares the host kernel.
Impact / Why This Matters
Without this, workloads that need VM semantics or root-level machine access must run outside OpenShell entirely, or accept container semantics on a shared host kernel. For multi-tenant clusters the container model is fine for isolation, but isolation is not the gap — Kata-style runtimeClassName already provides that with zero driver changes. The gap is VM semantics: agents that build or boot operating systems have no place to run inside OpenShell, which blocks that whole class of automation.
Proposed Design
A new runtime mode of the existing Kubernetes compute driver. The sandbox API, gateway, and security model are unchanged; only where the workload runs changes:
- Selecting
runtime = "kubevirt"in the Kubernetes driver config provisions each sandbox as a KubeVirtVirtualMachineinstead of a pod. - A sandbox reaches
Readyonly when the in-guest boundary connects to its supervisor — not when the VM merely runs. Exec, stop/start, delete, and policy evaluation behave exactly as on the pod runtime. - Each sandbox gets a per-sandbox egress fence (NetworkPolicy admitting only its supervisor), a one-shot bootstrap delivery, and a workspace volume that survives stop/start.
- Every guest runs its workload as root, and the guest — including the in-guest boundary — is untrusted: enforcement is deliberately outside the guest (fence, supervisor policy, driver checks), guest-supplied image policy is ignored, and guest-reported data is marked as guest-sourced in logs. Guest configuration (the in-guest traffic relay) is required for networking to work and is never relied on for security.
Notional configuration:
[openshell.drivers.kubernetes]
runtime = "kubevirt"
[openshell.drivers.kubernetes.kubevirt]
guest_image = "example.com/openshell/sandbox-vm:latest"
cpu_cores = 1
memory = "1Gi"
workload = "root"Acceptance Criteria
- With
runtime = "kubevirt",openshell sandbox createreaches Ready,execworks inside the guest, and stop/start preserve the VM and fence identities. - Direct egress from the guest is blocked except to its supervisor, verified at start.
-
sandbox deleteleaves zero managed objects. - In root mode, the workload runs as uid 0 and policy enforcement remains effective from outside the guest; guest-reported log data is distinguishable from supervisor-verified data.
- The mode, its configuration, and its security posture are documented.
Alternatives Considered
- Kata containers via
runtimeClassName: gives stronger isolation today with no driver changes, but not full VM semantics (no arbitrary guest OS, no root-machine access). Good complement, not a substitute. - The upstream Agent Sandbox CR: pod-shaped; there is no VM representation, so the PoC creates
VirtualMachineobjects directly. An upstream CR extension for VMs would be the long-term home and is worth discussing here. - The standalone libkrun VM driver: host-local VMs, not cluster-scheduled KubeVirt machines; a different deployment model.
- Extension points (interceptors, middleware, providers): none of them change where the workload runs, so they cannot close this gap.
Agent Investigation
A working proof of concept exists (design seeking review, not a merge request): branch kubevirt-compute-driver on my fork, rebased on current main. The 10-step acceptance demo and the root-mode attack-surface probes pass live on KubeVirt v1.9.1 (k3d). The trust model (nothing in-guest is a trust anchor; the guest is untrusted after the first workload runs in root mode), known limits, and demo transcripts are documented in e2e/kubevirt/README.md on that branch. A one-page explainer for reviewers: https://gist.github.com/jhjaggars/6814c43c9c732f2672637a9759bd7c89
A draft PR implementing this is ready to open once the issue is accepted.
Checklist
- I've reviewed existing issues and the published docs
- This is a design proposal, not a "please build this" request
Activity
johntmyers commentedon Nov 18, 2026
@jhjaggars we have this working for Firecracker and we are actively testing now. It'd be good to figure out if these efforts should be combined generally as a pattern on top of K8s.
regarding "and the ability to run non-Linux guests." - how would the isolation backend come into play here?
cc @drew @FrostGod - would like to see if this could generally be bundled together as "VMs on K8s" effort.
johntmyers commentedon Nov 18, 2026
@jhjaggars as a note your TS page isn't reachable through some corp proxies, attaching as an artifact might be better
jhjaggars commentedon Nov 18, 2026
The approach I've taken in my draft PR/PoC is that you only get the proxy features. So, none of the LSM controls are in place for example.