Skip to content

feat: KubeVirt runtime mode for the Kubernetes compute driver (VM-backed sandboxes) #5714

Open

Description

@jhjaggars

User Story

I run agent workloads on OpenShell on Kubernetes. I directly need sandbox workloads that get full virtual-machine semantics — a dedicated guest OS (Linux today; other guest operating systems are follow-up work), and a class of agent that needs low-level access: writing the base disk, loading a kernel, replacing a bootloader. This affects which workloads I can safely hand to a sandbox on a shared cluster.

Problem Statement

Today OpenShell's Kubernetes compute driver runs every sandbox as a pod: a container sharing the host kernel. That is fast, but the workload gets a Linux container and nothing else. Two capabilities are missing:

  • Full VM semantics: a dedicated per-sandbox machine and a guest OS selected per workload — Linux guests today. Non-Linux guests are a goal, not a current capability: each guest OS would need its own bootstrap and traffic-steering story (read the config disk, run the boundary, steer traffic to the supervisor), and that is listed as follow-up work.
  • A root-workload class: agents that need low-level access (base disk, kernel, bootloader). You cannot safely give a container that power, because a compromised root container shares the host kernel.

Impact / Why This Matters

Without this, workloads that need VM semantics or root-level machine access must run outside OpenShell entirely, or accept container semantics on a shared host kernel. For multi-tenant clusters the container model is fine for isolation, but isolation is not the gap — Kata-style runtimeClassName already provides that with zero driver changes. The gap is VM semantics: agents that build or boot operating systems have no place to run inside OpenShell, which blocks that whole class of automation.

Proposed Design

A new runtime mode of the existing Kubernetes compute driver. The sandbox API, gateway, and security model are unchanged; only where the workload runs changes:

  • Selecting runtime = "kubevirt" in the Kubernetes driver config provisions each sandbox as a KubeVirt VirtualMachine instead of a pod.
  • A sandbox reaches Ready only when the in-guest boundary connects to its supervisor — not when the VM merely runs. Exec, stop/start, delete, and policy evaluation behave exactly as on the pod runtime.
  • Each sandbox gets a per-sandbox egress fence (NetworkPolicy admitting only its supervisor), a one-shot bootstrap delivery, and a workspace volume that survives stop/start.
  • Every guest runs its workload as root, and the guest — including the in-guest boundary — is untrusted: enforcement is deliberately outside the guest (fence, supervisor policy, driver checks), guest-supplied image policy is ignored, and guest-reported data is marked as guest-sourced in logs. Guest configuration (the in-guest traffic relay) is required for networking to work and is never relied on for security.

Notional configuration:

[openshell.drivers.kubernetes]
runtime = "kubevirt"

[openshell.drivers.kubernetes.kubevirt]
guest_image = "example.com/openshell/sandbox-vm:latest"
cpu_cores   = 1
memory      = "1Gi"
workload    = "root"

Acceptance Criteria

  • With runtime = "kubevirt", openshell sandbox create reaches Ready, exec works inside the guest, and stop/start preserve the VM and fence identities.
  • Direct egress from the guest is blocked except to its supervisor, verified at start.
  • sandbox delete leaves zero managed objects.
  • In root mode, the workload runs as uid 0 and policy enforcement remains effective from outside the guest; guest-reported log data is distinguishable from supervisor-verified data.
  • The mode, its configuration, and its security posture are documented.

Alternatives Considered

  • Kata containers via runtimeClassName: gives stronger isolation today with no driver changes, but not full VM semantics (no arbitrary guest OS, no root-machine access). Good complement, not a substitute.
  • The upstream Agent Sandbox CR: pod-shaped; there is no VM representation, so the PoC creates VirtualMachine objects directly. An upstream CR extension for VMs would be the long-term home and is worth discussing here.
  • The standalone libkrun VM driver: host-local VMs, not cluster-scheduled KubeVirt machines; a different deployment model.
  • Extension points (interceptors, middleware, providers): none of them change where the workload runs, so they cannot close this gap.

Agent Investigation

A working proof of concept exists (design seeking review, not a merge request): branch kubevirt-compute-driver on my fork, rebased on current main. The 10-step acceptance demo and the root-mode attack-surface probes pass live on KubeVirt v1.9.1 (k3d). The trust model (nothing in-guest is a trust anchor; the guest is untrusted after the first workload runs in root mode), known limits, and demo transcripts are documented in e2e/kubevirt/README.md on that branch. A one-page explainer for reviewers: https://gist.github.com/jhjaggars/6814c43c9c732f2672637a9759bd7c89

A draft PR implementing this is ready to open once the issue is accepted.

Checklist

  • I've reviewed existing issues and the published docs
  • This is a design proposal, not a "please build this" request

Activity

  1. added
    state:acceptedA maintainer decided OpenShell should pursue this issue
    and removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Nov 18, 2026
  2. johntmyers commented on Nov 18, 2026

    @johntmyers
    Collaborator

    @jhjaggars we have this working for Firecracker and we are actively testing now. It'd be good to figure out if these efforts should be combined generally as a pattern on top of K8s.

    regarding "and the ability to run non-Linux guests." - how would the isolation backend come into play here?

    cc @drew @FrostGod - would like to see if this could generally be bundled together as "VMs on K8s" effort.

  3. johntmyers commented on Nov 18, 2026

    @johntmyers
    Collaborator

    @jhjaggars as a note your TS page isn't reachable through some corp proxies, attaching as an artifact might be better

  4. jhjaggars commented on Nov 18, 2026

    @jhjaggars
    ContributorAuthor

    The approach I've taken in my draft PR/PoC is that you only get the proxy features. So, none of the LSM controls are in place for example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:acceptedA maintainer decided OpenShell should pursue this issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions