Skip to content

feat(driver-kubernetes): KubeVirt runtime mode for VM-backed sandboxes - #5715

Draft
jhjaggars wants to merge 15 commits into
NVIDIA:mainfrom
jhjaggars:kubevirt-compute-driver
Draft

jhjaggars wants to merge 15 commits into
NVIDIA:mainfrom
jhjaggars:kubevirt-compute-driver

Conversation

@jhjaggars

@jhjaggars jhjaggars commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a runtime = "kubevirt" mode to the Kubernetes compute driver: each sandbox runs as a KubeVirt VirtualMachine instead of a pod, with a per-sandbox egress fence (NetworkPolicy), a one-shot bootstrap Secret delivered over cloud-init, and a per-sandbox workspace PVC shared over virtio-fs. The sandbox API, gateway, and security model are unchanged; only where the workload runs changes.

Every KubeVirt guest runs its workload as root, and the guest — including the in-guest openshell-sandbox boundary — is untrusted. The security boundary is everything outside the guest: the NetworkPolicy fence (only the supervisor is reachable), the supervisor's policy (decides what leaves), and the driver's checks of the VM UID and fence spec. Guest configuration (the nftables relay, the DNS relay) is required for networking to work and is never relied on for security. The supervisor does not trust the boundary for an untrusted guest: guest-supplied image policy is ignored, per-binary identity rules are rejected, provider material is refused, and every guest-reported event is marked data_origin=guest in OCSF logs. After the first boot the VM spec pins the guest image by digest; stop/start preserve VM and fence identities, and the fence is reconciled and re-verified on every start.

The earlier non-root ("sandboxed") implementation is kept on the kubevirt-compute-driver-sandboxed branch for comparison.

This is a proof of concept seeking design review: the guest image is purpose-built (Alpine + static supervisor + kernel, built by e2e/kubevirt/build-guest-image.sh), VMs are created directly rather than through an Agent Sandbox CR extension, and the known limits (fence is only as good as CNI NetworkPolicy enforcement; the bootstrap disk eject is hygiene, not a control) are documented in e2e/kubevirt/README.md. A one-page explainer for reviewers (design, trade-offs, trust model, limits): https://gist.github.com/jhjaggars/6814c43c9c732f2672637a9759bd7c89

Related Issue

Closes #5714

Changes

Fifteen commits, each building and passing: OCSF guest-sourced markers; privileged workload identity and default policy in isolation-interface/policy; sandbox-backend protocol extensions (privileged evidence validation, VM resource claims); the sandbox boundary's root-workload mode and nftables transparent network relay; supervisor privileged-mode validation and guest-reported exit events; the KubeVirt runtime mode in the Kubernetes driver (VM + fence + bootstrap Secret + PVC + supervisor topology); Helm RBAC for kubevirt.io and networking.k8s.io objects; the e2e acceptance demo and root-mode attack-surface probes; configuration and runtimes documentation. The sandbox's CPU and memory requests and limits map onto the VM (whole vCPUs for the guest, requests/limits on the virt-launcher container) with pod-path resource admission, and GPU requests are rejected (passthrough is not supported for VM sandboxes yet).

Release Notes

  • The Kubernetes compute driver accepts runtime = "kubevirt" to run sandboxes as KubeVirt virtual machines (experimental).

Testing

  • Appropriate checks pass: cargo fmt --all --check, clippy (-D warnings equivalent clean), cargo deny check, license/markdown/helm lint, helm unittests (486 + 7), docs build.
  • Unit tests pass: full workspace cargo test including openshell-server --features test-support.
  • E2E tests: not ported to the e2e/rust harness (see open questions). The PoC's scripted gates were verified live on KubeVirt v1.9.1 / k3d: the 10-step acceptance demo (e2e/kubevirt/demo.sh --demo) and the root-mode attack-surface probes (demo.sh --probes) both pass, including fence verification, stop/start identity, clean teardown, and the config-disk eject limit.

Checklist

  • Commits follow Conventional Commits and are DCO signed off.
  • Architecture/user docs updated in the same PR (configuration.mdx, runtimes.mdx).

…lders

Activity builders (process, network, HTTP, API activity, finding, config)
gain an unmapped-field map so callers can attach OCSF extensions, and the
process builder gains guest_sourced(): a data_origin=guest marking for
events whose payload was reported by an in-guest boundary rather than a
supervisor-verified observation. The shorthand formatter renders the
marker at a fixed position and escapes guest-controlled fields (process
names, command lines) with bracket escaping, so a forged name cannot
render a marker or start a forged line. Process events now require an
actor_process, matching the typestate builder.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
…leged VMs

ResolvedWorkloadIdentity::new refuses UID/GID zero, which is the right
invariant for container and sandboxed-VM workloads. The privileged
root-workload VM class deliberately runs the workload as root — kernel
separation is the boundary — so new_privileged() constructs the identity
for that class. It is gated on the workload mode at the call site.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
restrictive_default_policy_privileged() keeps the restrictive network
posture of restrictive_default_policy() but omits the filesystem,
Landlock, and process-identity sections: in-guest enforcement is
meaningless against a root workload, so the default must not promise
constraints that cannot be enforced. Policies that author those sections
are rejected by supervisor validation in privileged mode.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
…ndaries

Provisioning evidence gains validate_privileged(): a root-workload
boundary honestly reports no seccomp/Landlock/no_new_privs evidence
because none are enforced in that mode — the security boundary is
outside the guest — while kernel release and architecture must still be
present. Resource-claim constants mark KubeVirt guest generations
(kubevirt.generation), the workload mode (kubevirt.workload_mode, with
root selecting the privileged mode), and is_vm_guest() gives VM guests
VM policy-identity semantics (run_as_user/group must be sandbox or the
resolved guest uid/gid). The audit path and runtime records carry the
launch shape a VM boundary reports.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
The in-guest boundary (openshell-sandbox) learns two VM-mode behaviours:

- Root-workload mode: the boundary runs the workload as root without
  seccomp-notify interception, Landlock, or process identity; the
  network broker is either constructed as designed-unavailable (no
  pending channels, confirm_healthy succeeds by design) or, with the
  nftables transparent relay, feeds relayed connections through the
  same Mediate accept/decision/bridge flow the seccomp broker uses.
  The loopback DNS relay gains an admit_unregistered mode for the
  relay path (the seccomp broker that registers workload DNS sockets
  is absent).

- network_relay.rs: the nftables transparent TCP relay for privileged
  guests — capture ruleset plus listeners that hand relayed streams to
  the broker, so the supervisor's proxy and policy DNS run unchanged.

Guest-reported process exits are emitted with the OCSF guest-sourced
marker when running privileged.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
The supervisor learns the privileged (root-workload) mode end to end:

- validate_policy_enforceable_in_privileged_mode() rejects policies
  that author filesystem/Landlock/process-identity sections when the
  boundary reported the privileged workload mode — the default must not
  promise in-guest constraints that cannot be enforced.
- Main-process and adopted-exec exits are boundary-reported; in
  privileged mode those OCSF events carry the guest-sourced marker
  (the boundary is untrusted after start).
- The SSH access plane and boundary-access startup thread the
  privileged mode through; the supervisor pool records the topology.
- Policy DNS gains the mapping answers the guest's loopback resolver
  relies on in VM mode.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
A new runtime mode for the Kubernetes compute driver: set
runtime = "kubevirt" and each sandbox becomes a KubeVirt
VirtualMachine instead of a pod. The sandbox API, gateway, and security
model are unchanged; only where the workload runs changes.

The driver creates the VM halted first (to capture its identity for
security claims), plus the companions: a bootstrap Secret delivered to
the guest via cloud-init (one-shot: the guest init reads it once and
removes the PCI function backing the disk), a per-sandbox workspace PVC
shared over virtio-fs, an egress fence (NetworkPolicy admitting only
the supervisor), and — in the default dedicated topology — one
supervisor Pod per sandbox. The VM is only marked Running once the
boundary's session connects; stop/start preserve the VM and fence
identities; the fence is reconciled when it drifts and verified (UID +
spec) on every start.

Two workload modes: sandboxed (workload uid 10001, full in-guest
enforcement) and root (workload uid 0, in-guest enforcement demoted by
design — the fence and the supervisor's policy evaluation are the
boundary; the guest is untrusted after the first workload runs). After
the first boot the VM spec pins the guest image by digest.

Trust model: nothing in-guest is a trust anchor. dm-verity is
deliberately absent (a root workload may rewrite the base disk); the
root disk is wiped on every stop/start (RAM overlay over a read-only
base); guest-reported data is marked data_origin=guest in OCSF logs.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
The gateway role gains the rules the kubevirt runtime needs:
virtualmachines/virtualmachineinstances (get, list, watch, create,
patch, delete) from kubevirt.io, and networkpolicies (get, list, watch,
create, patch, delete) from networking.k8s.io for the egress fence and
its reconciler. RBAC unit tests cover the added rules.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
The PoC's e2e harness for the kubevirt runtime mode, run against a
local k3d cluster with KubeVirt deployed (see e2e/kubevirt/README.md):

- build-guest-image.sh builds the guest containerDisk (Alpine rootfs +
  static musl supervisor + guest init + kernel, repacked initramfs).
- demo.sh starts a host gateway against the cluster; --demo runs
  run-demo.sh, the 10-step acceptance demo (create/Ready, exec,
  L4/L7 policy, bypass blocked, stop/start identity, fence
  verification, dead-runtime Error, zero leftovers); --probes runs
  run-probes.sh, the root-workload attack-surface probes (in-guest
  reset, fake qemu-guest-agent, virtiofsd privileged operations,
  launcher-netns listener audit, and the config-disk eject limit:
  a root PCI rescan re-enumerates the ejected disk — the eject is
  hygiene, not a control).
- demo-policy.yaml / demo-policy-root.yaml: the policies the demo and
  probes run under.
- supervisor-values.yaml: shared-supervisor topology values for the
  gateway-topology variant.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Describe the runtime = "kubevirt" selection, its kubevirt.* TOML table
(guest image, pull policy, cores, memory, workload mode), the
supervisor topology options, and the security posture the mode
guarantees (enforced outside the guest; the guest is untrusted after
the first workload runs in root mode).

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 18, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 18, 2026
…trol

Guest configuration — the nftables relay, the seccomp broker's
steering, the config-disk eject — is required for networking to
function and is never relied on for security, because the guest
grants root. Every security property is enforced outside the guest:
the NetworkPolicy fence (only the supervisor is reachable), the
supervisor's policy (decides what leaves), and the driver's checks
of the VM UID and fence spec.

Reword the four places that counted the relay as a protection: the
demo's egress-fencing item (blocked 'at both the boundary and the
pod network'), the trust-model list ('NetworkPolicy + the nftables
relay's fail-closed deny'), the outside-boundary reliance ('the
fence and the relay's deny'), and the relay module's UDP-drop
comment ('defence in depth matching the fence'). State the principle
explicitly in the trust-model section and drop a stale plan-document
reference from the relay module doc comment.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
…guest class

Every KubeVirt guest is what root mode was: the workload runs as root
and the guest — including the in-guest openshell-sandbox boundary — is
untrusted. The security boundary is everything outside the guest: the
NetworkPolicy fence, the supervisor's policy, and the driver's
Kubernetes checks. Guest configuration (the nftables relay, the seccomp
broker's steering, the config-disk eject) is required for networking to
function and is never relied on for security.

The mode is gone as a concept: KubeVirtWorkloadMode, the workload
config key, the kubevirt.workload_mode claim, and the sandboxed guest
launch path are removed. The guest class is keyed on the
kubevirt.generation claim (is_kubevirt_guest): the driver renders a
uid-0 identity for every guest, the guest init always launches the
privileged boundary, and libkrun VM guests keep the trusted-boundary
model.

Supervisor gates that previously keyed on the privileged mode now key
on the untrusted-guest class, and none may trust the boundary:

- image policy discovery is disabled for untrusted guests (a policy
  planted at /etc/openshell/policy.yaml in the guest can never become
  the enforcement baseline or be synced to the gateway; the baseline is
  the gateway's policy or the restrictive default), with a regression
  test
- per-binary/ancestor/cmdline rules are rejected and the OPA engine is
  built with require_binary_identity=false
- provider files and environment are refused supervisor-side
- backend_audit validation requires only kernel release and
  architecture and is never treated as proof of enforcement
- every guest-reported event carries the OCSF guest-sourced mark
  (data_origin=guest), including network allow/deny events built from
  boundary-supplied identity; the mark is process-wide under the
  dedicated supervisor an untrusted guest requires

The kubevirt runtime now requires the dedicated supervisor topology
(a shared supervisor would let a parser bug reached by a hostile root
guest cross tenants).

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
The e2e harness and docs drop the sandboxed lane and the gateway-shared
supervisor topology (invalid for the kubevirt runtime):

- demo.sh renders one gateway config (dedicated supervisor, root
  workload); the OPENSHELL_KVPOC_WORKLOAD and OPENSHELL_KVPOC_TOPOLOGY
  switches and supervisor-values.yaml are removed
- demo-policy.yaml is now the root-class acceptance policy (network
  rules only); the sandboxed variant is deleted
- run-demo.sh's checks are single-path: exec as root, L4/L7 through the
  transparent relay, the fail-closed fence test after root deletes the
  relay ruleset, and the launcher-netns fence proof
- guest-init.sh always launches the privileged boundary and rejects any
  non-zero workload identity
- the README documents the single trust model, the rule that guest
  configuration is functional routing and never a security control,
  and that the boundary's confirmation claims are a consistency check
  anchored by the driver's Kubernetes reads; the transcript is refreshed
  from a live single-mode run
- the gateway configuration docs describe the dedicated-only topology
  and the untrusted-guest model

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
…ed objects

H5 (review finding): lifecycle checks fenced lookups but not the writes
that followed. Start checked the VM UID once at lookup and then patched
by name; create's read-modify-write helper re-read without rechecking
the UID; stop and the watch's auto-halt patched by name; delete used a
UID precondition on the VM but deleted companions by name. A VM
replaced under the same name between the check and the write was
booted, halted, or stripped by the wrong operation.

Every mutation is now pinned to the object that was checked:

- every VM patch (create's release, start's run-strategy patches,
  stop's halt, the watch's auto-halt) re-reads the VM, refuses a UID
  change (the checked object is gone — a Precondition abort, never a
  name-only fallback), and carries the expected metadata.uid and the
  resourceVersion from the same read, so the API server rejects a
  replaced or concurrently modified object; a resourceVersion conflict
  is retried from a fresh read
- the guest bootstrap Secret is read before start rewrites it: its
  ownerReference UID must be the checked VM, and the patch carries the
  Secret's UID and resourceVersion (a foreign object under the derived
  name is refused, not rewritten)
- the fence is re-verified immediately before the patch that boots the
  VM, so a fence widened or replaced during start's provisioning fails
  the start instead of releasing the guest
- companion deletes (bootstrap Secret, workspace PVC, egress fence)
  read the object first, require the VM's ownerReference UID, and carry
  the read UID as a DeleteOptions precondition; the supervisor Secret
  replace path and the stale-generation Secret cleanup delete with the
  read UID as a precondition

The fake API used by the unit tests now enforces the same contract as
the real API server: a patch whose pinned metadata.uid or
metadata.resourceVersion does not match the stored object is a 409
Conflict, and a delete whose DeleteOptions.preconditions.uid does not
match is refused. Four regression tests cover the finding: a VM
replaced between check and start is refused with no VM write landing;
a foreign-owned bootstrap Secret is refused and never rewritten; a
fence widened before release fails the start before the boot patch;
and a resourceVersion conflict retries from a fresh read and completes.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
H6 (review finding): the KubeVirt path never read the sandbox's
resource_requirements — every VM got the global kubevirt.cpu_cores and
kubevirt.memory with no limits, create skipped resource admission, and
capabilities() advertised GPU selection.

The sandbox's typed CPU and memory requests and limits now map onto the
VM (validated at admission, before anything is created):

- vCPU cores: the effective CPU (limit, else request, else the global
  kubevirt.cpu_cores) rounded up to whole vCPUs — a VM's vCPU count is
  integral; a fractional limit keeps its exact value on the
  virt-launcher container, so the guest sees the rounded count while
  the launcher is throttled to the fraction
- guest memory: the effective memory (limit, else request, else the
  global kubevirt.memory) as domain.memory.guest
- virt-launcher requests/limits: the sandbox's requests and limits
  verbatim on domain.resources (mirroring the pod path's rule that an
  omitted request defaults to the limit), so the launcher container is
  enforceable and KubeVirt's overhead lands on top
- validation: a limit below its request, memory below the safe guest
  minimum (256Mi), or an unparsable quantity is an InvalidArgument at
  admission

KubeVirt mode now runs the same resource admission as the pod path
(driver-config policy, workspace namespace rules, referenced-resource
admission) before anything is created. GPU requests are rejected at
admission with a clear message (GPU passthrough is not supported for VM
sandboxes yet; use the pod runtime), and capabilities() reports GPU
selection as unsupported in KubeVirt mode while the pod runtime keeps
it. CPU and memory limits are reported as supported in both modes.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: KubeVirt runtime mode for the Kubernetes compute driver (VM-backed sandboxes)

1 participant