Repository navigation
Description
User Story
I maintain OpenShell's E2E tests. Users keep reporting that real tools fail in sandboxes, such as Chromium, bubblewrap and Bazel, and we have no tests that run those tools. I want a suite where each test runs a real tool doing a realistic task. Then these bugs show up in our tests before users find them. Anyone building a new compute driver or isolation backend could also run the suite against their own gateway to check its edges.
Description
Many compatibility bugs come from users running real tools. They fall into two groups.
The tool fails to start, and one command reproduces it:
- bug(sandbox): namespace hardening prevents Chromium's guest sandbox from starting #4098: Chromium can't start with its own sandbox turned on. Chromium's sandbox needs a user namespace, and the workload seccomp filter blocks creating one. Users fall back to
--no-sandboxor run the browser outside OpenShell. - bug(sandbox): Claude Science cannot start inside a sandbox because its mandatory bubblewrap sandbox needs user namespaces #5370: Claude Science can't start, because its bubblewrap sandbox needs a user namespace. Comments report the same for Claude Code with
CLAUDE_CODE_SUBPROCESS_ENV_SCRUB=1, and on bug(sandbox): namespace hardening prevents Chromium's guest sandbox from starting #4098 for Codex. - bug(sandbox): restrictive seccomp filter stacking remains blocked in v0.1.8 (follow-up to #4097) #5467: workloads can't add their own seccomp filters. Found by reading the code; not yet reproduced with a real tool.
The tool starts, but fails while doing real work:
- bug(network): IPv4-mapped loopback connects (::ffff:127.0.0.1) are policy-evaluated and denied, while native 127.0.0.1 and ::1 bypass policy #5546: Bazel can't reach its own local server, because Java dials
::ffff:127.0.0.1. - feat: provide Java trust integration for sandbox TLS interception #2819 (closed): Bazel couldn't download dependencies, because Java didn't trust the sandbox's TLS certificate.
- Configurable sandbox concurrent-connection limit (MAX_NETWORK_STREAMS = 96), and a denial reason that names it #5486: Gradle with 4 parallel test processes hits the 96-connection limit.
- Sandbox supervisor keeps listening on loopback ports after the application closes them #5300: a tool that finds a free port by opening and closing it then sees every port as busy.
- feat(providers): add a Browserless example provider profile #5614: after 0.1.8, Playwright couldn't connect to a hosted browser until each WebSocket path had its own endpoint.
These tests don't fit conformance. Conformance checks a short, documented contract that every driver must meet, and TESTING.md says not to test undocumented behavior there. "Bazel works in a sandbox" is not part of that contract, and some workloads will only work on some drivers.
Proposal
A new test family, tests/suites/workloads, in its own workspace. Like conformance, it builds a program, openshell-workloads, with testkit::main(). So it has the same list, ready and run --junit commands, and anyone can run it against an existing gateway.
Each workload test has two steps:
- Start the tool, and check that it's ready. If the tool has its own protections, such as Chromium's sandbox or bubblewrap, start it with them on and check that they are on. A test must not pass because the tool quietly fell back to a weaker mode, such as Chromium with
--no-sandbox. This step covers the first group of bugs. - Do the task the bug reports describe, at the scale they describe, and check each step. For example, a build finishes, a request comes back, or 4 test processes connect. Then a failure says which part broke. This step covers the second group.
Tests don't need the internet by default:
- The image is pinned by digest and has the tool and its inputs built in.
- Sites and APIs the task talks to are served from
tests/support/host-server. - A test that needs the real internet says so in its name, so a run can leave it out.
Workload tests are slower and use larger images than conformance, so they don't run on every pull request. They run on demand, and on a schedule if e2e-suites.yml can do that.
The only way to skip a test today is #[ignore = "<issue link>"]. A workload that no driver supports yet uses that for now. Expected results for each environment (passes, known failure with an issue link, or not applicable) are a follow-up. Conformance needs the same thing.
First workloads
The first two are simple on purpose. They prove the suite itself works (the program, the image, the runner and the CI rows) before the harder tools from the bug reports.
1. Compile C with make (make_builds_a_c_program)
- The image has gcc, make, and a small C project: a few source files and a Makefile.
- The test runs
make -j4in the sandbox, runs the program it built, and checks its output. Then it runsmake -qto check that nothing needs rebuilding. - No network. It should pass on every driver today.
2. Run LiteLLM as a local proxy (litellm_proxies_a_chat_request)
host-serverserves a fake OpenAI-style chat API. A secondhost-serveron another port stands in for an API the policy doesn't allow.- The test starts LiteLLM inside the sandbox, listening on localhost, with two models: one pointing at the allowed fake API, one at the blocked one.
- A Python client in the same sandbox calls LiteLLM (
Sandbox::run_pythonandExecOutput::last_json, from test(testkit): share sandbox client helpers across testkit suites #5691). The test checks that:- the request to the allowed model comes back with the fake API's answer;
- the fake API received the request, with the model and message the client sent;
- the request to the blocked model fails, and the second server received nothing.
- This covers a tool talking to its own local server (as Bazel does in bug(network): IPv4-mapped loopback connects (::ffff:127.0.0.1) are policy-evaluated and denied, while native 127.0.0.1 and ::1 bypass policy #5546) and outbound traffic through policy.
To settle while building it
- The image. One image for both tests, built like
e2e/python/Dockerfile.workload: the same base andsandboxuser, plusbuild-essentialand a pinnedlitellm[proxy]. The testkit base policy usesrun_as_user: sandbox, and Podman rejects a named user the image doesn't have. The environments that run the suite should load the image in advance, as they doopenshell/e2e-python:dev. See test(testkit): run conformance against any gateway with one pullable test image #5352. - Where the C project lives. Built into the image, or kept in the suite and copied into the sandbox by the test. Keeping it in the suite means changing it doesn't need an image rebuild.
- LiteLLM going online. LiteLLM downloads a model price list from GitHub at startup unless
LITELLM_LOCAL_MODEL_COST_MAP=Trueis set. Check for other startup calls, such as telemetry, and turn them off so the test doesn't need the internet. - How LiteLLM reaches the proxy. On drivers without transparent TCP (Kubernetes, VM), LiteLLM's HTTP client has to use the sandbox's proxy settings. The client's call to LiteLLM on localhost must not go through the proxy.
- Which binary the policy names. LiteLLM runs as Python, so the policy allows the fake API's host and port for the Python binary.
- Which rows run it. Start with
ubuntu-docker-rootfulandfedora-podman-rootless. Add k3s once the image loads there.
Context
- Tracked under test: retire the old e2e test system onto testkit #5281 because it uses the same testkit and runner, but retiring
e2e/rustdoesn't depend on it. - Prior art:
- Kubernetes ships all of its E2E tests in one program. Only a reviewed subset is tagged
[Conformance], and runners pick the rest by name. - gVisor runs real language test suites and popular images in its sandbox. It keeps a list of known failures, each with a reason, and runs the excluded tests outside the sandbox to separate broken tests from sandbox gaps.
tests/suites/features/cdialready reads expected results that its environment provides.
- Kubernetes ships all of its E2E tests in one program. Only a reviewed subset is tagged
- Running the suite against other gateways depends on build(release): publish the existing conformance runner #5371 (publish the runner), test(testkit): run conformance against any gateway with one pullable test image #5352 (images any gateway can pull) and test(conformance): handle gateways whose sandboxes can't reach the test machine #5409 (sandboxes that can't reach the test host, which
host-serverneeds). - test(e2e): add cross-driver image-to-sandbox conformance #2422 (closed) proposed recording an expected result for each driver, which the follow-up would provide.
- Next workloads, from the bugs above, each in its own issue:
- a small Bazel build (bug(network): IPv4-mapped loopback connects (::ffff:127.0.0.1) are policy-evaluated and denied, while native 127.0.0.1 and ::1 bypass policy #5546, feat: provide Java trust integration for sandbox TLS interception #2819);
- bubblewrap with Claude Code (bug(sandbox): Claude Science cannot start inside a sandbox because its mandatory bubblewrap sandbox needs user namespaces #5370);
- Gradle with parallel test processes (Configurable sandbox concurrent-connection limit (MAX_NETWORK_STREAMS = 96), and a denial reason that names it #5486);
- Chromium (bug(sandbox): namespace hardening prevents Chromium's guest sandbox from starting #4098): one test that starts it with its sandbox on, ignored until bug(sandbox): namespace hardening prevents Chromium's guest sandbox from starting #4098 is fixed, and one that loads a page through policy with
--no-sandbox. Its image is about 2 GB and runs aspwuser, so it needs its own image work. - thread pinning in a parallel build (bug: some calls fail or are interrupted inside the sandbox but work natively (rt_sigqueueinfo, self-directed kill, sched_setaffinity, prlimit64, nice) #5185).
Definition of Done
-
tests/suites/workloadsexists as its own workspace, buildsopenshell-workloads, and is inALIGNED_WORKSPACES. - Its README says what belongs there rather than in conformance, describes the two-step shape, and explains how to add a workload.
TESTING.md's rules for where a test goes mention it. - The workloads image is pinned by digest, and the environments that run the suite load it before tests start.
-
make_builds_a_c_programandlitellm_proxies_a_chat_requestpass on the Docker and Podman rows without internet access. - The suite's rows in
.github/workflows/e2e-suites.ymldon't run on every pull request. - Follow-up issues cover the next workloads and expected results for each environment.
Activity