Skip to content

test(workloads): run real tools in sandboxes, starting with make and LiteLLM #5726

Open

Description

@matthewgrossman

User Story

I maintain OpenShell's E2E tests. Users keep reporting that real tools fail in sandboxes, such as Chromium, bubblewrap and Bazel, and we have no tests that run those tools. I want a suite where each test runs a real tool doing a realistic task. Then these bugs show up in our tests before users find them. Anyone building a new compute driver or isolation backend could also run the suite against their own gateway to check its edges.

Description

Many compatibility bugs come from users running real tools. They fall into two groups.

The tool fails to start, and one command reproduces it:

The tool starts, but fails while doing real work:

These tests don't fit conformance. Conformance checks a short, documented contract that every driver must meet, and TESTING.md says not to test undocumented behavior there. "Bazel works in a sandbox" is not part of that contract, and some workloads will only work on some drivers.

Proposal

A new test family, tests/suites/workloads, in its own workspace. Like conformance, it builds a program, openshell-workloads, with testkit::main(). So it has the same list, ready and run --junit commands, and anyone can run it against an existing gateway.

Each workload test has two steps:

  1. Start the tool, and check that it's ready. If the tool has its own protections, such as Chromium's sandbox or bubblewrap, start it with them on and check that they are on. A test must not pass because the tool quietly fell back to a weaker mode, such as Chromium with --no-sandbox. This step covers the first group of bugs.
  2. Do the task the bug reports describe, at the scale they describe, and check each step. For example, a build finishes, a request comes back, or 4 test processes connect. Then a failure says which part broke. This step covers the second group.

Tests don't need the internet by default:

  • The image is pinned by digest and has the tool and its inputs built in.
  • Sites and APIs the task talks to are served from tests/support/host-server.
  • A test that needs the real internet says so in its name, so a run can leave it out.

Workload tests are slower and use larger images than conformance, so they don't run on every pull request. They run on demand, and on a schedule if e2e-suites.yml can do that.

The only way to skip a test today is #[ignore = "<issue link>"]. A workload that no driver supports yet uses that for now. Expected results for each environment (passes, known failure with an issue link, or not applicable) are a follow-up. Conformance needs the same thing.

First workloads

The first two are simple on purpose. They prove the suite itself works (the program, the image, the runner and the CI rows) before the harder tools from the bug reports.

1. Compile C with make (make_builds_a_c_program)

  • The image has gcc, make, and a small C project: a few source files and a Makefile.
  • The test runs make -j4 in the sandbox, runs the program it built, and checks its output. Then it runs make -q to check that nothing needs rebuilding.
  • No network. It should pass on every driver today.

2. Run LiteLLM as a local proxy (litellm_proxies_a_chat_request)

To settle while building it

  • The image. One image for both tests, built like e2e/python/Dockerfile.workload: the same base and sandbox user, plus build-essential and a pinned litellm[proxy]. The testkit base policy uses run_as_user: sandbox, and Podman rejects a named user the image doesn't have. The environments that run the suite should load the image in advance, as they do openshell/e2e-python:dev. See test(testkit): run conformance against any gateway with one pullable test image #5352.
  • Where the C project lives. Built into the image, or kept in the suite and copied into the sandbox by the test. Keeping it in the suite means changing it doesn't need an image rebuild.
  • LiteLLM going online. LiteLLM downloads a model price list from GitHub at startup unless LITELLM_LOCAL_MODEL_COST_MAP=True is set. Check for other startup calls, such as telemetry, and turn them off so the test doesn't need the internet.
  • How LiteLLM reaches the proxy. On drivers without transparent TCP (Kubernetes, VM), LiteLLM's HTTP client has to use the sandbox's proxy settings. The client's call to LiteLLM on localhost must not go through the proxy.
  • Which binary the policy names. LiteLLM runs as Python, so the policy allows the fake API's host and port for the Python binary.
  • Which rows run it. Start with ubuntu-docker-rootful and fedora-podman-rootless. Add k3s once the image loads there.

Context

Definition of Done

  • tests/suites/workloads exists as its own workspace, builds openshell-workloads, and is in ALIGNED_WORKSPACES.
  • Its README says what belongs there rather than in conformance, describes the two-step shape, and explains how to add a workload. TESTING.md's rules for where a test goes mention it.
  • The workloads image is pinned by digest, and the environments that run the suite load it before tests start.
  • make_builds_a_c_program and litellm_proxies_a_chat_request pass on the Docker and Podman rows without internet access.
  • The suite's rows in .github/workflows/e2e-suites.yml don't run on every pull request.
  • Follow-up issues cover the next workloads and expected results for each environment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions