Skip to content

test(conformance): handle gateways whose sandboxes can't reach the test machine #5409

Open

Description

@matthewgrossman

User Story

I run openshell-conformance, or the Kubernetes E2E with OPENSHELL_E2E_KUBERNETES_FEATURES=e2e,e2e-kubernetes, against a remote Kubernetes or OpenShift gateway. Its sandboxes can't reach the machine running the tests. I want conformance to tell me that this environment requirement isn't met, instead of failing tests that say nothing about whether my gateway meets the contract.

Description

The credential_gating conformance tests (#5318) start servers on the machine that runs the tests (tests/support/host-server) and reach them from sandboxes as host.openshell.internal. When sandboxes can't reach that machine, for example on a remote cluster, these tests fail.

Before #5318, the same scenarios lived in e2e/rust behind the e2e-host-gateway feature, so OPENSHELL_E2E_KUBERNETES_FEATURES=e2e,e2e-kubernetes excluded them on a remote cluster. openshell-conformance has no feature selection, so it now runs them unconditionally. The compatibility review on #5318 flagged this, and the review accepted it as a tracked follow-up.

Options:

  1. Keep an opt-in. Gate the host-dependent conformance tests behind a declared environment capability, such as an env var or a openshell-conformance flag, as e2e-host-gateway did.
  2. Move them to a feature suite. Keep conformance free of tests that need the test machine, and run credential_gating from tests/suites/features/ on lanes that provide reachability.
  3. Check reachability up front. Before the host-dependent tests run, start a host server and have a sandbox connect to it through host.openshell.internal. If that fails, skip those tests with a clear message that names the missing requirement.

Context

  • tests/suites/conformance/src/credential_gating.rs, tests/support/host-server/.
  • TESTING.md (remote cluster section) and tests/suites/conformance/README.md state the requirement.
  • e2e/rust/e2e-kubernetes.sh: OPENSHELL_E2E_KUBERNETES_FEATURES, e2e-host-gateway.
  • tests/suites/features/host-gateway checks the same reachability on its own.
  • Other tests ported later (test(e2e): decide which e2e/rust tests to port and in what order #5297) that use the test machine as an upstream will have the same requirement.

Definition of Done

  • One option is chosen, with the reasoning recorded here.
  • openshell-conformance against a gateway whose sandboxes can't reach the test machine doesn't report the host-dependent tests as failures of the gateway, and its output names the missing requirement.
  • Lanes that provide reachability still run the host-dependent tests.
  • TESTING.md and tests/suites/conformance/README.md describe how to run conformance against a remote cluster.

Activity

  1. added
    topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.
    on Nov 9, 2026
  2. elezar commented on Nov 10, 2026

    @elezar
    Member

    Suggested framing for this issue: the question is where the host-dependent credential-gating tests belong, not only how conformance behaves on remote clusters.

    TESTING.md says conformance tests run on every compute driver, and that behaviour only some drivers have belongs in tests/suites/features. Four of the six credential-gating tests make a sandbox connect to a server on the test machine through host.openshell.internal: profile_rest_endpoint_keeps_body_placeholder, profile_websocket_endpoint_denies_binary_frames, policy_binding_rest_body_keeps_placeholder_without_rewrite and policy_binding_rest_body_rewrite_sends_secret_upstream. That is the behaviour features/host-gateway already covers as a feature.

    Proposal:

    • Move those four into a features suite next to host-gateway, gated and skipped the same way, so a gateway on a remote cluster passes conformance again.
    • Keep policy_binding_requires_inspection_or_opt_in and policy_binding_updates_provider_environment_live in conformance. Neither connects to the host server. It is started only for a free port. Consider a fixed port and the base image there, which would also drop the Python image dependency from conformance.
    • The helpers they share (create_sandbox, sandbox_with, run_python, install_gating_provider, the recorders) live in the conformance crate, so the move needs them in testkit or a support crate.
    • Check check-conformance-program.sh and the JUnit expectations for hard-coded test lists, and drop the README sentence saying the program rules out remote Kubernetes clusters.
  3. elezar commented on Nov 10, 2026

    @elezar
    Member

    One more point for the split: policy_binding_requires_inspection_or_opt_in, which stays in conformance, asserts on the gateway's error wording ("credentialed endpoint", "L4-only", and "tls:" with "skip"). The wording checks come from the #5264 tests. TESTING.md says conformance tests assert on the public contract and not on messages.

    If that wording is in the published docs or the proto/ comments, the assertions are fine. If it isn't, assert only the FailedPrecondition code, which the test already checks, or document the wording first.

  4. added
    area:driversCompute and secret drivers, host OS/architecture support, backend isolation, and lifecycle.
    on Nov 15, 2026
  5. matthewgrossman commented on Nov 18, 2026

    @matthewgrossman
    MemberAuthor

    I looked at where the host-dependent tests should go and how we could drop the dependency later. Proposal: for now, conformance assumes sandboxes can reach the test machine as host.openshell.internal, and this issue does option 3.

    Why keep the assumption for now

    • Docker, Podman and MicroVM provide host.openshell.internal by default. On Kubernetes it's opt-in through server.hostGatewayIP (empty by default); tmachine's k3s installer sets it.
    • The actual requirement is stronger than the alias: it has to point at the machine running the tests. That holds locally and in tmachine (one VM), not for a remote cluster.
    • Moving the host-dependent tests to a features suite (Evan's proposal above) was a good fit for the four credential_gating tests, but the network port (test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665) adds about 16 more that need an upstream: transparent egress, L7 REST/GraphQL/JSON-RPC, MCP profiles. Moving them all would empty most of the network side of conformance.

    So the work here:

    1. Check reachability up front (option 3). If a sandbox can't reach a host server, the host-dependent tests report "environment requirement not met: sandboxes can't reach the test machine as host.openshell.internal", not gateway failures.
    2. State the requirement in tests/suites/conformance/README.md and TESTING.md, including server.hostGatewayIP on Kubernetes.
    3. Evan's smaller points still apply: policy_binding_requires_inspection_or_opt_in and policy_binding_updates_provider_environment_live don't use the host server, and the error-wording assertions should only check documented wording.

    Ways to drop the dependency later

    1. Many tests don't need a server. Plenty of them ask what the supervisor decided, not what arrived. The supervisor reports that itself: the policy 403 and the OCSF events in Sandbox::logs (which also record allowed requests). Those tests can target a name that never resolves (*.invalid) or a documentation address (192.0.2.1), so nothing leaves the sandbox. That fits L4 denials and probably L7 allow/deny. To check: whether the supervisor makes L7 decisions before it connects upstream in transparent mode. Tests about what arrived upstream (credential placeholders and rewrites, middleware redaction, WebSocket frames, pipelining, reloads closing open connections, tls: skip) still need a server.

    2. Server in another sandbox, through service exposure. It would reuse OpenShell's own primitives and need nothing from the host, but it doesn't work today:

    • local gateways serve *.openshell.localhost (127.0.0.1) and accept plaintext only from loopback;
    • remote gateways need operator ingress and gateway authentication;
    • the gateway strips Authorization and re-frames requests, which breaks header, pipelining and raw-byte checks.

    It needs sandbox-to-sandbox support (#1049).

    3. Server in the same sandbox. No enforcement today. Workload sockets are pinned to loopback, and the broker connects loopback destinations locally, without policy (the schema docs say so). A self-hairpin would fix that: a reserved name the broker routes through the supervisor, which applies the normal egress policy and then delivers the connection to the sandbox's own loopback. The pieces exist: the broker already relays the Google metadata loopback address through the supervisor, and openshell forward and service exposure already reach a sandbox's loopback. Tests would be self-contained and driver-agnostic.

    The cons: it's a product feature, and it needs security review, because a policy that injects a real credential toward the sandbox itself hands the secret to the workload. It's also the simplest case of #1049, so it may be worth raising there.

    4. A server the environment provides. A stateless echo server deployed by the environment (a Service in the cluster, a container next to the gateway, or the test machine), with its address in an OPENSHELL_TEST_* variable. It replies with the request it received, so the test reads the result through exec and never talks to the server. This works on remote clusters without product changes, but conformance then depends on environment setup.

    Direction

    • New tests: assert supervisor decisions without a server where they can.
    • Tests about what arrived upstream: stay on host-server, behind the reachability check.
    • Revisit 3 (or 2) if Sandbox to Sandbox Communications #1049 moves, and 4 if conformance against remote clusters becomes a priority.
  6. matthewgrossman commented on Nov 18, 2026

    @matthewgrossman
    MemberAuthor

    A correction to option 1 above. I said the OCSF events in Sandbox::logs record the supervisor's decisions in a form tests can assert on. That's weaker than I made it sound: the supervisor pushes OCSF events to the gateway log stream as shorthand text with an empty fields map (crates/openshell-supervisor-process/src/log_push.rs). The docs describe the shorthand as a format for scanning, not a stable one, and the push channel can drop lines under load.

    The better signal for decision-only tests is the proxy's error body. docs/observability/logging.mdx documents a machine-readable error code (policy_denied, ssrf_denied, unsupported_l7_protocol, middleware_denied, upstream_unreachable, ...) plus method, path and rule_missing for L7 REST denials. So:

    • Decision-only tests should assert the status and the documented error code.
    • Log assertions should match finding names (for example openshell.middleware.traffic_uninspectable), not whole shorthand lines. Checking that secrets never appear in logs is still worth doing.
    • If structured log assertions become common, the product fix is to push the OCSF attributes in SandboxLogLine.fields, which the proto comment already describes ("e.g. dst_host, action").

    Using 502 upstream_unreachable to tell "allowed" from "denied" without a server is possible, but it only proves the policy didn't block the request, not that the request was forwarded intact. It also depends on the open question above about transparent-mode L7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:driversCompute and secret drivers, host OS/architecture support, backend isolation, and lifecycle.topic:testingUnit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions