Repository navigation
Description
User Story
I run openshell-conformance, or the Kubernetes E2E with OPENSHELL_E2E_KUBERNETES_FEATURES=e2e,e2e-kubernetes, against a remote Kubernetes or OpenShift gateway. Its sandboxes can't reach the machine running the tests. I want conformance to tell me that this environment requirement isn't met, instead of failing tests that say nothing about whether my gateway meets the contract.
Description
The credential_gating conformance tests (#5318) start servers on the machine that runs the tests (tests/support/host-server) and reach them from sandboxes as host.openshell.internal. When sandboxes can't reach that machine, for example on a remote cluster, these tests fail.
Before #5318, the same scenarios lived in e2e/rust behind the e2e-host-gateway feature, so OPENSHELL_E2E_KUBERNETES_FEATURES=e2e,e2e-kubernetes excluded them on a remote cluster. openshell-conformance has no feature selection, so it now runs them unconditionally. The compatibility review on #5318 flagged this, and the review accepted it as a tracked follow-up.
Options:
- Keep an opt-in. Gate the host-dependent conformance tests behind a declared environment capability, such as an env var or a
openshell-conformanceflag, ase2e-host-gatewaydid. - Move them to a feature suite. Keep conformance free of tests that need the test machine, and run
credential_gatingfromtests/suites/features/on lanes that provide reachability. - Check reachability up front. Before the host-dependent tests run, start a host server and have a sandbox connect to it through
host.openshell.internal. If that fails, skip those tests with a clear message that names the missing requirement.
Context
tests/suites/conformance/src/credential_gating.rs,tests/support/host-server/.TESTING.md(remote cluster section) andtests/suites/conformance/README.mdstate the requirement.e2e/rust/e2e-kubernetes.sh:OPENSHELL_E2E_KUBERNETES_FEATURES,e2e-host-gateway.tests/suites/features/host-gatewaychecks the same reachability on its own.- Other tests ported later (test(e2e): decide which e2e/rust tests to port and in what order #5297) that use the test machine as an upstream will have the same requirement.
Definition of Done
- One option is chosen, with the reasoning recorded here.
-
openshell-conformanceagainst a gateway whose sandboxes can't reach the test machine doesn't report the host-dependent tests as failures of the gateway, and its output names the missing requirement. - Lanes that provide reachability still run the host-dependent tests.
-
TESTING.mdandtests/suites/conformance/README.mddescribe how to run conformance against a remote cluster.
Activity
elezar commentedon Nov 10, 2026
Suggested framing for this issue: the question is where the host-dependent credential-gating tests belong, not only how conformance behaves on remote clusters.
TESTING.mdsays conformance tests run on every compute driver, and that behaviour only some drivers have belongs intests/suites/features. Four of the six credential-gating tests make a sandbox connect to a server on the test machine throughhost.openshell.internal:profile_rest_endpoint_keeps_body_placeholder,profile_websocket_endpoint_denies_binary_frames,policy_binding_rest_body_keeps_placeholder_without_rewriteandpolicy_binding_rest_body_rewrite_sends_secret_upstream. That is the behaviourfeatures/host-gatewayalready covers as a feature.Proposal:
host-gateway, gated and skipped the same way, so a gateway on a remote cluster passes conformance again.policy_binding_requires_inspection_or_opt_inandpolicy_binding_updates_provider_environment_livein conformance. Neither connects to the host server. It is started only for a free port. Consider a fixed port and the base image there, which would also drop the Python image dependency from conformance.create_sandbox,sandbox_with,run_python,install_gating_provider, the recorders) live in the conformance crate, so the move needs them intestkitor a support crate.check-conformance-program.shand the JUnit expectations for hard-coded test lists, and drop the README sentence saying the program rules out remote Kubernetes clusters.elezar commentedon Nov 10, 2026
One more point for the split:
policy_binding_requires_inspection_or_opt_in, which stays in conformance, asserts on the gateway's error wording ("credentialed endpoint", "L4-only", and "tls:" with "skip"). The wording checks come from the #5264 tests.TESTING.mdsays conformance tests assert on the public contract and not on messages.If that wording is in the published docs or the
proto/comments, the assertions are fine. If it isn't, assert only theFailedPreconditioncode, which the test already checks, or document the wording first.matthewgrossman commentedon Nov 18, 2026
I looked at where the host-dependent tests should go and how we could drop the dependency later. Proposal: for now, conformance assumes sandboxes can reach the test machine as
host.openshell.internal, and this issue does option 3.Why keep the assumption for now
host.openshell.internalby default. On Kubernetes it's opt-in throughserver.hostGatewayIP(empty by default); tmachine's k3s installer sets it.credential_gatingtests, but the network port (test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665) adds about 16 more that need an upstream: transparent egress, L7 REST/GraphQL/JSON-RPC, MCP profiles. Moving them all would empty most of the network side of conformance.So the work here:
host.openshell.internal", not gateway failures.tests/suites/conformance/README.mdandTESTING.md, includingserver.hostGatewayIPon Kubernetes.policy_binding_requires_inspection_or_opt_inandpolicy_binding_updates_provider_environment_livedon't use the host server, and the error-wording assertions should only check documented wording.Ways to drop the dependency later
1. Many tests don't need a server. Plenty of them ask what the supervisor decided, not what arrived. The supervisor reports that itself: the policy 403 and the OCSF events in
Sandbox::logs(which also record allowed requests). Those tests can target a name that never resolves (*.invalid) or a documentation address (192.0.2.1), so nothing leaves the sandbox. That fits L4 denials and probably L7 allow/deny. To check: whether the supervisor makes L7 decisions before it connects upstream in transparent mode. Tests about what arrived upstream (credential placeholders and rewrites, middleware redaction, WebSocket frames, pipelining, reloads closing open connections,tls: skip) still need a server.2. Server in another sandbox, through service exposure. It would reuse OpenShell's own primitives and need nothing from the host, but it doesn't work today:
*.openshell.localhost(127.0.0.1) and accept plaintext only from loopback;Authorizationand re-frames requests, which breaks header, pipelining and raw-byte checks.It needs sandbox-to-sandbox support (#1049).
3. Server in the same sandbox. No enforcement today. Workload sockets are pinned to loopback, and the broker connects loopback destinations locally, without policy (the schema docs say so). A self-hairpin would fix that: a reserved name the broker routes through the supervisor, which applies the normal egress policy and then delivers the connection to the sandbox's own loopback. The pieces exist: the broker already relays the Google metadata loopback address through the supervisor, and
openshell forwardand service exposure already reach a sandbox's loopback. Tests would be self-contained and driver-agnostic.The cons: it's a product feature, and it needs security review, because a policy that injects a real credential toward the sandbox itself hands the secret to the workload. It's also the simplest case of #1049, so it may be worth raising there.
4. A server the environment provides. A stateless echo server deployed by the environment (a Service in the cluster, a container next to the gateway, or the test machine), with its address in an
OPENSHELL_TEST_*variable. It replies with the request it received, so the test reads the result throughexecand never talks to the server. This works on remote clusters without product changes, but conformance then depends on environment setup.Direction
host-server, behind the reachability check.matthewgrossman commentedon Nov 18, 2026
A correction to option 1 above. I said the OCSF events in
Sandbox::logsrecord the supervisor's decisions in a form tests can assert on. That's weaker than I made it sound: the supervisor pushes OCSF events to the gateway log stream as shorthand text with an emptyfieldsmap (crates/openshell-supervisor-process/src/log_push.rs). The docs describe the shorthand as a format for scanning, not a stable one, and the push channel can drop lines under load.The better signal for decision-only tests is the proxy's error body.
docs/observability/logging.mdxdocuments a machine-readableerrorcode (policy_denied,ssrf_denied,unsupported_l7_protocol,middleware_denied,upstream_unreachable, ...) plusmethod,pathandrule_missingfor L7 REST denials. So:errorcode.openshell.middleware.traffic_uninspectable), not whole shorthand lines. Checking that secrets never appear in logs is still worth doing.SandboxLogLine.fields, which the proto comment already describes ("e.g. dst_host, action").Using
502 upstream_unreachableto tell "allowed" from "denied" without a server is possible, but it only proves the policy didn't block the request, not that the request was forwarded intact. It also depends on the open question above about transparent-mode L7.