Repository navigation
test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance - #5665
matthewgrossman wants to merge 37 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
🌿 Preview your docs: https://nvidia-preview-pr-5665.docs.buildwithfern.com/openshell |
|
/ok to test 858dbc6 |
|
Label |
|
/ok to test 45fd589 |
858dbc6 to
45fd589
Compare
|
/ok to test 251835b |
b76ff7b to
6eae90b
Compare
251835b to
3214f39
Compare
|
/ok to test 3214f39 |
|
/ok to test f99bacb |
3214f39 to
f99bacb
Compare
|
/ok to test afcf2f4 |
f99bacb to
afcf2f4
Compare
|
/ok to test 054ed4f |
|
/ok to test 31e7182 |
Drops the include of a conformance client script from another workspace. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Replace conformance's support module and generic client scripts with testkit's run_python, last_json, policy readers, providers and binaries_yaml, and the connect, http and http_raw probes. Protocol and specialized clients move to the sandbox-clients scripts module. The pipelined request tests now count every response in the reply; the 403 follows the first response's body with no line break, so the old line-based count missed it. The policy update tests use the Python image, which has the sandbox user the base policy runs as. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…denials Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
… the finding name Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…e-host-ip Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ca gateways Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
/ok to test 86a2569 |
bfcbcc7 to
86a2569
Compare
… arrives on loopback Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
/ok to test c67e467 |
…k-l7-policy/matthewgrossman Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Compatibility review (advisory)New or updated breaking changes; see the full report for all findings. 1. api-behavior — No explicit risk acceptance foundGateway mutation OCSF status_detail now appends ErrorInfo.reason, changing values such as rejected:aborted to rejected:aborted:RESOURCE_VERSION_CONFLICT.
Review details
Advisory review; publication gates are unchanged. |
Add host_server_support::allow_host, Context::python_sandbox and policy::try_set, and use them instead of local wrappers and repeated builder chains. The host servers share one accept loop and connection counter. policy_update_changes_only_the_selected_endpoint compares the unselected rules as one value, and the MCP policy is built as JSON. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Send each request with sandbox_clients_support::http and compare its outcome, reporting every mismatch at once, instead of running three Python clients that each reimplemented an HTTP client and printed a keyed result matrix. The sandbox sets no proxy, so the urllib and raw socket variants sent the same bytes over the same path; one request per case remains. The REST upgrade check uses one sandbox with an enforce and an audit endpoint on two ports. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Use allow_host, python_sandbox and policy::try_set, and delete the local allow wrappers. Define the keep-alive client's signal directory once and pass it as an argument, instead of repeating its /tmp paths in the script and the test. Drop the policy set that repeated the initial policy in the reload test, and say why the ambiguous-update test keeps it. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…k-l7-policy/matthewgrossman Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…k-l7-policy/matthewgrossman Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…k-l7-policy/matthewgrossman Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…pers Share one policy update helper between accepted and rejected updates, encode GraphQL GET queries with form_urlencoded, merge the MCP fixture's revision and metadata checks, and drive the reload keep-alive client through stdin and stdout instead of signal files. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Summary
Port the network, L7, policy-update, and port-forward tests from
e2e/rustto testkit, and delete the legacy files. Most tests move into the conformance suite. The IP-literal egress test moves to a newroutable-host-ipfeature package.Stack:
main← this PR ← #5695 and #5780; #5786 builds on #5780. #5691 (shared sandbox client helpers) has merged intomain, and this branch includes it.Related Issue
Closes #5655
Part of #5281
Follow-up: #5752 (the k3s-ha skip below).
Changes
Context::compute_driver,Context::python_sandbox,SandboxBuilder::driver_config,Sandbox::logs/logs_until,policy::set/try_set,policy::from_yaml, andpolicy::base_with.TcpServer,free_port, a loopbackexchangeclient, andallow_host, which lets the image's Python reach a host server. The HTTP, TCP and WebSocket servers share one accept loop.Sandbox::run_python,ExecOutput::last_json,Sandbox::policy_revisions/active_policy,SandboxBuilder::providersandpolicy::binaries_yaml, plushost_server_support::TEST_HOST. Conformance no longer has asupportmodule.sandbox_clients_support::{connect, http, http_raw}calls. The GraphQL and JSON-RPC tests are tables of named requests and expected outcomes, and report every mismatch at once. Credential tests read the provider placeholder withprintenvand send it withhttp.tests/support/sandbox-clients/scripts: the MCP, localhost, listener lifetime, preconnect owner, socket inheritance, reload keep-alive, echo server, WebSocket text and binary, and bound credential (bash) clients, exposed throughsandbox_clients_support::scripts.routable-host-ipno longer includes a conformance script from another workspace.network: bypass denial, reserved destinations, localhost without proxy env, listener lifetime.network_l7+network_l7/mcp: REST, GraphQL, JSON-RPC, and MCP.network_egress: reload, ambiguous update,tls: skip, middleware, pipelining.policy_updateandport_forward.credential_gatingadditions: header and body rewrite, WebSocket text rewrite, and static credentials bound to profile endpoints.tests/suites/features/routable-host-ip: the IP-literal endpoint test (explicitallowed_ipsand implicit). Policy DNS gives the sandbox a synthetic address forhost.openshell.internal, so the test reaches a host server by name first and uses the address the request arrived at (host-serverRequest::local_addr). When that's loopback, because the Docker and Podman supervisors default to the host network, it uses the host's outbound address instead. CI rows: Docker rootful (deb), Podman rootful (rpm),k3s, andk3s-ha.policy_denied, and REST upgrades and the fail-closed middlewareunsupported_l7_protocol, as documented under "Proxy Error Responses".profile_endpoints_bind_static_credentials,fail_closed_middleware_blocks_uninspectable_traffic) haveunsupported.yamlentries forubuntu-k3s/k3s-ha, so they're skipped there. test(conformance): run the two sandbox-log tests on k3s-ha again #5752 tracks removing them.live_policy_updateis removed frome2e:rust:push. That lane already runsopenshell-conformanceagainst a push-mode gateway, so it runs thepolicy_updatetests instead. The tmachineconfig-pushenvironment is left to test(conformance): run the conformance suite against a gateway that pushes configuration #5622.static_provider_credentials_are_bound_to_profile_endpointsand its helpers fromhost_gateway_alias.rs. The other two tests in that file stay until PR5.[[test]]entries,PODMAN_CI_TESTSentries, andartifacts.nixfollow-up binary.e2e:websocket-conformancetask and the now-unusedSandboxGuard::spawn_forward.TESTING.md(routable-host-ip), and a note onContext::namethat sandbox suffixes can have at most 5 characters (test(testkit): share sandbox client helpers across testkit suites #5691'sMAX_SANDBOX_NAME_LENcheck enforces it).Defaults applied
host.openshell.internal. The legacy private-rangeallowed_ipsentries are gone.Sandbox::execinsleep infinitysandboxes, using pipes. The legacy tests were also non-TTY, so PTY/pipe parity is unchanged.exec_untilhelper;Sandbox::poll_execalready covers it.policy::base(),python_image(), and the shared Python binaries list.jsonrpc_default_audit_mode_still_denies_response_frames.live_policy_updateround-trip and from-empty tests are merged intopolicy_set_versions_and_ignores_repeats.Deviations from the plan
port_forwardis its own module, not part ofcli.rs, to keep it separate from the CLI tests. It startsforward start -dwithcli_spawnand waits for the exit code, because the backgroundedsshkeeps the CLI's pipes open. A cleanup hook runsforward stop.credential_gating.rs. The unbound endpoint is a second port on the test host instead ofhost.docker.internal, so the 403 holds on every driver. The legacy Podmanconnect-deniedspecial case is gone.Rows that kept a container fixture
None. The IP-literal test needs a host address that the supervisor allows as an IP literal. Rootless Podman's address is link-local, which is always blocked. So instead of keeping a container fixture, that test is a feature package that does not run on rootless Podman.
Driver coverage changes
bypass_detection,forward_proxy_*,host_gateway_alias,live_policy_update, andno_proxy. It now runs all of the ported conformance tests, on rootful and rootless.host_gateway_aliasfrom this set. It now runs all of them throughopenshell-conformanceine2e-vm.sh. That is new coverage. If a newly covered row fails, I'll raise it here before skipping anything.Testing
cargo fmt --checkandcargo clippy --locked --all-targets -D warningsfortests/testkit,tests/support,tests/suites/conformance,tests/suites/features,tests/suites/continuitycargo testfortests/testkitandtests/support(new host-server unit tests included)tasks/scripts/check-conformance-program.sh(77 tests)routable-host-ipandhost-gatewayfeature testscargo check --testsfore2e/rustwith docker, podman, kubernetes, vm, and config-push features.cargo clippyonhost_gateway_aliasis clean.Checklist