Repository navigation
test(testkit): share sandbox client helpers across testkit suites - #5691
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/ok to test 70aaaf3 |
|
/ok to test dcf9567 |
|
/ok to test 70adbdc |
|
/ok to test 4fa87f9 |
|
Label |
|
/ok to test b76ff7b |
elezar
left a comment
There was a problem hiding this comment.
The design looks sound.
- The client runs inside the sandbox, so policy applies to it.
- The probe script only moves bytes and prints one JSON line, and HTTP parsing is unit-tested in Rust.
- The script is embedded in the test binary, so it can't drift from its parser.
- Failures include stderr in testkit reports.
I read the helper code and docs and skimmed the converted suites. I did not build or run anything.
To address
- Document the sandbox-script pattern in the testkit README. It lists the helpers but not why the client runs in the sandbox, the script contract (one JSON line, failures as results, argv input, stdlib only, parsing in Rust), or where a new script goes. Please also say that one-off scripts may stay beside a test and when to promote one to
sandbox-clients. Many legacye2e/rust/tests/files carry hand-rolled clients of this shape (bypass_detection.rs,forward_proxy_*_l7.rs), and DNS, TLS and listener probes will follow. - #5442 is merged, but the diff against
mainstill shows about 40 commits and 2.4k added lines. Could you rebase so this PR shows only the helper change? Closes #5689isn't registered as a closing reference (closingIssuesReferencesis empty), and #5689 doesn't havestate:accepted.
Suggestions
- A hint toward
testkit::python_image()whenrun_pythonfails withpython3: not found. TESTING.mdmentions Python only for the pytest-eraexec_python. A pointer torun_pythonwould avoid confusion.
For #5665: when it rewrites credential_gating.rs, it can drop the local run_python for Sandbox::run_python. BODY_CLIENT may fit the shared http helper if it can send a body and headers (not checked). WEBSOCKET_CLIENT is a candidate for a script file.
…nd binaries_yaml Shared helpers for suites that run client scripts in a sandbox: attach providers from the sandbox builder, run a Python script and read its JSON result, read a sandbox's policy revisions and active policy, and write a binaries list into hand-written policy YAML. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
A shared home for the clients tests run inside a sandbox. One probe script connects, optionally sends bytes, and reads until the connection ends; connect(), http() and http_raw() run it and return typed results. HTTP parsing happens in Rust and is unit-tested. Requests send Connection: close by default, and raw sends stop after two idle seconds, so a keep-alive server can't stall a read. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The test uses TEST_HOST and the sandbox-clients http probe in place of its own Python client. Its main changes from sh -c "exec sleep infinity" to sleep infinity, which leaves the same final process. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ands Tests create a long-running sandbox with the builder chain directly instead of a per-file ready_sandbox helper. The cli and policy-advisor mains change from sh -c "exec sleep infinity" to sleep infinity, which leaves the same final process. connect-ssh-supervision keeps its sh -lc mains. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
… keycloak tests The cli tests read sandbox get JSON through Sandbox::get_json, the cdi scenario creates its sandbox with cli_sandbox, and the keycloak refresh test installs its provider with install_provider instead of importing the profile and creating the provider by hand. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The e2e suites exercise the helpers. Keep only the HTTP parser tests for chunked bodies and non-HTTP bytes, which no e2e test reaches yet. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> #5657 makes the same install_provider change in that file along with its typed profile, so this branch leaves it unchanged to avoid a conflict.
The gateway limits sandbox names to 19 characters, which leaves 5 for a suffix after test-<run-id>-. cx.sandbox, cx.cli_sandbox and cx.track now panic at the test's call with that explanation, instead of the test failing later with the gateway's validation error. Context::name is unchanged, since provider and profile names have no such limit. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The testkit README now explains why clients run in the sandbox, the contract a client script follows, and when a script moves to sandbox-clients. tcp_probe.py points to it, sandbox-clients says it holds network clients only, TESTING.md points Rust tests to run_python, and run_python suggests python_image() when the sandbox has no python3. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
b76ff7b to
6eae90b
Compare
|
/ok to test 6eae90b |
|
Thanks @elezar. All addressed in
The two suggestions are in too: For #5665: it already drops its local |
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…errno Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
/ok to test 06b770c |
|
This looks good to ship. The shared probe and testkit helpers fit the stacked PRs well, and the sandbox name check will save a lot of confusing gateway errors. A few nits to consider, none blocking:
|
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
/ok to test e38f314 |
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
/ok to test a81fb32 |
|
Thanks for the review. All addressed:
|
Summary
Adds shared helpers for tests that run clients inside a sandbox, so later network, policy and credential suites don't each write their own. Converts the existing copies on this branch to use them. #5665 (PR2) and #5695 (PR5, issue #5676) stack on this. #5657 (PR3) is on main. #5624 is on main and adopts the helpers after this merges.
Related Issue
Closes #5689
Part of #5281
Stack: main ← #5691 ← #5665 ← #5695. #5442 has merged, and this branch is rebased onto
main.Changes
SandboxBuilder::providers(..),Sandbox::run_python(script, args)(fails on a nonzero exit),ExecOutput::last_json()(last JSON object or array line of stdout),Sandbox::policy_revisions()/active_policy(), andpolicy::binaries_yaml(..)for hand-written policy YAML.cx.sandbox,cx.cli_sandboxandcx.trackpanic at the test's call when a name exceeds the gateway's 19 characters (testkit::MAX_SANDBOX_NAME_LEN), which leaves 5 for a suffix. test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 found that 21 of its 22 E2E failures came from long names.Context::nameis unchanged, because provider and profile names have no limit, so test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 doesn't need to change it.TEST_HOST(host.openshell.internal).tests/support/sandbox-clientscrate (sandbox_clients_support): one Python probe script, plusconnect,httpandhttp_raw, which run it and return typed results (connect outcome with errno, or a separate DNS-failure flag when the name doesn't resolve; how the read ended, received bytes, parsed HTTP response). HTTP is parsed in Rust.httpsendsConnection: close;http_rawstops reading after 2 idle seconds, so a keep-alive server can't stall a test.sandbox-clients.tcp_probe.pyand the crate docs point to it,sandbox-clientssays it holds network clients only,TESTING.mdpoints Rust tests torun_python, andrun_pythonsuggestspython_image()when the sandbox has nopython3..image(testkit::python_image()).command(["sleep", "infinity"]); the testkit README shows this as the standard pattern.features/host-gatewayusesTEST_HOST. It keeps its inline client, because since fix(server): find a finished sandbox's supervisor on any replica #5667 the client runs as the sandbox's main process (to cover attaching to a finished process on another replica), and the probes run throughexec. No test on this branch callshttp(..)yet; test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 does.ready_sandboxhelpers inconformance/src/exec_cli.rs,conformance/src/ssh.rsandfeatures/connect-ssh-supervisionare removed and inlined. connect-ssh-supervision keeps itssh -lcmains.conformance/src/cli.rsreadssandbox getJSON throughSandbox::get_json(), the CDI scenario creates its sandbox withcli_sandbox. The Keycloak refresh test is left alone: test(e2e): port provider tests from e2e/rust to testkit #5657 converts it toinstall_providerwith its typed profile.sh -c "exec sleep infinity"mains (cli.rs,policy_advisor.rs) becomesleep infinity, which leaves the same final process.credential_gating.rsuses the sharedTEST_HOST; its clients take the host and port as arguments. test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665 rewrites the rest of that file.Testing
cargo fmt --checkandcargo clippy --locked --all-targets -- -D warningsfortests/testkit,tests/support,tests/suites/conformance,tests/suites/featuresandtests/suites/drivers/podmancargo nextest runfortests/testkitandtests/support. Unit tests for the new helpers are limited to two HTTP parser cases no e2e test reaches yet; the e2e suites cover the restfeatures/host-gatewaypass. CDI and Keycloak need a GPU or Keycloak and run in CIconnect-ssh-supervisionpasses clippy in a Linux containerChecklist