Repository navigation
test(e2e): port provider tests from e2e/rust to testkit - #5657
Draft
matthewgrossman wants to merge 11 commits into
Draft
matthewgrossman wants to merge 11 commits into
matthewgrossman wants to merge 11 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Member
Author
|
/ok to test b70615f |
|
Label |
Member
Author
|
/ok to test c7af9e2 |
5 tasks
Member
Author
|
/ok to test b9d5b3e |
10 of 11 tasks
matthewgrossman
added a commit
that referenced
this pull request
Nov 18, 2026
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> #5657 makes the same install_provider change in that file along with its typed profile, so this branch leaves it unchanged to avoid a conflict.
matthewgrossman
added a commit
that referenced
this pull request
Nov 18, 2026
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> #5657 makes the same install_provider change in that file along with its typed profile, so this branch leaves it unchanged to avoid a conflict.
matthewgrossman
changed the base branch from
main
to
test/5689-share-sandbox-helpers/matthewgrossman
November 18, 2026 18:26
create_provider and install_provider return a builder that creates the provider when awaited, so tests can chain .config(key, value) for provider create --config. Existing callers are unchanged. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Move provider_files and provider_auto_create from e2e/rust into a providers conformance suite. The auto-create test imports a run-unique profile instead of relying on a globally imported claude-code profile, so it no longer changes shared gateway state. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Requests go to the gateway's endpoint with the service URL's host as the Host header, which is how the gateway routes service traffic. Over TLS the client trusts the gateway CA and presents the registered gateway's client certificate, through openshell-crypto's rustls configuration. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
… suite Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…mage Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
matthewgrossman
force-pushed
the
test/5656-port-provider-tests/matthewgrossman
branch
from
November 18, 2026 18:37
b9d5b3e to
b5ef866
Compare
Member
Author
|
/ok to test b5ef866 |
matthewgrossman
changed the base branch from
test/5689-share-sandbox-helpers/matthewgrossman
to
main
November 18, 2026 18:38
4 tasks
…er-tests/matthewgrossman Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…lder::providers Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Member
Author
|
/ok to test d79cc8a |
laik
pushed a commit
to laik/OpenShell
that referenced
this pull request
Nov 19, 2026
…IDIA#5691) * test(testkit): add providers, run_python, last_json, policy readers and binaries_yaml Shared helpers for suites that run client scripts in a sandbox: attach providers from the sandbox builder, run a Python script and read its JSON result, read a sandbox's policy revisions and active policy, and write a binaries list into hand-written policy YAML. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(host-server): add TEST_HOST Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(support): add sandbox-clients crate with connect and http probes A shared home for the clients tests run inside a sandbox. One probe script connects, optionally sends bytes, and reads until the connection ends; connect(), http() and http_raw() run it and return typed results. HTTP parsing happens in Rust and is unit-tested. Requests send Connection: close by default, and raw sends stop after two idle seconds, so a keep-alive server can't stall a read. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(features): use shared helpers in host-gateway The test uses TEST_HOST and the sandbox-clients http probe in place of its own Python client. Its main changes from sh -c "exec sleep infinity" to sleep infinity, which leaves the same final process. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(suites): drop ready_sandbox wrappers and unify long-running commands Tests create a long-running sandbox with the builder chain directly instead of a per-file ready_sandbox helper. The cli and policy-advisor mains change from sh -c "exec sleep infinity" to sleep infinity, which leaves the same final process. connect-ssh-supervision keeps its sh -lc mains. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(testkit): list the shared sandbox helpers Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(suites): use shared sandbox and provider helpers in cli, cdi and keycloak tests The cli tests read sandbox get JSON through Sandbox::get_json, the cdi scenario creates its sandbox with cli_sandbox, and the keycloak refresh test installs its provider with install_provider instead of importing the profile and creating the provider by hand. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test: trim unit tests for the shared sandbox helpers The e2e suites exercise the helpers. Keep only the HTTP parser tests for chunked bodies and non-HTTP bytes, which no e2e test reaches yet. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(features): leave the keycloak refresh test to NVIDIA#5657 Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> NVIDIA#5657 makes the same install_provider change in that file along with its typed profile, so this branch leaves it unchanged to avoid a conflict. * test(testkit): fail early on sandbox names the gateway rejects The gateway limits sandbox names to 19 characters, which leaves 5 for a suffix after test-<run-id>-. cx.sandbox, cx.cli_sandbox and cx.track now panic at the test's call with that explanation, instead of the test failing later with the gateway's validation error. Context::name is unchanged, since provider and profile names have no such limit. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(testkit): document how to write a sandbox client The testkit README now explains why clients run in the sandbox, the contract a client script follows, and when a script moves to sandbox-clients. tcp_probe.py points to it, sandbox-clients says it holds network clients only, TESTING.md points Rust tests to run_python, and run_python suggests python_image() when the sandbox has no python3. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(conformance): use the shared TEST_HOST in credential gating Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(support): report DNS failures from the connect probe as dns, not errno Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(support): drop the unused is_dns_failure helper Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(support): parse HTTP replies with httparse Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(support): report a stalled send from the connect probe as a timeout Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(testkit): link the sandbox client guide and the server name limit Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Port the provider tests from
e2e/rustto testkit and delete the legacy files. The provider files and auto-create tests join the conformance suite. The service bearer passthrough test moves to a newfeatures/service-exposuresuite.Related Issue
Closes #5656
Part of #5281
Stack
Targets
main, which now includes #5691. The provider files test attaches its provider with #5691'sSandboxBuilder::providers(..), and the auto-create sandbox runssleep infinity. The other #5691 helpers don't apply: the service-exposure server is the sandbox's main process, which sandbox-clients probes can't be, and no test runs Python scripts, names the host gateway, or writes policy YAML by hand.Changes
create_providerandinstall_providerreturn aProviderBuilder. Chain.config(key, value)to pass--config; existing calls are unchanged.cx.service_get(url, headers)requests an exposed sandbox service through the gateway endpoint, with the service URL's authority asHost. It uses the gateway's mTLS bundle when the gateway has one, so tmachine environments need no extra routing.testkit::Profileandtestkit::Credentialbuild provider profiles in Rust instead offormat!YAML. Setters are named after the profile YAML fields, endpoints usepolicy::Endpoint(which gainsenforce()andpath()), andprofile.to_yaml()renders through openshell-providers, the CLI's export path. Unit tests compare built profiles with the YAML they name.providers.rsandfeatures/keycloak-provider-refreshuse it;credential_gating.rsis converted in test(e2e): port network, L7 and policy-update tests from e2e/rust to conformance #5665.providers.rsportsprovider_file_open_update_and_detachandauto_created_provider_credential_available_in_sandbox.service-exposurepackage withbearer_passthrough_preserves_authorization_header, plus rows in.github/workflows/e2e-suites.yml.provider_files.rs,provider_auto_create.rs,service_bearer_passthrough.rs, their[[test]]entries, theprovider_auto_createentry intests/tmachine/artifacts.nix, and the now-unusedrustlsandtokio-rustlse2e dependencies.Differences from the legacy tests
provider_filesran on Docker only, and tmachine lanes skippedprovider_auto_create. Service exposure runs on deb docker, rpm podman (rootful and rootless), k3s, and k3s-ha. Legacy ran it on Docker only.providers/*.yamlglobally and usedclaude-code. The port imports its own run-unique profile and passes the credential through the CLI environment. That removes the global import that deferred this move in test(conformance): migrate workspace, upload, and template e2e tests #4246./run/openshell/providers/<provider>/client.toml.Testing
cargo fmt --checkandcargo clippy -D warningsfor testkit, conformance, features, ande2e/rustservicetestsfeatures/service-exposurerows, all threefeatures/keycloak-provider-refreshrows, and the legacy lanes pass in Branch E2E Checks. The provider conformance tests also pass locally against a Docker gateway.Checklist