From 9959030f832460eaca76dd63f7bc7dc2d8a6266a Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 19 Nov 2026 15:54:41 -0800 Subject: [PATCH 01/12] docs(testkit): add the tests/support environment contract table Signed-off-by: Matthew Grossman --- tests/support/README.md | 42 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 tests/support/README.md diff --git a/tests/support/README.md b/tests/support/README.md new file mode 100644 index 0000000000..f2ca63d396 --- /dev/null +++ b/tests/support/README.md @@ -0,0 +1,42 @@ +# Test support crates + +Each crate here is a fixture for one concern, such as a service the +environment runs, and any suite can depend on it by path. See +[Adding a feature suite](../../TESTING.md#adding-a-feature-suite) for how an +environment and a fixture share a contract. + +## Environment contract + +These are the variables that an environment, the test runner or CI sets for +tests to read. A tmachine environment publishes its variables with +`tests/tmachine/ansible/playbooks/test-env.yaml`, and `openshell-test-runner` +passes them to the tests. To run a suite against your own deployment, set them +yourself. A fixture fails the test when a variable it needs is missing; it +never skips it. Add a row when you add a variable. + +| Variable | Set by | Read by | Meaning | +|---|---|---|---| +| `OPENSHELL_TEST_CONTAINER_ENGINE` | Every Docker and Podman base environment (`playbooks/docker.yaml`, `podman-rootful.yaml`, `podman-rootless.yaml`); `e2e:oci-image` tasks | `container-engine-support` | The command that reaches the gateway's container engine, split on whitespace, such as `docker` or `sudo -n podman`. | +| `OPENSHELL_TEST_PODMAN_USERNS` | `-userns-` (`playbooks/drivers/podman/userns.yaml`) | `podman-support` | The gateway's Podman user-namespace mode: `default`, `auto`, `keep-id` or `private`. | +| `OPENSHELL_TEST_CDI_SCENARIOS` | `-cdi` (`playbooks/cdi.yaml`) | `tests/suites/features/cdi` | The path of the document that names the CDI device kind, each scenario and the checks it expects. | +| `OPENSHELL_TEST_KEYCLOAK_ISSUER` | `-keycloak` (`playbooks/keycloak.yaml`) | `keycloak-support` | The OIDC issuer URL of the test realm, reachable from the test process. | +| `OPENSHELL_TEST_KEYCLOAK_CLIENT_ID` | `-keycloak` | `keycloak-support` | A client in that realm that allows password grants. | +| `OPENSHELL_TEST_KEYCLOAK_USERNAME` | `-keycloak` | `keycloak-support` | The realm user that tests sign in as. | +| `OPENSHELL_TEST_KEYCLOAK_PASSWORD` | `-keycloak` | `keycloak-support` | That user's password. | +| `OPENSHELL_TEST_BASE_IMAGE` | Optional; you, for example to use a mirror | `testkit::base_image()` | The default sandbox image. Unset, it is `nvcr.io/nvidia/base/ubuntu:24.04`. | +| `OPENSHELL_TEST_PYTHON_IMAGE` | Optional; `e2e/rust/e2e-kubernetes.sh` and `e2e-vm.sh` set a pullable image | `testkit::python_image()` | The workload image for tests that run `python3`, curl, git or openssl. Unset, it is `openshell/e2e-python:dev`, which the tmachine installers load. | +| `OPENSHELL_TEST_UNSUPPORTED` | CI, from `tests/suites/conformance/unsupported.yaml` (`tasks/scripts/conformance-unsupported.sh`), and `e2e/support/conformance.sh` for the host lanes; the runner passes it to the tests | testkit programs | Tests the environment can't pass yet, as `=` entries. See [Unsupported tests](../testkit/README.md#unsupported-tests). | +| `OPENSHELL_TEST_POSTGRES_URL` | `tasks/scripts/run-postgres-tests.sh` (`mise run test:rust:postgres`), or you | `openshell-server`'s PostgreSQL tests | A disposable PostgreSQL database. Never point it at one a gateway uses. | + +Other `OPENSHELL_TEST_*` names are internal test hooks and are left out on +purpose: unit-test child-process markers, the re-entry markers in +`tests/support/environment.rs`, Nix guest-cache settings, `install.sh` test +hooks and crate test settings. + +### Addresses the environment chooses + +When a test must listen on an address the environment picked, because the +gateway or another service is configured to call it, the environment +publishes it as `OPENSHELL_TEST__LISTEN` and the test binds it. The +test never picks the address itself. Tests that bind the same address can't +run at the same time. From 174cf4953f35f0602b4ee885b0c4609091899315 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 19 Nov 2026 15:54:41 -0800 Subject: [PATCH 02/12] test(tmachine): add a gateway_overrides role for environment gateway settings Signed-off-by: Matthew Grossman --- TESTING.md | 29 ++++++++--- .../playbooks/drivers/podman/userns.yaml | 42 ++++----------- .../roles/cdi_fixtures/tasks/main.yaml | 24 ++------- .../gateway_overrides/defaults/main.yaml | 5 ++ .../roles/gateway_overrides/tasks/main.yaml | 51 +++++++++++++++++++ .../openshell_external_driver/tasks/main.yaml | 17 ++----- .../files/merge_gateway_toml_test.py | 22 ++++++++ .../tasks/merge-settings.yaml | 7 +-- 8 files changed, 125 insertions(+), 72 deletions(-) create mode 100644 tests/tmachine/ansible/roles/gateway_overrides/defaults/main.yaml create mode 100644 tests/tmachine/ansible/roles/gateway_overrides/tasks/main.yaml diff --git a/TESTING.md b/TESTING.md index e234086747..7829434e28 100644 --- a/TESTING.md +++ b/TESTING.md @@ -324,7 +324,9 @@ service. clients there from the variables. A missing variable fails the test; it never skips it. - **Documentation.** Document the variables in the fixture's module docs, so - anyone can run the suite against their own deployment by setting them. + anyone can run the suite against their own deployment by setting them, and + add a row for each to the + [environment contract table](tests/support/README.md#environment-contract). The Keycloak fixture is `tests/support/keycloak`. Run the suite that uses it with: @@ -596,11 +598,26 @@ nix run .#openshell-test-runner -- run \ #### Gateway overrides in an environment -An environment overrides gateway settings by writing the tables to change to -`/etc/openshell-test/gateway-overrides.toml` in a playbook that runs before the -installer. The file holds whole tables, such as `[openshell.gateway]` or -`[openshell.drivers.podman]`, and is not a complete configuration. Each -installer applies it to the gateway's own file: +An environment overrides gateway settings with the `gateway_overrides` role, +which takes a name and the tables to change, such as `[openshell.gateway]` or +`[openshell.drivers.podman]`: + +```yaml +- name: Push configuration to supervisors + ansible.builtin.include_role: + name: gateway_overrides + vars: + gateway_overrides_name: config-push + gateway_overrides_content: | + [openshell.gateway] + config_delivery_mode = "push" +``` + +The role writes each name as its own marked block in +`/etc/openshell-test/gateway-overrides.toml`, so writers keep each other's +settings. Two blocks must not declare the same table. Don't write the file +directly. The file is not a complete configuration. Each installer applies it +to the gateway's own file: - `binaries` applies it to the `/etc/openshell/gateway.toml` it generates. - `deb` and `rpm` apply it to the service user's diff --git a/tests/tmachine/ansible/playbooks/drivers/podman/userns.yaml b/tests/tmachine/ansible/playbooks/drivers/podman/userns.yaml index 202270100f..dc74fe8a93 100644 --- a/tests/tmachine/ansible/playbooks/drivers/podman/userns.yaml +++ b/tests/tmachine/ansible/playbooks/drivers/podman/userns.yaml @@ -4,18 +4,16 @@ # Configure the Podman gateway's user-namespace mode and publish it. # # `podman_userns_mode` is default, auto, keep-id or private. Every mode except -# default writes its `[openshell.drivers.podman]` settings to -# `/etc/openshell-test/gateway-overrides.toml`, which every installer applies to the active -# gateway-overrides.toml (the openshell_gateway role for `binaries`, the -# openshell_packaged_gateway role for the packages); default removes the file. The -# Podman support crate (tests/support/podman) reads the published mode and -# runs Podman's own reference container in it. +# default adds its `[openshell.drivers.podman]` settings to the gateway +# overrides (the gateway_overrides role), which every installer applies to the +# gateway's configuration. Default mode adds nothing, so userns, uidmap and +# gidmap stay unset. The Podman support crate (tests/support/podman) reads the +# published mode and runs Podman's own reference container in it. --- - name: Podman user namespaces for tests hosts: all gather_facts: false vars: - podman_userns_driver_config: /etc/openshell-test/gateway-overrides.toml podman_userns_settings: default: "" auto: | @@ -43,34 +41,14 @@ {{ podman_userns_settings.keys() | join(', ') }}, not {{ podman_userns_mode | default('unset') }} - - name: Create the test configuration directory - become: true - ansible.builtin.file: - path: "{{ podman_userns_driver_config | dirname }}" - state: directory - owner: root - group: root - mode: "0755" - - name: Write the Podman driver user-namespace settings - become: true - ansible.builtin.copy: - dest: "{{ podman_userns_driver_config }}" - content: "{{ podman_userns_settings[podman_userns_mode] }}" - owner: root - group: root - mode: "0644" + ansible.builtin.include_role: + name: gateway_overrides + vars: + gateway_overrides_name: podman-userns + gateway_overrides_content: "{{ podman_userns_settings[podman_userns_mode] }}" when: podman_userns_mode != "default" - # The fragment is the only source of userns, uidmap and gidmap settings in - # the gateway's configuration, so default mode leaves them unset. - - name: Remove Podman driver user-namespace settings for default mode - become: true - ansible.builtin.file: - path: "{{ podman_userns_driver_config }}" - state: absent - when: podman_userns_mode == "default" - - name: Publish Podman user-namespace test variables ansible.builtin.include_tasks: ../../test-env.yaml vars: diff --git a/tests/tmachine/ansible/roles/cdi_fixtures/tasks/main.yaml b/tests/tmachine/ansible/roles/cdi_fixtures/tasks/main.yaml index 5f0c04586c..250629d90b 100644 --- a/tests/tmachine/ansible/roles/cdi_fixtures/tasks/main.yaml +++ b/tests/tmachine/ansible/roles/cdi_fixtures/tasks/main.yaml @@ -38,25 +38,11 @@ # The tests select the scenario device per sandbox through driver config, which # the gateway rejects unless an administrator allows it. The installers add # this to the active driver's settings. -- name: Create the test settings directory - become: true - ansible.builtin.file: - path: /etc/openshell-test - state: directory - owner: root - group: root - mode: "0755" - -# The gateway must accept a per-sandbox CDI device, which it does only when the -# driver allows caller driver config. Every installer merges this override into -# the gateway's own configuration (see openshell_gateway/tasks/merge-settings.yaml). - name: Allow caller driver config for the CDI tests - become: true - ansible.builtin.copy: - dest: /etc/openshell-test/gateway-overrides.toml - content: | + ansible.builtin.include_role: + name: gateway_overrides + vars: + gateway_overrides_name: cdi + gateway_overrides_content: | [openshell.drivers.{{ environment_driver }}] allow_driver_config = true - owner: root - group: root - mode: "0644" diff --git a/tests/tmachine/ansible/roles/gateway_overrides/defaults/main.yaml b/tests/tmachine/ansible/roles/gateway_overrides/defaults/main.yaml new file mode 100644 index 0000000000..73e224fb07 --- /dev/null +++ b/tests/tmachine/ansible/roles/gateway_overrides/defaults/main.yaml @@ -0,0 +1,5 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +--- +gateway_overrides_path: /etc/openshell-test/gateway-overrides.toml diff --git a/tests/tmachine/ansible/roles/gateway_overrides/tasks/main.yaml b/tests/tmachine/ansible/roles/gateway_overrides/tasks/main.yaml new file mode 100644 index 0000000000..5114e90e69 --- /dev/null +++ b/tests/tmachine/ansible/roles/gateway_overrides/tasks/main.yaml @@ -0,0 +1,51 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Add gateway settings to /etc/openshell-test/gateway-overrides.toml, which +# the `binaries`, `deb` and `rpm` installers apply to the gateway's +# configuration (openshell_gateway/tasks/merge-settings.yaml; k3s installers +# ignore it). +# +# Inputs: +# gateway_overrides_name names the marked block, such as `cdi`; writing +# the same name again replaces its block +# gateway_overrides_content whole TOML tables, such as +# `[openshell.drivers.podman]` +# +# Each name gets its own marked block, so writers keep each other's settings +# and may run in any order. Two blocks must not declare the same table: TOML +# rejects a table defined twice, and the installer fails on it. +# +# TODO: https://github.com/NVIDIA/OpenShell/pull/5489 reads gateway.d drop-in +# files. When it lands, write each name to its own drop-in here instead. +--- +- name: Require a gateway override name and content + ansible.builtin.assert: + that: + - gateway_overrides_name is defined + - gateway_overrides_name is match('^[a-z0-9][a-z0-9-]*$') + - gateway_overrides_content is defined + - gateway_overrides_content | trim | length > 0 + fail_msg: >- + gateway_overrides needs gateway_overrides_name (lowercase letters, + digits and dashes) and non-empty gateway_overrides_content + +- name: Create the test configuration directory + become: true + ansible.builtin.file: + path: "{{ gateway_overrides_path | dirname }}" + state: directory + owner: root + group: root + mode: "0755" + +- name: Write the {{ gateway_overrides_name }} gateway overrides + become: true + ansible.builtin.blockinfile: + path: "{{ gateway_overrides_path }}" + create: true + owner: root + group: root + mode: "0644" + marker: "# {mark} {{ gateway_overrides_name }}" + block: "{{ gateway_overrides_content }}" diff --git a/tests/tmachine/ansible/roles/openshell_external_driver/tasks/main.yaml b/tests/tmachine/ansible/roles/openshell_external_driver/tasks/main.yaml index a31363d57d..2f3d4e3872 100644 --- a/tests/tmachine/ansible/roles/openshell_external_driver/tasks/main.yaml +++ b/tests/tmachine/ansible/roles/openshell_external_driver/tasks/main.yaml @@ -108,18 +108,11 @@ ansible.builtin.fail: msg: The external driver socket did not appear -- name: Create the environment settings directory - become: true - ansible.builtin.file: - path: /etc/openshell-test - state: directory - mode: "0755" - - name: Point the gateway at the external driver - become: true - ansible.builtin.copy: - dest: /etc/openshell-test/gateway-overrides.toml - mode: "0644" - content: | + ansible.builtin.include_role: + name: gateway_overrides + vars: + gateway_overrides_name: external-driver + gateway_overrides_content: | [openshell.drivers.{{ openshell_external_driver_name }}] socket_path = "{{ openshell_external_driver_socket }}" diff --git a/tests/tmachine/ansible/roles/openshell_gateway/files/merge_gateway_toml_test.py b/tests/tmachine/ansible/roles/openshell_gateway/files/merge_gateway_toml_test.py index ba6926169a..3f37c15178 100644 --- a/tests/tmachine/ansible/roles/openshell_gateway/files/merge_gateway_toml_test.py +++ b/tests/tmachine/ansible/roles/openshell_gateway/files/merge_gateway_toml_test.py @@ -98,3 +98,25 @@ def test_creates_from_default_else_minimal(tmp_path): run, base = merge(other, None, '[openshell.drivers.podman]\nuserns = "auto"\n') parsed = tomllib.loads(base.read_text()) assert parsed["openshell"]["version"] == 2 + + +def test_applies_every_marked_block(tmp_path): + overrides = ( + "# BEGIN podman-userns\n[openshell.drivers.podman]\nuserns = \"auto\"\n# END podman-userns\n" + "# BEGIN config-push\n[openshell.gateway]\nconfig_delivery_mode = \"push\"\n# END config-push\n" + ) + run, base = merge(tmp_path, BASE, overrides) + assert run.returncode == 0, run.stderr + parsed = tomllib.loads(base.read_text()) + assert parsed["openshell"]["drivers"]["podman"]["userns"] == "auto" + assert parsed["openshell"]["gateway"]["config_delivery_mode"] == "push" + + +def test_rejects_blocks_that_declare_the_same_table(tmp_path): + overrides = ( + "# BEGIN cdi\n[openshell.drivers.podman]\nallow_driver_config = true\n# END cdi\n" + "# BEGIN podman-userns\n[openshell.drivers.podman]\nuserns = \"auto\"\n# END podman-userns\n" + ) + run, base = merge(tmp_path, BASE, overrides) + assert run.returncode != 0 + assert base.read_text() == BASE diff --git a/tests/tmachine/ansible/roles/openshell_gateway/tasks/merge-settings.yaml b/tests/tmachine/ansible/roles/openshell_gateway/tasks/merge-settings.yaml index 6d167a3ec8..2b3b8e3e22 100644 --- a/tests/tmachine/ansible/roles/openshell_gateway/tasks/merge-settings.yaml +++ b/tests/tmachine/ansible/roles/openshell_gateway/tasks/merge-settings.yaml @@ -1,10 +1,11 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -# An environment overrides gateway settings by writing the tables to change, +# An environment overrides gateway settings by adding the tables to change, # such as `[openshell.drivers.podman]`, to -# /etc/openshell-test/gateway-overrides.toml before the installer runs (for -# example playbooks/drivers/podman/userns.yaml). Apply them to +# /etc/openshell-test/gateway-overrides.toml with the gateway_overrides role +# before the installer runs (for example playbooks/drivers/podman/userns.yaml). +# Apply them to # `openshell_gateway_merge_target` without replacing the file; see # files/merge-gateway-toml.py. # From 2eb7ee9124fce3adbdaba5359b332964354c687d Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 19 Nov 2026 16:00:13 -0800 Subject: [PATCH 03/12] test(testkit): serve TLS from host servers with a test or environment CA Signed-off-by: Matthew Grossman --- tests/suites/conformance/Cargo.lock | 4 + tests/suites/features/Cargo.lock | 4 + tests/support/Cargo.lock | 4 + tests/support/README.md | 2 + tests/support/host-server/Cargo.toml | 4 + tests/support/host-server/src/http.rs | 72 ++++- tests/support/host-server/src/lib.rs | 5 +- tests/support/host-server/src/tls.rs | 308 +++++++++++++++++++++ tests/support/host-server/src/websocket.rs | 3 +- 9 files changed, 391 insertions(+), 15 deletions(-) create mode 100644 tests/support/host-server/src/tls.rs diff --git a/tests/suites/conformance/Cargo.lock b/tests/suites/conformance/Cargo.lock index 0d17b24979..53c4bdee9e 100644 --- a/tests/suites/conformance/Cargo.lock +++ b/tests/suites/conformance/Cargo.lock @@ -1285,9 +1285,13 @@ dependencies = [ name = "host-server-support" version = "0.0.0" dependencies = [ + "openshell-crypto", "openshell-testkit", + "rcgen", + "rustls", "serde_json", "tokio", + "tokio-rustls", "tungstenite", ] diff --git a/tests/suites/features/Cargo.lock b/tests/suites/features/Cargo.lock index 3a0b5a30b2..2f966d222c 100644 --- a/tests/suites/features/Cargo.lock +++ b/tests/suites/features/Cargo.lock @@ -1317,9 +1317,13 @@ dependencies = [ name = "host-server-support" version = "0.0.0" dependencies = [ + "openshell-crypto", "openshell-testkit", + "rcgen", + "rustls", "serde_json", "tokio", + "tokio-rustls", "tungstenite", ] diff --git a/tests/support/Cargo.lock b/tests/support/Cargo.lock index 044a44aa3e..440b9ec84b 100644 --- a/tests/support/Cargo.lock +++ b/tests/support/Cargo.lock @@ -1281,9 +1281,13 @@ dependencies = [ name = "host-server-support" version = "0.0.0" dependencies = [ + "openshell-crypto", "openshell-testkit", + "rcgen", + "rustls", "serde_json", "tokio", + "tokio-rustls", "tungstenite", ] diff --git a/tests/support/README.md b/tests/support/README.md index f2ca63d396..c13ba758e1 100644 --- a/tests/support/README.md +++ b/tests/support/README.md @@ -23,6 +23,8 @@ never skips it. Add a row when you add a variable. | `OPENSHELL_TEST_KEYCLOAK_CLIENT_ID` | `-keycloak` | `keycloak-support` | A client in that realm that allows password grants. | | `OPENSHELL_TEST_KEYCLOAK_USERNAME` | `-keycloak` | `keycloak-support` | The realm user that tests sign in as. | | `OPENSHELL_TEST_KEYCLOAK_PASSWORD` | `-keycloak` | `keycloak-support` | That user's password. | +| `OPENSHELL_TEST_ADDITIONAL_CA_CERT` | The additional-CA environment, which configures the gateway's `additional_ca_bundle` (added by #5729) | `host-server-support` (`TestCa::from_env`) | The path of the CA certificate, in PEM, that the gateway trusts. | +| `OPENSHELL_TEST_ADDITIONAL_CA_KEY` | The additional-CA environment | `host-server-support` (`TestCa::from_env`) | The path of that CA's private key, in PEM, readable by the test user. Tests sign server certificates with it. | | `OPENSHELL_TEST_BASE_IMAGE` | Optional; you, for example to use a mirror | `testkit::base_image()` | The default sandbox image. Unset, it is `nvcr.io/nvidia/base/ubuntu:24.04`. | | `OPENSHELL_TEST_PYTHON_IMAGE` | Optional; `e2e/rust/e2e-kubernetes.sh` and `e2e-vm.sh` set a pullable image | `testkit::python_image()` | The workload image for tests that run `python3`, curl, git or openssl. Unset, it is `openshell/e2e-python:dev`, which the tmachine installers load. | | `OPENSHELL_TEST_UNSUPPORTED` | CI, from `tests/suites/conformance/unsupported.yaml` (`tasks/scripts/conformance-unsupported.sh`), and `e2e/support/conformance.sh` for the host lanes; the runner passes it to the tests | testkit programs | Tests the environment can't pass yet, as `=` entries. See [Unsupported tests](../testkit/README.md#unsupported-tests). | diff --git a/tests/support/host-server/Cargo.toml b/tests/support/host-server/Cargo.toml index cf51fd6a4c..ccac94f0d1 100644 --- a/tests/support/host-server/Cargo.toml +++ b/tests/support/host-server/Cargo.toml @@ -12,9 +12,13 @@ repository = "https://github.com/NVIDIA/OpenShell" publish = false [dependencies] +openshell-crypto = { path = "../../../crates/openshell-crypto" } openshell-testkit = { path = "../../testkit/testkit" } +rcgen = { version = "0.14.10", default-features = false, features = ["pem"] } +rustls = { version = "0.23", default-features = false, features = ["std", "tls12"] } serde_json = "1" tokio = { version = "1.43", features = ["io-util", "net", "rt", "time"] } +tokio-rustls = { version = "0.26", default-features = false, features = ["tls12"] } tungstenite = "0.26" [dev-dependencies] diff --git a/tests/support/host-server/src/http.rs b/tests/support/host-server/src/http.rs index b93e6dddcc..8b1f830faa 100644 --- a/tests/support/host-server/src/http.rs +++ b/tests/support/host-server/src/http.rs @@ -7,10 +7,11 @@ use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::{Arc, Mutex, PoisonError}; use testkit::{Poll, Result}; -use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; -use tokio::net::TcpStream; +use tokio::io::{AsyncRead, AsyncReadExt as _, AsyncWrite, AsyncWriteExt as _}; use tokio::task::JoinHandle; +use tokio_rustls::TlsAcceptor; +use crate::tls::Identity; use crate::{MAX_REQUEST, READ_TIMEOUT, WAIT_INTERVAL, WAIT_TIMEOUT, bind, find, header}; /// A request an [`HttpRecorder`] read. A chunked body arrives decoded. @@ -171,6 +172,30 @@ impl HttpRecorder { /// Answer every request with `handler`'s response. pub async fn serve( handler: impl Fn(&Request) -> Response + Send + Sync + 'static, + ) -> Result { + Self::listen(None, handler).await + } + + /// Answer every request over TLS with `200` and `body`, presenting + /// `identity`'s certificate. + pub async fn start_tls(identity: &Identity, body: impl Into) -> Result { + let body = body.into(); + Self::serve_tls(identity, move |_| Response::text(body.clone())).await + } + + /// Answer every request over TLS with `handler`'s response, presenting + /// `identity`'s certificate. A connection that fails the TLS handshake + /// counts in [`connections`](Self::connections) but records no request. + pub async fn serve_tls( + identity: &Identity, + handler: impl Fn(&Request) -> Response + Send + Sync + 'static, + ) -> Result { + Self::listen(Some(identity.acceptor()?), handler).await + } + + async fn listen( + tls: Option, + handler: impl Fn(&Request) -> Response + Send + Sync + 'static, ) -> Result { let (listener, port) = bind().await?; let handler: Arc = Arc::new(handler); @@ -181,11 +206,21 @@ impl HttpRecorder { let task = tokio::spawn(async move { while let Ok((stream, _)) = listener.accept().await { task_connections.fetch_add(1, Ordering::AcqRel); - tokio::spawn(respond( - stream, - Arc::clone(&handler), - Arc::clone(&task_requests), - )); + let _ = stream.set_nodelay(true); + let local_addr = stream.local_addr().ok(); + let handler = Arc::clone(&handler); + let requests = Arc::clone(&task_requests); + let tls = tls.clone(); + tokio::spawn(async move { + match tls { + None => respond(stream, local_addr, handler, requests).await, + Some(tls) => { + if let Ok(stream) = tls.accept(stream).await { + respond(stream, local_addr, handler, requests).await; + } + } + } + }); } }); Ok(Self { @@ -239,10 +274,14 @@ impl Drop for HttpRecorder { /// Read requests, record each, and answer with `handler` until a response /// or request closes the connection. -async fn respond(mut stream: TcpStream, handler: Arc, requests: Arc>>) { - let _ = stream.set_nodelay(true); +async fn respond( + mut stream: impl AsyncRead + AsyncWrite + Unpin, + local_addr: Option, + handler: Arc, + requests: Arc>>, +) { let mut buffer = Vec::new(); - while let Some(request) = read_request(&mut stream, &mut buffer).await { + while let Some(request) = read_request(&mut stream, local_addr, &mut buffer).await { let response = handler(&request); let close = !response.keep_alive || request @@ -260,13 +299,18 @@ async fn respond(mut stream: TcpStream, handler: Arc, requests: Arc) -> Option { +/// `buffer`, and leave any bytes after it in `buffer`. `local_addr` is the +/// address the connection arrived on. +pub(crate) async fn read_request( + stream: &mut (impl AsyncRead + Unpin), + local_addr: Option, + buffer: &mut Vec, +) -> Option { let mut chunk = [0_u8; 4096]; loop { if let Some((mut request, length)) = parse_request(buffer) { buffer.drain(..length); - request.local_addr = stream.local_addr().ok(); + request.local_addr = local_addr; return Some(request); } match tokio::time::timeout(READ_TIMEOUT, stream.read(&mut chunk)).await { @@ -337,6 +381,8 @@ fn decode_chunked(bytes: &[u8]) -> Option<(Vec, usize)> { #[cfg(test)] mod tests { + use tokio::net::TcpStream; + use super::*; async fn connect(port: u16) -> TcpStream { diff --git a/tests/support/host-server/src/lib.rs b/tests/support/host-server/src/lib.rs index e9781bf502..374d3be419 100644 --- a/tests/support/host-server/src/lib.rs +++ b/tests/support/host-server/src/lib.rs @@ -11,7 +11,8 @@ //! host, such as one on a remote cluster, can't run tests that use them. //! //! - [`HttpRecorder`] answers HTTP requests with a fixed body or a handler's -//! [`Response`], and records each parsed [`Request`]. +//! [`Response`], and records each parsed [`Request`]. It serves HTTPS with +//! a certificate from a [`TestCa`]. //! - [`WebSocketRecorder`] completes the WebSocket handshake and records the //! frames that arrive. //! - [`TcpServer`] records raw bytes, and echoes them or answers once the @@ -21,6 +22,7 @@ mod http; mod tcp; +mod tls; mod websocket; use std::time::Duration; @@ -30,6 +32,7 @@ use tokio::net::TcpListener; pub use http::{HttpRecorder, Request, Response}; pub use tcp::{TcpServer, exchange}; +pub use tls::{ADDITIONAL_CA_CERT_ENV, ADDITIONAL_CA_KEY_ENV, Identity, TestCa}; pub use websocket::{TEXT_REPLY, WebSocketRecorder}; /// The name a sandbox reaches these servers by. diff --git a/tests/support/host-server/src/tls.rs b/tests/support/host-server/src/tls.rs new file mode 100644 index 0000000000..6b36293b92 --- /dev/null +++ b/tests/support/host-server/src/tls.rs @@ -0,0 +1,308 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Certificate authorities for TLS host servers. +//! +//! A test signs a server certificate with a [`TestCa`] and serves it with +//! [`HttpRecorder::serve_tls`](crate::HttpRecorder::serve_tls): +//! +//! - [`TestCa::new`] makes a CA that only the test knows, so nothing trusts it +//! unless the test hands over [`TestCa::cert_pem`]. +//! - [`TestCa::from_env`] loads the environment's additional CA, which the +//! gateway is configured to trust (`additional_ca_bundle`). The environment +//! publishes the paths of its PEM certificate and key in +//! `OPENSHELL_TEST_ADDITIONAL_CA_CERT` and `OPENSHELL_TEST_ADDITIONAL_CA_KEY`. + +use std::path::Path; +use std::sync::Arc; + +use openshell_crypto::pki::{self, KeyPair}; +use rcgen::{ + BasicConstraints, CertificateParams, DnType, ExtendedKeyUsagePurpose, IsCa, Issuer, + KeyUsagePurpose, +}; +use rustls::pki_types::pem::PemObject as _; +use rustls::pki_types::{CertificateDer, PrivateKeyDer}; +use testkit::{Error, Result}; +use tokio_rustls::TlsAcceptor; + +/// The path of the environment's additional CA certificate, in PEM. +pub const ADDITIONAL_CA_CERT_ENV: &str = "OPENSHELL_TEST_ADDITIONAL_CA_CERT"; +/// The path of the environment's additional CA private key, in PEM. +pub const ADDITIONAL_CA_KEY_ENV: &str = "OPENSHELL_TEST_ADDITIONAL_CA_KEY"; + +/// A certificate authority that signs server certificates for host servers. +pub struct TestCa { + cert_pem: String, + issuer: Issuer<'static, KeyPair>, +} + +impl TestCa { + /// A new CA that only this test knows. + pub fn new() -> Result { + let key = pki::generate_keypair().map_err(failed("generate a test CA key"))?; + let cert = pki::self_signed(ca_params(), &key).map_err(failed("sign the test CA"))?; + Ok(Self { + cert_pem: cert.pem(), + issuer: cert.into_issuer(key), + }) + } + + /// The environment's additional CA, from the files that + /// `OPENSHELL_TEST_ADDITIONAL_CA_CERT` and `OPENSHELL_TEST_ADDITIONAL_CA_KEY` + /// name. Fails if either is unset. + pub fn from_env() -> Result { + Self::from_vars(|name| std::env::var(name).ok()) + } + + fn from_vars(var: impl Fn(&str) -> Option) -> Result { + let path = |name: &str| { + var(name) + .filter(|value| !value.trim().is_empty()) + .ok_or_else(|| { + Error::msg(format!( + "{name} is not set; run this test in an environment with an additional CA" + )) + }) + }; + let cert = path(ADDITIONAL_CA_CERT_ENV)?; + let key = path(ADDITIONAL_CA_KEY_ENV)?; + Self::from_files(cert.as_ref(), key.as_ref()) + } + + /// A CA from its PEM certificate and private key files. + pub fn from_files(cert: &Path, key: &Path) -> Result { + let read = |path: &Path| { + std::fs::read_to_string(path) + .map_err(|error| Error::msg(format!("read {}: {error}", path.display()))) + }; + let cert_pem = read(cert)?; + let key_pem = read(key)?; + let der = CertificateDer::from_pem_slice(cert_pem.as_bytes()).map_err(failed(format!( + "parse the CA certificate {}", + cert.display() + )))?; + let key = KeyPair::from_pem(&key_pem) + .map_err(failed(format!("parse the CA key {}", key.display())))?; + let issuer = pki::issuer_from_der(&der, key) + .map_err(failed(format!("load the CA {}", cert.display())))?; + Ok(Self { cert_pem, issuer }) + } + + /// The CA certificate, in PEM, for a client that should trust it. + pub fn cert_pem(&self) -> &str { + &self.cert_pem + } + + /// A server certificate for `names`, signed by this CA. A name that parses + /// as an IP address becomes an IP address entry. + pub fn leaf(&self, names: &[&str]) -> Result { + let names: Vec = names.iter().map(ToString::to_string).collect(); + let first = names + .first() + .cloned() + .ok_or_else(|| Error::msg("a server certificate needs at least one name"))?; + let mut params = CertificateParams::new(names.clone()) + .map_err(failed(format!("server certificate names {names:?}")))?; + params.distinguished_name.push(DnType::CommonName, first); + params.key_usages = vec![KeyUsagePurpose::DigitalSignature]; + params.extended_key_usages = vec![ExtendedKeyUsagePurpose::ServerAuth]; + let key = pki::generate_keypair().map_err(failed("generate a server key"))?; + let cert = pki::signed_by_issuer(params, &key, &self.issuer) + .map_err(failed(format!("sign a server certificate for {names:?}")))?; + let key_der = key + .serialize_der() + .map_err(failed("export the server key"))?; + let key_pem = key + .serialize_pem() + .map_err(failed("export the server key"))?; + Ok(Identity { + cert_pem: cert.pem(), + key_pem, + cert_der: cert.der().clone(), + key_der, + }) + } +} + +/// A server certificate and its private key. +#[derive(Clone)] +pub struct Identity { + pub cert_pem: String, + pub key_pem: String, + cert_der: CertificateDer<'static>, + key_der: Vec, +} + +impl Identity { + pub(crate) fn acceptor(&self) -> Result { + let key = PrivateKeyDer::Pkcs8(self.key_der.clone().into()); + let mut config = openshell_crypto::tls::try_server_builder() + .map_err(failed("build a TLS server configuration"))? + .with_no_client_auth() + .with_single_cert(vec![self.cert_der.clone()], key) + .map_err(failed("load the server certificate"))?; + config.alpn_protocols = vec![b"http/1.1".to_vec()]; + Ok(TlsAcceptor::from(Arc::new(config))) + } +} + +impl std::fmt::Debug for Identity { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Identity") + .field("cert_pem", &self.cert_pem) + .finish_non_exhaustive() + } +} + +fn ca_params() -> CertificateParams { + let mut params = CertificateParams::default(); + params + .distinguished_name + .push(DnType::CommonName, "OpenShell test CA"); + params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); + params.key_usages = vec![ + KeyUsagePurpose::KeyCertSign, + KeyUsagePurpose::CrlSign, + KeyUsagePurpose::DigitalSignature, + ]; + params +} + +fn failed(context: impl std::fmt::Display) -> impl FnOnce(E) -> Error { + move |error| Error::msg(format!("{context}: {error}")) +} + +#[cfg(test)] +mod tests { + use std::path::PathBuf; + + use rustls::pki_types::ServerName; + use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; + use tokio::net::TcpStream; + use tokio_rustls::TlsConnector; + use tokio_rustls::client::TlsStream; + + use super::*; + use crate::{HttpRecorder, Response}; + + /// Open a TLS connection to `port` on loopback as `name`, trusting `ca`. + async fn connect(ca: &str, name: &str, port: u16) -> std::io::Result> { + let mut roots = rustls::RootCertStore::empty(); + roots + .add(CertificateDer::from_pem_slice(ca.as_bytes()).unwrap()) + .unwrap(); + let config = openshell_crypto::tls::try_client_builder() + .unwrap() + .with_root_certificates(roots) + .with_no_client_auth(); + #[allow( + clippy::disallowed_methods, + reason = "standalone test support has no openshell-core dependency and enables TCP_NODELAY below" + )] + let stream = TcpStream::connect(("127.0.0.1", port)).await?; + stream.set_nodelay(true)?; + TlsConnector::from(Arc::new(config)) + .connect(ServerName::try_from(name.to_string()).unwrap(), stream) + .await + } + + async fn get(stream: &mut TlsStream, path: &str) -> String { + stream + .write_all(format!("GET {path} HTTP/1.1\r\nHost: test\r\n\r\n").as_bytes()) + .await + .unwrap(); + let mut response = Vec::new(); + stream.read_to_end(&mut response).await.unwrap(); + String::from_utf8_lossy(&response).into_owned() + } + + fn write_ca(dir: &str) -> (PathBuf, PathBuf, String) { + let key = pki::generate_keypair().unwrap(); + let cert = pki::self_signed(ca_params(), &key).unwrap(); + let dir = std::env::temp_dir().join(format!("{dir}-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let (cert_path, key_path) = (dir.join("ca.pem"), dir.join("ca-key.pem")); + std::fs::write(&cert_path, cert.pem()).unwrap(); + std::fs::write(&key_path, key.serialize_pem().unwrap()).unwrap(); + (cert_path, key_path, cert.pem()) + } + + #[tokio::test] + async fn serve_tls_answers_a_client_that_trusts_the_ca() { + let ca = TestCa::new().unwrap(); + let identity = ca.leaf(&["localhost", "127.0.0.1"]).unwrap(); + let recorder = HttpRecorder::serve_tls(&identity, |request| { + Response::text(format!("tls {}", request.path())) + }) + .await + .unwrap(); + for name in ["localhost", "127.0.0.1"] { + let mut stream = connect(ca.cert_pem(), name, recorder.port).await.unwrap(); + let response = get(&mut stream, "/hello").await; + assert!(response.starts_with("HTTP/1.1 200 OK"), "{response}"); + assert!(response.ends_with("tls /hello"), "{response}"); + } + let requests = recorder.wait_for_requests(2).await.unwrap(); + assert_eq!(requests[0].path(), "/hello"); + } + + #[tokio::test] + async fn serve_tls_is_rejected_for_another_name_or_ca() { + let ca = TestCa::new().unwrap(); + let recorder = HttpRecorder::start_tls(&ca.leaf(&["localhost"]).unwrap(), "ok") + .await + .unwrap(); + let wrong_name = connect(ca.cert_pem(), "other.test", recorder.port).await; + assert!( + wrong_name.is_err(), + "a certificate for another name was accepted" + ); + let other = TestCa::new().unwrap(); + let wrong_ca = connect(other.cert_pem(), "localhost", recorder.port).await; + assert!( + wrong_ca.is_err(), + "a certificate from another CA was accepted" + ); + assert!(recorder.requests().is_empty()); + } + + #[tokio::test] + async fn from_vars_loads_the_environment_ca() { + let (cert, key, cert_pem) = write_ca("host-server-env-ca"); + let ca = TestCa::from_vars(|name| match name { + ADDITIONAL_CA_CERT_ENV => Some(cert.display().to_string()), + ADDITIONAL_CA_KEY_ENV => Some(key.display().to_string()), + _ => None, + }) + .unwrap(); + assert_eq!(ca.cert_pem(), cert_pem); + let recorder = HttpRecorder::start_tls(&ca.leaf(&["localhost"]).unwrap(), "env") + .await + .unwrap(); + let mut stream = connect(&cert_pem, "localhost", recorder.port) + .await + .unwrap(); + assert!(get(&mut stream, "/").await.ends_with("env")); + } + + #[test] + fn from_vars_fails_when_a_variable_is_missing() { + let error = TestCa::from_vars(|name| { + (name == ADDITIONAL_CA_CERT_ENV).then(|| "/nonexistent/ca.pem".to_string()) + }) + .err() + .unwrap(); + assert!(error.to_string().contains(ADDITIONAL_CA_KEY_ENV), "{error}"); + let error = TestCa::from_vars(|_| None).err().unwrap(); + assert!( + error.to_string().contains(ADDITIONAL_CA_CERT_ENV), + "{error}" + ); + } + + #[test] + fn leaf_needs_a_name() { + assert!(TestCa::new().unwrap().leaf(&[]).is_err()); + } +} diff --git a/tests/support/host-server/src/websocket.rs b/tests/support/host-server/src/websocket.rs index 4f38bdb8f3..11ac2606f4 100644 --- a/tests/support/host-server/src/websocket.rs +++ b/tests/support/host-server/src/websocket.rs @@ -105,7 +105,8 @@ impl Drop for WebSocketRecorder { async fn accept_websocket(mut stream: TcpStream, state: Arc) { let _ = stream.set_nodelay(true); - let Some(request) = read_request(&mut stream, &mut Vec::new()).await else { + let local_addr = stream.local_addr().ok(); + let Some(request) = read_request(&mut stream, local_addr, &mut Vec::new()).await else { return; }; let Some(key) = header(&request.head, "sec-websocket-key") else { From c71a6283bbde80366a433f01d915acef09f2971a Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 19 Nov 2026 16:30:33 -0800 Subject: [PATCH 04/12] test(testkit): let a host HTTP server listen on a chosen port Signed-off-by: Matthew Grossman --- tests/support/README.md | 5 +- tests/support/host-server/src/http.rs | 89 ++++++++++++++++++++-- tests/support/host-server/src/lib.rs | 18 +++-- tests/support/host-server/src/tcp.rs | 2 +- tests/support/host-server/src/websocket.rs | 2 +- 5 files changed, 101 insertions(+), 15 deletions(-) diff --git a/tests/support/README.md b/tests/support/README.md index c13ba758e1..a2c0a02eaa 100644 --- a/tests/support/README.md +++ b/tests/support/README.md @@ -39,6 +39,7 @@ hooks and crate test settings. When a test must listen on an address the environment picked, because the gateway or another service is configured to call it, the environment -publishes it as `OPENSHELL_TEST__LISTEN` and the test binds it. The -test never picks the address itself. Tests that bind the same address can't +publishes it as `OPENSHELL_TEST__LISTEN` and the test binds it, for +example with `HttpRecorder::builder().port(port)` from `host-server-support`. +The test never picks the address itself. Tests that bind the same address can't run at the same time. diff --git a/tests/support/host-server/src/http.rs b/tests/support/host-server/src/http.rs index 8b1f830faa..ce7d62110e 100644 --- a/tests/support/host-server/src/http.rs +++ b/tests/support/host-server/src/http.rs @@ -173,14 +173,23 @@ impl HttpRecorder { pub async fn serve( handler: impl Fn(&Request) -> Response + Send + Sync + 'static, ) -> Result { - Self::listen(None, handler).await + Self::builder().serve(handler).await + } + + /// A recorder on a chosen port or with TLS, such as on a port the + /// environment published: + /// + /// ```ignore + /// let upstream = HttpRecorder::builder().port(port).tls(&identity).start("ok").await?; + /// ``` + pub fn builder() -> HttpRecorderBuilder { + HttpRecorderBuilder::default() } /// Answer every request over TLS with `200` and `body`, presenting /// `identity`'s certificate. pub async fn start_tls(identity: &Identity, body: impl Into) -> Result { - let body = body.into(); - Self::serve_tls(identity, move |_| Response::text(body.clone())).await + Self::builder().tls(identity).start(body).await } /// Answer every request over TLS with `handler`'s response, presenting @@ -190,14 +199,15 @@ impl HttpRecorder { identity: &Identity, handler: impl Fn(&Request) -> Response + Send + Sync + 'static, ) -> Result { - Self::listen(Some(identity.acceptor()?), handler).await + Self::builder().tls(identity).serve(handler).await } async fn listen( + port: u16, tls: Option, handler: impl Fn(&Request) -> Response + Send + Sync + 'static, ) -> Result { - let (listener, port) = bind().await?; + let (listener, port) = bind(port).await?; let handler: Arc = Arc::new(handler); let requests = Arc::new(Mutex::new(Vec::new())); let connections = Arc::new(AtomicUsize::new(0)); @@ -266,6 +276,47 @@ impl HttpRecorder { } } +/// Options for an [`HttpRecorder`], from [`HttpRecorder::builder`]. +#[derive(Debug, Default)] +pub struct HttpRecorderBuilder { + port: u16, + tls: Option, +} + +impl HttpRecorderBuilder { + /// Listen on `port` instead of a random free one. Fails to start if + /// something already listens there. + #[must_use] + pub fn port(mut self, port: u16) -> Self { + self.port = port; + self + } + + /// Serve TLS, presenting `identity`'s certificate. A connection that + /// fails the TLS handshake counts in + /// [`connections`](HttpRecorder::connections) but records no request. + #[must_use] + pub fn tls(mut self, identity: &Identity) -> Self { + self.tls = Some(identity.clone()); + self + } + + /// Answer every request with `200` and `body`. + pub async fn start(self, body: impl Into) -> Result { + let body = body.into(); + self.serve(move |_| Response::text(body.clone())).await + } + + /// Answer every request with `handler`'s response. + pub async fn serve( + self, + handler: impl Fn(&Request) -> Response + Send + Sync + 'static, + ) -> Result { + let tls = self.tls.as_ref().map(Identity::acceptor).transpose()?; + HttpRecorder::listen(self.port, tls, handler).await + } +} + impl Drop for HttpRecorder { fn drop(&mut self) { self.task.abort(); @@ -442,6 +493,34 @@ mod tests { ); } + #[tokio::test] + async fn http_recorder_listens_on_a_chosen_port() { + let port = crate::free_port().await.unwrap(); + let recorder = HttpRecorder::builder() + .port(port) + .start("fixed") + .await + .unwrap(); + assert_eq!(recorder.port, port); + let mut stream = connect(port).await; + stream + .write_all(b"GET / HTTP/1.1\r\nHost: test\r\n\r\n") + .await + .unwrap(); + assert!(read_to_end(&mut stream).await.ends_with("fixed")); + + let error = HttpRecorder::builder() + .port(port) + .start("taken") + .await + .err() + .unwrap(); + assert!( + error.to_string().contains(&format!("port {port}")), + "{error}" + ); + } + #[tokio::test] async fn http_recorder_keeps_a_connection_alive_until_the_request_closes_it() { let recorder = diff --git a/tests/support/host-server/src/lib.rs b/tests/support/host-server/src/lib.rs index 374d3be419..b369cfac0c 100644 --- a/tests/support/host-server/src/lib.rs +++ b/tests/support/host-server/src/lib.rs @@ -12,7 +12,8 @@ //! //! - [`HttpRecorder`] answers HTTP requests with a fixed body or a handler's //! [`Response`], and records each parsed [`Request`]. It serves HTTPS with -//! a certificate from a [`TestCa`]. +//! a certificate from a [`TestCa`], and listens on a random free port unless +//! [`HttpRecorder::builder`] names one. //! - [`WebSocketRecorder`] completes the WebSocket handshake and records the //! frames that arrive. //! - [`TcpServer`] records raw bytes, and echoes them or answers once the @@ -30,7 +31,7 @@ use std::time::Duration; use testkit::{Result, ResultExt as _}; use tokio::net::TcpListener; -pub use http::{HttpRecorder, Request, Response}; +pub use http::{HttpRecorder, HttpRecorderBuilder, Request, Response}; pub use tcp::{TcpServer, exchange}; pub use tls::{ADDITIONAL_CA_CERT_ENV, ADDITIONAL_CA_KEY_ENV, Identity, TestCa}; pub use websocket::{TEXT_REPLY, WebSocketRecorder}; @@ -43,10 +44,15 @@ const WAIT_INTERVAL: Duration = Duration::from_millis(100); const READ_TIMEOUT: Duration = Duration::from_secs(10); const MAX_REQUEST: usize = 1024 * 1024; -async fn bind() -> Result<(TcpListener, u16)> { - let listener = TcpListener::bind(("0.0.0.0", 0)) +/// Listen on every host address at `port`, or at a random free port for 0. +async fn bind(port: u16) -> Result<(TcpListener, u16)> { + let listener = TcpListener::bind(("0.0.0.0", port)) .await - .context("bind a host server")?; + .context(if port == 0 { + "bind a host server".to_string() + } else { + format!("bind a host server on port {port}") + })?; let port = listener .local_addr() .context("read the host server's address")? @@ -56,7 +62,7 @@ async fn bind() -> Result<(TcpListener, u16)> { /// A port that nothing on the test host listened on when this returned. pub async fn free_port() -> Result { - let (_listener, port) = bind().await?; + let (_listener, port) = bind(0).await?; Ok(port) } diff --git a/tests/support/host-server/src/tcp.rs b/tests/support/host-server/src/tcp.rs index 1b6d566895..f0a93ef006 100644 --- a/tests/support/host-server/src/tcp.rs +++ b/tests/support/host-server/src/tcp.rs @@ -45,7 +45,7 @@ impl TcpServer { } async fn start(mode: Mode) -> Result { - let (listener, port) = bind().await?; + let (listener, port) = bind(0).await?; let received = Arc::new(Mutex::new(Vec::new())); let connections = Arc::new(AtomicUsize::new(0)); let task_received = Arc::clone(&received); diff --git a/tests/support/host-server/src/websocket.rs b/tests/support/host-server/src/websocket.rs index 11ac2606f4..5488006e83 100644 --- a/tests/support/host-server/src/websocket.rs +++ b/tests/support/host-server/src/websocket.rs @@ -34,7 +34,7 @@ struct WebSocketState { impl WebSocketRecorder { pub async fn start() -> Result { - let (listener, port) = bind().await?; + let (listener, port) = bind(0).await?; let state = Arc::new(WebSocketState::default()); let task_state = Arc::clone(&state); let task = tokio::spawn(async move { From 6d14aa7329f863a5d63ced1efc2b7d63f70bfbd9 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 19 Nov 2026 16:35:53 -0800 Subject: [PATCH 05/12] test(tmachine): add a corporate-proxy environment with a TLS-intercepting proxy Run one corporate proxy in the guest that serves TLS, requires Basic auth and intercepts the TLS in every tunnel with its own CA. It merges the three fixture proxies from the legacy corporate-proxy tests into one asyncio service, since TLS inside the listener's TLS needs asyncio's start_tls. Point the gateway's driver at it through the gateway_overrides role, with a port-qualified no_proxy entry, and publish OPENSHELL_TEST_CORPORATE_PROXY_{URL,CA_CERT,BYPASS_PORT,LOG}. Signed-off-by: Matthew Grossman --- .../ansible/playbooks/corporate-proxy.yaml | 25 +++ .../roles/corporate_proxy/defaults/main.yaml | 22 +++ .../corporate_proxy/files/corporate-proxy.py | 185 ++++++++++++++++++ .../roles/corporate_proxy/tasks/main.yaml | 179 +++++++++++++++++ .../openshell-corporate-proxy.service.j2 | 13 ++ tests/tmachine/config.nix | 6 + 6 files changed, 430 insertions(+) create mode 100644 tests/tmachine/ansible/playbooks/corporate-proxy.yaml create mode 100644 tests/tmachine/ansible/roles/corporate_proxy/defaults/main.yaml create mode 100644 tests/tmachine/ansible/roles/corporate_proxy/files/corporate-proxy.py create mode 100644 tests/tmachine/ansible/roles/corporate_proxy/tasks/main.yaml create mode 100644 tests/tmachine/ansible/roles/corporate_proxy/templates/openshell-corporate-proxy.service.j2 diff --git a/tests/tmachine/ansible/playbooks/corporate-proxy.yaml b/tests/tmachine/ansible/playbooks/corporate-proxy.yaml new file mode 100644 index 0000000000..6cb2431f1d --- /dev/null +++ b/tests/tmachine/ansible/playbooks/corporate-proxy.yaml @@ -0,0 +1,25 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Run a corporate proxy, point the gateway's driver at it and publish how +# tests read it. The corporate-proxy fixture (tests/support/corporate-proxy) +# reads the variables published below. +--- +- name: Corporate proxy for tests + hosts: all + gather_facts: false + pre_tasks: + - name: Wait for SSH + ansible.builtin.wait_for_connection: + roles: + - corporate_proxy + tasks: + - name: Publish corporate proxy test variables + ansible.builtin.include_tasks: test-env.yaml + vars: + test_env_name: corporate-proxy + test_env_vars: + OPENSHELL_TEST_CORPORATE_PROXY_URL: "https://{{ corporate_proxy_host }}:{{ corporate_proxy_port }}" + OPENSHELL_TEST_CORPORATE_PROXY_CA_CERT: "{{ corporate_proxy_config_dir }}/ca.pem" + OPENSHELL_TEST_CORPORATE_PROXY_BYPASS_PORT: "{{ corporate_proxy_bypass_port }}" + OPENSHELL_TEST_CORPORATE_PROXY_LOG: "{{ corporate_proxy_log }}" diff --git a/tests/tmachine/ansible/roles/corporate_proxy/defaults/main.yaml b/tests/tmachine/ansible/roles/corporate_proxy/defaults/main.yaml new file mode 100644 index 0000000000..2e42e192ce --- /dev/null +++ b/tests/tmachine/ansible/roles/corporate_proxy/defaults/main.yaml @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +--- +# Files the gateway and the tests read: the proxy's CA, its listener +# certificate and the credentials file. +corporate_proxy_config_dir: /etc/openshell-test/corporate-proxy +# Only the proxy reads the private keys. +corporate_proxy_key_dir: /etc/openshell-test/corporate-proxy/private +corporate_proxy_script: /usr/local/libexec/openshell-test/corporate-proxy.py +corporate_proxy_log: /var/log/openshell-test/corporate-proxy.jsonl +corporate_proxy_service: openshell-corporate-proxy.service +# The name sandboxes reach the guest by. The listener certificate and the +# re-signed tunnel certificates are issued for it. +corporate_proxy_host: host.openshell.internal +corporate_proxy_port: 3128 +# The destination port that no_proxy sends direct. Tests bind it themselves. +corporate_proxy_bypass_port: 39443 +# Throwaway credentials for a proxy that only this guest can reach. +corporate_proxy_username: proxyuser +corporate_proxy_password: proxypass +corporate_proxy_cert_days: 3650 diff --git a/tests/tmachine/ansible/roles/corporate_proxy/files/corporate-proxy.py b/tests/tmachine/ansible/roles/corporate_proxy/files/corporate-proxy.py new file mode 100644 index 0000000000..af16bbadcb --- /dev/null +++ b/tests/tmachine/ansible/roles/corporate_proxy/files/corporate-proxy.py @@ -0,0 +1,185 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""A corporate forward proxy for tests. + +It does what a strict corporate proxy does, all at once: + +- serves its listener over TLS, with a certificate signed by its own CA; +- requires Basic auth on every CONNECT; +- intercepts the TLS inside every tunnel, re-signing it with the same + certificate, and opens its own unverified TLS connection to the target. + +It appends one JSON line per CONNECT to the log, which tests read to see what +went through the proxy: + + {"time": 1.0, "target": "10.0.2.15:41234", "auth": "ok", "result": "intercepted"} + +`auth` is `ok`, `missing` or `wrong`. `result` is `rejected` (407), +`dial_failed` (502), `client_tls_failed` (the client didn't accept the +proxy's certificate), `upstream_tls_failed` or `intercepted`. The log never +holds credentials. +""" + +import argparse +import asyncio +import base64 +import json +import ssl +import time +from pathlib import Path + +MAX_HEAD = 65536 +DIAL_TIMEOUT = 10 +HANDSHAKE_TIMEOUT = 10 +CHUNK = 65536 + + +class Proxy: + def __init__(self, args): + credentials = Path(args.auth_file).read_text(encoding="utf-8").strip() + self.expected_auth = "Basic " + base64.b64encode(credentials.encode()).decode() + self.server_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + self.server_context.load_cert_chain(args.cert, args.key) + self.upstream_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) + self.upstream_context.check_hostname = False + self.upstream_context.verify_mode = ssl.CERT_NONE + self.log_file = Path(args.log).open("a", encoding="utf-8") # noqa: SIM115 + + def log(self, target, auth, result, **extra): + entry = {"time": time.time(), "target": target, "auth": auth, "result": result} + entry.update(extra) + self.log_file.write(json.dumps(entry) + "\n") + self.log_file.flush() + + async def handle(self, reader, writer): + try: + await self.connect(reader, writer) + except ( + OSError, + asyncio.IncompleteReadError, + asyncio.LimitOverrunError, + ssl.SSLError, + ): + pass + finally: + writer.close() + + async def connect(self, reader, writer): + head = await reader.readuntil(b"\r\n\r\n") + lines = head.decode("latin-1").split("\r\n") + parts = lines[0].split() + if len(parts) < 2 or parts[0].upper() != "CONNECT": + writer.write( + b"HTTP/1.1 405 Method Not Allowed\r\nContent-Length: 0\r\n\r\n" + ) + await writer.drain() + return + target = parts[1] + auth = "missing" + for line in lines[1:]: + name, _, value = line.partition(":") + if name.strip().lower() == "proxy-authorization": + auth = "ok" if value.strip() == self.expected_auth else "wrong" + if auth != "ok": + self.log(target, auth, "rejected") + writer.write( + b"HTTP/1.1 407 Proxy Authentication Required\r\n" + b'Proxy-Authenticate: Basic realm="corp"\r\n' + b"Content-Length: 0\r\n\r\n" + ) + await writer.drain() + return + + host, _, port = target.rpartition(":") + try: + upstream_reader, upstream_writer = await asyncio.wait_for( + asyncio.open_connection(host.strip("[]"), int(port)), DIAL_TIMEOUT + ) + except (OSError, ValueError, TimeoutError) as error: + self.log(target, auth, "dial_failed", error=repr(error)) + writer.write(b"HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\n\r\n") + await writer.drain() + return + + try: + writer.write(b"HTTP/1.1 200 Connection Established\r\n\r\n") + await writer.drain() + # TLS inside the listener's TLS. asyncio supports this from Python + # 3.11; ssl.SSLSocket can't be wrapped a second time. + try: + await writer.start_tls( + self.server_context, ssl_handshake_timeout=HANDSHAKE_TIMEOUT + ) + except (OSError, ssl.SSLError, TimeoutError) as error: + self.log(target, auth, "client_tls_failed", error=repr(error)) + return + try: + await upstream_writer.start_tls( + self.upstream_context, ssl_handshake_timeout=HANDSHAKE_TIMEOUT + ) + except (OSError, ssl.SSLError, TimeoutError) as error: + self.log(target, auth, "upstream_tls_failed", error=repr(error)) + return + self.log(target, auth, "intercepted") + await relay(reader, writer, upstream_reader, upstream_writer) + finally: + upstream_writer.close() + + +async def pipe(reader, writer): + while chunk := await reader.read(CHUNK): + writer.write(chunk) + await writer.drain() + + +async def relay(client_reader, client_writer, upstream_reader, upstream_writer): + tasks = [ + asyncio.ensure_future(pipe(client_reader, upstream_writer)), + asyncio.ensure_future(pipe(upstream_reader, client_writer)), + ] + # TLS streams can't half-close, so the tunnel ends when either side does. + _, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED) + for task in pending: + task.cancel() + await asyncio.gather(*tasks, return_exceptions=True) + + +def ignore_disconnects(loop, context): + # Probes and clients that drop the listener's TLS handshake would + # otherwise fill the journal. + if isinstance(context.get("exception"), (OSError, ssl.SSLError)): + return + loop.default_exception_handler(context) + + +async def main(): + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--listen", default="0.0.0.0") + parser.add_argument("--port", type=int, required=True) + parser.add_argument( + "--cert", required=True, help="PEM certificate chain, leaf first" + ) + parser.add_argument("--key", required=True) + parser.add_argument("--auth-file", required=True, help="a user:password file") + parser.add_argument("--log", required=True) + args = parser.parse_args() + + proxy = Proxy(args) + asyncio.get_running_loop().set_exception_handler(ignore_disconnects) + server = await asyncio.start_server( + proxy.handle, + args.listen, + args.port, + ssl=proxy.server_context, + ssl_handshake_timeout=HANDSHAKE_TIMEOUT, + limit=MAX_HEAD, + ) + print(f"corporate proxy listening on {args.listen}:{args.port}", flush=True) + async with server: + await server.serve_forever() + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/tests/tmachine/ansible/roles/corporate_proxy/tasks/main.yaml b/tests/tmachine/ansible/roles/corporate_proxy/tasks/main.yaml new file mode 100644 index 0000000000..e94d01dae5 --- /dev/null +++ b/tests/tmachine/ansible/roles/corporate_proxy/tasks/main.yaml @@ -0,0 +1,179 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Run a corporate proxy in the guest and point the gateway's driver at it. +# The proxy (files/corporate-proxy.py) serves TLS, requires Basic auth and +# intercepts the TLS in every tunnel, all with one CA of its own. +--- +- name: Require the environment's compute driver + ansible.builtin.assert: + that: [environment_driver is defined] + fail_msg: environment_driver names the gateway driver and comes from the environment's base in config.nix + +- name: Install OpenSSL and Python + become: true + ansible.builtin.package: + name: [openssl, python3] + state: present + +- name: Require Python 3.11 or later for TLS inside TLS + ansible.builtin.command: + argv: [/usr/bin/python3, -c, "import sys; sys.exit(sys.version_info < (3, 11))"] + changed_when: false + +- name: Create the corporate proxy directories + become: true + ansible.builtin.file: + path: "{{ item.path }}" + state: directory + owner: root + group: root + mode: "{{ item.mode }}" + loop: + - { path: "{{ corporate_proxy_config_dir }}", mode: "0755" } + - { path: "{{ corporate_proxy_key_dir }}", mode: "0700" } + - { path: "{{ corporate_proxy_script | dirname }}", mode: "0755" } + - { path: "{{ corporate_proxy_log | dirname }}", mode: "0755" } + +# The listener certificate must be a leaf: rustls rejects a CA certificate +# presented as a server's. Python 3.13 and later also require its authority +# key identifier. The same leaf re-signs every tunnel, since sandboxes +# reach every destination by corporate_proxy_host. +- name: Create the corporate proxy CA and listener certificate + become: true + ansible.builtin.shell: + cmd: | + set -eu + umask 077 + keys={{ corporate_proxy_key_dir }} + conf={{ corporate_proxy_config_dir }} + work=$(mktemp -d) + trap 'rm -rf "$work"' EXIT + openssl req -x509 -newkey rsa:2048 -nodes -days {{ corporate_proxy_cert_days }} \ + -keyout "$keys/ca-key.pem" -out "$work/ca.pem" \ + -subj '/CN=OpenShell test corporate proxy CA' \ + -addext 'basicConstraints=critical,CA:TRUE' \ + -addext 'keyUsage=critical,keyCertSign,cRLSign' + openssl req -newkey rsa:2048 -nodes -subj '/CN={{ corporate_proxy_host }}' \ + -keyout "$keys/proxy-key.pem" -out "$work/proxy.csr" + printf '%s\n' 'basicConstraints=critical,CA:FALSE' \ + 'subjectAltName=DNS:{{ corporate_proxy_host }}' \ + 'extendedKeyUsage=serverAuth' \ + 'keyUsage=critical,digitalSignature,keyEncipherment' \ + 'subjectKeyIdentifier=hash' 'authorityKeyIdentifier=keyid' > "$work/proxy.ext" + openssl x509 -req -in "$work/proxy.csr" -CA "$work/ca.pem" -CAkey "$keys/ca-key.pem" \ + -CAcreateserial -days {{ corporate_proxy_cert_days }} -extfile "$work/proxy.ext" \ + -out "$work/proxy.pem" + cat "$work/proxy.pem" "$work/ca.pem" > "$work/proxy-chain.pem" + install -m 0644 "$work/proxy-chain.pem" "$conf/proxy-chain.pem" + install -m 0644 "$work/ca.pem" "$conf/ca.pem" + creates: "{{ corporate_proxy_config_dir }}/ca.pem" + +# Every gateway user (root, tmachine or the package's) must read these. +- name: Write the corporate proxy credentials + become: true + ansible.builtin.copy: + dest: "{{ corporate_proxy_config_dir }}/auth" + content: "{{ corporate_proxy_username }}:{{ corporate_proxy_password }}\n" + owner: root + group: root + mode: "0644" + +- name: Install the corporate proxy + become: true + ansible.builtin.copy: + src: corporate-proxy.py + dest: "{{ corporate_proxy_script }}" + owner: root + group: root + mode: "0755" + +# Tests read the log as the tmachine user. +- name: Create the corporate proxy log + become: true + ansible.builtin.file: + path: "{{ corporate_proxy_log }}" + state: touch + owner: root + group: root + mode: "0644" + access_time: preserve + modification_time: preserve + +- name: Install the corporate proxy unit + become: true + ansible.builtin.template: + src: openshell-corporate-proxy.service.j2 + dest: /etc/systemd/system/{{ corporate_proxy_service }} + owner: root + group: root + mode: "0644" + +- name: Start the corporate proxy + block: + - name: Start the corporate proxy service + become: true + ansible.builtin.systemd_service: + name: "{{ corporate_proxy_service }}" + daemon_reload: true + enabled: true + state: restarted + + # Check what a sandbox's supervisor relies on: the listener's certificate + # verifies against the CA for corporate_proxy_host, and a CONNECT without + # credentials gets a 407. + - name: Check that the corporate proxy serves TLS and requires auth + ansible.builtin.command: + argv: + - /usr/bin/python3 + - -c + - | + import socket, ssl, sys, time + ca, host, port = sys.argv[1], sys.argv[2], int(sys.argv[3]) + context = ssl.create_default_context(cafile=ca) + deadline = time.monotonic() + 15 + while True: + try: + raw = socket.create_connection(("127.0.0.1", port), timeout=5) + break + except OSError: + if time.monotonic() > deadline: + raise + time.sleep(0.5) + with context.wrap_socket(raw, server_hostname=host) as conn: + conn.sendall(b"CONNECT 127.0.0.1:1 HTTP/1.1\r\nHost: 127.0.0.1:1\r\n\r\n") + status = conn.recv(4096).split(b"\r\n", 1)[0] + sys.exit(0 if b" 407 " in status else f"expected a 407, got {status!r}") + - "{{ corporate_proxy_config_dir }}/ca.pem" + - "{{ corporate_proxy_host }}" + - "{{ corporate_proxy_port }}" + changed_when: false + + rescue: + - name: Collect corporate proxy diagnostics + become: true + ansible.builtin.shell: | + systemctl status {{ corporate_proxy_service }} --no-pager -l + journalctl -u {{ corporate_proxy_service }} --no-pager -n 40 + changed_when: false + failed_when: false + register: corporate_proxy_diagnostics + + - name: Report why the corporate proxy did not start + ansible.builtin.fail: + msg: "{{ corporate_proxy_service }} did not start:\n{{ corporate_proxy_diagnostics.stdout }}\n{{ corporate_proxy_diagnostics.stderr }}" + +# Basic credentials go over the proxy's TLS listener, so the gateway needs no +# proxy_auth_allow_insecure. The CA covers the listener and the re-signed +# tunnels, for both the supervisor and the workload. +- name: Point the gateway's driver at the corporate proxy + ansible.builtin.include_role: + name: gateway_overrides + vars: + gateway_overrides_name: corporate-proxy + gateway_overrides_content: | + [openshell.drivers.{{ environment_driver }}] + https_proxy = "https://{{ corporate_proxy_host }}:{{ corporate_proxy_port }}" + proxy_auth_file = "{{ corporate_proxy_config_dir }}/auth" + proxy_ca_bundle = "{{ corporate_proxy_config_dir }}/ca.pem" + no_proxy = "{{ corporate_proxy_host }}:{{ corporate_proxy_bypass_port }}" diff --git a/tests/tmachine/ansible/roles/corporate_proxy/templates/openshell-corporate-proxy.service.j2 b/tests/tmachine/ansible/roles/corporate_proxy/templates/openshell-corporate-proxy.service.j2 new file mode 100644 index 0000000000..192b432420 --- /dev/null +++ b/tests/tmachine/ansible/roles/corporate_proxy/templates/openshell-corporate-proxy.service.j2 @@ -0,0 +1,13 @@ +{# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. #} +{# SPDX-License-Identifier: Apache-2.0 #} +[Unit] +Description=OpenShell test corporate proxy +After=network-online.target +Wants=network-online.target + +[Service] +ExecStart=/usr/bin/python3 {{ corporate_proxy_script }} --port {{ corporate_proxy_port }} --cert {{ corporate_proxy_config_dir }}/proxy-chain.pem --key {{ corporate_proxy_key_dir }}/proxy-key.pem --auth-file {{ corporate_proxy_config_dir }}/auth --log {{ corporate_proxy_log }} +Restart=always + +[Install] +WantedBy=multi-user.target diff --git a/tests/tmachine/config.nix b/tests/tmachine/config.nix index 5a41618bf4..5caaea6383 100644 --- a/tests/tmachine/config.nix +++ b/tests/tmachine/config.nix @@ -112,6 +112,12 @@ let bases = dockerBases ++ podmanBases; setup.playbooks = [ "ansible/playbooks/cdi.yaml" ]; }; + # A TLS-intercepting corporate proxy with Basic auth that the gateway's + # driver sends sandbox egress through, for the corporate-proxy suite. + corporate-proxy = { + bases = dockerBases ++ podmanBases; + setup.playbooks = [ "ansible/playbooks/corporate-proxy.yaml" ]; + }; }; featureEnvironments = pkgs.lib.concatMap ( feature: map (base: withSetup base feature features.${feature}.setup) features.${feature}.bases From cbd58a5bb0e50bf71afe9231aa60d927e08ab0e6 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 19 Nov 2026 16:39:07 -0800 Subject: [PATCH 06/12] test(support): add a corporate-proxy fixture that reads the proxy's CONNECT log Read OPENSHELL_TEST_CORPORATE_PROXY_* and parse the proxy's JSON-lines log from a cursor, so a test sees only the CONNECTs its own sandbox made. Signed-off-by: Matthew Grossman --- tests/support/Cargo.lock | 10 + tests/support/Cargo.toml | 2 +- tests/support/corporate-proxy/Cargo.toml | 20 ++ tests/support/corporate-proxy/src/lib.rs | 354 +++++++++++++++++++++++ 4 files changed, 385 insertions(+), 1 deletion(-) create mode 100644 tests/support/corporate-proxy/Cargo.toml create mode 100644 tests/support/corporate-proxy/src/lib.rs diff --git a/tests/support/Cargo.lock b/tests/support/Cargo.lock index 440b9ec84b..ea0368c598 100644 --- a/tests/support/Cargo.lock +++ b/tests/support/Cargo.lock @@ -566,6 +566,16 @@ version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f8f80099a98041a3d1622845c271458a2d73e688351bf3cb999266764b81d48" +[[package]] +name = "corporate-proxy-support" +version = "0.0.0" +dependencies = [ + "openshell-testkit", + "serde", + "serde_json", + "tempfile", +] + [[package]] name = "cpufeatures" version = "0.2.17" diff --git a/tests/support/Cargo.toml b/tests/support/Cargo.toml index e3dae4234d..7c1840a58a 100644 --- a/tests/support/Cargo.toml +++ b/tests/support/Cargo.toml @@ -5,4 +5,4 @@ # environment runs. Suites depend on them by path. [workspace] resolver = "2" -members = ["container-engine", "host-server", "keycloak", "podman", "sandbox-clients"] +members = ["container-engine", "corporate-proxy", "host-server", "keycloak", "podman", "sandbox-clients"] diff --git a/tests/support/corporate-proxy/Cargo.toml b/tests/support/corporate-proxy/Cargo.toml new file mode 100644 index 0000000000..3363668906 --- /dev/null +++ b/tests/support/corporate-proxy/Cargo.toml @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "corporate-proxy-support" +description = "The environment's corporate proxy and the CONNECTs it logged, for test suites" +version = "0.0.0" +edition = "2024" +rust-version = "1.94" +license = "Apache-2.0" +repository = "https://github.com/NVIDIA/OpenShell" +publish = false + +[dependencies] +openshell-testkit = { path = "../../testkit/testkit" } +serde = { version = "1", features = ["derive"] } +serde_json = "1" + +[dev-dependencies] +tempfile = "3" diff --git a/tests/support/corporate-proxy/src/lib.rs b/tests/support/corporate-proxy/src/lib.rs new file mode 100644 index 0000000000..271d87becc --- /dev/null +++ b/tests/support/corporate-proxy/src/lib.rs @@ -0,0 +1,354 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! The environment's corporate proxy, for suites that check sandbox egress +//! through it. +//! +//! The environment runs one proxy and configures the gateway's driver to send +//! sandbox egress through it. The proxy serves TLS, requires Basic auth and +//! intercepts the TLS in every tunnel, all with one CA of its own. The +//! gateway's `no_proxy` sends one destination port, the bypass port, direct. +//! The environment publishes: +//! +//! | Variable | Meaning | +//! |---|---| +//! | `OPENSHELL_TEST_CORPORATE_PROXY_URL` | The proxy URL the gateway uses, e.g. `https://host.openshell.internal:3128` | +//! | `OPENSHELL_TEST_CORPORATE_PROXY_CA_CERT` | The path of the proxy's CA certificate, in PEM | +//! | `OPENSHELL_TEST_CORPORATE_PROXY_BYPASS_PORT` | The destination port the gateway's `no_proxy` sends direct | +//! | `OPENSHELL_TEST_CORPORATE_PROXY_LOG` | The path of the proxy's log, readable by the tests | +//! +//! Under `openshell-test-runner`, the tmachine `-corporate-proxy` +//! environments do that (`tests/tmachine/ansible/playbooks/corporate-proxy.yaml`). +//! A missing variable fails the test; it never skips it. +//! +//! The log has one JSON line per CONNECT and lasts as long as the +//! environment, so a test takes a [`LogCursor`] before its sandbox starts +//! and reads only what came after it, for the ports it started servers on: +//! +//! ```ignore +//! let proxy = CorporateProxy::from_env()?; +//! let cursor = proxy.log_cursor()?; +//! // ... run a sandbox that reaches a server on `port` ... +//! let connects = proxy.connects_to(&cursor, port)?; +//! ``` +//! +//! Tests that bind the bypass port can't run at the same time. + +use std::fmt; +use std::io::{Read as _, Seek as _, SeekFrom}; +use std::path::{Path, PathBuf}; + +use serde::Deserialize; +use testkit::{Error, Result}; + +pub const URL_ENV: &str = "OPENSHELL_TEST_CORPORATE_PROXY_URL"; +pub const CA_CERT_ENV: &str = "OPENSHELL_TEST_CORPORATE_PROXY_CA_CERT"; +pub const BYPASS_PORT_ENV: &str = "OPENSHELL_TEST_CORPORATE_PROXY_BYPASS_PORT"; +pub const LOG_ENV: &str = "OPENSHELL_TEST_CORPORATE_PROXY_LOG"; + +/// How to reach the environment's corporate proxy and read its log. +#[derive(Clone, Debug)] +pub struct CorporateProxy { + url: String, + ca_cert: PathBuf, + bypass_port: u16, + log: PathBuf, +} + +/// A point in the proxy's log. [`CorporateProxy::connects_since`] reads the +/// CONNECTs logged after it. +#[derive(Clone, Copy, Debug)] +pub struct LogCursor(u64); + +/// One CONNECT the proxy received. +#[derive(Clone, Debug, Deserialize, PartialEq, Eq)] +pub struct Connect { + /// The CONNECT target, `host:port`. The supervisor sends a validated IP + /// unless the gateway sets `proxy_connect_by_hostname`. + pub target: String, + pub auth: Auth, + pub result: Outcome, + /// Why the CONNECT failed, for failure messages. + #[serde(default)] + pub error: Option, +} + +/// Whether the CONNECT carried the proxy's credentials. +#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum Auth { + Ok, + Missing, + Wrong, +} + +/// What the proxy did with the CONNECT. +#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum Outcome { + /// Answered 407, because the credentials were missing or wrong. + Rejected, + /// Answered 502, because the target refused or timed out. + DialFailed, + /// The client rejected the proxy's re-signed certificate, or dropped the + /// tunnel's TLS handshake. + ClientTlsFailed, + /// The target's TLS handshake failed. + UpstreamTlsFailed, + /// The proxy terminated the tunnel's TLS and relayed it to the target. + Intercepted, +} + +impl Connect { + /// The CONNECT target's host, without IPv6 brackets. + pub fn host(&self) -> &str { + self.target + .rsplit_once(':') + .map_or(self.target.as_str(), |(host, _)| host) + .trim_start_matches('[') + .trim_end_matches(']') + } + + /// The CONNECT target's port. + pub fn port(&self) -> Option { + self.target.rsplit_once(':')?.1.parse().ok() + } +} + +impl fmt::Display for Connect { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "CONNECT {} auth={:?} result={:?}", + self.target, self.auth, self.result + )?; + if let Some(error) = &self.error { + write!(f, " error={error}")?; + } + Ok(()) + } +} + +impl CorporateProxy { + /// Read `OPENSHELL_TEST_CORPORATE_PROXY_*`. Fails naming the first + /// missing variable. + pub fn from_env() -> Result { + Self::from_vars(|name| std::env::var(name).ok()) + } + + fn from_vars(var: impl Fn(&str) -> Option) -> Result { + let required = |name: &str| { + var(name) + .map(|value| value.trim().to_string()) + .filter(|value| !value.is_empty()) + .ok_or_else(|| { + Error::msg(format!( + "{name} is not set; this suite needs a corporate proxy (see the corporate-proxy-support crate docs)" + )) + }) + }; + let url = required(URL_ENV)?; + let ca_cert = required(CA_CERT_ENV)?.into(); + let bypass_port = required(BYPASS_PORT_ENV)?; + let bypass_port = bypass_port.parse().map_err(|error| { + Error::msg(format!( + "{BYPASS_PORT_ENV} is {bypass_port:?}, not a port: {error}" + )) + })?; + let log = required(LOG_ENV)?.into(); + Ok(Self { + url, + ca_cert, + bypass_port, + log, + }) + } + + /// The proxy URL the gateway uses. + pub fn url(&self) -> &str { + &self.url + } + + /// The path of the proxy's CA certificate, in PEM. + pub fn ca_cert(&self) -> &Path { + &self.ca_cert + } + + /// The destination port the gateway's `no_proxy` sends direct. + pub fn bypass_port(&self) -> u16 { + self.bypass_port + } + + /// The end of the log now. + pub fn log_cursor(&self) -> Result { + let metadata = std::fs::metadata(&self.log) + .map_err(|error| Error::msg(format!("read {}: {error}", self.log.display())))?; + Ok(LogCursor(metadata.len())) + } + + /// The CONNECTs logged after `cursor`, oldest first. A line the proxy is + /// still writing is left out. + pub fn connects_since(&self, cursor: &LogCursor) -> Result> { + let read = + |error: std::io::Error| Error::msg(format!("read {}: {error}", self.log.display())); + let mut file = std::fs::File::open(&self.log).map_err(read)?; + let len = file.metadata().map_err(read)?.len(); + testkit::ensure!( + len >= cursor.0, + "{} is {len} bytes, shorter than the cursor at {}; was it truncated?", + self.log.display(), + cursor.0 + ); + file.seek(SeekFrom::Start(cursor.0)).map_err(read)?; + let mut text = String::new(); + file.read_to_string(&mut text).map_err(read)?; + parse(&text).map_err(|error| Error::msg(format!("{}: {error}", self.log.display()))) + } + + /// The CONNECTs to `port` logged after `cursor`. + pub fn connects_to(&self, cursor: &LogCursor, port: u16) -> Result> { + Ok(self + .connects_since(cursor)? + .into_iter() + .filter(|connect| connect.port() == Some(port)) + .collect()) + } +} + +/// Parse the complete lines of `text`. +fn parse(text: &str) -> std::result::Result, String> { + let complete = text.rfind('\n').map_or("", |end| &text[..end]); + complete + .lines() + .filter(|line| !line.trim().is_empty()) + .map(|line| { + serde_json::from_str(line).map_err(|error| format!("bad log line {line:?}: {error}")) + }) + .collect() +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + use std::io::Write as _; + + use super::*; + + fn vars(pairs: &[(&str, &str)]) -> impl Fn(&str) -> Option { + let map: HashMap = pairs + .iter() + .map(|(key, value)| ((*key).to_string(), (*value).to_string())) + .collect(); + move |name| map.get(name).cloned() + } + + const ALL: &[(&str, &str)] = &[ + (URL_ENV, "https://host.openshell.internal:3128"), + (CA_CERT_ENV, "/etc/ca.pem"), + (BYPASS_PORT_ENV, "39443"), + (LOG_ENV, "/var/log/proxy.jsonl"), + ]; + + #[test] + fn reads_every_variable() { + let proxy = CorporateProxy::from_vars(vars(ALL)).unwrap(); + assert_eq!(proxy.url(), "https://host.openshell.internal:3128"); + assert_eq!(proxy.ca_cert(), Path::new("/etc/ca.pem")); + assert_eq!(proxy.bypass_port(), 39443); + } + + #[test] + fn missing_or_empty_variables_fail_naming_them() { + for name in [URL_ENV, CA_CERT_ENV, BYPASS_PORT_ENV, LOG_ENV] { + let without: Vec<_> = ALL + .iter() + .copied() + .filter(|(key, _)| *key != name) + .collect(); + let error = CorporateProxy::from_vars(vars(&without)).unwrap_err(); + assert!(error.to_string().contains(name), "{error}"); + + let mut empty = without.clone(); + empty.push((name, " ")); + let error = CorporateProxy::from_vars(vars(&empty)).unwrap_err(); + assert!(error.to_string().contains(name), "{error}"); + } + } + + #[test] + fn bad_bypass_port_fails() { + let mut pairs = ALL.to_vec(); + pairs.retain(|(key, _)| *key != BYPASS_PORT_ENV); + pairs.push((BYPASS_PORT_ENV, "70000")); + let error = CorporateProxy::from_vars(vars(&pairs)).unwrap_err(); + assert!(error.to_string().contains("not a port"), "{error}"); + } + + #[test] + fn connect_target_parts() { + let connect = |target: &str| Connect { + target: target.into(), + auth: Auth::Ok, + result: Outcome::Intercepted, + error: None, + }; + assert_eq!(connect("10.0.2.15:443").host(), "10.0.2.15"); + assert_eq!(connect("10.0.2.15:443").port(), Some(443)); + assert_eq!(connect("[fd00::1]:8443").host(), "fd00::1"); + assert_eq!(connect("[fd00::1]:8443").port(), Some(8443)); + assert_eq!(connect("nonsense").port(), None); + } + + #[test] + fn reads_only_complete_lines_after_the_cursor() { + let mut log = tempfile::NamedTempFile::new().unwrap(); + writeln!( + log, + r#"{{"time": 1.0, "target": "10.0.0.1:1", "auth": "missing", "result": "rejected"}}"# + ) + .unwrap(); + let proxy = CorporateProxy { + log: log.path().to_owned(), + ..CorporateProxy::from_vars(vars(ALL)).unwrap() + }; + let cursor = proxy.log_cursor().unwrap(); + writeln!( + log, + r#"{{"time": 2.0, "target": "10.0.0.1:443", "auth": "ok", "result": "intercepted"}}"# + ) + .unwrap(); + writeln!( + log, + r#"{{"time": 3.0, "target": "10.0.0.1:444", "auth": "ok", "result": "dial_failed", "error": "refused"}}"# + ) + .unwrap(); + write!(log, r#"{{"time": 4.0, "target": "10.0"#).unwrap(); + log.flush().unwrap(); + + let connects = proxy.connects_since(&cursor).unwrap(); + assert_eq!( + connects, + [ + Connect { + target: "10.0.0.1:443".into(), + auth: Auth::Ok, + result: Outcome::Intercepted, + error: None, + }, + Connect { + target: "10.0.0.1:444".into(), + auth: Auth::Ok, + result: Outcome::DialFailed, + error: Some("refused".into()), + }, + ] + ); + assert_eq!(proxy.connects_to(&cursor, 443).unwrap(), connects[..1]); + } + + #[test] + fn bad_lines_fail() { + let error = parse("not json\n").unwrap_err(); + assert!(error.contains("not json"), "{error}"); + } +} From ff65c346ba14d6047bc35a26fbbc870db59f6a94 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 19 Nov 2026 16:39:36 -0800 Subject: [PATCH 07/12] test(support): check whether a named Podman secret exists Signed-off-by: Matthew Grossman --- tests/support/container-engine/src/lib.rs | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/tests/support/container-engine/src/lib.rs b/tests/support/container-engine/src/lib.rs index 7d47f2d33f..922cb93042 100644 --- a/tests/support/container-engine/src/lib.rs +++ b/tests/support/container-engine/src/lib.rs @@ -146,6 +146,15 @@ impl Engine { .await } + /// Whether a secret named exactly `name` exists. Podman only: Docker + /// lists secrets only in swarm mode. + pub async fn secret_exists(&self, name: &str) -> Result { + let names = self + .output(["secret", "ls", "--format", "{{.Name}}"]) + .await?; + Ok(names.lines().any(|line| line.trim() == name)) + } + /// Run the engine with `args` and return its trimmed standard output. /// A non-zero exit is an error that includes the engine's output. pub async fn output(&self, args: I) -> Result From 75a7586169e5bd4ab8f3a959d28276739149b2c7 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 19 Nov 2026 16:48:00 -0800 Subject: [PATCH 08/12] test(support): publish the corporate proxy bypass port as a _LISTEN variable Follow the environment contract's OPENSHELL_TEST__LISTEN name for an address the environment picks and a test binds. Signed-off-by: Matthew Grossman --- tests/support/corporate-proxy/src/lib.rs | 16 ++++++++-------- .../ansible/playbooks/corporate-proxy.yaml | 2 +- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/tests/support/corporate-proxy/src/lib.rs b/tests/support/corporate-proxy/src/lib.rs index 271d87becc..fc0eff575b 100644 --- a/tests/support/corporate-proxy/src/lib.rs +++ b/tests/support/corporate-proxy/src/lib.rs @@ -14,7 +14,7 @@ //! |---|---| //! | `OPENSHELL_TEST_CORPORATE_PROXY_URL` | The proxy URL the gateway uses, e.g. `https://host.openshell.internal:3128` | //! | `OPENSHELL_TEST_CORPORATE_PROXY_CA_CERT` | The path of the proxy's CA certificate, in PEM | -//! | `OPENSHELL_TEST_CORPORATE_PROXY_BYPASS_PORT` | The destination port the gateway's `no_proxy` sends direct | +//! | `OPENSHELL_TEST_CORPORATE_PROXY_BYPASS_LISTEN` | The port a test listens on for a destination that the gateway's `no_proxy` sends direct | //! | `OPENSHELL_TEST_CORPORATE_PROXY_LOG` | The path of the proxy's log, readable by the tests | //! //! Under `openshell-test-runner`, the tmachine `-corporate-proxy` @@ -43,7 +43,7 @@ use testkit::{Error, Result}; pub const URL_ENV: &str = "OPENSHELL_TEST_CORPORATE_PROXY_URL"; pub const CA_CERT_ENV: &str = "OPENSHELL_TEST_CORPORATE_PROXY_CA_CERT"; -pub const BYPASS_PORT_ENV: &str = "OPENSHELL_TEST_CORPORATE_PROXY_BYPASS_PORT"; +pub const BYPASS_LISTEN_ENV: &str = "OPENSHELL_TEST_CORPORATE_PROXY_BYPASS_LISTEN"; pub const LOG_ENV: &str = "OPENSHELL_TEST_CORPORATE_PROXY_LOG"; /// How to reach the environment's corporate proxy and read its log. @@ -149,10 +149,10 @@ impl CorporateProxy { }; let url = required(URL_ENV)?; let ca_cert = required(CA_CERT_ENV)?.into(); - let bypass_port = required(BYPASS_PORT_ENV)?; + let bypass_port = required(BYPASS_LISTEN_ENV)?; let bypass_port = bypass_port.parse().map_err(|error| { Error::msg(format!( - "{BYPASS_PORT_ENV} is {bypass_port:?}, not a port: {error}" + "{BYPASS_LISTEN_ENV} is {bypass_port:?}, not a port: {error}" )) })?; let log = required(LOG_ENV)?.into(); @@ -245,7 +245,7 @@ mod tests { const ALL: &[(&str, &str)] = &[ (URL_ENV, "https://host.openshell.internal:3128"), (CA_CERT_ENV, "/etc/ca.pem"), - (BYPASS_PORT_ENV, "39443"), + (BYPASS_LISTEN_ENV, "39443"), (LOG_ENV, "/var/log/proxy.jsonl"), ]; @@ -259,7 +259,7 @@ mod tests { #[test] fn missing_or_empty_variables_fail_naming_them() { - for name in [URL_ENV, CA_CERT_ENV, BYPASS_PORT_ENV, LOG_ENV] { + for name in [URL_ENV, CA_CERT_ENV, BYPASS_LISTEN_ENV, LOG_ENV] { let without: Vec<_> = ALL .iter() .copied() @@ -278,8 +278,8 @@ mod tests { #[test] fn bad_bypass_port_fails() { let mut pairs = ALL.to_vec(); - pairs.retain(|(key, _)| *key != BYPASS_PORT_ENV); - pairs.push((BYPASS_PORT_ENV, "70000")); + pairs.retain(|(key, _)| *key != BYPASS_LISTEN_ENV); + pairs.push((BYPASS_LISTEN_ENV, "70000")); let error = CorporateProxy::from_vars(vars(&pairs)).unwrap_err(); assert!(error.to_string().contains("not a port"), "{error}"); } diff --git a/tests/tmachine/ansible/playbooks/corporate-proxy.yaml b/tests/tmachine/ansible/playbooks/corporate-proxy.yaml index 6cb2431f1d..13bb529900 100644 --- a/tests/tmachine/ansible/playbooks/corporate-proxy.yaml +++ b/tests/tmachine/ansible/playbooks/corporate-proxy.yaml @@ -21,5 +21,5 @@ test_env_vars: OPENSHELL_TEST_CORPORATE_PROXY_URL: "https://{{ corporate_proxy_host }}:{{ corporate_proxy_port }}" OPENSHELL_TEST_CORPORATE_PROXY_CA_CERT: "{{ corporate_proxy_config_dir }}/ca.pem" - OPENSHELL_TEST_CORPORATE_PROXY_BYPASS_PORT: "{{ corporate_proxy_bypass_port }}" + OPENSHELL_TEST_CORPORATE_PROXY_BYPASS_LISTEN: "{{ corporate_proxy_bypass_port }}" OPENSHELL_TEST_CORPORATE_PROXY_LOG: "{{ corporate_proxy_log }}" From d5425121097ea72d741bd8d242118f7d382380c4 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 19 Nov 2026 16:48:00 -0800 Subject: [PATCH 09/12] test(features): port the corporate-proxy e2e tests to testkit Check sandbox egress through the environment's corporate proxy: approved requests are proxied with credentials and a validated IP target, denied ones never reach the proxy, the no_proxy port is dialed direct, and on Podman the per-sandbox credentials secret goes away with the sandbox. A second test checks that the supervisor and the workload both trust the intercepting proxy's CA. Signed-off-by: Matthew Grossman --- tests/suites/features/Cargo.lock | 20 ++ tests/suites/features/Cargo.toml | 2 +- .../features/corporate-proxy/Cargo.toml | 19 + .../corporate-proxy/tests/corporate_proxy.rs | 334 ++++++++++++++++++ 4 files changed, 374 insertions(+), 1 deletion(-) create mode 100644 tests/suites/features/corporate-proxy/Cargo.toml create mode 100644 tests/suites/features/corporate-proxy/tests/corporate_proxy.rs diff --git a/tests/suites/features/Cargo.lock b/tests/suites/features/Cargo.lock index 2f966d222c..9cc9c36d5f 100644 --- a/tests/suites/features/Cargo.lock +++ b/tests/suites/features/Cargo.lock @@ -583,6 +583,26 @@ version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f8f80099a98041a3d1622845c271458a2d73e688351bf3cb999266764b81d48" +[[package]] +name = "corporate-proxy" +version = "0.0.0" +dependencies = [ + "container-engine-support", + "corporate-proxy-support", + "host-server-support", + "openshell-testkit", + "serde_json", +] + +[[package]] +name = "corporate-proxy-support" +version = "0.0.0" +dependencies = [ + "openshell-testkit", + "serde", + "serde_json", +] + [[package]] name = "cpufeatures" version = "0.2.17" diff --git a/tests/suites/features/Cargo.toml b/tests/suites/features/Cargo.toml index 7c7fe40ae7..d1e0903c2d 100644 --- a/tests/suites/features/Cargo.toml +++ b/tests/suites/features/Cargo.toml @@ -3,4 +3,4 @@ [workspace] resolver = "2" -members = ["cdi", "connect-ssh-supervision", "docker-updates", "host-gateway", "keycloak-provider-refresh", "kubernetes-updates", "oci-image", "routable-host-ip"] +members = ["cdi", "connect-ssh-supervision", "corporate-proxy", "docker-updates", "host-gateway", "keycloak-provider-refresh", "kubernetes-updates", "oci-image", "routable-host-ip"] diff --git a/tests/suites/features/corporate-proxy/Cargo.toml b/tests/suites/features/corporate-proxy/Cargo.toml new file mode 100644 index 0000000000..ebb77379c1 --- /dev/null +++ b/tests/suites/features/corporate-proxy/Cargo.toml @@ -0,0 +1,19 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "corporate-proxy" +description = "Features suite: sandbox egress through an authenticated, TLS-intercepting corporate proxy" +version = "0.0.0" +edition = "2024" +rust-version = "1.94" +license = "Apache-2.0" +repository = "https://github.com/NVIDIA/OpenShell" +publish = false + +[dependencies] +container-engine-support = { path = "../../../support/container-engine" } +corporate-proxy-support = { path = "../../../support/corporate-proxy" } +host-server-support = { path = "../../../support/host-server" } +openshell-testkit = { path = "../../../testkit/testkit" } +serde_json = "1" diff --git a/tests/suites/features/corporate-proxy/tests/corporate_proxy.rs b/tests/suites/features/corporate-proxy/tests/corporate_proxy.rs new file mode 100644 index 0000000000..9c00269850 --- /dev/null +++ b/tests/suites/features/corporate-proxy/tests/corporate_proxy.rs @@ -0,0 +1,334 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Sandbox egress through a corporate proxy that the gateway's driver is +//! configured with. +//! +//! The environment runs one proxy (see `corporate-proxy-support`). It serves +//! TLS, requires Basic auth and intercepts the TLS in every tunnel, re-signing +//! it with its own CA, and logs each CONNECT. The gateway's `no_proxy` sends +//! one destination port, the bypass port, direct. +//! +//! The upstreams are TLS servers on the test host with certificates from a +//! [`TestCa`] that nothing trusts, so only the proxy's re-signed certificate +//! can satisfy a client that verifies. +//! +//! Ported from `e2e/rust/tests/docker_corporate_proxy.rs` and +//! `podman_corporate_proxy.rs`, which started their own proxy per test and +//! rewrote the gateway's configuration. + +use std::time::Duration; + +use container_engine_support::Engine; +use corporate_proxy_support::{Auth, Connect, CorporateProxy, Outcome}; +use host_server_support::{HttpRecorder, Identity, TEST_HOST, TestCa}; +use testkit::policy::{self, Endpoint as _}; +use testkit::sdk::raw::proto; +use testkit::{Context, PYTHON_BINARIES, Poll, ResultExt as _, Sandbox}; + +/// Requests `https://:/` for each `::` +/// after `argv[2]` and prints one JSON object of results by name. With +/// `argv[1]` `verify`, the client verifies the server against the image's +/// trust store, which the driver adds the proxy's CA to; with `insecure`, it +/// doesn't verify. A failed request is retried, since a policy reload while +/// the sandbox starts can deny it once. +const CLIENT: &str = r#" +import json, ssl, sys, time, urllib.request + +mode, host = sys.argv[1], sys.argv[2] +if mode == "verify": + context = ssl.create_default_context() +else: + context = ssl._create_unverified_context() + +def fetch(port, attempts): + result = {"status": -1, "error": "not attempted"} + for attempt in range(attempts): + if attempt: + time.sleep(1) + try: + url = f"https://{host}:{port}/" + with urllib.request.urlopen(url, timeout=30, context=context) as response: + return {"status": response.status, "body": response.read().decode()} + except Exception as error: + result = {"status": -1, "error": f"{type(error).__name__}: {error}"} + return result + +results = {} +for spec in sys.argv[3:]: + name, port, attempts = spec.split(":") + results[name] = fetch(int(port), int(attempts)) +print(json.dumps(results)) +"#; + +/// The address ranges a sandbox may reach `host.openshell.internal` at, +/// across drivers. +const HOST_RANGES: &[&str] = &["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "fc00::/7"]; + +/// How long the driver may take to remove a deleted sandbox's secret. +const SECRET_REMOVAL_TIMEOUT: Duration = Duration::from_mins(1); + +/// A policy-approved request goes through the proxy with its credentials and +/// a validated IP as the CONNECT target. A denied one never reaches the +/// proxy, and one to the bypass port is dialed direct. On Podman, the +/// credentials reach the supervisor in a per-sandbox secret that deleting +/// the sandbox removes. +/// +/// The workload's tunnels skip the supervisor's TLS inspection and don't +/// verify the server, so the proxy's interception doesn't matter here. +#[testkit::test] +async fn approved_egress_uses_authenticated_proxy(cx: &mut Context) -> testkit::Result { + let proxy = CorporateProxy::from_env()?; + let identity = TestCa::new()?.leaf(&[TEST_HOST])?; + let allowed = upstream(&identity, format!("allowed-{}", cx.run_id()), None).await?; + let denied = upstream(&identity, format!("denied-{}", cx.run_id()), None).await?; + let bypass = upstream( + &identity, + format!("bypass-{}", cx.run_id()), + Some(proxy.bypass_port()), + ) + .await?; + + let mut policy = policy::base(); + for port in [allowed.port, bypass.port] { + allow(&mut policy, port).tls_skip(); + } + let cursor = proxy.log_cursor()?; + let sandbox = python_sandbox(cx, "cpegr", policy).await?; + let results = fetch( + &sandbox, + "insecure", + &[ + ("allowed", allowed.port, 6), + ("bypass", bypass.port, 6), + ("denied", denied.port, 1), + ], + ) + .await?; + let connects = proxy.connects_since(&cursor)?; + let report = log_report(&connects); + + expect_body(&results, "allowed", &allowed, &report)?; + let to_allowed = connects_to(&connects, allowed.port); + testkit::ensure!( + to_allowed + .iter() + .any(|connect| connect.auth == Auth::Ok && connect.host() != TEST_HOST), + "the proxy saw no authenticated CONNECT to a validated IP on port {}:\n{report}", + allowed.port + ); + testkit::ensure!( + connects.iter().all(|connect| connect.auth == Auth::Ok), + "every CONNECT should carry the proxy's credentials:\n{report}" + ); + + expect_denied(&results, &denied, &connects, &report)?; + + expect_body(&results, "bypass", &bypass, &report)?; + testkit::ensure!( + connects_to(&connects, bypass.port).is_empty(), + "the bypass port {} is in no_proxy, so its requests must not reach the proxy:\n{report}", + bypass.port + ); + + if cx.compute_driver().await? == "podman" { + expect_secret_removed_with(&sandbox).await?; + } + Ok(()) +} + +/// The proxy's CA, which the driver adds to the sandbox's trust bundle, is +/// trusted both by the supervisor when it inspects TLS and by the workload +/// when its TLS goes through the tunnel unchanged. +#[testkit::test] +async fn intercepting_proxy_ca_is_trusted_by_supervisor_and_workload( + cx: &mut Context, +) -> testkit::Result { + let proxy = CorporateProxy::from_env()?; + let identity = TestCa::new()?.leaf(&[TEST_HOST])?; + let inspected = upstream(&identity, format!("inspected-{}", cx.run_id()), None).await?; + let raw = upstream(&identity, format!("raw-{}", cx.run_id()), None).await?; + let denied = upstream(&identity, format!("denied-{}", cx.run_id()), None).await?; + + let mut policy = policy::base(); + allow(&mut policy, inspected.port); + allow(&mut policy, raw.port).tls_skip(); + let cursor = proxy.log_cursor()?; + let sandbox = python_sandbox(cx, "cpca", policy).await?; + let results = fetch( + &sandbox, + "verify", + &[ + ("inspected", inspected.port, 6), + ("raw", raw.port, 6), + ("denied", denied.port, 1), + ], + ) + .await?; + let connects = proxy.connects_since(&cursor)?; + let report = log_report(&connects); + + expect_body(&results, "inspected", &inspected, &report) + .context("the supervisor should trust the proxy's re-signed certificate")?; + expect_body(&results, "raw", &raw, &report) + .context("the workload should trust the proxy's re-signed certificate")?; + for server in [&inspected, &raw] { + testkit::ensure!( + connects_to(&connects, server.port) + .iter() + .any(|connect| connect.result == Outcome::Intercepted), + "the proxy intercepted no tunnel to port {}:\n{report}", + server.port + ); + } + expect_denied(&results, &denied, &connects, &report) +} + +/// A TLS server on the test host that answers every request with its +/// marker. +struct Upstream { + server: HttpRecorder, + marker: String, + port: u16, +} + +/// Start an [`Upstream`] answering with `marker`, on `port` or a random free +/// one. +async fn upstream( + identity: &Identity, + marker: String, + port: Option, +) -> testkit::Result { + let mut builder = HttpRecorder::builder().tls(identity); + if let Some(port) = port { + builder = builder.port(port); + } + let server = builder.start(marker.clone()).await?; + Ok(Upstream { + port: server.port, + server, + marker, + }) +} + +/// Let the image's Python reach `TEST_HOST` on `port`, under a network +/// policy of its own. +fn allow(policy: &mut proto::SandboxPolicy, port: u16) -> &mut proto::NetworkEndpoint { + let endpoint = policy::allow( + policy, + &format!("upstream_{port}"), + TEST_HOST, + port, + PYTHON_BINARIES, + ); + endpoint.allowed_ips = HOST_RANGES.iter().map(ToString::to_string).collect(); + endpoint +} + +async fn python_sandbox( + cx: &Context, + suffix: &str, + policy: proto::SandboxPolicy, +) -> testkit::Result { + cx.sandbox(suffix) + .image(testkit::python_image()) + .command(["sleep", "infinity"]) + .policy(policy) + .create_ready() + .await +} + +/// Run [`CLIENT`] in `sandbox` for each `(name, port, attempts)`. +async fn fetch( + sandbox: &Sandbox, + mode: &str, + targets: &[(&str, u16, u32)], +) -> testkit::Result { + let mut args = vec![mode.to_string(), TEST_HOST.to_string()]; + args.extend( + targets + .iter() + .map(|(name, port, attempts)| format!("{name}:{port}:{attempts}")), + ); + let args: Vec<&str> = args.iter().map(String::as_str).collect(); + sandbox.run_python(CLIENT, &args).await?.last_json() +} + +fn connects_to(connects: &[Connect], port: u16) -> Vec<&Connect> { + connects + .iter() + .filter(|connect| connect.port() == Some(port)) + .collect() +} + +fn log_report(connects: &[Connect]) -> String { + if connects.is_empty() { + return "the proxy logged no CONNECT".to_string(); + } + let lines: Vec = connects.iter().map(ToString::to_string).collect(); + format!("the proxy logged:\n{}", lines.join("\n")) +} + +/// Fail unless the request `name` got `upstream`'s response. +fn expect_body( + results: &serde_json::Value, + name: &str, + upstream: &Upstream, + report: &str, +) -> testkit::Result { + let result = &results[name]; + testkit::ensure!( + result["status"] == 200 && result["body"].as_str() == Some(upstream.marker.as_str()), + "the {name} request to port {} failed: {result}\n{report}", + upstream.port + ); + Ok(()) +} + +/// The policy denied the request to `denied`, so it neither reached the +/// proxy nor the server. +fn expect_denied( + results: &serde_json::Value, + denied: &Upstream, + connects: &[Connect], + report: &str, +) -> testkit::Result { + testkit::ensure!( + results["denied"]["status"] != 200 && denied.server.requests().is_empty(), + "the policy-denied request to port {} reached the server: {}", + denied.port, + results["denied"] + ); + testkit::ensure!( + connects_to(connects, denied.port).is_empty(), + "the policy-denied port {} must not reach the proxy:\n{report}", + denied.port + ); + Ok(()) +} + +/// Podman's driver gives the supervisor the proxy credentials in a +/// per-sandbox secret: it exists while the sandbox runs, and deleting the +/// sandbox removes it. +async fn expect_secret_removed_with(sandbox: &Sandbox) -> testkit::Result { + let engine = Engine::from_env()?; + let secret = format!("openshell-proxy-auth-{}", sandbox.id()); + testkit::ensure!( + engine.secret_exists(&secret).await?, + "Podman secret {secret} should hold the running sandbox's proxy credentials" + ); + sandbox.delete().await?; + testkit::poll_until( + &format!("removal of Podman secret {secret}"), + SECRET_REMOVAL_TIMEOUT, + Duration::from_secs(1), + async || { + Ok(if engine.secret_exists(&secret).await? { + Poll::Pending(format!("{secret} still exists")) + } else { + Poll::Ready(()) + }) + }, + ) + .await +} From 66f87d4169448d9787e038a5178a091d7f7c91b3 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 19 Nov 2026 16:50:27 -0800 Subject: [PATCH 10/12] test(e2e): run the corporate-proxy suite on Docker and Podman environments Add features/corporate-proxy rows for the rootful Docker and the rootful and rootless Podman corporate-proxy environments, and list the environment's OPENSHELL_TEST_CORPORATE_PROXY_* variables in the contract table. Signed-off-by: Matthew Grossman --- .github/workflows/e2e-suites.yml | 3 + e2e/rust/tests/docker_corporate_proxy.rs | 1133 ---------------------- e2e/rust/tests/podman_corporate_proxy.rs | 918 ------------------ tests/support/README.md | 4 + 4 files changed, 7 insertions(+), 2051 deletions(-) delete mode 100644 e2e/rust/tests/docker_corporate_proxy.rs delete mode 100644 e2e/rust/tests/podman_corporate_proxy.rs diff --git a/.github/workflows/e2e-suites.yml b/.github/workflows/e2e-suites.yml index 4936fd7952..7e88f430bc 100644 --- a/.github/workflows/e2e-suites.yml +++ b/.github/workflows/e2e-suites.yml @@ -95,6 +95,9 @@ jobs: {"environment":"ubuntu-docker-rootful-cdi","installer":"deb","family":"features","package":"cdi","gateway":"openshell","runner":"ubuntu-24.04"}, {"environment":"fedora-podman-rootful-cdi","installer":"rpm","family":"features","package":"cdi","gateway":"openshell","runner":"ubuntu-24.04"}, {"environment":"fedora-podman-rootless-cdi","installer":"rpm","family":"features","package":"cdi","gateway":"openshell","runner":"ubuntu-24.04"}, + {"environment":"ubuntu-docker-rootful-corporate-proxy","installer":"deb","family":"features","package":"corporate-proxy","gateway":"openshell","runner":"ubuntu-24.04"}, + {"environment":"fedora-podman-rootful-corporate-proxy","installer":"rpm","family":"features","package":"corporate-proxy","gateway":"openshell","runner":"ubuntu-24.04"}, + {"environment":"fedora-podman-rootless-corporate-proxy","installer":"rpm","family":"features","package":"corporate-proxy","gateway":"openshell","runner":"ubuntu-24.04"}, {"environment":"ubuntu-docker-rootful-keycloak","installer":"deb","family":"features","package":"keycloak-provider-refresh","gateway":"openshell","runner":"ubuntu-24.04"}, {"environment":"ubuntu-docker-rootful","installer":"deb","family":"features","package":"oci-image","gateway":"openshell","runner":"ubuntu-24.04"}, {"environment":"fedora-podman-rootful","installer":"rpm","family":"features","package":"oci-image","gateway":"openshell","runner":"ubuntu-24.04"}, diff --git a/e2e/rust/tests/docker_corporate_proxy.rs b/e2e/rust/tests/docker_corporate_proxy.rs deleted file mode 100644 index 6865ae80ff..0000000000 --- a/e2e/rust/tests/docker_corporate_proxy.rs +++ /dev/null @@ -1,1133 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -#![cfg(feature = "e2e-docker")] - -//! Cross-layer E2E coverage for corporate forward-proxy egress from Docker -//! sandboxes (issue #3545). -//! -//! The Docker counterpart of `podman_corporate_proxy.rs`. It drives the whole -//! chain end to end: -//! -//! gateway TOML → Docker driver config → companion supervisor archive → -//! supervisor CLI parsing → policy evaluation → proxied CONNECT -//! -//! and asserts the properties only a real run can establish: -//! -//! 1. A policy-approved HTTPS request reaches its destination *through* the -//! proxy, with a validated IP as the CONNECT target. -//! 2. A policy-denied destination never reaches the proxy at all. -//! 3. Credentials arrive through the supervisor-volume file — the proxy answers -//! 407 to an unauthenticated CONNECT, so a 200 proves delivery. -//! 4. A port-qualified `no_proxy` entry bypasses the proxy for that port only. -//! 5. An `https://` proxy works when its CA is supplied via `proxy_ca_bundle`. -//! 6. A TLS-intercepting proxy CA is trusted by both inspected supervisor -//! upstream TLS and a raw workload TLS connection. -//! -//! Fixtures run as host processes and are reached by the host-networked -//! supervisor. `host.openshell.internal` is normalized to the gateway host -//! address, including the CI job-container address when Docker runs on the -//! host daemon. - -use std::fmt::Write as _; -use std::io::Write as _; -use std::path::PathBuf; -use std::time::Duration; - -use openshell_e2e::harness::cli::wait_for_healthy; -use openshell_e2e::harness::gateway::ManagedGateway; -use openshell_e2e::harness::host_process::HostPythonFixture; -use openshell_e2e::harness::port::find_free_port; -use openshell_e2e::harness::sandbox::SandboxGuard; -use serial_test::serial; -use tempfile::NamedTempFile; - -/// The OpenShell alias for the gateway host. -const HOST_ALIAS: &str = "host.openshell.internal"; -const PROXY_USER: &str = "proxyuser"; -const PROXY_PASS: &str = "proxypass"; - -const ALLOWED_MARKER: &str = "docker-corp-proxy-e2e-allowed-upstream"; -const DENIED_MARKER: &str = "docker-corp-proxy-e2e-denied-upstream"; -const BYPASS_MARKER: &str = "docker-corp-proxy-e2e-bypass-upstream"; -const READY_MARKER: &str = "docker-corp-proxy-e2e-workload-done"; - -/// Ports the fixtures bind and the workload addresses them by. -struct FixturePorts { - proxy: u16, - allowed: u16, - denied: u16, - bypass: u16, -} - -impl FixturePorts { - fn pick() -> Self { - Self { - proxy: find_free_port(), - allowed: find_free_port(), - denied: find_free_port(), - bypass: find_free_port(), - } - } -} - -/// Python helper retained by the shared proxy fixture shape. -const HOST_REWRITE: &str = " -def dial_host(host): - return host -"; - -/// A forward proxy that requires Basic auth and logs every CONNECT it sees. -/// -/// The log lines are the test's evidence: they record the exact CONNECT target -/// (proving validated-IP form, and which destination ports were proxied at -/// all) and whether credentials arrived. -fn proxy_script(port: u16) -> String { - format!( - r#" -import base64, select, socket, threading -{HOST_REWRITE} -EXPECTED = 'Basic ' + base64.b64encode(b'{PROXY_USER}:{PROXY_PASS}').decode() - -def log(msg): - print(msg, flush=True) - -def read_head(conn): - data = b'' - while b'\r\n\r\n' not in data: - chunk = conn.recv(4096) - if not chunk: - return None - data += chunk - if len(data) > 65536: - return None - return data - -def pipe(a, b): - try: - while True: - ready, _, _ = select.select([a, b], [], []) - for sock in ready: - chunk = sock.recv(65536) - if not chunk: - return - (b if sock is a else a).sendall(chunk) - except OSError: - return - -def handle(conn): - try: - head = read_head(conn) - if head is None: - # Readiness probes connect and close without sending a request. - return - lines = head.decode('latin-1').split('\r\n') - parts = lines[0].split() - if len(parts) < 2 or parts[0].upper() != 'CONNECT': - log('NON_CONNECT %s' % lines[0]) - conn.sendall(b'HTTP/1.1 405 Method Not Allowed\r\nContent-Length: 0\r\n\r\n') - return - target = parts[1] - auth = None - for line in lines[1:]: - if line.lower().startswith('proxy-authorization:'): - auth = line.split(':', 1)[1].strip() - if auth != EXPECTED: - log('CONNECT %s auth=fail' % target) - conn.sendall(b'HTTP/1.1 407 Proxy Authentication Required\r\n' - b'Proxy-Authenticate: Basic realm="corp"\r\n' - b'Content-Length: 0\r\n\r\n') - return - host, _, port = target.rpartition(':') - host = host.strip('[]') - try: - upstream = socket.create_connection((dial_host(host), int(port)), timeout=10) - except OSError: - log('CONNECT %s auth=ok dial=fail' % target) - conn.sendall(b'HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\n\r\n') - return - log('CONNECT %s auth=ok' % target) - conn.sendall(b'HTTP/1.1 200 Connection Established\r\n\r\n') - pipe(conn, upstream) - upstream.close() - except OSError: - pass - finally: - conn.close() - -server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) -server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) -server.bind(('0.0.0.0', {port})) -server.listen(64) -log('proxy-listening') -while True: - client, _ = server.accept() - threading.Thread(target=handle, args=(client,), daemon=True).start() -"# - ) -} - -// Delimits the proxy's CA certificate in its output, so the test can recover it -// and hand it back as the corporate CA bundle. -const CA_BEGIN: &str = "---PROXY-CA-BEGIN---"; -const CA_END: &str = "---PROXY-CA-END---"; - -/// A forward proxy that terminates TLS with a CA-signed certificate and logs -/// every CONNECT it sees. -/// -/// It mints a corporate CA and a listener leaf signed by it (SAN = the host -/// alias, which is the name the supervisor uses for SNI), serves the leaf, and -/// prints the CA between [`CA_BEGIN`]/[`CA_END`] so the test can trust it via -/// `proxy_ca_bundle`. The listener certificate must be a leaf: rustls rejects a -/// `CA:TRUE` certificate presented as an end-entity certificate. No Basic auth -/// here — this test isolates the `https://` proxy plus corporate-CA path. -/// Mint a corporate CA plus a listener leaf signed by it, and print the CA. -/// -/// Split out of [`tls_proxy_script`] to keep that function within clippy's -/// line budget. The listener certificate must be a leaf: rustls rejects a -/// `CA:TRUE` certificate presented as an end-entity certificate, so a single -/// self-signed `openssl req -x509` certificate cannot serve as both the -/// anchor and the listener identity. Signing a leaf also matches what a real -/// intercepting proxy does. -fn tls_proxy_pki_preamble() -> String { - format!( - r" -workdir = tempfile.mkdtemp() -ca_key = os.path.join(workdir, 'ca-key.pem') -ca_crt = os.path.join(workdir, 'ca.pem') -leaf_key = os.path.join(workdir, 'leaf-key.pem') -leaf_csr = os.path.join(workdir, 'leaf.csr') -leaf_crt = os.path.join(workdir, 'leaf.pem') -chain = os.path.join(workdir, 'chain.pem') -ext = os.path.join(workdir, 'leaf.ext') - -def run(*args): - subprocess.run(args, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) - -run('openssl', 'req', '-x509', '-newkey', 'rsa:2048', '-nodes', - '-keyout', ca_key, '-out', ca_crt, '-days', '1', - '-subj', '/CN=docker-corp-proxy-e2e-ca', - '-addext', 'basicConstraints=critical,CA:TRUE', - '-addext', 'keyUsage=critical,keyCertSign,cRLSign') - -with open(ext, 'w') as fh: - fh.write('basicConstraints=critical,CA:FALSE\n' - 'subjectAltName=DNS:{HOST_ALIAS}\n' - 'extendedKeyUsage=serverAuth\n' - 'keyUsage=critical,digitalSignature,keyEncipherment\n') -run('openssl', 'req', '-newkey', 'rsa:2048', '-nodes', - '-keyout', leaf_key, '-out', leaf_csr, '-subj', '/CN={HOST_ALIAS}') -run('openssl', 'x509', '-req', '-in', leaf_csr, '-CA', ca_crt, '-CAkey', ca_key, - '-CAcreateserial', '-out', leaf_crt, '-days', '1', '-extfile', ext) - -with open(chain, 'w') as out: - for part in (leaf_crt, ca_crt): - with open(part) as fh: - out.write(fh.read()) - -with open(ca_crt) as fh: - print('{CA_BEGIN}\n' + fh.read() + '{CA_END}', flush=True) -" - ) -} - -fn tls_proxy_script(port: u16) -> String { - let pki = tls_proxy_pki_preamble(); - format!( - r" -import os, select, socket, ssl, subprocess, tempfile, threading -{HOST_REWRITE} -{pki} - -def log(msg): - print(msg, flush=True) - -def read_head(conn): - data = b'' - while b'\r\n\r\n' not in data: - chunk = conn.recv(4096) - if not chunk: - return None - data += chunk - if len(data) > 65536: - return None - return data - -def pipe(a, b): - try: - while True: - ready, _, _ = select.select([a, b], [], []) - for sock in ready: - chunk = sock.recv(65536) - if not chunk: - return - (b if sock is a else a).sendall(chunk) - except OSError: - return - -def handle(conn): - try: - head = read_head(conn) - if head is None: - return - lines = head.decode('latin-1').split('\r\n') - parts = lines[0].split() - if len(parts) < 2 or parts[0].upper() != 'CONNECT': - log('NON_CONNECT %s' % lines[0]) - conn.sendall(b'HTTP/1.1 405 Method Not Allowed\r\nContent-Length: 0\r\n\r\n') - return - target = parts[1] - host, _, port = target.rpartition(':') - host = host.strip('[]') - try: - upstream = socket.create_connection((dial_host(host), int(port)), timeout=10) - except OSError: - log('CONNECT %s dial=fail' % target) - conn.sendall(b'HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\n\r\n') - return - log('CONNECT %s ok' % target) - conn.sendall(b'HTTP/1.1 200 Connection Established\r\n\r\n') - pipe(conn, upstream) - upstream.close() - except OSError: - pass - finally: - conn.close() - -ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) -ctx.load_cert_chain(chain, leaf_key) -server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) -server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) -server.bind(('0.0.0.0', {port})) -server.listen(64) -log('tls-proxy-listening') -while True: - raw, _ = server.accept() - try: - conn = ctx.wrap_socket(raw, server_side=True) - except OSError: - # Readiness probes open a bare TCP connection; the failed handshake is - # expected and must not spam the log. - raw.close() - continue - threading.Thread(target=handle, args=(conn,), daemon=True).start() -" - ) -} - -/// A plain HTTP CONNECT proxy that intercepts tunneled destination TLS. -/// -/// The proxy presents a corporate-CA-signed certificate for -/// `host.openshell.internal`, then opens an unverified TLS connection to the -/// real fixture. This lets one test prove both consumers of the configured CA: -/// inspected traffic is verified by the supervisor, while `tls: skip` traffic -/// is verified directly by the workload through its combined trust bundle. -fn intercepting_proxy_script(port: u16) -> String { - let pki = tls_proxy_pki_preamble(); - format!( - r" -import select, socket, ssl, subprocess, tempfile, threading, os -{HOST_REWRITE} -{pki} - -def log(msg): - print(msg, flush=True) - -def read_head(conn): - data = b'' - while b'\r\n\r\n' not in data: - chunk = conn.recv(4096) - if not chunk: - return None - data += chunk - if len(data) > 65536: - return None - return data - -def relay_http(client, upstream): - # select() cannot safely drive SSLSocket application data: TLS control - # records can make the fd readable while recv() still waits for payload. - # This fixture serves one bodyless HTTP request per intercepted tunnel, so - # relay that exchange directly and let the upstream HTTP/1.0 close delimit - # the response. - request = read_head(client) - if request is None: - return - upstream.sendall(request) - while True: - chunk = upstream.recv(65536) - if not chunk: - return - client.sendall(chunk) - -server_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) -server_context.load_cert_chain(chain, leaf_key) -upstream_context = ssl._create_unverified_context() - -def handle(conn): - target = '' - try: - head = read_head(conn) - if head is None: - return - lines = head.decode('latin-1').split('\r\n') - parts = lines[0].split() - if len(parts) < 2 or parts[0].upper() != 'CONNECT': - conn.sendall(b'HTTP/1.1 405 Method Not Allowed\r\nContent-Length: 0\r\n\r\n') - return - target = parts[1] - host, _, target_port = target.rpartition(':') - host = host.strip('[]') - try: - upstream_raw = socket.create_connection( - (dial_host(host), int(target_port)), timeout=10) - upstream = upstream_context.wrap_socket( - upstream_raw, server_hostname='{HOST_ALIAS}') - except OSError as err: - log('CONNECT %s mitm=dial-fail error=%r' % (target, err)) - conn.sendall(b'HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\n\r\n') - return - conn.sendall(b'HTTP/1.1 200 Connection Established\r\n\r\n') - try: - client = server_context.wrap_socket(conn, server_side=True) - except OSError as err: - log('CONNECT %s mitm=client-tls-fail error=%r' % (target, err)) - return - log('CONNECT %s mitm=ok' % target) - relay_http(client, upstream) - upstream.close() - client.close() - except OSError as err: - log('CONNECT %s mitm=error error=%r' % (target, err)) - finally: - try: - conn.close() - except OSError: - pass - -server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) -server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) -server.bind(('0.0.0.0', {port})) -server.listen(64) -log('intercepting-proxy-listening') -while True: - conn, _ = server.accept() - threading.Thread(target=handle, args=(conn,), daemon=True).start() -" - ) -} - -/// Extract the proxy's CA certificate PEM from its output. -fn ca_cert_from_logs(logs: &str) -> Result { - let start = logs - .find(CA_BEGIN) - .ok_or_else(|| format!("proxy CA begin marker not found in logs:\n{logs}"))? - + CA_BEGIN.len(); - let end = logs[start..] - .find(CA_END) - .ok_or_else(|| format!("proxy CA end marker not found in logs:\n{logs}"))?; - Ok(logs[start..start + end].trim().to_string()) -} - -/// A TLS server with a self-signed certificate, serving one identifying marker. -/// -/// The workload uses an unverified TLS context, so the certificate only needs -/// to exist — but the handshake itself must be real, because it is what proves -/// bytes flowed end to end through the tunnel. -fn tls_upstream_script(marker: &str, port: u16) -> String { - format!( - r#" -import http.server, os, ssl, subprocess, tempfile - -workdir = tempfile.mkdtemp() -key = os.path.join(workdir, 'key.pem') -crt = os.path.join(workdir, 'cert.pem') -subprocess.run( - ['openssl', 'req', '-x509', '-newkey', 'rsa:2048', '-nodes', - '-keyout', key, '-out', crt, '-days', '1', '-subj', '/CN={HOST_ALIAS}'], - check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) - -class Handler(http.server.BaseHTTPRequestHandler): - def do_GET(self): - body = b'{{"upstream":"{marker}"}}' - self.send_response(200) - self.send_header('Content-Type', 'application/json') - self.send_header('Content-Length', str(len(body))) - self.end_headers() - self.wfile.write(body) - - def log_message(self, fmt, *args): - pass - -class Server(http.server.HTTPServer): - # Readiness probes open a bare TCP connection and close it; the failed - # handshake is expected and must not spam the log. - def handle_error(self, request, client_address): - pass - -ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) -ctx.load_cert_chain(crt, key) -server = Server(('0.0.0.0', {port}), Handler) -server.socket = ctx.wrap_socket(server.socket, server_side=True) -print('tls-upstream-listening', flush=True) -server.serve_forever() -"# - ) -} - -/// Workload: one approved HTTPS request through the proxy, one policy-denied, -/// and one that `no_proxy` should send direct. -/// -fn workload_script(ports: &FixturePorts) -> String { - let (allowed, denied, bypass) = (ports.allowed, ports.denied, ports.bypass); - format!( - r" -import json, ssl, time, urllib.request - -ctx = ssl._create_unverified_context() - -def fetch(url, retries): - last = {{'status': -1, 'error': 'not attempted'}} - for attempt in range(retries): - try: - with urllib.request.urlopen(url, timeout=30, context=ctx) as resp: - return {{'status': resp.status, 'body': resp.read().decode()}} - except Exception as err: - last = {{'status': -1, 'error': str(err)}} - time.sleep(1) - return last - -# The approved requests are retried: policy reload during sandbox startup can -# transiently surface as a 403 in the forward proxy. -print('ALLOWED_RESULT ' + json.dumps( - fetch('https://{HOST_ALIAS}:{allowed}/', 6)), flush=True) -print('BYPASS_RESULT ' + json.dumps( - fetch('https://{HOST_ALIAS}:{bypass}/', 6)), flush=True) -# The denied request must fail, so a single attempt is enough. -print('DENIED_RESULT ' + json.dumps( - fetch('https://{HOST_ALIAS}:{denied}/', 1)), flush=True) -print('{READY_MARKER}', flush=True) -" - ) -} - -/// Workload variant that uses the system/default trust store. -/// -/// The inspected endpoint proves the supervisor accepts the corporate -/// re-signing CA for upstream TLS. The `tls: skip` endpoint proves the same CA -/// reached the workload trust bundle because Python verifies it directly. -fn trusted_workload_script(ports: &FixturePorts) -> String { - let (inspected, raw, denied) = (ports.allowed, ports.bypass, ports.denied); - format!( - r" -import json, ssl, time, urllib.request - -ctx = ssl.create_default_context() - -def fetch(url, retries): - last = {{'status': -1, 'error': 'not attempted'}} - for attempt in range(retries): - try: - with urllib.request.urlopen(url, timeout=30, context=ctx) as resp: - return {{'status': resp.status, 'body': resp.read().decode()}} - except Exception as err: - last = {{'status': -1, 'error': str(err)}} - time.sleep(1) - return last - -print('INSPECTED_RESULT ' + json.dumps( - fetch('https://{HOST_ALIAS}:{inspected}/', 6)), flush=True) -print('RAW_RESULT ' + json.dumps( - fetch('https://{HOST_ALIAS}:{raw}/', 6)), flush=True) -print('DENIED_RESULT ' + json.dumps( - fetch('https://{HOST_ALIAS}:{denied}/', 1)), flush=True) -print('{READY_MARKER}', flush=True) -" - ) -} - -/// Policy allowing the proxied and the bypassed upstream, but not the denied -/// one. `tls: skip` keeps the tunnel raw so the workload's TLS session runs end -/// to end, which is what makes the proxied CONNECT path observable. -fn policy_yaml(ports: &FixturePorts) -> String { - let (allowed, bypass) = (ports.allowed, ports.bypass); - format!( - r#"version: 1 - -filesystem_policy: - include_workdir: true - read_only: - - /usr - - /lib - - /proc - - /dev/urandom - - /app - - /etc - - /var/log - read_write: - - /sandbox - - /tmp - - /dev/null - -landlock: - compatibility: best_effort - -process: - run_as_user: sandbox - run_as_group: sandbox - -network_policies: - docker_corporate_proxy_e2e: - name: docker_corporate_proxy_e2e - endpoints: - - host: {HOST_ALIAS} - port: {allowed} - tls: skip - enforcement: enforce - allowed_ips: - - "10.0.0.0/8" - - "172.0.0.0/8" - - "192.168.0.0/16" - - "fc00::/7" - - host: {HOST_ALIAS} - port: {bypass} - tls: skip - enforcement: enforce - allowed_ips: - - "10.0.0.0/8" - - "172.0.0.0/8" - - "192.168.0.0/16" - - "fc00::/7" - binaries: - - path: /usr/bin/curl - - path: /usr/bin/python* - - path: /usr/local/bin/python* - - path: /sandbox/.venv/bin/python* - - path: /sandbox/.uv/python/*/bin/python* -"# - ) -} - -/// Allow one auto-inspected TLS endpoint and one raw `tls: skip` endpoint. -fn interception_policy_yaml(ports: &FixturePorts) -> String { - let (inspected, raw) = (ports.allowed, ports.bypass); - format!( - r#"version: 1 - -filesystem_policy: - include_workdir: true - read_only: [/usr, /lib, /proc, /dev/urandom, /app, /etc, /var/log] - read_write: [/sandbox, /tmp, /dev/null] - -landlock: - compatibility: best_effort - -process: - run_as_user: sandbox - run_as_group: sandbox - -network_policies: - docker_corporate_proxy_interception_e2e: - name: docker_corporate_proxy_interception_e2e - endpoints: - - host: {HOST_ALIAS} - port: {inspected} - enforcement: enforce - allowed_ips: ["10.0.0.0/8", "172.0.0.0/8", "192.168.0.0/16", "fc00::/7"] - - host: {HOST_ALIAS} - port: {raw} - tls: skip - enforcement: enforce - allowed_ips: ["10.0.0.0/8", "172.0.0.0/8", "192.168.0.0/16", "fc00::/7"] - binaries: - - path: /usr/bin/python* - - path: /usr/local/bin/python* - - path: /sandbox/.venv/bin/python* - - path: /sandbox/.uv/python/*/bin/python* -"# - ) -} - -/// Inserts corporate-proxy keys into the harness-generated Docker table and -/// restores the original file when dropped. -struct GatewayProxyConfig { - config_path: PathBuf, - original: Vec, - restored: bool, -} - -fn insert_docker_driver_settings(config: &str, extra: &str) -> Result { - let table_header = "[openshell.drivers.docker]"; - let table_start = config - .find(table_header) - .ok_or_else(|| format!("gateway config has no {table_header} table"))?; - let table_body = table_start + table_header.len(); - let insert_at = config[table_body..] - .find("\n[") - .map_or(config.len(), |offset| table_body + offset); - - let mut updated = String::with_capacity(config.len() + extra.len() + 2); - updated.push_str(&config[..insert_at]); - if !updated.ends_with('\n') { - updated.push('\n'); - } - updated.push_str(extra); - if !extra.ends_with('\n') { - updated.push('\n'); - } - updated.push_str(&config[insert_at..]); - Ok(updated) -} - -#[test] -fn proxy_settings_are_inserted_before_nested_driver_tables() { - let config = r#"[openshell.drivers.docker] -socket_path = "/var/run/docker.sock" - -[openshell.drivers.docker.resource_admission] -enabled = true -"#; - - let updated = insert_docker_driver_settings(config, "https_proxy = \"http://proxy\"\n") - .expect("insert proxy settings"); - let proxy_position = updated - .find("https_proxy = \"http://proxy\"") - .expect("proxy setting is present"); - let nested_table_position = updated - .find("[openshell.drivers.docker.resource_admission]") - .expect("nested table is preserved"); - - assert!(proxy_position < nested_table_position); - assert!(updated.contains("enabled = true")); -} - -impl GatewayProxyConfig { - /// Locate the gateway's `--config` path from the wrapper's args file. - fn config_path_from_args() -> Result { - let args_file = std::env::var("OPENSHELL_E2E_GATEWAY_ARGS_FILE") - .map_err(|_| "OPENSHELL_E2E_GATEWAY_ARGS_FILE must be set".to_string())?; - let raw = std::fs::read(&args_file) - .map_err(|err| format!("read gateway args file '{args_file}': {err}"))?; - let args: Vec = raw - .split(|byte| *byte == 0) - .filter(|arg| !arg.is_empty()) - .map(|arg| String::from_utf8_lossy(arg).into_owned()) - .collect(); - args.iter() - .position(|arg| arg == "--config") - .and_then(|index| args.get(index + 1)) - .map(PathBuf::from) - .ok_or_else(|| format!("no --config argument in gateway args file '{args_file}'")) - } - - /// Insert raw TOML lines into `[openshell.drivers.docker]` and restart the - /// gateway, without waiting for it to become healthy. - /// - /// Used directly by the fail-closed case, which expects the gateway *not* - /// to come up. - fn apply_raw(extra: &str) -> Result { - let config_path = Self::config_path_from_args()?; - let original = std::fs::read(&config_path) - .map_err(|err| format!("read gateway config '{}': {err}", config_path.display()))?; - - let config = std::str::from_utf8(&original).map_err(|err| { - format!( - "gateway config '{}' is not UTF-8: {err}", - config_path.display() - ) - })?; - let updated = insert_docker_driver_settings(config, extra) - .map_err(|err| format!("gateway config '{}': {err}", config_path.display()))?; - std::fs::write(&config_path, updated.as_bytes()) - .map_err(|err| format!("write gateway config '{}': {err}", config_path.display()))?; - - let guard = Self { - config_path, - original, - restored: false, - }; - let gateway = ManagedGateway::from_env()? - .ok_or_else(|| "managed gateway metadata disappeared".to_string())?; - gateway.stop()?; - gateway.start()?; - Ok(guard) - } - - /// Point the Docker driver at a corporate proxy and wait for the gateway to - /// come back healthy. - async fn apply( - proxy_url: &str, - auth_file: Option<&str>, - ca_bundle: Option<&str>, - no_proxy: Option<&str>, - ) -> Result { - let mut extra = format!("https_proxy = \"{proxy_url}\"\n"); - if let Some(auth_file) = auth_file { - let _ = write!( - extra, - "proxy_auth_file = \"{auth_file}\"\nproxy_auth_allow_insecure = true\n" - ); - } - if let Some(ca_bundle) = ca_bundle { - let _ = writeln!(extra, "proxy_ca_bundle = \"{ca_bundle}\""); - } - if let Some(no_proxy) = no_proxy { - let _ = writeln!(extra, "no_proxy = \"{no_proxy}\""); - } - let guard = Self::apply_raw(&extra)?; - wait_for_healthy(Duration::from_secs(120)).await?; - Ok(guard) - } - - /// Restore the original config and restart the gateway. - async fn restore(&mut self) -> Result<(), String> { - if self.restored { - return Ok(()); - } - std::fs::write(&self.config_path, &self.original).map_err(|err| { - format!( - "restore gateway config '{}': {err}", - self.config_path.display() - ) - })?; - let gateway = ManagedGateway::from_env()? - .ok_or_else(|| "managed gateway metadata disappeared".to_string())?; - gateway.stop()?; - gateway.start()?; - wait_for_healthy(Duration::from_secs(120)).await?; - self.restored = true; - Ok(()) - } -} - -impl Drop for GatewayProxyConfig { - fn drop(&mut self) { - if self.restored { - return; - } - // Panic path: put the original config back and synchronously restart - // the gateway so later test binaries in this run inherit neither the - // proxy settings on disk nor the configuration still loaded in the - // running process. - let _ = std::fs::write(&self.config_path, &self.original); - if let Ok(Some(gateway)) = ManagedGateway::from_env() { - let _ = gateway.stop(); - let _ = gateway.start(); - } - } -} - -/// Skip unless this run owns a Docker gateway it can reconfigure. -fn should_run(label: &str) -> bool { - if std::env::var("OPENSHELL_E2E_DRIVER").as_deref() != Ok("docker") { - eprintln!("Skipping {label}: e2e driver is not docker"); - return false; - } - match ManagedGateway::from_env() { - Ok(Some(_)) => true, - Ok(None) => { - eprintln!("Skipping {label}: e2e gateway is not managed by this test run"); - false - } - Err(err) => panic!("load managed e2e gateway metadata: {err}"), - } -} - -/// Write a temp file and return its path as an owned `String`. -fn temp_file_with(contents: &str, label: &str) -> (NamedTempFile, String) { - let mut file = NamedTempFile::new().unwrap_or_else(|err| panic!("create {label}: {err}")); - file.write_all(contents.as_bytes()) - .unwrap_or_else(|err| panic!("write {label}: {err}")); - file.flush() - .unwrap_or_else(|err| panic!("flush {label}: {err}")); - let path = file - .path() - .to_str() - .unwrap_or_else(|| panic!("{label} path should be utf-8")) - .to_string(); - (file, path) -} - -/// Assert the workload's results and the proxy's own record of what it saw. -fn assert_proxied_egress(output: &str, proxy_logs: &str, ports: &FixturePorts) { - assert!( - output.contains(READY_MARKER), - "workload did not finish; output:\n{output}" - ); - - // The approved request succeeded and its body came from the upstream, - // proving bytes traversed the tunnel rather than the proxy short-circuiting. - assert!( - output.contains("ALLOWED_RESULT") && output.contains(ALLOWED_MARKER), - "approved HTTPS request should have reached the upstream through the proxy:\n{output}" - ); - - // The CONNECT target is a validated resolved address, not the hostname, - // and it arrived authenticated -- which is only possible if the credential - // staged into the overlay reached the supervisor. - assert!( - proxy_logs.lines().any(|line| { - line.starts_with("CONNECT ") - && line.contains(&format!(":{} auth=ok", ports.allowed)) - && !line.contains(HOST_ALIAS) - }), - "proxy should have seen an authenticated validated-IP CONNECT to the approved upstream:\n{proxy_logs}" - ); - assert!( - !proxy_logs.contains("auth=fail"), - "the proxy credential should have been sent on the first CONNECT:\n{proxy_logs}" - ); - - // The denied destination failed inside the sandbox, and never reached the - // proxy: policy stops it before the upstream dial. - assert!( - !proxy_logs.contains(&format!(":{}", ports.denied)), - "policy-denied destination must never reach the proxy:\n{proxy_logs}" - ); - assert!( - !output.contains(DENIED_MARKER), - "policy-denied upstream body must never reach the workload:\n{output}" - ); - - // The no_proxy entry is port-qualified, so this destination was dialed - // directly while the approved one above still went through the proxy. - assert!( - output.contains("BYPASS_RESULT") && output.contains(BYPASS_MARKER), - "no_proxy destination should have been reached by a direct dial:\n{output}" - ); - assert!( - !proxy_logs.contains(&format!(":{}", ports.bypass)), - "no_proxy destination must never reach the proxy:\n{proxy_logs}" - ); -} - -#[tokio::test] -#[serial(docker_corporate_proxy)] -async fn docker_corporate_proxy_routes_approved_tls_egress() { - if !should_run("corporate proxy test") { - return; - } - - let ports = FixturePorts::pick(); - - // ── Host fixtures, reached by the host supervisor ── - let proxy = HostPythonFixture::start(&proxy_script(ports.proxy), ports.proxy) - .await - .expect("start fake corporate proxy"); - let _allowed = HostPythonFixture::start( - &tls_upstream_script(ALLOWED_MARKER, ports.allowed), - ports.allowed, - ) - .await - .expect("start approved TLS upstream"); - let _denied = HostPythonFixture::start( - &tls_upstream_script(DENIED_MARKER, ports.denied), - ports.denied, - ) - .await - .expect("start denied TLS upstream"); - let _bypass = HostPythonFixture::start( - &tls_upstream_script(BYPASS_MARKER, ports.bypass), - ports.bypass, - ) - .await - .expect("start no_proxy TLS upstream"); - - // ── Point the Docker driver at the corporate proxy ──────────────────── - let (_auth_file, auth_path) = - temp_file_with(&format!("{PROXY_USER}:{PROXY_PASS}\n"), "proxy auth file"); - - let mut gateway_config = GatewayProxyConfig::apply( - &format!("http://{HOST_ALIAS}:{}", ports.proxy), - Some(&auth_path), - None, - // Port-qualified, so only this destination port bypasses the proxy. - Some(&format!("{HOST_ALIAS}:{}", ports.bypass)), - ) - .await - .expect("apply corporate proxy gateway config"); - - // ── Run the workload ────────────────────────────────────────────── - let (_policy, policy_path) = temp_file_with(&policy_yaml(&ports), "policy file"); - let script = workload_script(&ports); - let mut sandbox = - SandboxGuard::create(&["--policy", &policy_path, "--", "python3", "-c", &script]) - .await - .expect("create Docker sandbox behind the corporate proxy"); - - assert_proxied_egress( - &sandbox.create_output, - &proxy.logs().expect("read fake proxy logs"), - &ports, - ); - - sandbox.cleanup().await; - - gateway_config - .restore() - .await - .expect("restore gateway config"); -} - -#[tokio::test] -#[serial(docker_corporate_proxy)] -async fn docker_corporate_proxy_trusts_ca_bundle_for_https_proxy() { - if !should_run("https corporate proxy test") { - return; - } - - let ports = FixturePorts::pick(); - - let proxy = HostPythonFixture::start(&tls_proxy_script(ports.proxy), ports.proxy) - .await - .expect("start fake https corporate proxy"); - let _allowed = HostPythonFixture::start( - &tls_upstream_script(ALLOWED_MARKER, ports.allowed), - ports.allowed, - ) - .await - .expect("start approved TLS upstream"); - let _denied = HostPythonFixture::start( - &tls_upstream_script(DENIED_MARKER, ports.denied), - ports.denied, - ) - .await - .expect("start denied TLS upstream"); - let _bypass = HostPythonFixture::start( - &tls_upstream_script(BYPASS_MARKER, ports.bypass), - ports.bypass, - ) - .await - .expect("start second approved TLS upstream"); - - // The proxy prints its CA on startup; the supervisor must trust it to - // complete the TLS handshake with the proxy at all. - let ca_pem = ca_cert_from_logs(&proxy.logs().expect("read https proxy logs")) - .expect("recover corporate CA from proxy output"); - let (_ca_file, ca_path) = temp_file_with(&ca_pem, "corporate CA bundle"); - - let mut gateway_config = GatewayProxyConfig::apply( - &format!("https://{HOST_ALIAS}:{}", ports.proxy), - None, - Some(&ca_path), - None, - ) - .await - .expect("apply https corporate proxy gateway config"); - - let (_policy, policy_path) = temp_file_with(&policy_yaml(&ports), "policy file"); - let script = workload_script(&ports); - let mut sandbox = - SandboxGuard::create(&["--policy", &policy_path, "--", "python3", "-c", &script]) - .await - .expect("create Docker sandbox behind the https corporate proxy"); - - let proxy_logs = proxy.logs().expect("read https proxy logs"); - assert!( - sandbox.create_output.contains(ALLOWED_MARKER), - "approved upstream body missing -- egress did not complete through the https proxy:\n{}", - sandbox.create_output - ); - assert!( - proxy_logs.lines().any(|line| { - line.starts_with("CONNECT ") - && line.contains(&format!(":{} ok", ports.allowed)) - && !line.contains(HOST_ALIAS) - }), - "https proxy should have seen a validated-IP CONNECT to the approved upstream:\n{proxy_logs}" - ); - assert!( - !proxy_logs.contains(&format!(":{}", ports.denied)), - "policy-denied destination must never reach the https proxy:\n{proxy_logs}" - ); - - sandbox.cleanup().await; - - gateway_config - .restore() - .await - .expect("restore gateway config"); -} - -#[tokio::test] -#[serial(docker_corporate_proxy)] -async fn docker_corporate_proxy_trusts_intercepted_destination_tls() { - if !should_run("TLS-intercepting corporate proxy test") { - return; - } - - let ports = FixturePorts::pick(); - let proxy = HostPythonFixture::start(&intercepting_proxy_script(ports.proxy), ports.proxy) - .await - .expect("start TLS-intercepting corporate proxy"); - let _inspected = HostPythonFixture::start( - &tls_upstream_script(ALLOWED_MARKER, ports.allowed), - ports.allowed, - ) - .await - .expect("start inspected TLS upstream"); - let _denied = HostPythonFixture::start( - &tls_upstream_script(DENIED_MARKER, ports.denied), - ports.denied, - ) - .await - .expect("start denied TLS upstream"); - let _raw = HostPythonFixture::start( - &tls_upstream_script(BYPASS_MARKER, ports.bypass), - ports.bypass, - ) - .await - .expect("start raw workload TLS upstream"); - - let ca_pem = ca_cert_from_logs(&proxy.logs().expect("read intercepting proxy logs")) - .expect("recover intercepting proxy CA"); - let (_ca_file, ca_path) = temp_file_with(&ca_pem, "intercepting proxy CA bundle"); - let mut gateway_config = GatewayProxyConfig::apply( - &format!("http://{HOST_ALIAS}:{}", ports.proxy), - None, - Some(&ca_path), - None, - ) - .await - .expect("apply TLS-intercepting proxy configuration"); - - let (_policy, policy_path) = temp_file_with( - &interception_policy_yaml(&ports), - "interception policy file", - ); - let script = trusted_workload_script(&ports); - let mut sandbox = - SandboxGuard::create(&["--policy", &policy_path, "--", "python3", "-c", &script]) - .await - .expect("create Docker sandbox behind TLS-intercepting proxy"); - - let output = &sandbox.create_output; - let proxy_logs = proxy.logs().expect("read TLS-intercepting proxy logs"); - assert!( - output.contains("INSPECTED_RESULT") && output.contains(ALLOWED_MARKER), - "supervisor-inspected TLS should trust the corporate re-signing CA:\n\ - workload output:\n{output}\nproxy output:\n{proxy_logs}" - ); - assert!( - output.contains("RAW_RESULT") && output.contains(BYPASS_MARKER), - "raw workload TLS should trust the corporate CA through the combined bundle:\n{output}" - ); - assert!( - proxy_logs.contains(&format!(":{} mitm=ok", ports.allowed)), - "intercepting proxy should see inspected upstream TLS:\n{proxy_logs}" - ); - assert!( - proxy_logs.contains(&format!(":{} mitm=ok", ports.bypass)), - "intercepting proxy should see raw workload TLS:\n{proxy_logs}" - ); - assert!( - !proxy_logs.contains(&format!(":{}", ports.denied)) && !output.contains(DENIED_MARKER), - "policy-denied destination must not reach the intercepting proxy:\n{proxy_logs}\n{output}" - ); - - sandbox.cleanup().await; - gateway_config - .restore() - .await - .expect("restore gateway config"); -} diff --git a/e2e/rust/tests/podman_corporate_proxy.rs b/e2e/rust/tests/podman_corporate_proxy.rs deleted file mode 100644 index 36906d8ada..0000000000 --- a/e2e/rust/tests/podman_corporate_proxy.rs +++ /dev/null @@ -1,918 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -#![cfg(feature = "e2e-podman")] - -//! Cross-layer E2E coverage for corporate forward-proxy egress (issue #1792). -//! -//! Every other test for this feature builds a config struct or calls a CONNECT -//! helper directly, so none of them can catch a break in the wiring *between* -//! layers. This test drives the whole chain end to end: -//! -//! gateway TOML → Podman driver config → supervisor argv + secret mount → -//! supervisor CLI parsing → policy evaluation → proxied CONNECT -//! -//! against a fake authenticated forward proxy, and asserts the four properties -//! that only a real run can establish: -//! -//! 1. A policy-approved HTTPS request reaches its destination *through* the -//! proxy, and the CONNECT target is a validated IP rather than a hostname. -//! 2. A policy-denied destination never reaches the proxy at all. -//! 3. Credentials arrive via the mounted Podman secret — the proxy answers 407 -//! to an unauthenticated CONNECT, so a 200 response proves delivery. -//! 4. Deleting the sandbox removes the per-sandbox Podman secret. - -use std::io::Write as _; -use std::os::unix::fs::PermissionsExt as _; -use std::path::PathBuf; -use std::time::Duration; - -use openshell_e2e::harness::cli::wait_for_healthy; -use openshell_e2e::harness::container::{ContainerEngine, HostSupportContainer}; -use openshell_e2e::harness::gateway::ManagedGateway; -use openshell_e2e::harness::port::find_free_port; -use openshell_e2e::harness::sandbox::SandboxGuard; -use serial_test::serial; -use tempfile::NamedTempFile; - -const HOST_ALIAS: &str = "host.openshell.internal"; - -const PROXY_USER: &str = "proxyuser"; -const PROXY_PASS: &str = "proxypass"; - -const ALLOWED_MARKER: &str = "corp-proxy-e2e-allowed-upstream"; -const DENIED_MARKER: &str = "corp-proxy-e2e-denied-upstream"; -const READY_MARKER: &str = "corp-proxy-e2e-workload-done"; - -const SECRET_PREFIX: &str = "openshell-proxy-auth-"; - -/// A forward proxy that requires Basic auth and logs every CONNECT it sees. -/// -/// The log lines are the test's evidence: they record the exact CONNECT target -/// (proving validated-IP vs hostname form) and whether credentials arrived. -fn proxy_script(port: u16) -> String { - format!( - r#" -import base64, select, socket, threading - -EXPECTED = 'Basic ' + base64.b64encode(b'{PROXY_USER}:{PROXY_PASS}').decode() - -def log(msg): - print(msg, flush=True) - -def read_head(conn): - data = b'' - while b'\r\n\r\n' not in data: - chunk = conn.recv(4096) - if not chunk: - return None - data += chunk - if len(data) > 65536: - return None - return data - -def pipe(a, b): - try: - while True: - ready, _, _ = select.select([a, b], [], []) - for sock in ready: - chunk = sock.recv(65536) - if not chunk: - return - (b if sock is a else a).sendall(chunk) - except OSError: - return - -def handle(conn): - try: - head = read_head(conn) - if head is None: - # Readiness probes connect and close without sending a request. - return - lines = head.decode('latin-1').split('\r\n') - parts = lines[0].split() - if len(parts) < 2 or parts[0].upper() != 'CONNECT': - log('NON_CONNECT %s' % lines[0]) - conn.sendall(b'HTTP/1.1 405 Method Not Allowed\r\nContent-Length: 0\r\n\r\n') - return - target = parts[1] - auth = None - for line in lines[1:]: - if line.lower().startswith('proxy-authorization:'): - auth = line.split(':', 1)[1].strip() - if auth != EXPECTED: - log('CONNECT %s auth=fail' % target) - conn.sendall(b'HTTP/1.1 407 Proxy Authentication Required\r\n' - b'Proxy-Authenticate: Basic realm="corp"\r\n' - b'Content-Length: 0\r\n\r\n') - return - host, _, port = target.rpartition(':') - host = host.strip('[]') - try: - upstream = socket.create_connection((host, int(port)), timeout=10) - except OSError: - log('CONNECT %s auth=ok dial=fail' % target) - conn.sendall(b'HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\n\r\n') - return - log('CONNECT %s auth=ok' % target) - conn.sendall(b'HTTP/1.1 200 Connection Established\r\n\r\n') - pipe(conn, upstream) - upstream.close() - except OSError: - pass - finally: - conn.close() - -server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) -server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) -server.bind(('0.0.0.0', {port})) -server.listen(64) -log('proxy-listening') -while True: - client, _ = server.accept() - threading.Thread(target=handle, args=(client,), daemon=True).start() -"# - ) -} - -// Delimits the proxy's CA certificate in its stdout, so the test can recover -// it and hand it back as the corporate CA bundle. -const CA_BEGIN: &str = "---PROXY-CA-BEGIN---"; -const CA_END: &str = "---PROXY-CA-END---"; - -/// A forward proxy that terminates TLS with a CA-signed certificate and logs -/// every CONNECT it sees. -/// -/// The container generates a corporate CA and a separate listener leaf signed -/// by it (SAN = the proxy alias, which is the name the supervisor uses for -/// SNI), serves the leaf, and prints the CA between [`CA_BEGIN`]/[`CA_END`] so -/// the test can trust it via `proxy_ca_bundle`. The listener certificate must -/// be a leaf: rustls rejects a `CA:TRUE` certificate presented as an -/// end-entity certificate (`CaUsedAsEndEntity`), so a single self-signed -/// `openssl req -x509` certificate — which is a CA by default — cannot serve -/// as both the anchor and the listener identity. Signing a leaf also matches -/// what a real intercepting proxy does. No Basic auth: this test isolates the -/// `https://` proxy + corporate-CA path; credential delivery is covered by the -/// plaintext-proxy test. -fn tls_proxy_script(port: u16) -> String { - format!( - r" -import os, select, socket, ssl, subprocess, tempfile, threading - -workdir = tempfile.mkdtemp() -ca_key = os.path.join(workdir, 'ca-key.pem') -ca_crt = os.path.join(workdir, 'ca.pem') -leaf_key = os.path.join(workdir, 'leaf-key.pem') -leaf_csr = os.path.join(workdir, 'leaf.csr') -leaf_crt = os.path.join(workdir, 'leaf.pem') -chain = os.path.join(workdir, 'chain.pem') -ext = os.path.join(workdir, 'leaf.ext') - -def run(*args): - subprocess.run(args, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) - -run('openssl', 'req', '-x509', '-newkey', 'rsa:2048', '-nodes', - '-keyout', ca_key, '-out', ca_crt, '-days', '1', - '-subj', '/CN=corp-proxy-e2e-ca') - -with open(ext, 'w') as fh: - fh.write('basicConstraints=critical,CA:FALSE\n' - 'subjectAltName=DNS:{HOST_ALIAS}\n' - 'extendedKeyUsage=serverAuth\n') -run('openssl', 'req', '-newkey', 'rsa:2048', '-nodes', - '-keyout', leaf_key, '-out', leaf_csr, '-subj', '/CN={HOST_ALIAS}') -run('openssl', 'x509', '-req', '-in', leaf_csr, '-CA', ca_crt, '-CAkey', ca_key, - '-CAcreateserial', '-out', leaf_crt, '-days', '1', '-extfile', ext) - -with open(chain, 'w') as out: - for part in (leaf_crt, ca_crt): - with open(part) as fh: - out.write(fh.read()) - -with open(ca_crt) as fh: - print('{CA_BEGIN}\n' + fh.read() + '{CA_END}', flush=True) - -def log(msg): - print(msg, flush=True) - -def read_head(conn): - data = b'' - while b'\r\n\r\n' not in data: - chunk = conn.recv(4096) - if not chunk: - return None - data += chunk - if len(data) > 65536: - return None - return data - -def pipe(a, b): - try: - while True: - ready, _, _ = select.select([a, b], [], []) - for sock in ready: - chunk = sock.recv(65536) - if not chunk: - return - (b if sock is a else a).sendall(chunk) - except OSError: - return - -def handle(conn): - try: - head = read_head(conn) - if head is None: - return - lines = head.decode('latin-1').split('\r\n') - parts = lines[0].split() - if len(parts) < 2 or parts[0].upper() != 'CONNECT': - log('NON_CONNECT %s' % lines[0]) - conn.sendall(b'HTTP/1.1 405 Method Not Allowed\r\nContent-Length: 0\r\n\r\n') - return - target = parts[1] - host, _, port = target.rpartition(':') - host = host.strip('[]') - try: - upstream = socket.create_connection((host, int(port)), timeout=10) - except OSError: - log('CONNECT %s dial=fail' % target) - conn.sendall(b'HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\n\r\n') - return - log('CONNECT %s ok' % target) - conn.sendall(b'HTTP/1.1 200 Connection Established\r\n\r\n') - pipe(conn, upstream) - upstream.close() - except OSError: - pass - finally: - conn.close() - -ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) -ctx.load_cert_chain(chain, leaf_key) -server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) -server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) -server.bind(('0.0.0.0', {port})) -server.listen(64) -log('tls-proxy-listening') -while True: - raw, _ = server.accept() - try: - conn = ctx.wrap_socket(raw, server_side=True) - except OSError: - # Readiness probes open a bare TCP connection; the failed handshake is - # expected and must not spam the log. - raw.close() - continue - threading.Thread(target=handle, args=(conn,), daemon=True).start() -" - ) -} - -/// Extract the proxy's CA certificate PEM from its stdout. -fn ca_cert_from_logs(logs: &str) -> Result { - let start = logs - .find(CA_BEGIN) - .ok_or_else(|| format!("proxy CA begin marker not found in logs:\n{logs}"))? - + CA_BEGIN.len(); - let end = logs[start..] - .find(CA_END) - .ok_or_else(|| format!("proxy CA end marker not found in logs:\n{logs}"))?; - Ok(logs[start..start + end].trim().to_string()) -} - -/// A TLS server with a self-signed certificate, serving one identifying marker. -/// -/// The sandbox workload uses an unverified TLS context, so the certificate only -/// needs to exist — but the TLS handshake itself must be real, because it is -/// what proves bytes flowed end to end through the tunnel. -fn tls_upstream_script(common_name: &str, marker: &str, port: u16) -> String { - format!( - r#" -import http.server, os, ssl, subprocess, tempfile - -workdir = tempfile.mkdtemp() -key = os.path.join(workdir, 'key.pem') -crt = os.path.join(workdir, 'cert.pem') -subprocess.run( - ['openssl', 'req', '-x509', '-newkey', 'rsa:2048', '-nodes', - '-keyout', key, '-out', crt, '-days', '1', '-subj', '/CN={common_name}'], - check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) - -class Handler(http.server.BaseHTTPRequestHandler): - def do_GET(self): - body = b'{{"upstream":"{marker}"}}' - self.send_response(200) - self.send_header('Content-Type', 'application/json') - self.send_header('Content-Length', str(len(body))) - self.end_headers() - self.wfile.write(body) - - def log_message(self, fmt, *args): - pass - -class Server(http.server.HTTPServer): - # Readiness probes open a bare TCP connection and close it; the failed - # handshake is expected and must not spam the log. - def handle_error(self, request, client_address): - pass - -ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) -ctx.load_cert_chain(crt, key) -server = Server(('0.0.0.0', {port}), Handler) -server.socket = ctx.wrap_socket(server.socket, server_side=True) -print('tls-upstream-listening', flush=True) -server.serve_forever() -"# - ) -} - -/// Workload: one approved HTTPS request, one denied, then idle so the test can -/// inspect the live sandbox's Podman secret before deleting it. -fn workload_script(allowed_port: u16, denied_port: u16) -> String { - format!( - r" -import json, ssl, time, urllib.request - -ctx = ssl._create_unverified_context() - -def fetch(url, retries): - last = {{'status': -1, 'error': 'not attempted'}} - for attempt in range(retries): - try: - with urllib.request.urlopen(url, timeout=30, context=ctx) as resp: - return {{'status': resp.status, 'body': resp.read().decode()}} - except Exception as err: - last = {{'status': -1, 'error': str(err)}} - time.sleep(1) - return last - -# The approved request is retried: policy reload during sandbox startup can -# transiently surface as a 403 in the forward proxy. -print('ALLOWED_RESULT ' + json.dumps( - fetch('https://{HOST_ALIAS}:{allowed_port}/', 6)), flush=True) -# The denied request must fail, so a single attempt is enough. -print('DENIED_RESULT ' + json.dumps( - fetch('https://{HOST_ALIAS}:{denied_port}/', 1)), flush=True) -print('{READY_MARKER}', flush=True) -while True: - time.sleep(1) -" - ) -} - -/// Policy allowing only the approved upstream. `tls: skip` keeps the tunnel raw -/// so the workload's TLS session runs end to end to the upstream, which is what -/// makes the proxied CONNECT path observable. -fn policy_yaml(allowed_port: u16) -> String { - format!( - r#"version: 1 - -filesystem_policy: - include_workdir: true - read_only: - - /usr - - /lib - - /proc - - /dev/urandom - - /app - - /etc - - /var/log - read_write: - - /sandbox - - /tmp - - /dev/null - -landlock: - compatibility: best_effort - -process: - run_as_user: sandbox - run_as_group: sandbox - -network_policies: - corporate_proxy_e2e: - name: corporate_proxy_e2e - endpoints: - - host: {HOST_ALIAS} - port: {allowed_port} - tls: skip - enforcement: enforce - allowed_ips: - - "10.0.0.0/8" - - "172.0.0.0/8" - - "192.168.0.0/16" - - "fc00::/7" - binaries: - - path: /usr/bin/curl - - path: /usr/bin/python* - - path: /usr/local/bin/python* - - path: /sandbox/.venv/bin/python* - - path: /sandbox/.uv/python/*/bin/python* -"# - ) -} - -/// Appends corporate-proxy keys to the harness-generated gateway TOML and -/// restores the original file when dropped. -/// -/// The `[openshell.drivers.podman]` table is the last one the harness writes, -/// so appending bare keys lands in that table without introducing a duplicate -/// table header. -struct GatewayProxyConfig { - config_path: PathBuf, - original: Vec, - restored: bool, -} - -impl GatewayProxyConfig { - /// Locate the gateway's `--config` path from the wrapper's args file. - fn config_path_from_args() -> Result { - let args_file = std::env::var("OPENSHELL_E2E_GATEWAY_ARGS_FILE") - .map_err(|_| "OPENSHELL_E2E_GATEWAY_ARGS_FILE must be set".to_string())?; - let raw = std::fs::read(&args_file) - .map_err(|err| format!("read gateway args file '{args_file}': {err}"))?; - let args: Vec = raw - .split(|byte| *byte == 0) - .filter(|arg| !arg.is_empty()) - .map(|arg| String::from_utf8_lossy(arg).into_owned()) - .collect(); - args.iter() - .position(|arg| arg == "--config") - .and_then(|index| args.get(index + 1)) - .map(PathBuf::from) - .ok_or_else(|| format!("no --config argument in gateway args file '{args_file}'")) - } - - /// Rewrite the gateway config with corporate-proxy settings and restart the - /// gateway so the Podman driver picks them up. - async fn apply(proxy_url: &str, auth_file: &str) -> Result { - Self::apply_with(proxy_url, Some(auth_file), None).await - } - - /// Like [`apply`], but with optional proxy credentials and an optional - /// corporate CA bundle path (for an `https://` proxy). - async fn apply_with( - proxy_url: &str, - auth_file: Option<&str>, - ca_bundle: Option<&str>, - ) -> Result { - let config_path = Self::config_path_from_args()?; - let original = std::fs::read(&config_path) - .map_err(|err| format!("read gateway config '{}': {err}", config_path.display()))?; - - let mut proxy_config = format!("https_proxy = \"{proxy_url}\"\n"); - if let Some(auth_file) = auth_file { - proxy_config.push_str(&format!( - "proxy_auth_file = \"{auth_file}\"\n\ - proxy_auth_allow_insecure = true\n" - )); - } - if let Some(ca_bundle) = ca_bundle { - proxy_config.push_str(&format!("proxy_ca_bundle = \"{ca_bundle}\"\n")); - } - - // Insert the proxy fields at the end of the Podman driver table. The - // generated config may add driver subtables (for example - // `resource_admission`) before the gateway JWT and auth tables, so a - // gateway-table marker is not a reliable boundary. - let podman_table = b"[openshell.drivers.podman]"; - let table_start = original - .windows(podman_table.len()) - .position(|window| window == podman_table) - .ok_or_else(|| { - format!( - "gateway config '{}' has no Podman driver table", - config_path.display() - ) - })?; - let table_body = table_start + podman_table.len(); - let next_table = b"\n["; - let offset = original[table_body..] - .windows(next_table.len()) - .position(|window| window == next_table) - .map_or(original.len(), |relative| table_body + relative); - let mut updated = Vec::with_capacity(original.len() + proxy_config.len() + 1); - updated.extend_from_slice(&original[..offset]); - if !updated.ends_with(b"\n") { - updated.push(b'\n'); - } - updated.extend_from_slice(proxy_config.as_bytes()); - updated.extend_from_slice(&original[offset..]); - std::fs::write(&config_path, &updated) - .map_err(|err| format!("write gateway config '{}': {err}", config_path.display()))?; - - let guard = Self { - config_path, - original, - restored: false, - }; - restart_gateway().await?; - Ok(guard) - } - - /// Restore the original config and restart the gateway. - async fn restore(&mut self) -> Result<(), String> { - if self.restored { - return Ok(()); - } - std::fs::write(&self.config_path, &self.original).map_err(|err| { - format!( - "restore gateway config '{}': {err}", - self.config_path.display() - ) - })?; - restart_gateway().await?; - self.restored = true; - Ok(()) - } -} - -impl Drop for GatewayProxyConfig { - fn drop(&mut self) { - if self.restored { - return; - } - // Panic path: put the original config back and synchronously restart the - // gateway so later test binaries in this run inherit neither the proxy - // settings on disk nor the temporary configuration still loaded in the - // running process. Nothing else restarts it here: the only - // `ManagedGateway` is the short-lived one inside `restart_gateway`, and - // its `Drop` only calls `start`, which does not reload config for an - // already-running gateway. - let _ = std::fs::write(&self.config_path, &self.original); - if let Ok(Some(gateway)) = ManagedGateway::from_env() { - let _ = gateway.stop(); - let _ = gateway.start(); - } - } -} - -async fn restart_gateway() -> Result<(), String> { - let gateway = ManagedGateway::from_env()? - .ok_or_else(|| "managed gateway metadata disappeared".to_string())?; - gateway.stop()?; - gateway.start()?; - wait_for_healthy(Duration::from_mins(2)).await -} - -/// Names of the per-sandbox corporate proxy credential secrets. -fn proxy_auth_secret_names() -> Result, String> { - let engine = ContainerEngine::from_env()?; - let output = engine - .command() - .args(["secret", "ls", "--format", "{{.Name}}"]) - .output() - .map_err(|err| format!("list container secrets: {err}"))?; - if !output.status.success() { - return Err(format!( - "secret ls failed (exit {:?}):\n{}", - output.status.code(), - String::from_utf8_lossy(&output.stderr) - )); - } - Ok(String::from_utf8_lossy(&output.stdout) - .lines() - .map(str::trim) - .filter(|name| name.starts_with(SECRET_PREFIX)) - .map(ToOwned::to_owned) - .collect()) -} - -async fn wait_for_secret_removal(secret: &str, timeout: Duration) -> Result<(), String> { - let start = std::time::Instant::now(); - loop { - if !proxy_auth_secret_names()?.iter().any(|name| name == secret) { - return Ok(()); - } - if start.elapsed() > timeout { - return Err(format!( - "proxy-auth secret '{secret}' still present {}s after sandbox deletion", - timeout.as_secs() - )); - } - tokio::time::sleep(Duration::from_secs(2)).await; - } -} - -/// Assert the workload's results and the proxy's own record of what it saw. -/// -/// `output` is the sandbox's stdout; `proxy_logs` is the fake proxy's log, -/// which is the only place the CONNECT target form and the credential outcome -/// are observable. -fn assert_proxied_egress(output: &str, proxy_logs: &str, allowed_port: u16, denied_port: u16) { - // The approved request succeeded and its body came from the upstream, - // proving bytes traversed the tunnel rather than the proxy short-circuiting. - assert!( - output.contains("ALLOWED_RESULT") && output.contains(ALLOWED_MARKER), - "approved HTTPS request should have reached the upstream through the proxy:\n{output}" - ); - assert!( - output.contains(r#""status": 200"#), - "approved HTTPS request should have returned 200:\n{output}" - ); - - // The denied destination failed inside the sandbox. Depending on where the - // isolated Podman path rejects it, this is either the supervisor's HTTP 403 - // or a fail-closed socket permission error. The proxy-log assertion below - // rules out a vacuous failure after upstream contact. - let denied_line = output - .lines() - .find(|line| line.contains("DENIED_RESULT")) - .unwrap_or_default(); - assert!( - denied_line.contains(r#""status": -1"#) - && (denied_line.contains("403") || denied_line.contains("Permission denied")), - "denied destination should have been refused by policy:\n{output}" - ); - assert!( - !output.contains(DENIED_MARKER), - "denied destination's body must never reach the sandbox:\n{output}" - ); - - // Credentials arrived through the mounted secret: the proxy answers 407 - // without them, so a completed CONNECT is proof of delivery. - assert!( - proxy_logs.lines().any(|line| { - line.starts_with("CONNECT ") && line.contains(&format!(":{allowed_port} auth=ok")) - }), - "proxy should have seen an authenticated validated-IP CONNECT to \ - host port {allowed_port}.\nProxy logs:\n{proxy_logs}\nSandbox output:\n{output}" - ); - assert!( - !proxy_logs.contains("auth=fail"), - "proxy must never see an unauthenticated CONNECT:\n{proxy_logs}" - ); - - // The CONNECT target is a validated IP, not a hostname — the proxy performs - // no DNS resolution of its own, so the tunnel stays bound to the address - // that passed SSRF and allowed_ips validation. - assert!( - !proxy_logs.contains(HOST_ALIAS), - "CONNECT should target a validated IP, not the hostname:\n{proxy_logs}" - ); - - // The denied destination never reached the proxy: policy denial happens - // before any upstream contact. Its dedicated host port makes this - // observable even though both fixtures use the driver-neutral host alias. - assert!( - !proxy_logs - .lines() - .any(|line| line.starts_with("CONNECT ") && line.contains(&format!(":{denied_port} "))), - "policy-denied host port {denied_port} must never reach the proxy:\n{proxy_logs}" - ); -} - -#[tokio::test] -#[serial(corporate_proxy)] -async fn podman_corporate_proxy_routes_approved_tls_egress() { - if std::env::var("OPENSHELL_E2E_DRIVER").as_deref() != Ok("podman") { - eprintln!("Skipping corporate proxy test: e2e driver is not podman"); - return; - } - if ManagedGateway::from_env() - .expect("load managed e2e gateway metadata") - .is_none() - { - eprintln!("Skipping corporate proxy test: e2e gateway is not managed by this test run"); - return; - } - - // The split Podman supervisor uses host networking and cannot resolve - // container-network aliases. Run each fixture on a distinct host port and - // reach it through the driver-neutral host alias. - let proxy_port = find_free_port(); - let proxy = - HostSupportContainer::start_python_on_host_network(&proxy_script(proxy_port), proxy_port) - .await - .expect("start fake corporate proxy"); - let allowed_port = find_free_port(); - let allowed = HostSupportContainer::start_python_on_host_network( - &tls_upstream_script(HOST_ALIAS, ALLOWED_MARKER, allowed_port), - allowed_port, - ) - .await - .expect("start approved TLS upstream"); - let denied_port = find_free_port(); - let denied = HostSupportContainer::start_python_on_host_network( - &tls_upstream_script(HOST_ALIAS, DENIED_MARKER, denied_port), - denied_port, - ) - .await - .expect("start denied TLS upstream"); - - assert_ne!( - allowed.port, denied.port, - "approved and denied upstreams must have distinct host ports" - ); - - // ── Point the gateway at the corporate proxy ────────────────────── - let mut auth_file = NamedTempFile::new().expect("create proxy auth file"); - writeln!(auth_file, "{PROXY_USER}:{PROXY_PASS}").expect("write proxy credentials"); - auth_file.flush().expect("flush proxy credentials"); - let auth_path = auth_file - .path() - .to_str() - .expect("proxy auth path should be utf-8") - .to_string(); - - let secrets_before = proxy_auth_secret_names().expect("snapshot proxy-auth secrets"); - - let mut gateway_config = - GatewayProxyConfig::apply(&format!("http://{HOST_ALIAS}:{}", proxy.port), &auth_path) - .await - .expect("apply corporate proxy gateway config"); - - // ── Run the workload ────────────────────────────────────────────── - let mut policy = NamedTempFile::new().expect("create policy file"); - policy - .write_all(policy_yaml(allowed.port).as_bytes()) - .expect("write policy file"); - policy.flush().expect("flush policy file"); - let policy_path = policy - .path() - .to_str() - .expect("policy path should be utf-8") - .to_string(); - - let script = workload_script(allowed.port, denied.port); - let mut sandbox = SandboxGuard::create_keep_with_args( - &["--policy", &policy_path], - &["python3", "-c", &script], - READY_MARKER, - ) - .await - .expect("create sandbox behind the corporate proxy"); - - assert_proxied_egress( - &sandbox.create_output, - &proxy.logs().expect("read fake proxy logs"), - allowed.port, - denied.port, - ); - - // ── Secret lifecycle ────────────────────────────────────────────── - let secrets_live = proxy_auth_secret_names().expect("list proxy-auth secrets while sandbox up"); - let new_secrets: Vec<&String> = secrets_live - .iter() - .filter(|name| !secrets_before.contains(name)) - .collect(); - assert_eq!( - new_secrets.len(), - 1, - "exactly one proxy-auth secret should exist for the running sandbox. \ - Before: {secrets_before:?}, now: {secrets_live:?}" - ); - let sandbox_secret = new_secrets[0].clone(); - - sandbox.cleanup().await; - - wait_for_secret_removal(&sandbox_secret, Duration::from_mins(1)) - .await - .expect("sandbox deletion should remove the proxy-auth secret"); - - gateway_config - .restore() - .await - .expect("restore gateway config"); -} - -/// Assert the approved destination reached the `https://` proxy over the -/// tunnel and the denied one never did. The TLS proxy logs `CONNECT -/// : ok` (no auth dimension in this test). -fn assert_https_proxied_egress( - output: &str, - proxy_logs: &str, - allowed_port: u16, - denied_port: u16, -) { - assert!( - output.contains(READY_MARKER), - "workload did not finish; output:\n{output}" - ); - assert!( - output.contains(&format!( - "\"body\": \"{{\\\"upstream\\\":\\\"{ALLOWED_MARKER}\\\"" - )), - "approved upstream body missing — egress did not complete through the https proxy:\n{output}" - ); - assert!( - proxy_logs.lines().any(|line| { - line.starts_with("CONNECT ") && line.contains(&format!(":{allowed_port} ok")) - }), - "https proxy should have seen a validated-IP CONNECT to approved host port {allowed_port}:\n{proxy_logs}" - ); - assert!( - !proxy_logs - .lines() - .any(|line| line.starts_with("CONNECT ") && line.contains(&format!(":{denied_port} "))), - "policy-denied host port {denied_port} must never reach the https proxy:\n{proxy_logs}" - ); -} - -#[tokio::test] -#[serial(corporate_proxy)] -async fn podman_corporate_proxy_trusts_ca_bundle_for_https_proxy() { - if std::env::var("OPENSHELL_E2E_DRIVER").as_deref() != Ok("podman") { - eprintln!("Skipping https corporate proxy test: e2e driver is not podman"); - return; - } - if ManagedGateway::from_env() - .expect("load managed e2e gateway metadata") - .is_none() - { - eprintln!( - "Skipping https corporate proxy test: e2e gateway is not managed by this test run" - ); - return; - } - - let proxy_port = find_free_port(); - let proxy = HostSupportContainer::start_python_on_host_network( - &tls_proxy_script(proxy_port), - proxy_port, - ) - .await - .expect("start fake https corporate proxy"); - let allowed_port = find_free_port(); - let allowed = HostSupportContainer::start_python_on_host_network( - &tls_upstream_script(HOST_ALIAS, ALLOWED_MARKER, allowed_port), - allowed_port, - ) - .await - .expect("start approved TLS upstream"); - let denied_port = find_free_port(); - let denied = HostSupportContainer::start_python_on_host_network( - &tls_upstream_script(HOST_ALIAS, DENIED_MARKER, denied_port), - denied_port, - ) - .await - .expect("start denied TLS upstream"); - - assert_ne!( - allowed.port, denied.port, - "approved and denied upstreams must have distinct host ports" - ); - - // Recover the proxy's self-signed CA from its logs and hand it back as the - // corporate CA bundle the supervisor must trust for the proxy handshake. - let ca_pem = ca_cert_from_logs(&proxy.logs().expect("read proxy logs for CA")) - .expect("extract proxy CA certificate"); - let mut ca_file = NamedTempFile::new().expect("create proxy CA bundle file"); - ca_file - .write_all(ca_pem.as_bytes()) - .expect("write proxy CA bundle"); - ca_file.flush().expect("flush proxy CA bundle"); - ca_file - .as_file() - .set_permissions(std::fs::Permissions::from_mode(0o644)) - .expect("make proxy CA bundle readable by the non-root supervisor"); - let ca_path = ca_file - .path() - .to_str() - .expect("CA bundle path should be utf-8") - .to_string(); - - // ── Point the gateway at the https:// proxy with the CA bundle ───── - let mut gateway_config = GatewayProxyConfig::apply_with( - &format!("https://{HOST_ALIAS}:{}", proxy.port), - None, - Some(&ca_path), - ) - .await - .expect("apply https corporate proxy gateway config"); - - // ── Run the workload ────────────────────────────────────────────── - let mut policy = NamedTempFile::new().expect("create policy file"); - policy - .write_all(policy_yaml(allowed.port).as_bytes()) - .expect("write policy file"); - policy.flush().expect("flush policy file"); - let policy_path = policy - .path() - .to_str() - .expect("policy path should be utf-8") - .to_string(); - - let script = workload_script(allowed.port, denied.port); - let mut sandbox = SandboxGuard::create_keep_with_args( - &["--policy", &policy_path], - &["python3", "-c", &script], - READY_MARKER, - ) - .await - .expect("create sandbox behind the https corporate proxy"); - - assert_https_proxied_egress( - &sandbox.create_output, - &proxy.logs().expect("read fake proxy logs"), - allowed.port, - denied.port, - ); - - sandbox.cleanup().await; - - gateway_config - .restore() - .await - .expect("restore gateway config"); -} diff --git a/tests/support/README.md b/tests/support/README.md index a2c0a02eaa..66edccf489 100644 --- a/tests/support/README.md +++ b/tests/support/README.md @@ -23,6 +23,10 @@ never skips it. Add a row when you add a variable. | `OPENSHELL_TEST_KEYCLOAK_CLIENT_ID` | `-keycloak` | `keycloak-support` | A client in that realm that allows password grants. | | `OPENSHELL_TEST_KEYCLOAK_USERNAME` | `-keycloak` | `keycloak-support` | The realm user that tests sign in as. | | `OPENSHELL_TEST_KEYCLOAK_PASSWORD` | `-keycloak` | `keycloak-support` | That user's password. | +| `OPENSHELL_TEST_CORPORATE_PROXY_URL` | `-corporate-proxy` (`playbooks/corporate-proxy.yaml`) | `corporate-proxy-support` | The corporate proxy URL that the gateway's driver sends sandbox egress through, such as `https://host.openshell.internal:3128`. | +| `OPENSHELL_TEST_CORPORATE_PROXY_CA_CERT` | `-corporate-proxy` | `corporate-proxy-support` | The path of the proxy's CA certificate, in PEM. It signs the proxy's listener certificate and the certificates it re-signs tunnels with. | +| `OPENSHELL_TEST_CORPORATE_PROXY_BYPASS_LISTEN` | `-corporate-proxy` | `corporate-proxy-support` | The port a test listens on for a destination that the gateway's `no_proxy` sends direct. | +| `OPENSHELL_TEST_CORPORATE_PROXY_LOG` | `-corporate-proxy` | `corporate-proxy-support` | The path of the proxy's log, one JSON line per CONNECT, readable by the test user. | | `OPENSHELL_TEST_ADDITIONAL_CA_CERT` | The additional-CA environment, which configures the gateway's `additional_ca_bundle` (added by #5729) | `host-server-support` (`TestCa::from_env`) | The path of the CA certificate, in PEM, that the gateway trusts. | | `OPENSHELL_TEST_ADDITIONAL_CA_KEY` | The additional-CA environment | `host-server-support` (`TestCa::from_env`) | The path of that CA's private key, in PEM, readable by the test user. Tests sign server certificates with it. | | `OPENSHELL_TEST_BASE_IMAGE` | Optional; you, for example to use a mirror | `testkit::base_image()` | The default sandbox image. Unset, it is `nvcr.io/nvidia/base/ubuntu:24.04`. | From 301c0b9ec2b8f8cf9a0045e2f7ca58aedcbf6889 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 19 Nov 2026 16:50:27 -0800 Subject: [PATCH 11/12] test(e2e): remove the legacy Docker and Podman corporate-proxy tests features/corporate-proxy replaces them. Their TOML-editing helper test checked only the file's own helper, so it goes with the file. Signed-off-by: Matthew Grossman --- TESTING.md | 2 +- e2e/rust/Cargo.toml | 10 ---------- e2e/rust/e2e-podman.sh | 1 - tests/tmachine/artifacts.nix | 3 +-- 4 files changed, 2 insertions(+), 14 deletions(-) diff --git a/TESTING.md b/TESTING.md index 7829434e28..9c2e55ff22 100644 --- a/TESTING.md +++ b/TESTING.md @@ -889,7 +889,7 @@ cargo test --manifest-path e2e/rust/Cargo.toml --features e2e --test sync Run a single Docker-only test directly with cargo: ```shell -cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test docker_corporate_proxy +cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test gateway_start ``` The harness (`e2e/rust/src/harness/`) provides: diff --git a/e2e/rust/Cargo.toml b/e2e/rust/Cargo.toml index 89d52187d5..2f3e86de6c 100644 --- a/e2e/rust/Cargo.toml +++ b/e2e/rust/Cargo.toml @@ -93,11 +93,6 @@ name = "vm_bind_mount" path = "tests/vm_bind_mount.rs" required-features = ["e2e-vm"] -[[test]] -name = "docker_corporate_proxy" -path = "tests/docker_corporate_proxy.rs" -required-features = ["e2e-docker"] - [[test]] name = "additional_ca" path = "tests/additional_ca.rs" @@ -138,11 +133,6 @@ name = "ocsf_jsonl_conformance" path = "tests/ocsf_jsonl_conformance.rs" required-features = ["e2e-podman"] -[[test]] -name = "podman_corporate_proxy" -path = "tests/podman_corporate_proxy.rs" -required-features = ["e2e-podman"] - [[test]] name = "podman_resource_limits" path = "tests/podman_resource_limits.rs" diff --git a/e2e/rust/e2e-podman.sh b/e2e/rust/e2e-podman.sh index bd34cf2d77..c131e61513 100755 --- a/e2e/rust/e2e-podman.sh +++ b/e2e/rust/e2e-podman.sh @@ -29,7 +29,6 @@ PODMAN_CI_TESTS=( local_driver_token_restart ocsf_jsonl_conformance otlp_relay - podman_corporate_proxy podman_gateway_start podman_host_gateway provider_token_exchange diff --git a/tests/tmachine/artifacts.nix b/tests/tmachine/artifacts.nix index d8b743b803..e5a226ea1a 100644 --- a/tests/tmachine/artifacts.nix +++ b/tests/tmachine/artifacts.nix @@ -91,11 +91,10 @@ let # `enable_bind_mounts`), which the tmachine gateway does not enable. Wrapper # lane. "driver_config_volume" - # The next three self-skip without wrapper-owned gateway metadata (restart + # The next two self-skip without wrapper-owned gateway metadata (restart # and configuration control), so exclude them rather than report false # passes. Wrapper lane. "local_driver_token_restart" - "podman_corporate_proxy" "podman_gateway_start" # Its move to conformance is deferred until the isolation work in #4120; see # #4246. From b4835690291aa1bd2bbd53ba8653709e04219295 Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Thu, 19 Nov 2026 22:57:19 -0800 Subject: [PATCH 12/12] ci(e2e): drop the rootless Podman corporate-proxy row Sandbox creation fails there because Podman can't relabel the root-owned proxy CA bind mount. Tracked in #5795. Signed-off-by: Matthew Grossman --- .github/workflows/e2e-suites.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/e2e-suites.yml b/.github/workflows/e2e-suites.yml index 7e88f430bc..ca6c82584e 100644 --- a/.github/workflows/e2e-suites.yml +++ b/.github/workflows/e2e-suites.yml @@ -97,7 +97,6 @@ jobs: {"environment":"fedora-podman-rootless-cdi","installer":"rpm","family":"features","package":"cdi","gateway":"openshell","runner":"ubuntu-24.04"}, {"environment":"ubuntu-docker-rootful-corporate-proxy","installer":"deb","family":"features","package":"corporate-proxy","gateway":"openshell","runner":"ubuntu-24.04"}, {"environment":"fedora-podman-rootful-corporate-proxy","installer":"rpm","family":"features","package":"corporate-proxy","gateway":"openshell","runner":"ubuntu-24.04"}, - {"environment":"fedora-podman-rootless-corporate-proxy","installer":"rpm","family":"features","package":"corporate-proxy","gateway":"openshell","runner":"ubuntu-24.04"}, {"environment":"ubuntu-docker-rootful-keycloak","installer":"deb","family":"features","package":"keycloak-provider-refresh","gateway":"openshell","runner":"ubuntu-24.04"}, {"environment":"ubuntu-docker-rootful","installer":"deb","family":"features","package":"oci-image","gateway":"openshell","runner":"ubuntu-24.04"}, {"environment":"fedora-podman-rootful","installer":"rpm","family":"features","package":"oci-image","gateway":"openshell","runner":"ubuntu-24.04"},