diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 71fe4a7af1..e7a66ac157 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -211,6 +211,55 @@ See [`values.yaml`](values.yaml) for source defaults. Selected overlays: - [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants - [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development +### Sandbox service routing + +Gateway-mediated sandbox service routing is disabled by default in this Kubernetes +chart to avoid an unintended alternate application access path. Control-plane API +access is unchanged. + +For service ingress directly through supervisors, enable +`[openshell.supervisor.app_listener].enabled` and pair it with an ingress solution +that routes requests to the app listener on the supervisor owning each sandbox: + +```yaml +gatewayConfig: + openshell.gateway: + enable_gateway_service_routing: false + openshell.supervisor.app_listener: + enabled: true + port: 9443 + base_url: https://services.example.com + openshell.drivers.kubernetes.sandbox_runtime.app_listener: + tls_secret_name: openshell-app-tls +``` + +Provision `openshell-app-tls` with a certificate for `*.services.example.com`. +Pair the listener with a TLS-passthrough proxy or a compatible Kubernetes +ingress/Gateway API integration. The integration consumes service discovery to +map each advertised hostname to the current owning supervisor's IP and configured +app-listener port, preserving the application hostname as Host and TLS SNI. +Configure DNS and NetworkPolicy for that path. Enabling the listener alone does +not provision DNS or external ingress routes, and the chart does not enable it +automatically. + +To use gateway-mediated service routing instead, explicitly enable forwarding; +that path does not require the app listener: + +```yaml +gatewayConfig: + openshell.gateway: + enable_gateway_service_routing: true +``` + +Existing Helm installations that rely on gateway service URLs must set this +explicitly before upgrading. Missing saved values receive the disabled chart +default; explicit choices are preserved. If neither routing mode is enabled, +startup logs warn that sandbox service routing is disabled. + +The core runtime continues to default gateway routing to enabled, preserving +Docker, Podman, and VM deployments outside this chart. Gateway service +authentication remains independently opt-in. + ### Additional destination CAs Create an operator-owned ConfigMap with a `ca.crt` key when sandboxes must reach @@ -589,7 +638,8 @@ for peer identity policies, multi-node registrations, and authorization. | gateway.image.repository | string | `"openshell/gateway"` | Gateway image repository. | | gateway.image.tag | string | `""` | Gateway image tag. Defaults to the chart appVersion when empty. | | gateway.listenAddress | string | `"0.0.0.0"` | Gateway listener IP. Use `::` for IPv6. | -| gatewayConfig | object | `{"openshell":{"version":2}}` | Non-secret gateway application configuration. Top-level keys name TOML tables and are rendered into the mounted gateway.toml file. Kubernetes resource inputs remain outside this map; template expressions derive the corresponding runtime values from their resource owner. | +| gatewayConfig | object | `{"openshell":{"version":2},"openshell.gateway":{"enable_gateway_service_routing":false}}` | Non-secret gateway application configuration. Top-level keys name TOML tables and are rendered into the mounted gateway.toml file. Kubernetes resource inputs remain outside this map; template expressions derive the corresponding runtime values from their resource owner. | +| gatewayConfig."openshell.gateway".enable_gateway_service_routing | bool | `false` | Opt in to gateway-mediated sandbox service URLs and forwarding. Disabled for Kubernetes; configure application-listener ingress or explicitly enable this path. | | global.image.pullPolicy | string | `"IfNotPresent"` | Shared OpenShell image pull policy. Individual image pull policies take precedence. | | global.image.registry | string | `"ghcr.io/nvidia"` | Shared OpenShell image registry. Individual image registries take precedence. | | global.image.tag | string | `""` | Shared OpenShell image tag. Defaults to the chart appVersion when empty. | diff --git a/deploy/helm/openshell/README.md.gotmpl b/deploy/helm/openshell/README.md.gotmpl index 7c7a246bf6..24f531a74a 100644 --- a/deploy/helm/openshell/README.md.gotmpl +++ b/deploy/helm/openshell/README.md.gotmpl @@ -212,6 +212,55 @@ See [`values.yaml`](values.yaml) for source defaults. Selected overlays: - [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants - [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development +### Sandbox service routing + +Gateway-mediated sandbox service routing is disabled by default in this Kubernetes +chart to avoid an unintended alternate application access path. Control-plane API +access is unchanged. + +For service ingress directly through supervisors, enable +`[openshell.supervisor.app_listener].enabled` and pair it with an ingress solution +that routes requests to the app listener on the supervisor owning each sandbox: + +```yaml +gatewayConfig: + openshell.gateway: + enable_gateway_service_routing: false + openshell.supervisor.app_listener: + enabled: true + port: 9443 + base_url: https://services.example.com + openshell.drivers.kubernetes.sandbox_runtime.app_listener: + tls_secret_name: openshell-app-tls +``` + +Provision `openshell-app-tls` with a certificate for `*.services.example.com`. +Pair the listener with a TLS-passthrough proxy or a compatible Kubernetes +ingress/Gateway API integration. The integration consumes service discovery to +map each advertised hostname to the current owning supervisor's IP and configured +app-listener port, preserving the application hostname as Host and TLS SNI. +Configure DNS and NetworkPolicy for that path. Enabling the listener alone does +not provision DNS or external ingress routes, and the chart does not enable it +automatically. + +To use gateway-mediated service routing instead, explicitly enable forwarding; +that path does not require the app listener: + +```yaml +gatewayConfig: + openshell.gateway: + enable_gateway_service_routing: true +``` + +Existing Helm installations that rely on gateway service URLs must set this +explicitly before upgrading. Missing saved values receive the disabled chart +default; explicit choices are preserved. If neither routing mode is enabled, +startup logs warn that sandbox service routing is disabled. + +The core runtime continues to default gateway routing to enabled, preserving +Docker, Podman, and VM deployments outside this chart. Gateway service +authentication remains independently opt-in. + ### Additional destination CAs Create an operator-owned ConfigMap with a `ca.crt` key when sandboxes must reach diff --git a/deploy/helm/openshell/templates/_toml.tpl b/deploy/helm/openshell/templates/_toml.tpl index 6ae006de65..a1e2d1f839 100644 --- a/deploy/helm/openshell/templates/_toml.tpl +++ b/deploy/helm/openshell/templates/_toml.tpl @@ -102,6 +102,10 @@ Top-level lists represent TOML arrays of tables and preserve their YAML order. * {{- $config := deepCopy (.Values.gatewayConfig | default dict) -}} {{- $legacyServer := .Values.server | default dict -}} {{- $gateway := get $config "openshell.gateway" | default dict -}} +{{/* Keep the Kubernetes routing default when saved values omit it or use null. */}} +{{- if or (not (hasKey $gateway "enable_gateway_service_routing")) (eq (get $gateway "enable_gateway_service_routing") nil) -}} +{{- $_ := set $gateway "enable_gateway_service_routing" false -}} +{{- end -}} {{- if not (hasKey $gateway "name") -}}{{- $_ := set $gateway "name" (get $legacyServer "name" | default (include "openshell.fullname" .)) -}}{{- end -}} {{/* The Service and probes own listener ports, so keep runtime listeners aligned. */}} {{- $listenHost := include "openshell.listenHost" .Values.gateway.listenAddress -}} diff --git a/deploy/helm/openshell/tests/gateway_config_test.yaml b/deploy/helm/openshell/tests/gateway_config_test.yaml index 7aa3a8a8a7..de4ea98448 100644 --- a/deploy/helm/openshell/tests/gateway_config_test.yaml +++ b/deploy/helm/openshell/tests/gateway_config_test.yaml @@ -13,19 +13,40 @@ release: namespace: my-namespace tests: - - it: leaves gateway service routing enabled by the runtime default when omitted + - it: disables gateway service routing by default on Kubernetes template: templates/gateway-config.yaml asserts: - - notMatchRegex: + - matchRegex: path: data["gateway.toml"] - pattern: 'enable_gateway_service_routing\s*=' + pattern: '(?m)^enable_gateway_service_routing\s*=\s*false$' + + - it: allows explicitly enabling gateway service routing + template: templates/gateway-config.yaml + set: + gatewayConfig: + openshell.gateway: + enable_gateway_service_routing: true + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '(?m)^enable_gateway_service_routing\s*=\s*true$' + + - it: defaults gateway routing off when saved configuration omits the gateway table + template: templates/gateway-config.yaml + set: + gatewayConfig: + openshell.gateway: null + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '(?m)^enable_gateway_service_routing\s*=\s*false$' - - it: disables gateway service routing through canonical configuration + - it: defaults gateway routing off when saved configuration omits the flag template: templates/gateway-config.yaml set: gatewayConfig: openshell.gateway: - enable_gateway_service_routing: false + enable_gateway_service_routing: null asserts: - matchRegex: path: data["gateway.toml"] diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 7129244d59..7187333afe 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -415,6 +415,9 @@ gatewayConfig: # acquisition over mTLS. Configure node_local_port or endpoint, an exact or # cluster-scoped peer identity, the Workload API socket, and the stable sandbox # SPIFFE ID prefix. SPIRE must authorize the gateway and attest its Sandbox references. + openshell.gateway: + # -- Opt in to gateway-mediated sandbox service URLs and forwarding. Disabled for Kubernetes; configure application-listener ingress or explicitly enable this path. + enable_gateway_service_routing: false # Server configuration server: diff --git a/docs/how-it-works/gateways/configuration.mdx b/docs/how-it-works/gateways/configuration.mdx index f09e04c8b0..b34a7b0c0c 100644 --- a/docs/how-it-works/gateways/configuration.mdx +++ b/docs/how-it-works/gateways/configuration.mdx @@ -607,7 +607,9 @@ and `https://--./` for the unnamed endpoint. The base domain is the first wildcard DNS SAN in `server_sans`; gateways without SANs use the built-in `openshell.localhost`. -`enable_gateway_service_routing` defaults to `true`. Set it to `false` to stop publishing +The core runtime defaults `enable_gateway_service_routing` to `true`; Docker, Podman, +and VM gateway behavior is unchanged. The Kubernetes Helm chart defaults it to +`false`. Set it to `false` to stop publishing gateway service URLs and reject gateway-mediated application HTTP requests and WebSocket upgrades with `404`. Control-plane APIs, service declarations, direct supervisor listeners, and discovery remain available. The setting also accepts @@ -621,6 +623,26 @@ gatewayConfig: enable_gateway_service_routing: false ``` +For Kubernetes service ingress directly through supervisors, set +`[openshell.supervisor.app_listener].enabled = true` and configure the +[supervisor application listener](#supervisor-application-listener). Pair it with +an ingress solution, such as a TLS-passthrough proxy or a compatible Kubernetes +ingress/Gateway API integration, that consumes discovery and routes each +advertised hostname to the current owning supervisor's app-listener IP and port. +Preserve the application hostname as Host and TLS SNI. Enabling the listener +alone does not provision DNS or external ingress routes; the chart does not +enable either the listener or the ingress integration automatically. + +If both routing modes are disabled, startup logs warn that sandbox service +routing is disabled. Gateway-mediated forwarding does not require the app +listener. + +When upgrading a Helm installation that uses gateway service URLs, explicitly +set `enable_gateway_service_routing: true` in the canonical table above before the +upgrade. Missing saved values receive the chart's `false` default; explicit +`true` values remain enabled. This changes application forwarding only, not +control-plane API access or the service-authentication default. + With gateway routing disabled, `gateway_url` is empty and `gateway_service_urls` is empty. A configured application origin supplies the preferred URL; without one, preferred URLs are empty. Idempotent retries preserve all recorded URL fields together. Use service diff --git a/e2e/kubernetes/app-listener/README.md b/e2e/kubernetes/app-listener/README.md index 93dc17c0b6..b428010d01 100644 --- a/e2e/kubernetes/app-listener/README.md +++ b/e2e/kubernetes/app-listener/README.md @@ -31,8 +31,9 @@ differs from the repository's mise environment. `OPENSHELL_E2E_APP_DELIVERY` accepts `push` (default) or `poll`; `OPENSHELL_E2E_APP_TAKEOVER` accepts `graceful` (default) or `abrupt` for shared supervisors. Dedicated runs include gateway-outage expiry. Every run also -disables the listener and verifies absent app sockets/TLS mounts/Services and -continued gateway application ingress. Logs, events, and resource summaries +disables the listener, explicitly enables gateway routing for the fallback +check, and verifies absent app sockets/TLS mounts/Services and continued gateway +application ingress. Logs, events, and resource summaries are written under `results/app-listener/`, without TLS private keys or Secrets. GitHub Actions runs four matrix cases under the existing `test:e2e` label: diff --git a/e2e/kubernetes/app-listener/run.py b/e2e/kubernetes/app-listener/run.py index a5f6bf7583..67711d0a78 100644 --- a/e2e/kubernetes/app-listener/run.py +++ b/e2e/kubernetes/app-listener/run.py @@ -279,7 +279,8 @@ def wait_for_shared_hosts(): json.dumps( { "gatewayConfig": { - "openshell.supervisor.app_listener": {"enabled": False} + "openshell.supervisor.app_listener": {"enabled": False}, + "openshell.gateway": {"enable_gateway_service_routing": True}, } } ) diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index a318f8bf74..f7c30d02b2 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -1820,7 +1820,11 @@ Watches recheck bearer credentials and membership every second. Allow one refres interval for withdrawals; route lease expiry remains independently enforced. For incorrect public service URLs, inspect -`[openshell.gateway].enable_gateway_service_routing` first. When false, gateway URL fields +`[openshell.gateway].enable_gateway_service_routing` first. The Helm chart defaults it +to false; the core runtime and non-Kubernetes deployments retain true. To retain +gateway forwarding in a Helm deployment or upgrade, set +`gatewayConfig."openshell.gateway".enable_gateway_service_routing: true` explicitly. +When false, gateway URL fields are empty and gateway application requests return `404`; direct listener and discovery behavior is independent. Then inspect `[openshell.supervisor.app_listener].base_url`. It is the external HTTPS base origin