Skip to content

feat(gateway): broker stable sandbox SPIFFE provider tokens - #4732

Merged
TaylorMutch merged 38 commits into
mainfrom
feat/4731-stable-sandbox-spiffe/drew
Nov 18, 2026
Merged

TaylorMutch merged 38 commits into
mainfrom
feat/4731-stable-sandbox-spiffe/drew

Conversation

@drew

@drew drew commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Enable shared-supervisor Kubernetes sandboxes to obtain provider access tokens using stable sandbox SPIFFE identities. The gateway obtains sandbox JWT-SVIDs through SPIRE's experimental SPIFFE Broker API and performs provider-token exchanges. Supervisors authenticate with each sandbox generation's credentials and receive only final provider access tokens; workloads receive no SPIFFE socket or credentials.

Related Issue

Closes #4731

Changes

  • Add GetSandboxProviderAccessToken to the OpenShell service. Validate the signed session against persisted runtime generation, credential epoch, token lineage, and active state. Resolve provider attachment, credential key, endpoint, audience, and scopes from gateway configuration.
  • Derive <prefix>/gateway/<gateway-id>/sandbox/<sandbox-id> and request identity using the Sandbox resource's persisted Kubernetes UID. Callers cannot supply workload references or selectors.
  • Authenticate to the Broker over mTLS using fresh gateway X.509-SVID credentials from the CSI Workload API. Verify the Broker's certificate chain and configured exact identity or cluster-scoped SPIFFE path prefix through openshell-crypto, then verify returned JWT-SVID signatures, audience, expiry, and exact sandbox subject.
  • Perform RFC 7523 grants or both RFC 8693 exchange stages on the gateway. Bind cached access tokens to the sandbox generation, credential epoch, provider revision, configured grant, and resolved subject. Revalidate configuration and session authorization before serving cached or newly acquired tokens.
  • Reuse authenticated supervisor channels with bearer rotation and revocation checks. Bound identity acquisition at 10 seconds, each provider-token request at 30 seconds, the gateway operation at 120 seconds, and the supervisor operation at 135 seconds.
  • Separate shared SPIFFE types into openshell-spiffe and the Broker adapter into openshell-spire. Update published configuration docs, Helm guidance, troubleshooting, and integration fixtures.

Application flow

This shows a cache miss. Gateway-to-SPIRE traffic stays on the gateway's node: the Workload API uses its CSI-mounted Unix socket, and the Broker API uses mTLS directly to the node address supplied by Kubernetes' Downward API. No Kubernetes Broker Service is used in node-local mode. Workloads and shared supervisors can run on other nodes.

sequenceDiagram
    participant W as Sandbox workload
    participant S as Shared supervisor
    participant G as Gateway
    participant A as Gateway node's SPIRE agent
    participant K as Kubernetes API
    participant T as Provider token service
    participant U as Upstream API

    W->>S: Make an HTTP request
    S->>G: Request token using sandbox session JWT
    G->>G: Authorize session, provider and configured grant
    G->>A: Local Workload API: obtain gateway identity and trust bundles
    G->>A: Node-local Broker over mTLS: request sandbox identity by CR UID
    A->>A: Check allowlisted gateway SPIFFE identity
    A->>K: Attest Sandbox CR and check gateway authorization
    K-->>A: Attestation and authorization results
    A-->>G: Sandbox JWT-SVID and verification bundles
    G->>G: Verify sandbox identity, signature, audience and expiry
    G->>T: Perform configured OAuth2 grant or token exchanges
    T-->>G: Final provider access token
    G->>G: Recheck configuration and session authorization
    G-->>S: Final access token and remaining lifetime
    S->>U: Forward HTTP request with injected token
    U-->>S: Response
    S-->>W: Response
Loading

The gateway performs the RFC 7523 grant or both RFC 8693 exchange stages. Shared supervisors receive only the final provider token; sandbox workloads and shared supervisors receive no SPIFFE credentials or Workload API mount.

Configuration and scope

The feature is disabled unless [openshell.gateway.spiffe_broker] is configured; there is no compile-time feature flag. Gateway session JWTs with a positive TTL are required.

Operators must enable SPIRE's experimental Broker endpoint, authorize the gateway identity, configure Kubernetes attestation with access_policy = "enforced", and provision registrations for sandbox identities. The agent needs Sandbox lookup and SubjectAccessReview permissions; the gateway identity needs narrowly scoped impersonate-via-spire authorization. Authorize the Broker peer with an exact SPIFFE ID or a cluster-scoped SPIFFE path prefix. Configure node_local_port to reach the gateway node’s agent directly, with OPENSHELL_NODE_IP injected from status.hostIP. This path uses no Broker Service. An explicit HTTPS endpoint remains available as an alternative. Use a node alias to distribute sandbox registrations to every node where a gateway replica may run.

The gateway uses a CSI Workload API mount. Shared supervisors and sandbox workloads require no SPIFFE mounts. RFC 8693 profiles use the same configured spiffe_broker.workload_api_socket_path for the gateway's component JWT-SVID.

This change supports Kubernetes shared supervision. It does not automatically provision SPIRE registrations or enable other compute drivers. Already-issued provider bearer tokens retain their normal expiration. Kubernetes Restricted Pod Security admission was validated; OpenShift restricted-v2 SCC compatibility has not been validated.

The public RPC adds a method to generated Rust server traits and the full TypeScript client interface; downstream handwritten implementations and mocks may require updates.

Connection pooling, broader cache improvements, generated upstream Broker bindings, and inclusion of the SPIRE fixture in default CI remain follow-ups.

Testing

Local validation of the node-local changes at 14a87e4eb:

  • All eight targeted SPIFFE tests passed against the four-node GKE spiffe-poc cluster. Coverage includes gateway relocation and fresh acquisition on every node without a Broker Service, node-local Broker egress restrictions, local-agent replacement, abrupt supervisor failure, credential rotation, and stale-session fencing.
  • All 481 Helm tests passed. Focused Rust configuration and SPIRE tests, Clippy, generated Helm documentation, and fixture shell syntax checks passed. Endpoint validation covers IPv4 and IPv6; the live GKE run used IPv4.
  • GitHub CI for the pushed commits is tracked in the PR checks.

Validation of the implementation at 17c8b7b2c:

  • 36 Kubernetes conformance tests passed.
  • Eight SPIFFE Broker/provider tests passed on both a fresh cluster and the retained development cluster, including credential rotation, supervisor failover, and stale-session fencing.
  • Configuration-disabled shared-supervisor coverage passed.
  • Eight independent real-SPIRE protocol checks covered successful issuance and rejection of unauthorized references, stale UIDs, unauthorized callers, incorrect peer identities, and untrusted certificate chains.
  • 284 focused Rust tests, 473 Helm tests, and 206 TypeScript SDK tests passed.
  • Formatting, Clippy, Helm lint, crypto dependency policy, and lockfile checks passed.
  • Required GitHub CI passed: Branch Checks and E2E. The GPU gate passed with the GPU tests skipped.

The subsequent commit 4532524dc only corrects troubleshooting advice and a test description; git diff --check passed.

@copy-pr-bot

copy-pr-bot Bot commented Oct 20, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@drew
drew added this pull request to stack #4698 October 20, 2026 02:05
@github-actions

Copy link
Copy Markdown

@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from db063a9 to 8dfbd78 Compare October 20, 2026 16:24
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch 2 times, most recently from a876a1a to 437619a Compare October 20, 2026 18:39
@drew drew changed the title feat(supervisor): delegate stable sandbox SPIFFE identities feat(gateway): broker stable sandbox SPIFFE provider tokens Oct 20, 2026
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from 3da8000 to 5fbece5 Compare October 20, 2026 22:10
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from 5fbece5 to 1899b5b Compare October 20, 2026 22:38
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch 2 times, most recently from 96ba1b7 to e0b6cec Compare October 20, 2026 22:53
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch 3 times, most recently from b4c1940 to b733887 Compare October 21, 2026 00:20
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from b733887 to 8e6786b Compare October 21, 2026 00:55
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from 8e6786b to 6dd2cbb Compare October 21, 2026 01:05
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from 6dd2cbb to 85cba03 Compare October 21, 2026 01:48
@drew
drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from a78134c to 03c6d6e Compare October 22, 2026 00:58
Preserve the SPIFFE RPC alongside service discovery and update the combined public schema inventory without changing durable fingerprints.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

@EmilienM EmilienM left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another iteration needed: 4 major, 19 minor (inline, 1 below)

The Broker API rework removed the hostPath and node-coupling problems from round 1, and the cache-binding fixes hold up. The main problem now is that the Broker connection pins one SPIRE agent's ID, so the documented Service setup only works on a single-node cluster. The other majors are per-request costs on the shared-supervisor path (catalog scans, the global gRPC rate limiter) and the wire format of the new RPC's credential_key.

Outside the diff:

  • Minor: docs/how-it-works/providers/profiles.mdx:705, still says token grants need the sandbox supervisor's own Workload API socket, as do docs/kubernetes/access-control.mdx:37 and deploy/helm/openshell/README.md.gotmpl:535; a short note linking to the new Broker section would cover shared supervisors.
How this was reviewed
  • Commit d7b0c95, round 2: a full pass, since the branch was rebased after round 1's 9b60d8d
  • Deep review, because it touches AGENTS.md, .agents/skills/helm-dev-environment/SKILL.md and crates/openshell-core/src/dynamic_credential_key.rs
  • Passes: architecture, performance, security, scalability, rust, kubernetes and supply-chain, then correctness, then an advanced defensive review
  • Independent skeptics checked each candidate finding (3 with a reproducer for blocking and major ones, 2 for architecture and scalability ones, 1 for minor ones); 19 of 23 held up
  • Repo rules read from main: AGENTS.md, CLAUDE.md, CONTRIBUTING.md, docs/CONTRIBUTING.mdx
  • Not covered: nothing skipped

Comment thread crates/openshell-spire/src/broker.rs Resolved
Comment thread proto/openshell.proto Resolved
Comment thread crates/openshell-spire/src/broker.rs Resolved
Comment thread e2e/with-kube-gateway.sh Resolved
Comment thread crates/openshell-server/Cargo.toml Resolved
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
…n errors

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
… grants

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
… teardown

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
…ation

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@TaylorMutch

Copy link
Copy Markdown
Collaborator

Pushed the review fixes and node-local Broker changes through 14a87e4.

Validated the node-local path against a four-node GKE cluster. The targeted SPIFFE test binary passed 8/8 tests (one live integration scenario and seven shared-helper tests), with no failures or skips. The live scenario covered:

  • Gateway relocation across all four nodes, forcing a fresh provider-token acquisition on each node while preserving the sandbox SPIFFE identity.
  • No Broker Kubernetes Service, with TCP 8443 egress restricted to the gateway's current node address.
  • Replacement of the local SPIRE agent and successful acquisition without re-registering the sandbox.
  • Abrupt supervisor SIGKILL, takeover, stale-session/credential-epoch fencing, and fresh provider-token acquisition.
  • Acquisition after workload X.509-SVID rotation, and rejection of an unregistered/recreated sandbox identity.

The remote SIGKILL fixture uses an unprivileged sidecar sharing the supervisor Pod's process namespace and UID; it requires no host PID namespace, runtime socket, or additional capabilities.

Additional local checks passed:

  • 481 Helm tests across 34 suites.
  • Focused Rust Broker configuration and SPIRE adapter tests, including IPv4/IPv6 endpoint formatting, missing/invalid node addresses, and mutually exclusive transport settings.
  • Clippy for the modified Rust packages and E2E target, generated Helm documentation checks, and fixture shell syntax checks.

The live GKE run used IPv4. IPv6 endpoint handling is covered by unit tests; OpenShift restricted-v2 SCC compatibility has not been validated live. These results are separate from GitHub CI for the pushed commits.

The PR description now shows the application flow and explains that gateway-to-SPIRE traffic stays node-local. The GKE cluster and deployed fixtures have been left in place.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at be94c73. Branch Checks, Helm Lint, E2E, and the GPU gate are green. Review threads are resolved, including the acquisition-failure fan-out and lazy-channel fixes. The small Kubernetes runtime-format duplication remains an accepted scope decision.

I accept the documented Rust generated-server-trait and TypeScript injected-client source-compatibility impacts of adding GetSandboxProviderAccessToken to OpenShell. The experimental SPIRE Broker prerequisite and outstanding live OpenShift validation remain documented.

Multi-node GKE validation: #4732 (comment)

@TaylorMutch
TaylorMutch added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit df6a799 Nov 18, 2026
161 of 164 checks passed
@TaylorMutch
TaylorMutch deleted the feat/4731-stable-sandbox-spiffe/drew branch November 18, 2026 22:30
mrunalp added a commit that referenced this pull request Nov 18, 2026
…ecrets/mrunalp

Resolve the schema inventory conflict with #4732 by combining the RPC
counts and recomputing the reviewed public fingerprint.

Adapt to #5668, which added Basic client authentication to OAuth refresh:
count a client_secret supplied by a workspace secret as present when
validating token_endpoint_auth_method, and test a Basic mint whose client
secret comes from a workspace secret. Update the new request_token test
and refresh request literal for the merged signatures.

Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. test:e2e Requires end-to-end coverage test:e2e-kubernetes Requires Kubernetes end-to-end coverage topic:api Public APIs, gRPC/protobuf, SDK interfaces, versioning, compatibility, and API design standards. topic:architecture System design, component boundaries, data/control flow, architectural decisions, and RFCs. topic:authnz Authentication, authorization, identity, sessions, credential protection, and policy integrity. topic:documentation Architecture docs, READMEs, project website updates, and automated documentation generation. topic:kubernetes Kubernetes operators, CRDs, deployment controllers, and native Kubernetes API integration. topic:operability Installing/configuring deployments, upgrades, backup/restore, diagnostics, and maintenance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: delegate stable sandbox SPIFFE identities from the shared supervisor

6 participants