Repository navigation
Description
User Story
I run OpenShell with extension services registered at both extension points: a gateway interceptor for sandbox admission and supervisor middleware for egress requests, both with fail_closed. Agent Policy Fabric is one example.
An extension service restarts whenever it is upgraded or its configuration changes. During each restart, OpenShell denies the CreateSandbox calls and egress requests that arrive, and users see them as intermittent permission errors that succeed on retry.
I need a restart of an extension service to be invisible to the agents running in sandboxes. Graceful shutdown and overlapping instances on the service side do not achieve this: calls that OpenShell had not yet delivered when the old connection closed are still failed and denied.
Problem Statement
While a registered gateway interceptor or supervisor middleware service is restarting, OpenShell fails every extension call that arrives, including calls the service never received. With fail_closed, each failed call becomes a denied gateway operation or a denied egress request. This affects Evaluate, SnapshotProviderProfiles and Describe on gateway interceptors, and EvaluateHttpRequest, Describe and ValidateConfig on supervisor middleware.
This happens even when the service hands over with no downtime. OpenShell keeps one HTTP/2 connection per extension service; when the service closes it gracefully, calls still queued in OpenShell's client, or refused by the shutdown notice, fail without reaching the service. The service cannot recover them because it never received them.
Impact / Why This Matters
- Every restart of an extension service can deny sandbox creations and egress requests. An agent sees a permission error on a request its policy allows.
- The failures are intermittent and clear on retry, so they are hard to attribute to the restart.
- The workaround is to schedule extension-service restarts for idle periods or to switch the registration to fail open. The first is not possible for long-running agents, and the second removes the enforcement the extension exists to provide.
Observed on OpenShell 0.1.9 on Kubernetes, with one sandbox sending about 45 egress requests per second and about 4 openshell sandbox create calls per second:
| Extension service rollout | Restarts with failed calls | Failed egress requests | Failed sandbox creations |
|---|---|---|---|
| Pod replaced, about 10 s without a ready instance | 2 of 2 | 42 of 2,344 | 27 of 247 |
| New pod ready before the old one stops, graceful drain | 4 of 40 | 3 of 80,690 | 1 of 6,412 |
Acceptance Criteria
- A unary extension call that the service did not receive is answered by the restarted service when it returns within the call's timeout.
- No call is delivered to the extension service more than once.
- A call to a service that stays down still fails under the configured failure policy, no later than its timeout.
- The behavior during extension-service restarts is documented.
Reproduction Steps
- Install the OpenShell Helm chart with one
openshell.gateway.interceptorsentry and oneopenshell.supervisor.middlewareentry, bothfail_closed, pointing at an extension service running as a Deployment in the same cluster. - Create a sandbox whose policy attaches the middleware to an endpoint:
openshell sandbox create --name egress --detach -- sleep 36000. - Send requests to that endpoint from the sandbox in a loop:
openshell sandbox exec egress -- sh -c 'while true; do curl -s -o /dev/null -w "%{http_code}\n" <endpoint>; done'. - In a second loop, run
openshell sandbox create. - Restart the extension service:
kubectl rollout restart deployment/<extension-service>. - Observe that egress requests return
403withmiddleware_failedand that sandbox creations fail withfailed closedduring the restart, and that the same calls succeed once it completes.
Environment
- OpenShell: 0.1.9 (CLI, gateway, supervisor and sandbox images; Helm chart 0.1.9)
- OS: macOS on Apple silicon, running the cluster in Docker
- OpenShell deployment mode and runtime: Kubernetes compute driver on a single-node kind cluster, gateway as a StatefulSet with one replica, plaintext development gateway
Logs
# openshell sandbox create, extension service has no ready instance
gateway interceptor '<name>' failed closed: interceptor transport error: evaluation timed out
# openshell sandbox create, extension service handing over with no downtime
provider profile source 'interceptor/<name>' snapshot failed: interceptor transport error: SnapshotProviderProfiles failed for '<name>': code: 'Internal error', message: "h2 protocol error: http2 error", source: tonic::transport::Error(Transport, hyper::Error(Http2, Error { kind: GoAway(...
# egress request from the sandbox
{"error":"middleware_failed","detail":"Request could not be processed by configured middleware","policy":"<policy>","layer":"l7","method":"GET","path":"/x", ...}
Activity