Skip to content

bug: extension calls fail closed while a gateway interceptor or supervisor middleware service restarts #5713

Open

Description

@dvavili

User Story

I run OpenShell with extension services registered at both extension points: a gateway interceptor for sandbox admission and supervisor middleware for egress requests, both with fail_closed. Agent Policy Fabric is one example.

An extension service restarts whenever it is upgraded or its configuration changes. During each restart, OpenShell denies the CreateSandbox calls and egress requests that arrive, and users see them as intermittent permission errors that succeed on retry.

I need a restart of an extension service to be invisible to the agents running in sandboxes. Graceful shutdown and overlapping instances on the service side do not achieve this: calls that OpenShell had not yet delivered when the old connection closed are still failed and denied.

Problem Statement

While a registered gateway interceptor or supervisor middleware service is restarting, OpenShell fails every extension call that arrives, including calls the service never received. With fail_closed, each failed call becomes a denied gateway operation or a denied egress request. This affects Evaluate, SnapshotProviderProfiles and Describe on gateway interceptors, and EvaluateHttpRequest, Describe and ValidateConfig on supervisor middleware.

This happens even when the service hands over with no downtime. OpenShell keeps one HTTP/2 connection per extension service; when the service closes it gracefully, calls still queued in OpenShell's client, or refused by the shutdown notice, fail without reaching the service. The service cannot recover them because it never received them.

Impact / Why This Matters

  • Every restart of an extension service can deny sandbox creations and egress requests. An agent sees a permission error on a request its policy allows.
  • The failures are intermittent and clear on retry, so they are hard to attribute to the restart.
  • The workaround is to schedule extension-service restarts for idle periods or to switch the registration to fail open. The first is not possible for long-running agents, and the second removes the enforcement the extension exists to provide.

Observed on OpenShell 0.1.9 on Kubernetes, with one sandbox sending about 45 egress requests per second and about 4 openshell sandbox create calls per second:

Extension service rollout Restarts with failed calls Failed egress requests Failed sandbox creations
Pod replaced, about 10 s without a ready instance 2 of 2 42 of 2,344 27 of 247
New pod ready before the old one stops, graceful drain 4 of 40 3 of 80,690 1 of 6,412

Acceptance Criteria

  • A unary extension call that the service did not receive is answered by the restarted service when it returns within the call's timeout.
  • No call is delivered to the extension service more than once.
  • A call to a service that stays down still fails under the configured failure policy, no later than its timeout.
  • The behavior during extension-service restarts is documented.

Reproduction Steps

  1. Install the OpenShell Helm chart with one openshell.gateway.interceptors entry and one openshell.supervisor.middleware entry, both fail_closed, pointing at an extension service running as a Deployment in the same cluster.
  2. Create a sandbox whose policy attaches the middleware to an endpoint: openshell sandbox create --name egress --detach -- sleep 36000.
  3. Send requests to that endpoint from the sandbox in a loop: openshell sandbox exec egress -- sh -c 'while true; do curl -s -o /dev/null -w "%{http_code}\n" <endpoint>; done'.
  4. In a second loop, run openshell sandbox create.
  5. Restart the extension service: kubectl rollout restart deployment/<extension-service>.
  6. Observe that egress requests return 403 with middleware_failed and that sandbox creations fail with failed closed during the restart, and that the same calls succeed once it completes.

Environment

  • OpenShell: 0.1.9 (CLI, gateway, supervisor and sandbox images; Helm chart 0.1.9)
  • OS: macOS on Apple silicon, running the cluster in Docker
  • OpenShell deployment mode and runtime: Kubernetes compute driver on a single-node kind cluster, gateway as a StatefulSet with one replica, plaintext development gateway

Logs

# openshell sandbox create, extension service has no ready instance
gateway interceptor '<name>' failed closed: interceptor transport error: evaluation timed out

# openshell sandbox create, extension service handing over with no downtime
provider profile source 'interceptor/<name>' snapshot failed: interceptor transport error: SnapshotProviderProfiles failed for '<name>': code: 'Internal error', message: "h2 protocol error: http2 error", source: tonic::transport::Error(Transport, hyper::Error(Http2, Error { kind: GoAway(...

# egress request from the sandbox
{"error":"middleware_failed","detail":"Request could not be processed by configured middleware","policy":"<policy>","layer":"l7","method":"GET","path":"/x", ...}

Activity

  1. added
    state:acceptedA maintainer decided OpenShell should pursue this issue
    and removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:acceptedA maintainer decided OpenShell should pursue this issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions