refactor(inference): remove managed inference routes - #3195
Conversation
|
🌿 Preview your docs: https://nvidia-preview-pr-3195.docs.buildwithfern.com/openshell |
Docker E2E Test AttestationLocal Docker E2E tests passed. CI does not currently run this lane, so this comment serves as the verification record.
Test SummaryThe Tests Executed
|
Python E2E Test AttestationLocal Python E2E tests passed against a Docker-backed gateway. CI does not currently run this lane, so this comment serves as the verification record.
Test SummaryThe 81 tests under Tests Executed |
Closes #3172 Remove the inference route control plane, inference.local data path, built-in router crate, and SDK surface. Move inference workloads to explicitly imported provider profiles and native endpoints, with migration cleanup and updated tests and documentation. Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
6599bfe to
36b3abd
Compare
johntmyers
left a comment
johntmyers
left a comment
There was a problem hiding this comment.
gator-agent
PR Review Status
The accepted issue and migration plan make this project-valid, and the broad removal is accompanied by updated SDKs, Fern docs, migrations, and replacement provider E2E coverage. One policy regression must be fixed before test dispatch: alternate-upstream legacy providers now grant access to the built-in public vendor endpoint even though their credential binding is withheld.
Action required: restore the endpoint-activation guard in provider policy composition and its OpenAI/Anthropic regression test.
Blocking findings:
GATOR-36b3abdf-01: alternate-upstream providers widen sandbox egress to the built-in public vendor endpoint
Carried findings:
- None
Non-blocking suggestions:
- None
Gator metadata
- Validation: linked accepted issue #3172 explicitly authorizes this cross-cutting managed-inference removal
- Docs: Fern docs and migration guidance updated
- Checks: current branch and Helm gates are green; required E2E has not yet been dispatched
- E2E:
test:e2erequired after review feedback is resolved - Head SHA:
36b3abdf4b313699c01f226b46836bfe2f6ec449 - Base SHA:
c96b9bff47f719e6f6bec21edd9ed1c33e3ba3ca - Merge base SHA:
8605943a2d5fec5707251fc67bba718cb9dcacb2 - Patch ID:
c581a82c7882eb073fa155abb32823afbce1deea - Gator payload:
8 - Review mode:
initial - Previous reviewed SHA: none
- Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:in-review
johntmyers
added
the
gator:in-review
Restore the provider policy activation guard so legacy OpenAI and Anthropic providers configured for alternate base URLs do not grant egress to the built-in public vendor endpoints. Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
|
Label |
johntmyers
left a comment
johntmyers
left a comment
There was a problem hiding this comment.
gator-agent
PR Review Status
Thanks @johntmyers. I checked your update restoring provider_profile_endpoints_are_active at the shared provider-policy composition point and the OpenAI/Anthropic alternate-upstream regression coverage. The prior vendor-egress finding is resolved, its review thread is closed, and the bounded follow-up review found no new blockers. Required E2E has been dispatched for this head.
Blocking findings:
- No blocking findings remain
Carried findings:
- None;
GATOR-36b3abdf-01is resolved by this head
Gator metadata
- Validation: linked accepted issue #3172 explicitly authorizes this cross-cutting managed-inference removal
- Docs: Fern docs and migration guidance updated in the reviewed PR
- Checks: branch and Helm gates are green; required current-head E2E attempt is queued
- E2E:
test:e2eapplied; E2E Label Help requested re-run all jobs; run33984167072was rerun and is queued - Head SHA:
3045866bde58a3402792ba3762ff5d4cd1dbe183 - Base SHA:
c96b9bff47f719e6f6bec21edd9ed1c33e3ba3ca - Merge base SHA:
8605943a2d5fec5707251fc67bba718cb9dcacb2 - Patch ID:
c139a1f471d12a710f86968cf8a5b05b9eb92ebb - Gator payload:
8 - Review mode:
follow_up - Previous reviewed SHA:
36b3abdf4b313699c01f226b46836bfe2f6ec449 - Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:watch-pipeline
johntmyers
added
gator:watch-pipeline
2 participants
Summary
Remove OpenShell's managed inference routing control plane and
inference.localdata path. Inference workloads now use explicitly imported, use-case-specific provider profiles attached to sandboxes and call provider-native endpoints directly;inference_capableremains informational for future observability hooks.Related Issue
Closes #3172
Changes
proto/,crates/openshell-core/,crates/openshell-server/, and SDKs: remove inference-route APIs, commands, generated clients, auth scopes, and persisted route behavior.crates/openshell-supervisor-network/andcrates/openshell-sandbox/: removeinference.local, request-shape matching, route refresh/caching, and inference-route supervisor configuration.crates/openshell-router/: remove the built-in privacy router crate and its packaging/dependency surface.crates/openshell-server/migrations/: delete legacyinference_routeobjects during SQLite and PostgreSQL upgrades.crates/openshell-providers/: retain provider aliases and Vertex constants, keepinference_capableinformational, and preserve endpoint credential-binding guards.e2e/: replace managed-route tests with explicitly imported OpenAI-style and Anthropic-style profile fixtures against native endpoints, including a negativeinference.localassertion.docs/,architecture/,examples/local-inference/, and agent skills: document the breaking migration and provide human-readable Now/After workflows.Deviations from Plan
None — implemented as planned in the approved issue comment and follow-up decisions.
Testing
mise run pre-commitpassesmise run testpassesmise run cipassesmise run e2e:dockerpassesmise run e2e:pythonpasses (86 passed, 81 OIDC-only skipped)Tests added:
x-api-keyprofiles calling native endpoints, secret substitution/model body assertions, and negativeinference.localbehavior.Checklist
Documentation updated:
docs/sandboxes/inference-routing.mdx: provider-backed inference workflow, breaking migration, and Now/After UX.docs/get-started/tutorials/andexamples/local-inference/: native endpoint examples using explicitly imported profiles.architecture/, provider/security/observability/reference docs, and agent skills: remove managed-route assumptions and align troubleshooting/workflows.