Skip to content

feat(kubernetes): configure workload and supervisor placement - #5629

Draft
TaylorMutch wants to merge 9 commits into
mainfrom
feat/5619-kubernetes-scheduling/TaylorMutch
Draft

TaylorMutch wants to merge 9 commits into
mainfrom
feat/5619-kubernetes-scheduling/TaylorMutch

Conversation

@TaylorMutch

@TaylorMutch TaylorMutch commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Prefer separate nodes for sandbox workloads and supervisors, including independently deployed shared supervisors. Operators can require separation or opt out, and can optionally select distinct node pools for gateways, supervisors, and workloads.

Related Issue

Closes #5619

Changes

  • Apply the consistent openshell.ai/runtime-role label to workload, supervisor, and gateway pods without changing Service or controller selectors.
  • Add gateway-owned workload placement and role anti-affinity with explicit namespace scope; default to preferred separation across all namespaces.
  • Reconcile placement on stop/start, preserve independently authored affinity and caller settings, normalize Kubernetes toleration defaults, reject conflicting selectors, and validate required placement after admission.
  • Configure reciprocal shared-supervisor rules through Helm, add opt-in supervisor selectors/tolerations, and place PKI hooks with the gateway.
  • Give scheduled pods the configured startup grace after a capacity wait; add one-to-three-node development support, E2E fixtures, and a Kubernetes pod placement guide with scenario diagrams.

Release Notes

Workloads and supervisors prefer separate nodes by default. Node selectors and tolerations remain opt-in. Existing pods acquire role labels and placement rules when recreated.

Testing

  • Checks appropriate to the affected code and behavior pass: driver and E2E Clippy, Rust formatting, Helm lint variants, generated chart docs, rendered-config preflight, resource coherence, ownership/scaling checks, Markdown/Mermaid lint, Fern validation, and docs navigation.
  • Unit tests added/updated: 347 Kubernetes-driver tests and 480 Helm tests pass. Coverage includes policy/scope transitions, placement conflicts, admission validation, and capacity-recovery timing.
  • E2E tests added/updated: sandbox create and stop/start pass with one eligible node using preferred anti-affinity, two eligible nodes using required separation across managed namespaces, and three distinct role pools with tainted supervisor/workload nodes.

The focused live test is mise run e2e:kubernetes:scheduling. It also passes with operator omitted from an operator-configured toleration. Its dedicated Cargo feature excludes the test from the general Kubernetes suite. Test fixtures use matching gateway, sandbox, and supervisor images built from this branch. The isolated validation cluster was removed after testing.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated where applicable: driver README, published gateway configuration, chart docs, and deployment skills.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 16, 2026
@github-actions

Copy link
Copy Markdown

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
…tart

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@TaylorMutch
TaylorMutch marked this pull request as ready for review November 18, 2026 18:33
@TaylorMutch
TaylorMutch requested a review from FrostGod November 18, 2026 18:34
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. helm — No explicit risk acceptance found

Helm previously rendered arbitrary gateway podLabels, but now rejects podLabels["openshell.ai/runtime-role"] unless its value is "gateway".

  • Impact: Existing releases using another value cannot render or upgrade, including when generated anti-affinity is disabled.
  • Action: Keep existing gateway podLabels valid in v0.1.10 and defer reserving this key to a minor release with instructions for migrating custom labels and their selectors.
  • PR: feat(kubernetes): configure workload and supervisor placement #5629

2. configuration — No explicit risk acceptance found

runtime_bootstrap_grace_seconds previously triggered reclamation of stale bootstraps, but released bootstraps with Unschedulable pods now bypass that deadline indefinitely.

  • Impact: Operators relying on the configured cleanup window can retain stranded sandbox resources instead of having failed creations deleted or failed restarts suspended.
  • Action: Preserve the existing TOML cleanup deadline in v0.1.10 and make extended or unlimited scheduling waits a separate opt-in setting.
  • PR: feat(kubernetes): configure workload and supervisor placement #5629

3. cli — No explicit risk acceptance found

HELM_K3S_AGENT_COUNT=2 mise run helm:k3s:create previously created three nodes, but now creates one unless callers switch to HELM_K3S_NODES=3 mise run helm:k3s:create.

  • Impact: Developer automation using the documented variable silently loses worker nodes, and existing SPIRE tasks fail their three-node requirement.
  • Action: Retain HELM_K3S_AGENT_COUNT as a backward-compatible fallback, emit only one --agents argument, and document HELM_K3S_NODES=3 as the replacement for HELM_K3S_AGENT_COUNT=2.
  • PR: feat(kubernetes): configure workload and supervisor placement #5629
Review details
  • Execution: complete
  • Candidate: PR #5629 (8c5337693e3948c26ecc4a73ea08e1f673f6026b)
  • Baseline: df6a799eb9f2ef92042363ad5b7e0d687c4ec03c (df6a799eb9f2ef92042363ad5b7e0d687c4ec03c)
  • Reviewed: rust-sdk, python-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@TaylorMutch
TaylorMutch marked this pull request as draft November 19, 2026 01:03

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(kubernetes): prefer workload separation from shared supervisors with common role labels

1 participant