Repository navigation
Sync upstream sandbox, snapshot, and HTTPS egress updates - #69
Merged
Merged
Conversation
…ate#2244) Part of agent-substrate#2243 A worker can hold more than one actor, so two resumes can race for its last free slot. The loser is retried transparently, but each losing attempt was recorded on `ate.scheduler.assignment.duration` as `ate.scheduler.outcome="error"`. One resume gave two records instead of one. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ORTER=prometheus as none where they serve /metrics (agent-substrate#2190) Follow-up to agent-substrate#1878 (agent-substrate#1877). Uses the shared exporter parser from agent-substrate#2193, as agreed in agent-substrate#2067. - `OTEL_METRICS_EXPORTER=prometheus` stops the OTLP push on the components that serve `/metrics`, since the specification defines it as the pull exporter they already provide. Before, it warned and kept the push, so each series arrived twice. `otlp,prometheus` keeps the push. ateom serves no `/metrics`, so there `prometheus` is skipped as unknown with a warning and the push stays on. - `OTEL_TRACES_EXPORTER` is honored with the same rules, knowing `otlp` (the default) and `none`. With `none` the provider keeps sampling and propagating trace context but exports nothing. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…2300) This PR builds on the fake data plane skeleton (agent-substrate#2272) so actors can actually be placed and resumed on fake Workers. With the skeleton, every resume failed with `no_capacity`. All changes are benchmark-only, under `cmd/benchmarking/isolate/`. **No production code changes.** atelet, ateom, ate-api-server and atecontroller are untouched, and nothing here runs in a normal install. The fake data plane should cost ate-api-server what a real one does on every call a benchmark exercises. Otherwise control-plane throughput measured on it would look better than it really is. Each commit closes one gap where a fake Worker behaved differently from a real one. ## What's in this PR Six commits, best reviewed one at a time: 1. **fake-atelet: write a placeholder object for each external snapshot.** ate-api-server copies an actor's snapshot when it creates a tag, golden tags included, and refuses to copy an empty one. On an external `Checkpoint` and on `UploadPausedCheckpoint`, fake-atelet now writes one empty object under the snapshot URI. It writes to the backend atelet would use, chosen by `ATE_STORAGE_BACKEND`. The write runs inside the configured delay. If the write fails, the call fails, because a later tag of that snapshot would fail too. 2. **Report fake Worker capacity through fake-atelet.** This fixes the known limitation from agent-substrate#2272. ate-api-server accepts a capacity report only from the atelet on the Worker's node. So fake-workersync plays each fake Worker's ateom and sends the report to the fake-atelet on that Worker's node, which relays `RegisterWorker` under its atelet identity. - **Same values a real Worker reports:** cpu and memory come from the pool's `template.resources.limits`, or from the node's allocatable when no limit is set, which is what the downward API gives ateom. Max actors is ateom's default of 1000 unless the `ate.dev/fake-max-actors` annotation overrides it. The hardware identity, which ate-api-server requires, comes from `hardware.ProbeHost()`, as ateom's does. Each Worker reports once, when it is made, and keeps that capacity, as ateom reads its limits once at startup. - **`readyReplicas` means usable:** a pool counts a Worker ready only once ate-api-server has accepted its capacity. - **A bad annotation doesn't remove the pool:** if a pool's annotation doesn't parse, or is out of range (1 to 2³¹−1), the controller reports an error and leaves the pool's Workers as they are. - **Relay security:** the relay listens on its own port (`--relay-port`, default 8086) and implements only `RegisterWorker`. Both ends use mutual TLS with the pod-identity certificates and check the peer's SPIFFE ID. fake-atelet accepts only fake-workersync, and fake-workersync accepts only the atelet identity. Without that check, any caller could set any Worker's capacity. Both ends reread the trust bundle at each handshake, as the herder does, so a CA rotation during a long run doesn't need a restart. - **Relay connections:** fake-workersync finds each node's relay by listing the fake-atelet pods and keeps one connection per relay. A connection is closed once its relay leaves the pod list. - **Tests:** `fakeworkertest` issues certificates shaped like pod-identity ones, so both ends of the relay are tested over real TLS handshakes, including a trust bundle rotation. 3. **fake-atelet: mint each actor's certificate on activation.** In production, every `Run` and `Restore` with tunneled egress costs ate-api-server a certificate mint. The default install turns on tunneled egress. fake-atelet now mints under its atelet identity, and the call fails if the mint fails, just as tunneled egress fails closed in ateom. The mint time is added on top of `--delay` rather than counted inside it, so if mints slow down under load, resumes slow down by the same amount. `--mint-actor-certificate=false` leaves the mint out, to measure what it costs. 4. **fake-workersync: drain a Worker before deleting it.** Before this change, a Worker that no pool wanted was deleted at once, even with Actors still on it. Now it is retired the way a real worker pod is: drained, then deleted once `ListWorkerActorAssignments` finds no Actors on it. In production, ateom stops a terminating pod's Actors, but nothing does that here. So after `--drain-grace` (default 1m) the Worker is deleted anyway, and `DeleteWorker` releases the Actors still on it, as the real syncer's delete does. A draining Worker doesn't count as a replica. After a restart, `adopt` reads the draining state back from ate-api-server. 5. **fake-workersync: follow WorkerPool label and template edits.** These match what the real worker syncer and a Deployment rollout do: - A label edit on a pool is written onto its Workers in place with `UpdateWorker`. The Worker's version is the precondition, and a conflicting write is retried on the next pass. - An edit to the sandbox class, `template.resources.limits` or the `ate.dev/fake-max-actors` annotation replaces the pool's Workers. `sandbox_class` can't be changed on a Worker, and a real Worker's capacity never changes, since ateom reads its limits and actor count once at startup. A change in a node's allocatable leaves its Workers as they are, as in production. - Edits made while fake-workersync was down are still applied after it restarts. An adopted Worker keeps the capacity ate-api-server recorded and isn't reported again; one whose recorded capacity no longer matches its pool is replaced. 6. **fake-workersync: register each replacement as a new Worker.** A real Worker is named after its pod's UID, so a replacement pod is always a new Worker. Before this change, a replaced fake Worker came back under the same name, pod UID and IP. Then anything ate-api-server keyed by the old Worker could carry over to its successor, which production never allows. - **New identity for each Worker:** each fake Worker is now named with a fresh UUID, which is also its pod UID. `fakeworker.Name` becomes the stand-in pod name, shared by every Worker that fills the same index of a pool, and `adopt` finds this run's Workers by that pod name. - **Replacement while the old Worker drains:** a replaced pod gets its new Worker in the same pass, as a Deployment creates a replacement pod without waiting for the old one to terminate. This covers sandbox class, limits and max-actors edits, scaling down and back up, and Workers whose node has left the benchmark set; those are replaced on a benchmark node, as a pod is when its node goes away. If no benchmark node is listed at all, Workers aren't moved off their nodes, since nothing could replace them. - **Status counts only usable Workers:** a pool's status counts only the Worker currently serving each wanted pod. A Worker made from an older template (sandbox class, limits or max actors), past `spec.replicas`, or off the benchmark nodes doesn't count as a replica even if its drain failed. It gets no label write or capacity report while the drain is retried. ## Known limitations - **Other template edits:** only sandbox class, limits and max-actors edits replace fake Workers. Edits to the image, env or template labels roll a real pool's pods but leave fake Workers in place. - **One mint per activation:** fake-atelet mints an actor certificate once per `Run`/`Restore`. Real ateom also renews it while the actor runs, so multi-hour runs with long-lived actors under-count mint load. - **List calls while draining:** each draining Worker costs one `ListWorkerActorAssignments` per pass until its Actors leave or `--drain-grace` runs out. Large scale-downs therefore add some load during a measurement. - **Storage client at startup:** fake-atelet creates its object-storage client when it starts and exits if that fails. It runs as the atelet service account, which already needs storage access. - **Relay node check:** the relay client checks the relay's SPIFFE ID and chain, but not that the relay is on the Worker's node, which `internal/ateletdial` does check. A misrouted report is rejected by ate-api-server and retried on the next pass. ## Not in this PR The deploy manifest and the `benchmarking/workloads/deploy.sh --fake-data-plane` mode come in a follow-up PR. Deploying these binaries needs the following, which that PR adds: - the fake-atelet relay port - the servicedns trust bundle mount, for fake-atelet to verify ate-api-server - a Role that lets fake-workersync list the fake-atelet pods in `ate-system` Addresses agent-substrate#2261 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Part of agent-substrate#1748, split out of agent-substrate#1984: the gVisor ateom emits the actor usage records, using agent-substrate#2005's building blocks. agent-substrate#1984 keeps the atelet side. - **Sampling.** The ateom samples its actors every `--usage-sample-interval` (default 1m) and writes one `ate.actor.usage_sampled` record per actor per sample. - **Order.** Per activation: `initial` when it starts, then `periodic`, then `final` after a checkpoint or terminate succeeds. The final reading is taken before the snapshot, because a Full checkpoint stops the sandbox's cgroup. - **Discovery read.** `GetActiveWorkloadStats` serves the latest sample, so atelet's poll reads no cgroup. - **Re-host.** A read goes through the activation's own state, so a re-host during it cannot mix two activations' numbers. **Until agent-substrate#1984 lands**, atelet still writes its own stdout usage record, so each gVisor actor has two usage records, with different messages and keys. **Tested:** unit tests. The combined change was tested on ate-dev earlier, with `initial`, `periodic`, and `final` per epoch for both checkpoint and terminate.
Part of agent-substrate#1748, split out of agent-substrate#1984: the micro-VM ateom emits the actor usage records, using agent-substrate#2005's building blocks. agent-substrate#1984 keeps the atelet side. - **Sampling.** The ateom samples its actors every `--usage-sample-interval` (default 1m) and writes one `ate.actor.usage_sampled` record per actor per sample. - **Order.** Per activation: `initial` once the guest answers, then `periodic`, then `final` after a checkpoint or terminate succeeds. The final record reuses the newest sample, so the checkpoint waits on no guest read. - **Discovery read.** `GetActiveWorkloadStats` serves the latest sample, so atelet's poll reads no guest. - **Re-host.** Every guest read goes through the activation's own state, so a re-host during it cannot mix two activations' numbers. - **Behaviour change.** A guest-stats target that names another actor now logs and reports the actor pending instead of failing the discovery RPC. - **Initial read.** It runs off the resume path and recovers from a panic, as a sweep does. **Until agent-substrate#1984 lands**, atelet still writes its own stdout usage record, so each micro-VM actor has two usage records, with different messages and keys. **Tested:** unit tests. The combined change was tested on ate-dev earlier: CPU rises within an epoch and restarts at zero for resumed guests.
…agent-substrate#2322) The field name doesn't follow the convention and ends with "ref", but more importantly, it's not really used. Part of agent-substrate#1378
…bstrate#2236) When the caller's context ends during an atelet call, for example because its deadline passes during a slow restore, the control API keeps the actor assigned but returns `Internal` to the caller. So `ResumeActor` can leave an actor `RESUMING` on a worker whose restore already succeeded while the caller receives a server-failure status. When the workflow context has ended, return its error, so the caller gets `Canceled` or `DeadlineExceeded` (the `apierror` convention for context errors). The actor keeps its worker assignment, so a retry can finish the operation. A `Canceled` or `DeadlineExceeded` from atelet with a live context still returns `Internal`, as on main, so the caller is not told that it cancelled. `Unavailable` keeps the mapping from agent-substrate#2196. Permanent errors keep their crash handling. This follows the shape that @laoj2 suggested in review. The unit tests cover a cancelled and an expired workflow context, atelet `Canceled` and `DeadlineExceeded` with a live context, and the existing permanent failures. The gRPC functional test checks `Unavailable`, atelet `Canceled` and `DeadlineExceeded`, and a caller deadline that passes during a slow restore. In each case the actor stays `RESUMING` with its worker assignment, and a retry reaches `RUNNING` on that worker. A permanent local restore failure still returns `Internal`, records a crash, clears the assignment, and releases worker capacity. ### Validation On main `0b91488d79637052458a29a6ba6c757bcf7a9292`, on 2026-10-08, the `controlapi` (51.4 s) and `functionaltest` (64.9 s, `REQUIRE_DOCKER=true`) race suites pass. Linux-targeted `go vet` and `gofmt` are clean. Without the context-error branch, three `TestHandleAteletError` subtests fail. ### Earlier lab evidence These trials used the first version of this PR, which also mapped atelet's own `Canceled` and `DeadlineExceeded` with a live context. With the current version, the second and third rows return `Internal` again. The actor still stays assigned and the retry still works. A separate two-worker ARM64 lab ran three trials per error code. A diagnostic interceptor replaced the reply only after the real atelet `Restore` handler succeeded. It tested selected error replies, rather than physical packet loss. | Injected reply code | Before: client code | With this mapping: client code | Fresh trials with the fix | | --- | --- | --- | ---: | | `Unavailable` | `Unavailable` | `Unavailable` | 3 passed | | `Canceled` | `Internal` | `Canceled` | 3 passed | | `DeadlineExceeded` | `Internal` | `DeadlineExceeded` | 3 passed | All nine trials passed 72 assertions. The actor retained its worker assignment and one observed live guest; explicit retry reached `RUNNING` with the same actor UID, guest execution ID, worker Pod UID, and checkpoint from reply loss through retry. A local AX outcome-reporting candidate consequently retained an uncertain result instead of reporting a definitive failure. Those trials used the unchanged AX candidate `55602feda4accf71d17705d5e6c48bc2c1a5ff1e` on `be7b1a8`, and Substrate mapping commit `438291462203f253366b1e5f8fc4f5b3fd63af9f` on an earlier lab base containing the availability candidate from agent-substrate#2055. This PR contains only the three-file mapping/test patch, freshly checked on current upstream main; the earlier trials are supporting evidence, not a live run of this rebased branch. The richer retry contract discussed in agent-substrate#1562 and automatic worker recovery remain separate work.
agent-substrate#2302) ### ateom-gvisor: log per-actor checkpoint and restore phase breakdowns Follow-up to agent-substrate#1941, which gave ateom-microvm the per-phase `Checkpoint timing breakdown` / `Restore timing breakdown` records. This PR gives ateom-gvisor the same, so the report in agent-substrate#2009 covers both sandbox classes. #### Why it need On a gVisor pool the breakdown stops at atelet's `ateom_restore` / `ateom_checkpoint` phase. That phase is most of a gVisor wake — roughly 165 ms of a 190 ms resume in local measurements — and nothing inside it is timed, so there is no way to say whether a slow resume was the network setup, the pause container, the app container's `runsc restore`, or the wakeup probe. gVisor is the default sandbox class, so this is the larger blind spot of the two. #### What - **`cmd/ateom-gvisor`**: `CheckpointWorkload` and `RestoreWorkload` emit one joinable record per operation from a `defer`, with the actor identity, snapshot scope, `error.type` on failure, and one float-seconds field per phase under `ateom.actor.{checkpoint,restore}.duration.<phase>`. Both records are written on failure too, carrying the phases that completed and the failing step's elapsed time. - restore phases: `prep`, `egress_prepare`, `egress_join`, `net_setup`, `durable_dir`, `pause_rootfs`, `pause_create`, `pause_restore`, `app_rootfs`, `app_create`, `app_restore`, `wakeup_probe`, `activate`, `total` — sequential, they partition the total. The restore record also carries the container count. - checkpoint phases: `pause`, `checkpoint`, `resume`, `teardown`, `total`. A small `restoreTiming` / `checkpointTiming` pair and a `lap()` helper stamp the phases as the handler advances; `phaselog_test.go` covers them. - **`internal/ateomphaselog`** (new): the record rendering both ateoms now share — `Phase`, `SnapshotPhaseAttrs`, `LogSnapshotPhases`, `ScopeLogValue` — moved out of `cmd/ateom-microvm/phaselog.go` with its tests. Phase *names* stay private to each binary, as in agent-substrate#1941: they are implementation details of one runtime, not values of the `ate.snapshot.phase` metric enum, and this PR adds nothing to the registry. - **`cmd/ateom-microvm`**: a mechanical refactor onto the shared package; the micro-VM records are unchanged. - **`docs/observability.md`**: the ateom paragraph now describes both runtimes' records and their failure semantics. No new instruments, no registry change, no behavior change. #### Example ```json {"msg":"Restore timing breakdown","ate.actor.uid":"8f2a…","ate.snapshot.scope":"full", "ateom.actor.restore.duration.prep":0.004, "ateom.actor.restore.duration.net_setup":0.011, "ateom.actor.restore.duration.pause_restore":0.038, "ateom.actor.restore.duration.app_restore":0.097, "ateom.actor.restore.duration.wakeup_probe":0.021, "ateom.actor.restore.duration.total":0.183, "container_count":2} ``` Joined with atelet's record for the same actor (which carries `ate.sandbox.class=gvisor`), `phase_report.py` from agent-substrate#2009 lists the gVisor rows alongside the micro-VM ones. The analyzer needs a small follow-up to stop assuming every ateom record is micro-VM (class from the paired atelet record, residual rule by class); that goes with agent-substrate#2148. ### Testing - `go test ./internal/ateomphaselog/...` — 4 tests (record shape, seconds not nanoseconds, scope mapping, `error.type` for gRPC / context / plain errors).
Be consistent with other sub-resources (e.g. EgressPolicy) and validate in the RPC layer and let store honor what the client specifies.
…te#2140) Part of agent-substrate#1756. - Adds `kubectl ate admin add-jwt-key`, `activate-jwt-key`, and `remove-jwt-key`, so rotating the actor JWT signing key no longer means editing the pool Secret by hand. A new key is published but signs nothing until activated, which gives relying parties time to fetch it. - Adds `list-jwt-keys`, which prints each key's ID and algorithm and marks the active one, so finding the key to remove doesn't mean decoding a Secret full of private keys. - `ConcretePool` gains `AddAuthority`, `Activate`, `RemoveAuthority`, and `ActiveID`. They refuse a duplicate or empty key ID, activating a key the pool doesn't hold, and removing the key that signs. - Each command writes the Secret back conditional on the resourceVersion it read and retries on a conflict, so a concurrent edit isn't lost. - Generated RS256 keys are now 4096-bit RSA instead of 2048-bit. RS256 stays only for relying parties that require it, such as Azure workload identity federation; ES256 is still the default. - The pool commands share one Kubernetes client helper, and make-jwt-pool's flag variables are renamed so the new commands can reuse them. The commands don't wait for relying parties; the operator decides when to activate and remove. The `--key-id` help describes the PKIX key IDs from agent-substrate#2132, which main doesn't use until that PR merges. Testing: unit tests for the pool methods, and fake-clientset tests for a full rotation, a conflict retry, refused changes, and the listing, plus a test that generated RS256 keys are 4096-bit. Not run against a cluster. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…nt-substrate#2315) Validate CDI device node paths under /dev via os.OpenRoot so symlinks cannot escape the device root, and require CDI mount HostPaths (and ELF DT_SONAME reads during SONAME symlink staging) to resolve within an explicit AllowedHostDirs allowlist via os.OpenRoot). Fixes agent-substrate#2091 This code isn't used at the moment, but we know we want to add back device injection in the future, and we know AI security scans will find it. For now I think that's cheaper than deleting and re-introducing (and possibly bringing back this bug)
…e. (agent-substrate#2354) Also add a resource-model.md markdown file that goes over the current resources and reference it from AGENTS.md in API proto directory.
…ubstrate#2314) Adds `--postgres-storage-class` / `ATE_INSTALL_POSTGRES_STORAGE_CLASS`. Unset keeps the cluster default, since the claim template is immutable on existing installs. Fixes agent-substrate#2298 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…t-substrate#2224) Part of agent-substrate#1827 Move external snapshot transfers in atelet and ate-api-server behind the `objectstoresnapshot.v1` plugin API, and ship a reference plugin for GCS and S3 that runs as a sidecar next to each. ## What changes | Commit | Change | |---|---| | objectstoreplugin: add object-store plugin and snapshot-plugin binary | `internal/objectstoreplugin` implements `NodeProvider` and `ControlProvider` on GCS/S3. `cmd/snapshot-plugin` serves one of them on a Unix socket (`node` or `control` mode). A `healthcheck` mode checks a running plugin's gRPC health service, for the sidecar's startup probe. | | atelet: transfer snapshots through NodeProvider | atelet fetches and uploads external snapshots through the node plugin instead of calling object storage. | | ateapi: clean up and copy snapshots through ControlProvider | ate-api-server deletes and copies external snapshots through the control plugin instead of its own storage client. | | manifests: run snapshot-plugin as atelet and ate-api-server sidecars | Native sidecars next to atelet (node mode) and ate-api-server (control mode), each sharing a pod-local emptyDir for the socket. Each sidecar has an exec `startupProbe` running `snapshot-plugin healthcheck` against its socket, so the main container starts only once the plugin serves. | ## Reviewing - **Moved code.** The node plugin's per-file upload and download are atelet's code moved as is, so it can be found in the old location on `main`. The control plugin wraps the existing `objectstore.DeletePrefix` and `CopyPrefix`. New code is limited to the gRPC glue, request validation, the startup health check, the binary, and the manifests. - **Unchanged object layout.** Files are stored as `<file>.zstd` with an uncompressed `manifest.json`, so snapshots written before and after this change are interchangeable. The manifest still goes last, in its own call, and remains the commit marker. - **Validation added at the plugin boundary.** The plugin now serves a socket API, so it checks every request: the snapshot URI, that the local directory is inside `--root` (`/var/lib/ate`), and that file names are base names with no duplicates. Every directory atelet passes today is under `/var/lib/ate`, and the file-name checks are the ones atelet already applied before transferring. - **A plugin that never comes up fails fast.** Clients dial with `WaitForReady`, so without a check the first transfer would hang. atelet and ate-api- server run `objectstoreplugin.WaitReady` (a gRPC health check) at startup with a one-minute bound and exit if the plugin is not serving. The sidecar's `startupProbe` runs that same health check by executing `snapshot-plugin healthcheck` inside the container, so the kubelet does not start atelet or ate-api-server until the plugin serves. Transfer RPCs get no fixed deadlines, since one would be wrong for large snapshots - **Shared backend selection.** atelet's `ATE_STORAGE_BACKEND` selection moves to `objectstorage.NewFromEnv` in `pkg/objectstorage`, along with `UsesS3` and `NewS3ClientFromEnv`, so atelet and both plugin modes select the backend the same way. These are new exported functions in `pkg/`. - **Package location.** The plugin lives in `internal/objectstoreplugin`, not `internal/plugins/`, because atelet and ate-api-server tests import its in-process clients (`objectstoreplugintest`) and it imports `internal/objectstore`. ## Not in this PR - `actor_jwt` is not set yet. Per-actor authorization is agent-substrate#777. ## Breaking change The object storage settings (`ATE_STORAGE_BACKEND`, `AWS_*`) move from the `ate-api-server` container to its snapshot-plugin` sidecar, which is now the only process in that pod that touches object storage for snapshots. ate-api-server exits at startup if `ATE_STORAGE_BACKEND` is still set on it. An install that patches these settings onto the `ate-api-server` container must patch the sidecar instead; see the kind overlay for an S3 example. ## Testing - `make test` and `make verify`. - New unit tests for the plugins (`internal/objectstoreplugin`), backend selection (`pkg/objectstorage`), and the shared validation helpers (`internal/resources`). - E2E on kind runs in CI (gVisor and micro-VM, S3 via rustfs). - E2E on GKE (GCS)
Add an e2e suite for the actor isolation boundary Tests added (internal/e2e/suites/security): - TestCrossTenantIsolation: the victim tenant writes a marker to its durable-dir volume, reads it back, and is suspended; the attacker tenant then resumes on the same worker. Controls confirm the probe reports missing files as errors and can read its own durable data. - VictimDurableMarkerNotVisibleThroughAttackerMount: the victim's marker is not readable through the attacker's own /data mount. - VictimHostStateNotReachableFromAttackerSandbox: the victim's durable directory, projected actor identity, and sandbox asset record are not readable at their host paths (taken from internal/ateompath), and the attacker cannot write into the victim's durable directory. - TagScopedToAtespaceCannotSeedAnotherTenant: an ATESPACE-scoped Tag created from the attacker's snapshot cannot seed an Actor in the other atespace. The test asserts the tag-scope rejection message, not just FailedPrecondition, since other preconditions on that path return the same code. A same-atespace clone from the tag restores the source's durable data, proving the tag was usable. Fixtures (internal/e2e/fixtures/security): a pool-bearing tenant, a namespace-only second tenant that must schedule onto the first's pool, and a shared probe-based ActorTemplate with a system-info volume and a durable-dir volume. The pool-bearing half is registered in sandbox_test.go's fixture rendering checks. Fixes #<issue_number_goes_here> > It's a good idea to open an issue first for discussion. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…-substrate#2333) A SandboxConfig can now carry several sandbox versions, with `spec.defaultVersion` selecting the one actors boot with: ```yaml kind: SandboxConfig metadata: name: gvisor-default annotations: sandboxconfig.ate.dev/is-class-default: "true" spec: sandboxClass: gvisor defaultVersion: v1 versions: - name: v1 state: Enabled pauseImage: "registry.k8s.io/pause:3.10.2@sha256:..." assets: {...} - name: v0 state: Disabled pauseImage: "registry.k8s.io/pause:3.10.2@sha256:..." assets: {...} ``` - `spec.pauseImage` and `spec.assets` move into `spec.versions[]`. Each entry has `name` (a DNS label, unique in the list), `state` (`Enabled` by default, or `Disabled`), `pauseImage` and `assets`. A config holds 1–16 versions. - CEL requires `defaultVersion` to name an existing version that is not `Disabled`, and applies the per-class pauseImage rule to every version. - The ValidatingAdmissionPolicy checks the per-class required assets on every version. - The shipped `gvisor-default` and `microvm` configs now use `defaultVersion: v1` and carry the `sandboxconfig.ate.dev/is-class-default` annotation. More than one config of a class may carry it. Nothing reads it yet; a later change resolves it newest-wins. - ate-api and atelet prewarm read only the default version for now. #### Breaking change Every SandboxConfig must use the new shape. The apiVersion stays `v1alpha1` and there is no conversion, so once the CRD is updated, `spec.pauseImage` and `spec.assets` are pruned from existing objects, and ate-api rejects actors that name them (FailedPrecondition) until they are re-applied with `defaultVersion` and `versions`. The install re-applies `gvisor-default` and `microvm`; custom SandboxConfigs must be rewritten by hand. Roll out the CRD together with ate-api: an older ate-api reading a new-shape config gets an empty pause image. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Part of agent-substrate#1660. - Adds the `egress/actorjwt` package, whose `Minter` mints actor JWTs through `Control.MintActorJWT`. Tokens are cached by actor, sorted audiences, and lifetime, keyed by the deterministic protobuf encoding of the mint request, and each is reused until a third of its lifetime remains. - Concurrent misses share one mint. It keeps running if the request that started it goes away and has no timeout of its own, so a slow mint still fills the cache. - The gateway knows the actor only by name, so it sends no UID (agent-substrate#2207). An actor recreated under the same name can get its predecessor's token until the refresh point. Nothing calls the minter yet, so `actor_jwt` entries still get a 501. The next PR in the series wires it into `applyEffects`. Testing: unit tests call a fake Control server over bufconn and cover reuse, keying, errors, and concurrent and canceled callers. They pass under `-race`. E2E hasn't run. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (no user-facing change yet; docs come later in the agent-substrate#1660 series)
Today atelet wrote each container's `config.json`, and each ateom loaded it and reshaped it for its runtime. To better decouple atelet and ateom, the ateom Container message now carries each container's `argv`, `environment`, `capabilities` and resource limits next to its volume mounts. atelet still resolves argv and env, because it pulled the image and has the image config. It also still resolves the capability set from the template. `ocispec.Build` now takes the ateom Container and the actor's directories. - ateom-gvisor builds and saves the spec before runsc create or restore, including the pause container's. - ateom-microvm builds it in memory. For the bundle, atelet only pulls images and writes the overlay spec. The pause container also starts with an empty environment instead of the pause image's ENV and a default PATH. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…-substrate#2372) see: agent-substrate#2077 prevents path traversal. we append to the base path here with a random actor UID so it should be hard to exploit this but let's prevent path traversals anyhow. still discussing if we should make other changes, but this is cheap to do now.
…ferred_fidelity (agent-substrate#2364) **This is a rename, not a behavior change.** `SnapshotConfig.onCommit` becomes `preferredFidelity`, and the two-valued `SnapshotContentScope` enum becomes the ordered `SnapshotFidelity` enum, with the same vocabulary carried through atelet, ateom, and the metric label. `MEMORY` is exactly what `FULL` was and `VOLUMES` is exactly what `DATA` was: what gets captured on pause and suspend, what gets restored, the golden-actor override, and the paused-upload narrowing are all unchanged. The point is to put the lifecycle-v2 names in place before step III adds `required_fidelity` and `resumed_fidelity` on top of them. The one addition is `ROOTFS`, defined in the enum so it is complete but rejected everywhere, since no runtime can produce it yet. Part of agent-substrate#2102, step II as tracked in agent-substrate#2351. Step I (agent-substrate#2309) removed `on_pause`. Two commits, meant to be read in order. ## 1. ateapi: `SnapshotFidelity` and `preferred_fidelity` - New ordered enum `SnapshotFidelity { VOLUMES = 1, ROOTFS = 2, MEMORY = 3 }`, each level including the ones below it. `SnapshotContentScope` is deleted. MEMORY is the old FULL and VOLUMES the old DATA, so behavior is unchanged. - `SnapshotConfig.on_commit` becomes `preferred_fidelity`, documented as a best-effort ceiling: the system never packages more than this level and may package less when a layer cannot be captured or is too expensive at the time. A process resumed in place keeps its memory regardless. Defaults to MEMORY. Field numbers rearranged to `storage_location = 1`, `preferred_fidelity = 2`. - `ExternalSnapshot` and `LocalSnapshot` record `fidelity` instead of `content_scope`. - ROOTFS is in the enum so it is complete, but no sandbox runtime captures rootfs without memory yet, so a new field-level validator rejects it at template admission. Actors only take their fidelity from a template, so none can carry it. - kubectl-ate tag output, demos, benchmarking manifests, glossary, API guide, and architecture doc use the new vocabulary. The obsolete apitool exemption is removed. ## 2. atelet, ateom: the same enum on the wire - Both internal protos replace `SnapshotScope` with a `SnapshotFidelity` mirroring the public one; request fields become `fidelity` and `desired_fidelity`. Every hop maps level by level, ROOTFS included, and nothing is promoted: an unexpected value goes out as UNSPECIFIED for the next hop to reject. - Defense in depth for ROOTFS: atelet rejects it in checkpoint, restore, and paused-upload validation, and both ateom runtimes reject it on `CheckpointWorkload` and `RestoreWorkload` through a shared `resources.ValidateSnapshotFidelity`. - A golden snapshot must record MEMORY (UNSPECIFIED is no longer tolerated). - The metric attribute `ate.snapshot.scope` (`full`/`data`/`unknown`) becomes `ate.snapshot.fidelity` (`volumes`/`rootfs`/`memory`/`unknown`) in the OTel registry, `ateattr`, the ateom phase logs, and the snapshot manifest atelet writes and reads back when narrowing a paused upload. Dashboards keyed on the old label need updating. ## Behavior changes - **Breaking API change.** Templates that set `onCommit` or use `SNAPSHOT_CONTENT_SCOPE_*` values are rejected. Per the pre-v1.0 policy there is no compatibility shim, and previously stored templates and snapshots are not expected to exist. - Snapshot manifests written with the old `scope` key read back with no fidelity, which the upload path already refuses with a clear message. - Unchanged: what MEMORY and VOLUMES capture and restore, the golden-actor MEMORY override, and the paused-upload narrowing from MEMORY to VOLUMES. ## Relationship to agent-substrate#2289 agent-substrate#2289 restructures the snapshot status into `Snapshot` and `SnapshotStorage`. If it lands first, the `fidelity` field in this PR moves onto `SnapshotStorage` (per copy), which is where it belongs long-term; the enum, the template field, and the data-plane change are the same either way. ## Follow-ups (step III) `required_fidelity` with `required <= preferred` validation and the CRASHED rule, `ResumeActorResponse.resumed_fidelity`, and ROOTFS support in the micro-VM runtime. ## Verification - `go test` for ateapi (unit and functional), kubectl-ate, atelet, ateom-microvm, ateom-gvisor, ateomphaselog, ateattr, resources, and apitool pass; new validator at 100% coverage. - golangci-lint, kube-api-linter, python-protos, boilerplate verifiers pass; codegen, proto-fmt, licenses, go.mod show no drift. - Not run locally: the Weaver metrics-registry check and shellcheck, which need Docker; relying on CI for those. e2e suites compile but were not run against a cluster. - [x] Breaking change - [x] Not backward compatible 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…gent-substrate#2332) WorkerPool now declares its sandbox class as `spec.sandboxClasses`, with an optional reference to a SandboxConfig: ```yaml spec: sandboxClasses: - name: gvisor configRef: name: gvisor-default ``` - `spec.sandboxClass` is replaced by `spec.sandboxClasses`, a list of `{name, configRef}`. The list is required and must have exactly one entry for now. `name` is `gvisor` or `microvm` and has no default. - `configRef` is optional and names a cluster-scoped SandboxConfig. Nothing reads it yet: actors still get their sandbox binaries from the ActorTemplate's SandboxConfig. - Code that needs the pool's class calls `WorkerPoolSpec.DefaultSandboxClass()`. #### Breaking change Every WorkerPool must now set `spec.sandboxClasses`. Once the CRD is updated, `spec.sandboxClass` is pruned, so re-apply each existing pool with `sandboxClasses` set (`- name: gvisor` for pools that used the old default). - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR
… is down (agent-substrate#2380) Calls to the snapshot plugin used `WaitForReady(true)`, so once the sidecar went down after startup a call waited out the caller's deadline (10m in ate-api-server, unbounded without one). - Calls wait at most 30s for the connection, then fail with `Unavailable`. Calls already running aren't cut short, and a sidecar restart within the window is ridden through. - atelet and ate-api-server now return a plugin `Unavailable` as `Unavailable` instead of `Internal`, so ate-api-server retries instead of crashing the actor. - The upload right after `CheckpointWorkload` still waits, since the workload is already gone and failing there would lose state. Tested against a real Unix-socket plugin: a 40s outage fails at 30s, a 12s outage succeeds. - [x] Tests pass --------- Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
Remove the extra `spec` wrapper from the SystemInfo volume examples and use the `ACTOR_METADATA_FIELD_*` enum names, so the examples can be parsed by `kubectl ate create actor-template`. To reproduce, copy the `actorMetadata` YAML example from the SystemInfo Volumes section of [docs/api-guide.md](https://github.com/agent-substrate/substrate/blob/760e577a38e66c0de51d05b3dad9a4db7dd0ff51/docs/api-guide.md#L188) into `actor-template.yaml`, then run from the repository root: ```bash go run ./cmd/kubectl-ate create actor-template -f actor-template.yaml ``` This fails with `unknown field "spec"`. Remove the top-level `spec` key and unindent its contents, then run the same command again. It now fails with `invalid value for enum field field: "name"`.
…once (agent-substrate#1984) The ateoms now write the actor usage records and serve their latest sample (agent-substrate#2194, agent-substrate#2195), so atelet stops writing its own and counts CPU per activation. - **atelet CPU.** Baselines are keyed by actor and epoch. A new activation counts in full if it began after atelet started, and a worker that does not answer keeps its baselines. This fixes the undercount on micro-VM restores and the double charge when a silent worker's baselines were dropped. - **atelet records removed.** The ateoms' `ate.actor.usage_sampled` records replace them, ending the period with two usage records per actor. - **Timeout.** `GetActiveWorkloadStats` answers from memory, so atelet's `statsRPCTimeout` drops from 55s to 5s. The 50s poll floor stays. - **Cleanup.** `UsageEmitter.EmitFinal` and its flush, atelet's stdout `SyncedWriter`, and the exported `actorlog.LabelsKey` go; nothing uses them now. - **Docs.** The proto comments describe the cached discovery read. The registry and the guide say when the final record is written: whenever the ateom tears an activation down (checkpoint, terminate, graceful shutdown, or a micro-VM re-host), read just before the teardown. #### Known limitation and future work atelet's CPU counter only sees samples of actors still hosted, so it loses each activation's tail: the CPU from the last sample atelet read to the teardown, up to one poll interval plus one sample interval. An activation shorter than that, the common checkpoint-when-idle case, can lose most or all of its CPU in the metric. The usage records keep it, through the final record. To fix it later, the ateoms can own the counter: each ateom adds an activation's CPU increase at every reading, initial, periodic, and final, and exports it through the existing OTLP relay. No gRPC change is needed. atelet then drops the baseline map, its start time, and the silent-worker rule; this also removes the baselines a crashed worker's leftover directory keeps. The cost is cardinality: the series become one per worker pod per label set instead of one per node, still bounded by pods rather than actors, plus a short-lived series per replaced pod. A collector can drop the pod attributes if that matters. #### Breaking change The stdout usage record moves from atelet to the ateoms, with flat `ate.*` keys and the message `Actor usage sampled`. Log filters on atelet's `Actor usage sample` record stop matching: filter on `jsonPayload.msg="Actor usage sampled"` and the `ate.*` keys instead. Identity is no longer in the label group, so on GKE it does not land in `LogEntry.labels`. Part of agent-substrate#1748 (steps 4 to 6). - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…t-substrate#2258) Part of agent-substrate#2212 (o11y side of agent-substrate#2211). If the gVisor sandbox dies after start (sentry crash, kill, OOM kill), nothing notices today. The actor stays RUNNING and its usage samples keep reading like an idle actor's, because the pause cgroup leaf and its counters outlive the processes in it. This makes the death visible: - `cgroupstats` also reads `populated` from `cgroup.events` and `oom_kill` from `memory.events`. If either file is missing, it never reads as a dead sandbox. - The ateom's usage sweep (`sweepUsage`, every `--usage-sample-interval`, 1m by default) logs one WARN per activation when the leaf is empty. Since agent-substrate#2194, `GetActiveWorkloadStats` serves the sweep's cached sample and reads no cgroup, so the check lives in the sweep, the only periodic cgroup read; `measure` hands back the raw `cgroupstats.Sample` for it. The record uses `ateattr.ActorLogAttrs` plus `ate.sandbox.oom_kills`, so filtering on `ate.actor.uid` finds it with the rest of the actor's logs. - Actors with a lifecycle RPC holding or waiting on their lock are skipped, since checkpoint and terminate empty the cgroup on purpose. `actorlock` gets a non-blocking `Busy` for this. An activation whose final record is written (checkpoint, terminate, drain) is no longer swept at all. - `readKeyedUint` walks lines in place instead of using a `bufio.Scanner`, which allocated 4 KiB per file. With the two extra reads, reading 1000 actors' cgroups takes about 9% longer than before and allocates half the bytes. It does not change actor state, RPC responses or the stats samples. Moving the actor to CRASHED is agent-substrate#2211. It also does not catch an app that exits while the sentry stays up. Tested on kind and on GKE (substrate-relay) with the branch image on the egress pool, before the rebase onto agent-substrate#2194 (the check then ran in `GetActiveWorkloadStats`). Unit tests pass at the rebased head; live re-verification on kind is pending: | Case | kind | GKE | |---|---|---| | OOM kill via `memory.max` | 1 WARN, `oom_kills=3` | 1 WARN, `oom_kills=1` | | `kill -9` the sentry | 1 WARN, `oom_kills=0` | 1 WARN, `oom_kills=0` | | Normal suspend | none | none | | Suspend then resume | none | none | | Idle actors | none (5) | none (3) | Each warning showed up on the first sweep after the fault and was not repeated. On GKE it lands in Cloud Logging as severity WARNING with the `ate.*` fields. The stats pipeline was unchanged. One caveat: `oom_kill` counts over the cgroup's lifetime, so a leaf left behind by an earlier activation of the same actor (agent-substrate#2107) carries its count forward. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
) Partially implements the proposed API changes in: https://docs.google.com/document/d/1lQ6fhrRb2sLCeFrvhsfytaoKVbA8X4w0TEwj84pr8uU/edit?tab=t.0 Depends on (and temporily includes) agent-substrate#2222, review that one first. Part of agent-substrate#2069 This PR tries not to make major changes to existing semantics. They will be done as a followup. Other things in the proposal that are not implemented in this PR (tracked separately) - hardware information (agent-substrate#2246) - sandbox information - combining pause/resume and introducing detach (agent-substrate#798) #### Breaking change The Snapshots API has been completely rewritten and replaces - ActorStatus in_progress_snapshot_uri, external_snapshot, local_snapshot, in_progress_local_snapshot_name fields. - TagStatus snapshot, storage_location fields ---- - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
agent-substrate#2009) ## benchmarking/analysis: aggregate suspend/resume phase breakdown log records ### Why agent-substrate#1941 makes atelet and ateom-microvm write one structured `Checkpoint timing breakdown` / `Restore timing breakdown` record per operation per layer, so that `SuspendActor` / `ResumeActor` latency can be attributed to the phases inside it. A log line is not a benchmark result, though: something has to get those records off the cluster and turn them into percentiles. Today nothing in `benchmarking/` reads node-side logs — the runner keeps its own `logs.txt`/`traces.txt` and the orchestrator tails the locust job. This PR adds that consumer, so a run can answer "where did the 6 s go?" from data rather than from one anecdote. ### What Everything lives under `benchmarking/analysis/`; no production code changes, and **no new log output** — this only reads what the binaries already write. - **`collect_logs.sh`** — runs `kubectl logs` for every atelet pod (`ate-system`) and every worker pod (`benchmark-workloads`) for the run's window and writes them to `--dest`, next to the locust artifacts. Two records per lifecycle operation is the entire volume it collects. - **`phase_report.py`** — parses the records from those dumps (tolerant of `kubectl` line prefixes and unrelated lines) and prints: - **Phase percentiles** per layer, operation, scope and phase (count, p50, p90, p95, max), with the atelet `ateom_checkpoint` / `ateom_restore` bucket split further by the ateom rows. - An **`unattributed`** row for the two checkpoint layers: the total minus what the logged phases account for, counting the three concurrent captures once. It is the time the instrumentation does not yet name. - **Waterfalls** of the slowest N operations, with the ateom record nested under the atelet phase it decomposes and the gap between the two (RPC and queueing between the layers). - Failed operations are excluded from the percentiles via the `error.type` marker on the record. - `--csv` writes `phase_percentiles.csv` for run-over-run comparison. - **`test_phase_report.py`** — 10 unittest tests covering parsing of both layers, nanosecond timestamps, prefixed lines, failure exclusion, the residual arithmetic and the waterfall join. - **`README.md`** — the record shapes, how to collect and aggregate, how to read each section and its concurrency caveats. `benchmarking/README.md` gains a short section pointing at it. - docs/dev/suspend-resume-phase-breakdown.md — an end-to-end runbook for a dev cluster: deploy the benchmark workloads on a micro-VM pool, drive glutton suspend/resume load with a sized working set, collect (kubectl or Cloud Logging), report, read the tables, and the failure modes that come up in practice. Linked from CONTRIBUTING.md beside the local micro-VM guide. ### Scope and follow-ups - *Standalone tool*. Nothing invokes it yet. benchmarking/automation/orchestrator.py deletes the workload and ate-system pods right after each test, so the collection has to run before that teardown. The follow-up is to call collect_logs.sh and phase_report.py from run_test() before the teardown and write the phase percentiles into the run's --dest as stats.jsonl rows (the suite's existing result format: timestamp, tag, test_name, metric, measurements), so they travel with the rest of the run's artifacts. **Follow-up (agent-substrate#2148):** the automated runs will consume this through the locust runner, which reads the pod logs itself before the orchestrator's teardown and appends the percentiles to stats.jsonl — that lands in a separate PR stacked on this one (branch phase-logs-runner; PR link to follow), so this PR stays the tool and its docs.
…nd CONNECT (agent-substrate#2308) Reject ingress requests that carry multiple `ate-target-actor` header entries across HTTP (`404 Not Found`), `CONNECT` (`%CEL(request.headers['ate-target-actor'])%` filter state + outer `CONNECT` precedence), and `atunnel` (`421 Misdirected Request`), preventing header smuggling mismatches between upstream Envoy RBAC/CEL authorization policies and `ext_proc` actor routing. Fixes agent-substrate#2307 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Add Substrate Security documentation covering security philosophy, scope, extensibility, and defensible multi-tenancy principles across identity, storage, networking, authorization, and auditability. Update README.md to reference the new security guide.
Naming the field `volume_name` is redundant. Rename to to `name`. Also rename `actor_volumes` to `external_volumes`. Part of agent-substrate#1378
…gent-substrate#2337) Part of agent-substrate#294. Takes over the half that agent-substrate#1067 left open and supersedes the stale agent-substrate#311. ## Summary `kubectl ate logs actors` gains `--source` to pick a class of lines from an actor's stream: | Flag | Shows | |---|---| | `--source=all` (default) | everything, unchanged | | `--source=containers` | every container's output with the lifecycle events removed | | `--source=lifecycle` | only the lifecycle events (`Actor starting/started`, `checkpointing/checkpointed`, `restoring/restored`, `terminated`) | | `-c NAME` | unchanged; implies container output, so it combines with `--source=containers` and is rejected with `--source=lifecycle` | The selection is client-side, as the issue intended: lifecycle events are the lines ateom emits with no `ate.actor.container.name` label (`ActorLogLabels` omits the key when the container name is empty). Named "lifecycle" rather than "supervisor" or "ateom" because the code and docs already use that word for these events and it leaks no implementation detail. ## Testing **Unit** (`go test ./cmd/kubectl-ate/...`): new filter table cases for every source, `TestNewLogLineFilter` for the flag validation, and `TestLogsActorRunner_Run_OneShot_SourceFilter` covering each source against an interleaved stream plus the two rejected combinations, which must fail before any logs are fetched. **End to end** on a kind cluster (`kind-ate-dev`) against the counter demo actor, using the plugin built from this branch. One-shot. Lines starting with `#` are annotations added for review; `$` lines are the commands as run. Two commands are negative cases and are **expected** to exit 1. ``` # The main commit this branch was built on; the plugin binary under test was built from the branch with make build-atectl. $ git rev-parse --short HEAD df78882 # Precondition: the actor is running, so its current worker's log stream is available. $ kubectl ate get actors -a ate-demo-counter ATESPACE NAME TEMPLATE STATE WORKER POD WORKER IPS VERSION AGE ate-demo-counter learner-one ate-demo-counter/counter ACTOR_STATE_RUNNING ate-demo-counter/counter-79b587778f-8lh4h 10.244.0.22 19 14h # Expected: only lifecycle events, no container output. Each suspend/resume cycle on this worker shows as a checkpointing/checkpointed or restoring/restored pair. $ kubectl ate logs actors learner-one -a ate-demo-counter --source=lifecycle {"time":"2026-10-07T21:31:23.367965976Z","message":"Actor restoring","span_id":"31bf9ef08d13d4d6","trace_flags":"01","trace_id":"61eb2ac56531db2b0308c31c564e6379"} {"time":"2026-10-07T21:31:23.550431306Z","message":"Actor restored","span_id":"31bf9ef08d13d4d6","trace_flags":"01","trace_id":"61eb2ac56531db2b0308c31c564e6379"} {"time":"2026-10-07T21:31:48.648599369Z","message":"Actor checkpointing","span_id":"c475c334c8df2dc4","trace_flags":"01","trace_id":"e617b6130fcc28674cc9899ec702e596"} {"time":"2026-10-07T21:31:48.791435032Z","message":"Actor checkpointed","span_id":"c475c334c8df2dc4","trace_flags":"01","trace_id":"e617b6130fcc28674cc9899ec702e596"} {"time":"2026-10-07T21:31:59.772566601Z","message":"Actor restoring","span_id":"a68c5afa10e86252","trace_flags":"01","trace_id":"b387404020eaca71d604b8f0c81d43c1"} {"time":"2026-10-07T21:31:59.96129419Z","message":"Actor restored","span_id":"a68c5afa10e86252","trace_flags":"01","trace_id":"b387404020eaca71d604b8f0c81d43c1"} {"time":"2026-10-07T21:38:25.962753328Z","message":"Actor checkpointing","span_id":"e181224bf8792b12","trace_flags":"01","trace_id":"52963b1666af63486d765bc1527fc8bc"} {"time":"2026-10-07T21:38:26.096296577Z","message":"Actor checkpointed","span_id":"e181224bf8792b12","trace_flags":"01","trace_id":"52963b1666af63486d765bc1527fc8bc"} {"time":"2026-10-07T21:43:14.718178039Z","message":"Actor restoring","span_id":"718a8f92f8735ed3","trace_flags":"00","trace_id":"42123dfac30d07fa86164199ecdbf0ac"} {"time":"2026-10-07T21:43:14.920089778Z","message":"Actor restored","span_id":"718a8f92f8735ed3","trace_flags":"00","trace_id":"42123dfac30d07fa86164199ecdbf0ac"} {"time":"2026-10-07T21:44:04.990909876Z","message":"Actor checkpointing","span_id":"107b6f12dd7f0e8b","trace_flags":"01","trace_id":"0bb09c29559dd8cfedb20f44a24bda4a"} {"time":"2026-10-07T21:44:05.134534519Z","message":"Actor checkpointed","span_id":"107b6f12dd7f0e8b","trace_flags":"01","trace_id":"0bb09c29559dd8cfedb20f44a24bda4a"} {"time":"2026-10-07T21:44:07.73397801Z","message":"Actor restoring","span_id":"7d062e8b5207afe9","trace_flags":"00","trace_id":"42f6d2926dd56687c222b1de832a072e"} {"time":"2026-10-07T21:44:07.978274791Z","message":"Actor restored","span_id":"7d062e8b5207afe9","trace_flags":"00","trace_id":"42f6d2926dd56687c222b1de832a072e"} {"time":"2026-10-07T21:44:11.552735135Z","message":"Actor checkpointing","span_id":"86954a4e96fbf250","trace_flags":"01","trace_id":"fb7199b8ee3491dcf561a25a8f60b1df"} {"time":"2026-10-07T21:44:11.69238189Z","message":"Actor checkpointed","span_id":"86954a4e96fbf250","trace_flags":"01","trace_id":"fb7199b8ee3491dcf561a25a8f60b1df"} {"time":"2026-10-07T21:44:13.26693018Z","message":"Actor restoring","span_id":"a896d4c30c7b664a","trace_flags":"00","trace_id":"607e6dbd891d7faa115e061dc4f473dc"} {"time":"2026-10-07T21:44:13.516304563Z","message":"Actor restored","span_id":"a896d4c30c7b664a","trace_flags":"00","trace_id":"607e6dbd891d7faa115e061dc4f473dc"} [exit 0] # Expected: default output is unchanged (every line). Baseline for the two counts below. $ kubectl ate logs actors learner-one -a ate-demo-counter | wc -l 5484 # Expected: baseline minus the lifecycle events. $ kubectl ate logs actors learner-one -a ate-demo-counter --source=containers | wc -l 5466 # Expected: 0, i.e. no lifecycle event survives --source=containers. $ kubectl ate logs actors learner-one -a ate-demo-counter --source=containers | grep -c '"message":"Actor ' 0 # Expected: containers + lifecycle = baseline (5466 + 18 = 5484). $ kubectl ate logs actors learner-one -a ate-demo-counter --source=lifecycle | wc -l 18 # Expected: same as --source=containers, since this actor has a single container (off by one only because the actor logs a line every 10s). $ kubectl ate logs actors learner-one -a ate-demo-counter -c counter | wc -l 5467 # Expected: ordinary application output, unchanged in shape. $ kubectl ate logs actors learner-one -a ate-demo-counter --source=containers | tail -3 {"time":"2026-10-08T12:22:35.319622976Z","count":5322,"fshash":"dorEZDUO6NCvAZmlU346thAgz9cSxuVYFZ2yOLq8rqg","level":"INFO","msg":"Count"} {"time":"2026-10-08T12:22:45.322833947Z","count":5323,"fshash":"dorEZDUO6NCvAZmlU346thAgz9cSxuVYFZ2yOLq8rqg","level":"INFO","msg":"Count"} {"time":"2026-10-08T12:22:55.316121067Z","count":5324,"fshash":"dorEZDUO6NCvAZmlU346thAgz9cSxuVYFZ2yOLq8rqg","level":"INFO","msg":"Count"} # NEGATIVE CASE, error is expected: -c and --source=lifecycle can never match the same line, so the CLI rejects the combination with exit 1 before fetching any logs. $ kubectl ate logs actors learner-one -a ate-demo-counter -c counter --source=lifecycle Error: --container cannot be combined with --source=lifecycle: lifecycle events are not emitted by a container [exit 1] # NEGATIVE CASE, error is expected: an unknown --source value is rejected with exit 1 and the list of valid values. $ kubectl ate logs actors learner-one -a ate-demo-counter --source=supervisor Error: invalid --source "supervisor": must be one of all, containers, lifecycle [exit 1] # Expected: the new flag is documented in --help next to --container. $ kubectl ate logs actors --help | grep -A1 -E -- "--source|--container" -c, --container string Show only logs from this container. -f, --follow Specify if the logs should be streamed. -- --source string Which lines to show: all, containers (every container's output, no lifecycle events), or lifecycle (only the actor's lifecycle events). (default "all") ``` Follow mode, with a suspend and a resume issued from another shell while `-f --source=lifecycle` was streaming. The stream shows only the lifecycle pair for each transition and reconnects once the actor is running again (on the same worker pod here, since the kind cluster has one). ``` $ kubectl ate logs actors learner-one -a ate-demo-counter -f --source=lifecycle # started, then suspend + resume in another shell Actor is currently running on pod ate-demo-counter/counter-79b587778f-8lh4h {"time":"2026-10-08T12:25:08.096184297Z","message":"Actor checkpointing","span_id":"b8dd6fc306a3ccf9","trace_flags":"01","trace_id":"45cae2a35d684ba8f1d6305c09333eb3"} {"time":"2026-10-08T12:25:08.291926175Z","message":"Actor checkpointed","span_id":"b8dd6fc306a3ccf9","trace_flags":"01","trace_id":"45cae2a35d684ba8f1d6305c09333eb3"} {"time":"2026-10-08T12:25:08.096184297Z","message":"Actor checkpointing","span_id":"b8dd6fc306a3ccf9","trace_flags":"01","trace_id":"45cae2a35d684ba8f1d6305c09333eb3"} {"time":"2026-10-08T12:25:08.291926175Z","message":"Actor checkpointed","span_id":"b8dd6fc306a3ccf9","trace_flags":"01","trace_id":"45cae2a35d684ba8f1d6305c09333eb3"} {"time":"2026-10-08T12:25:33.660662951Z","message":"Actor restoring","span_id":"d01fcf2c7f734ff2","trace_flags":"01","trace_id":"dfae910c43670918c5ab1cb3354863e3"} {"time":"2026-10-08T12:25:33.968364046Z","message":"Actor restored","span_id":"d01fcf2c7f734ff2","trace_flags":"01","trace_id":"dfae910c43670918c5ab1cb3354863e3"} [follow exited 124] ``` ``` [12:25:07Z] $ kubectl ate suspend actor learner-one -a ate-demo-counter ATESPACE NAME TEMPLATE STATE WORKER POD WORKER IPS VERSION AGE ate-demo-counter learner-one ate-demo-counter/counter ACTOR_STATE_SUSPENDED <none> 21 14h [12:25:33Z] $ kubectl ate resume actor learner-one -a ate-demo-counter ATESPACE NAME TEMPLATE STATE WORKER POD WORKER IPS VERSION AGE ate-demo-counter learner-one ate-demo-counter/counter ACTOR_STATE_RUNNING ate-demo-counter/counter-79b587778f-8lh4h 10.244.0.22 23 14h [12:26:04Z] $ kubectl ate get actors -a ate-demo-counter ATESPACE NAME TEMPLATE STATE WORKER POD WORKER IPS VERSION AGE ate-demo-counter learner-one ate-demo-counter/counter ACTOR_STATE_RUNNING ate-demo-counter/counter-79b587778f-8lh4h 10.244.0.22 23 14h ``` The repeated `checkpointing`/`checkpointed` pair after the suspend is the existing follow-mode reconnect replay (the resume cursor is a second-granularity `sinceTime`); it is unchanged by this PR. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Part of agent-substrate#1660. - `actor_jwt` entries in `replace_headers` are now injected on `http` and `https` rules alike, like `credential_uri` entries since agent-substrate#2158, when the request carries the header, instead of being denied with 501. The gateway mints a token for the actor behind the tunnel, with the policy's audiences and lifetime, and writes the prefix plus the token into the header. - Actor JWTs need no credential provider, so a gateway without one still injects them. A `credential_uri` entry still denies the request when no provider is configured. - A failed mint denies the request: 403 if the actor was deleted, 503 if ateapi is unavailable or the mint timed out, and 500 otherwise. - The `credential_uri` path moved into its own function. Each injection failure is wrapped with its header, and the credential URI for provider entries, so the router's log line says which injection denied the request. With `--experimental-enable-authz` set on ateapi, the gateway can only mint once agent-substrate#2026 lands. Testing: unit tests cover injection with no provider, the skip when the header is absent, and each denial on both legs. The e2e check is in agent-substrate#2256, stacked on this PR. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR (the user guide lands later in the agent-substrate#1660 series)
* Reduces go-routine and socket usage per Actor * Moves limits to per-Actor instead of across all Actors (noisy neighbor) * Clean-up of logic that can support later extension if needed. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…erver requires it (agent-substrate#2257) Part of agent-substrate#1756. - ate-api-server's `--actor-jwt-issuer` is now required instead of defaulting to `https://idp.<pod namespace>.svc`, so the issuer comes from one setting that ate-idp-server can share rather than from two binaries' defaults agreeing. - ate-setup takes the issuer from a new `actorJWT.issuer` setting (`ACTOR_JWT_ISSUER`, `--actor-jwt-issuer`, or the config file), defaulting to `https://idp.<namespace>.svc`, and validates it before touching the cluster. - ate-setup writes it to the `ate-api-server-envvars` ConfigMap as `ATE_API_ACTOR_JWT_ISSUER`, and the manifest passes `--actor-jwt-issuer=@env`, the same path the PostgreSQL settings take. Changing it rolls ate-api-server through the existing env-hash annotation. Anything that applies `ate-api-server.yaml` without ate-setup has to add `ATE_API_ACTOR_JWT_ISSUER` to that ConfigMap, or ate-api-server exits at startup. Testing: unit tests for the required flag, the `@env` lookup, ate-setup's default and override, and the ConfigMap write. I didn't deploy it by hand; the e2e install in CI is its first real run. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…e#2421) The field is repeated (so it should be plural), and it's also not aligned with other messages that call it `audiences`. Fixes agent-substrate#2408
Our entry points for building images and installing are the scripts under hack/ and cmd/ate-setup. Removing the Makefile cleans up a dev entrypoint we are no longer using.
…-substrate#2432) Renames the object-store plugin proto package from `objectstoresnapshot.v1` to `objectstore.v1`. It is split out of agent-substrate#2385 as its own refactor, since that PR adds a non-snapshot service to the same API. - Moves `pkg/proto/objectstoresnapshotpb/v1` to `pkg/proto/objectstorepb/v1` and `objectstoresnapshot.proto` to `objectstore.proto`, regenerated with `hack/update/codegen.sh`. - Updates every in-tree caller, the protoredact pinned field names, and `docs/architecture.md`. - Services, messages and fields keep their names. The `cmd/snapshot-plugin` binary, image and manifests are unchanged. Open agent-substrate#1907 will need a rebase. It adds no references to the old package, so only its existing conflicts with main apply. #### Breaking change gRPC method names change from `/objectstoresnapshot.v1.<Service>/<Method>` to `/objectstore.v1.<Service>/<Method>`. Out-of-tree plugins must regenerate from `objectstore.proto` and roll out with atelet and ate-api-server. Please add the `breaking-change` label. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…gent-substrate#2353) ### Summary Every replica receives every PodCertificateRequest, but only the replica chosen by rendezvous hashing should sign it. The others return `ErrNotAssigned` and were requeued with `AddRateLimited`, which treats a normal condition as a failure: - consumes the shared failure rate limiter (10 QPS bucket), and - grows the per-item exponential backoff (up to 1000s). This slows retries of genuinely assigned PCRs and delays takeover when a replica dies (the PCRs it owned may already be deep in backoff on the surviving replicas). Unassigned PCRs are now requeued with `AddAfter(rendezvous.LeaseDuration)`. Assignments only change when a replica joins or its lease expires, so re-checking more often gains nothing. `LeaseDuration` is exported from `rendezvous` for this purpose; its value is unchanged. A follow-up could make this event-driven (re-enqueue pending PCRs when the active replica set changes) instead of polling. ### Testing `go test ./cmd/podcertcontroller/...` - [x] Tests pass - [ ] Documentation updated (n/a)
We should have this section clearly describe the best practice. Part of agent-substrate#1686 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…e#2381) Fixes agent-substrate#2348 Restoring a sandboxed actor from a checkpointed snapshot requires strict compatibility between the worker that created the snapshot and the worker restoring it — across the sandbox class (`gvisor`, `microvm`), the host hardware (`architecture`, `cpu_features`), and the sandbox runtime assets (`gvisor_asset_hash`, microVM kernel/rootfs digests). - **Unified compatibility identity (`SandboxRuntime` & `VersionedSandboxCompat`)**: Combines `sandbox_class`, an informational version `name`, and a `compat_version` (`schema_version` + `repeated AttributeEntry attributes`) into a single self-contained message. When an actor is checkpointed, the control plane can copy the worker's `default_runtime` directly onto the snapshot, and during warm restore the scheduler can match the snapshot's `SandboxRuntime` against candidate workers without inspecting runtime-specific fields. - **Zero-downtime runtime rollouts (`default_runtime` vs. `restorable_runtimes`)**: Workers report both a `default_runtime` (used for cold starts and new checkpoints) and `restorable_runtimes` (other enabled runtime versions on the worker that can still warm-restore existing snapshots while a new default version rolls out). - **Schema-versioned key-value attributes (`schema_version` + `AttributeEntry`)**: `compat_version.schema_version` (`"v1"`) versions the compatibility attribute schema itself, while `attributes` uses a list-map of `AttributeEntry` (`+k8s:listType=map`, `+k8s:listMapKey=key`) so declarative validation enforces required keys, values, and length bounds on every entry. #### Scope of This PR vs. Follow-ups - **In this PR**: Establishes the `SandboxRuntime` / `VersionedSandboxCompat` proto schema and declarative validation, updates `internal/hardware` (`ProbeHost` and `Matches`), and wires end-to-end registration of `default_runtime` (with `sandbox_class` and host architecture) and `restorable_runtimes` through `ateom`, `atelet`, `ate-api-server`, and `fake-workersync`. - **Follow-ups**: - Enrich `default_runtime` (`name`, `gvisor_asset_hash`) and `restorable_runtimes` from `SandboxConfig` once `SandboxConfig` moves to `WorkerPool`. - Probe CPU feature attributes (`cpu_features`) in `hardware.ProbeHost()` (agent-substrate#2345 - WIP) - Stamp `worker.status.default_runtime` onto snapshots at pause/checkpoint time and filter candidate workers with `hardware.Matches` during warm-restore scheduling. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…trate#2437) We now have technical controls that obsolete some of the content in COLLABORATING.md: * We use branch protection rules to prevent creation of arbitrary branches (even by maintainers and owners). * We now require GitHub reviews and use the GitHub merge queue functionality, so we no longer need to caution not to merge your own PRs.
agent-substrate#2443) Binds the atenet-router's xDS server (ADS/CDS/LDS/RDS/SDS on `--port-xds`) to `127.0.0.1` instead of all interfaces. It is plaintext and unauthenticated, so any pod that could reach the router pod IP could stream the dataplane's listeners, routes, clusters and secret paths. Envoy in the same pod already dials `127.0.0.1:18000`, so nothing changes for it. Also drops the `xds` containerPort from the Deployment. Fixes agent-substrate#2276 - [x] Tests: `xds_listen_test.go` checks the bound address is loopback (including a real listen), and that the shipped manifest's Envoy bootstrap dials loopback on `--port-xds` with no container declaring that port. Both tests fail against the old code. - [ ] Docs: n/a
…rate#1634) Addresses agent-substrate#1017 by adding E2E tests for HTTPS and WebSocket egress. Added scope after the egress policy was finalized: [review comment](agent-substrate#1634 (comment)). Updates the gateways to reject HTTP/1.1 WebSocket upgrades with 403. PRs merge order/stack: agent-substrate#1945 agent-substrate#1944 and then this PR
EItanya
force-pushed
the
sync/upstream-main-20261009
branch
from
October 10, 2026 12:40
a03bdd3 to
2559ec0
Compare
EItanya
marked this pull request as ready for review
October 10, 2026 12:42
Integrate upstream sandbox versioning, snapshot plugins, actor identity, worker compatibility, and HTTPS egress changes with the fork. Adapt Helm charts, release workflows, gateway routing, and CA rotation fixtures while preserving fork behavior. Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
EItanya
force-pushed
the
sync/upstream-main-20261009
branch
from
October 12, 2026 13:10
2559ec0 to
5cf1c24
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Syncs the fork with upstream main at f2006302, including versioned sandbox configs, worker compatibility reporting, the snapshot plugin and lifecycle API, actor JWT injection, and HTTPS egress fixes.
Updates the Helm charts and release images for the snapshot plugin, preserves the actor JWT issuer default and fork workflow deadlines, and aligns agentgateway tunnel routing with upstream. Both install paths reject duplicate actor headers and use the gateway’s built-in HTTP upgrade rejection; Helm E2E also installs the service-DNS trust bundle for local HTTPS origins. Replaces removed Makefile commands in the fork workflows and fixes two CA-rotation tests that depended on filesystem timestamp granularity.
Breaking change
substrate-crdsbefore the application chart. Custom SandboxConfigs needspec.versions[]andspec.defaultVersion; WorkerPools usespec.sandboxClasses[].MintActorJWTRequest.audiences.snapshotPlugin.extraEnv; the API server no longer acceptsATE_STORAGE_BACKEND. The chart upgrade notes cover the Helm changes.Please apply the
breaking-changelabel. Merge with a merge commit to preserve upstream ancestry.The branch retains all imported upstream commits and combines the fork integration into one merge commit.
Validation: Helm lint and rendering pass. Live probes against the pinned agentgateway image verify that identical and conflicting duplicate actor headers return 404 for plain HTTP, gRPC, and CONNECT, while single-header requests reach actor resolution. The same duplicate-header probes reach actor resolution without the new policy. Regression cases cover identical and conflicting duplicates in the E2E suite.
All four CI checks pass on
5cf1c245c: unit tests and repository verification, full agentgateway E2E suites on gVisor and microVM, Helm E2E, and Helm verification. The first agentgateway attempt timed out contactinghttpbin.orgin the HTTP credential-injection test; rerunning the same commit passed that case in both sandbox lanes.Focused race tests for
internal/e2eand the networking suite pass locally, along with generated-code, CRD parity, Go module, and formatting checks. The full local race build and linter hit the shared temporary filesystem limit; CI completed the full unit and verification jobs successfully.